Version 5 F6 of syslog-ng PE supports Elasticsearch version 2.0 and newer. Because of compatibility reasons, syslog-ng PE has a separate destination (elasticsearch2()) that you can use with Elasticsearch version 2.0 and newer.

X-Pack security (Elasticsearch Shield) is supported for both Elasticsearch 1.x and 2.x destinations, allowing you to authenticate your syslog-ng PE clients on the Elasticsearch server.

Version 5.4 of syslog-ng PE can directly send log messages to Elasticsearch, allowing you to search and analyze your data in real time, and visualize it with Kibana.

The syslog-ng PE application allows you to publish your log data to your Apache Kafka message bus, where subscribers can access them, making it easy to integrate your log data into a big data solution.

Version 5.3 of syslog-ng PE can send plain-text log files to the Hadoop Distributed File System (HDFS), allowing you to store your log data on a distributed, scalable file system. This is especially useful if you have huge amount of log messages that would be difficult to store otherwise, or if you want to process your messages using Hadoop tools (for example, Apache Pig).

MongoDB is a schema-free, document-oriented database, ideal to collect log messages. Since it does not require a predetermined schema, it gives you much more flexibility than the SQL databases, making it it easy to store name-value pairs extracted from log messages. That way, you can conveniently store metadata received in a log message (for example, in the SDATA part of RFC5424-formatted log messages), or other data parsed from the body of the log messages (for example, usernames parsed from login/logout messages).

An important aspect of logging is alerting on important but rare events. A common way to do that is via e-mail, sent either to people or to services which process them further. The SMTP destination driver allows you to send e-mails based on incoming log messages to one or more addresses.

JavaScript Object Notation (JSON) is a text-based open standard designed for human-readable data interchange. It is used primarily to transmit data between a server and web application, serving as an alternative to XML. The syslog-ng PE application can separate parts of JSON-encoded log messages to name-value pairs, allowing you to receive structured log messages from such applications, store them, and convert them to other format if needed.

The systemd journal is a new type of system log storage. This is used, or will be used on most Linux distributions, such as RHEL (from RHEL7), Fedora, CentOS, and so on. The journal can store name-value pairs instead of the traditional system log entries. syslog-ng PE 6 LTS can directly read log messages from the journal file of platforms using systemd.

syslog-ng PE version 5 F6 includes a new template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format message.

The syslog-ng PE application can convert messages or selected value-pairs into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format, or to forward JSON messages to external applications.

The syslog-ng PE application allows you to select and construct name-value pairs from any information already available about the log message, or extracted from the message itself. You can directly use this structured information, for example, in the following places:

Using the groupset rewrite rule, you can rewrite multiple macros at the same time, making it easy to modify the values of fields parsed using patterndb or from JSON. You can specify the list of macros manually, or also as a glob pattern.

The syslog-ng PE application can compare the value of a macro to a list of strings. Earlier, if a specific macro had several different values, filtering on the macro values required several filter statements. Now you can specify every expected value in a file, and use the contents of that file in a filter.

With the hash template function, you can create message digests from parts of the log message.

HOSTID is a 32-bit number generated by a cryptographically secure pseudorandom number generator. Its purpose is to identify the syslog-ng PE host, thus it is the same for every message of the host. It can be accessed via the $HOSTID macro (which cannot be rewritten).

When the global option use-uniqid(yes) is set, syslog-ng PE generates this practically unique id for every received or locally generated message. You can add this ID to your messages using the $UNIQID macro.

The syslog-ng PE application uses a license in server mode to determine the maximum number of hosts that are allowed to connect. Use the syslog-ng-ctl show-license-info command to display license-related information the number of hosts currently logging to your server. This helps you to plan your capacity, to check your license usage, and to detect client misconfiguration that can result in a license miscount anomaly. Note that in client or relay mode, syslog-ng PE does not require a license.

To simplify the management of large-scale syslog-ng PE deployments, you can now centrally manage your syslog-ng PE hosts from Puppet. The syslog-ng Premium Edition Puppet module allows you to perform the following tasks.

The Puppet module supports the following platforms: Red Hat Enterprise Linux (RHEL), Oracle Linux, CentOS, Ubuntu, and Debian. Other Linux platforms based on .deb and .rpm packages might also work, but are not tested.

So far, you could access statistics only in unstructured format, using the syslog-ng-ctl stats command. Now you can query information from a running syslog-ng PE instance using the new syslog-ng-query utility. This tool allows you to access selected statistics in a controlled way, making it easy to process or monitor the results. This is a first step in a new statistics framework that aims to improve the how syslog-ng PE instances can be monitored.

Note that this new framework might decrease the performance of syslog-ng PE under very high load. If you experience any issues, contact the Balabit Support Team and let us know the details of your use case, so we can correct the problem.

In addition to Red Hat Enterprise Linux 6.5, syslog-ng PE now supports SELinux on Red Hat Enterprise Linux 5, as well as on 6.0-6.4. The CentOS platforms corresponding to the supported RHEL versions are supported as well.