Syslog-ng to send data to ELK post parsing

Syslog-ng is collecting cisco switch , routers and firewall data, which needs to be sent to ELK.

Before sending, how can i format logs into JSON

  • Hi ajay,

    For the problems you mentioned above, it is recommended to use Elasticsearch driver in syslog-ng PE, set template with (format-json) can help your problem.

    reference document: https://www.syslog-ng.com/community/b/blog/posts/using-syslog-ng-with-the-elastic-stack

    Hope it helps you.

    Pong,

    Regards

  • Hi Pong,

    So, i am a bit confused on how the template option works(since i am not very familiar with syslog-ng).

    Can u please explain how can i configure it for cisco and firewall logs. Also, do i need template for each log (since log patterns are different)

  • To make the pipeline more transparent, you can try having syslog-ng write to a file. Logstash can then consume the file and send to Elastic. For us, that's been more sustainable than direct Elastic output from syslog-ng, because we upgrade Elastic often. It's still cohesive, but more loosely coupled. If there's one thing you learn upgrading Elasticsearch often, it's handling breaking changes.

    One of the benefits in writing to a local file is that you can see parsing errors and get a fix in the syslog-ng parsing or even logstash parsing where required. You can also upgrade Logstash and Elasticsearch without thinking about dependencies in syslog-ng. On the downside, you'd need to handle log rotation.

    To output to a json file,  you could try something like this. There are far more examples in the syslog-ng documentation, including the elastic link mentioned previously.

    destination d_cisco {
    file("/var/log/hosts/cisco/$HOST.log"
    create_dirs(yes)
    mark-mode(periodical)
    log-fifo-size(2000)
    template("$(format-json --scope all-nv-pairs @timestamp=${S_ISODATE} ) \n") );
    };

    And then adjust until the output looks right. One template might work for multiple logs, but you'll need to check each one. Some log sources need more help than others. It's far easier than starting going directly to Elastic, because Elastic, once a field is typed, will not accept differently typed fields. Get a timestamp wrong and you'll need to not just fix syslog-ng but also delete the index from Elastic or re-name that particular field.