A few years ago I wrote a handy little tool to solve a simple problem: test automated ticketing & alerting based on syslog messages.

The problem was presented to me while working for a large telecommunications provider who had outsourced their fault monitoring to an off-shore consulting company. This particular company had a habit of hiring the cheapest employees, who consequently, were beginners at computers. This lead to many problems where a device would report a failure, but something would be mis-configured and the alarm wouldn’t work. For example, Cisco N7ks sends a message:

May   8 12:34:00.814 Nexus7000 PLATFORM-2-PS_FAIL Power supply 2 failed or shut down

when their power supplies fail. Any PLATFORM-2 messages were supposed to be evaluated by this company’s tools, and generate tickets for the on-site network admins. In reality, this rarely worked, and after every outage, there was an investigation and the question from management “how can outages like this can happen when we have many signs of an impending outage”.

We needed a quick and easy way to test that the automatic ticketing and alerting is correctly functioning, so my solution is to “fake” a syslog message. Since Syslog is UDP you can — within limitations — generate a packet from anywhere and appear to be somewhere else. Firewalls with anti-spoofing enabled will drop these packets if they ingress an interface on which they don’t belong. The biggest limitation: the faker must be executed nearby to the destination that you want to send the message, to avoid triggering anti-spoofing on firewalls.

The solution is a set of simple scripts based on sendip, which allows you to fake packets. We setup a server near our syslog collectors, and we can simulate any messages easily and repeatedly, which allows us to test our service providers to ensure they haven’t misconfigured their systems again. The tool started with a command line & interactive script for faking a message, but grew to a webpage to allow anyone from our first line team to assist in testing the chain.

The command line script is also now used for a continuous test, where a non-existent device sends a test message every 20 minutes. This test message generates a ticket to a test queue, where a script verifies regular alerts in the queue. This ensures continuous monitoring of a very complicated chain of tools managed by many teams, and provides metrics on exactly how long the message forwarding & parsing is taking.

This script has also proven useful when I moved into the Security Operations Center. Here my task was to board new devices into their SIEM environment. ArcSight is a nasty beast and has many inconsistencies in how you can process, change and parse messages. If we are lucky, projects provide a samples of logs from their devices. It was up to us to interpret the logs and parse them correctly. My little script I developed four years ago comes in handy again. We are using it to re-play messages from production systems in our test environment, where we then try to teach ArcSight how to understand the incoming messages & catch bad guys.

You can download the scripts, which I have setup here: syslog_faker_linux-x64_solaris-sun4v.  This includes compiled binaries for 64-bit Linux, and for Solaris (sun4v) **Major Warning** sendip must send messages as root (so it can mangle with the low-level packet headers). The PHP script must do a shellexec of a suid executable, so this should NEVER EVER EVER be available on a production system outside of your testing areas!