Saturday, December 03, 2011

Quickstatd: A simple tool to get performance data into Graphite

I recently wanted to start tracking OS-level performance metrics for a group of servers, and see the results charted in Graphite. My initial thought was to do this using collectd, with the graphite plugin. In many cases, using collectd is a great way to go -- it's been well tested and has a robust feature set.

But collectd wasn't a good fit for my environment. My servers do not have internet access. I was not able to install by copying the .rpm over to the servers, due to unmet dependancies. I decided I wanted something dead simple to install and operate, with no external dependencies.

The result was quickstatd: a small set of bash scripts, easily extendable, that forwards the performance data generated from standard system tools like vmstat, iostat, and sar, to a running instance of Graphite. I am providing it here in the hopes that it will be useful to others.

Here are two graph examples.  Graphite generated these using data provided by quickstatd's vmstat plugin.

Quickstatd has no external dependencies. Just extract the tarball, run the installer, and you're good to go. You can configure the services you want to record in /etc/quickstatd.conf. The tool is started and stopped via /etc/init.d/quickstatd.
Quickstatd is tested and works on CentOS (Fedora) and Ubuntu (Debian). With small modifications it could be made to work on OS X, Solaris, etc.
The vmstat plugin should work on most versions of Linux, but the systat package must be installed if you wish to use the 'sar' and 'iostat' monitoring tools on Debian/Ubuntu.


Anonymous said...
This comment has been removed by the author.
Anonymous said...

Great little framework - I'm currently modifying it to suit my needs.

One question: Is it meant to be run from a cronjob? Starting it as a service only gets you a one-time run and there doesn't seem to be any code to support keeping itself alive. When I saw the monitoring interval I assumed this was for quickstatd but that doesn't appear to be the case.

Travis Bear said...

There is no cron job. Once quickstatd is launched as a service, it runs forever, because the one-time commands it invokes (for example, 'vmstat 10') repeat on their own, indefinitely.

It would be possible to add support for non-recurring commands as well (for example, 'df -h'), and I intend to get around to this at some point. But that's future work.

Travis Bear said...

Today I checked in support for non-recurring jobs (such as 'df -h'), and included a disk usage metric as an example.

Unknown said...

Nice idea, and I do want to use it on Solaris 10 x86_64 just so I don't have to fight getting collectd compiled on there. Just a couple of issues though:

1. Doesn't explain it depends on netcat (nc) to send data.

2. gawk isn't on solaris 10, and so far /usr/xpg4/bin/awk isn't cutting it either. Blech.

Time to starting some hacking, both of this stuff and collectd and see which I get working first.

Digital said...

Can it be used with InfluxDB instead of graphite?