Centralized Analysis of Scale Data

The Grinder is a good tool for generating large levels of load against a server. But after the run is over, the love ends. There are no post-run analysis capabilities built in. No graphs of server-side performance metrics, or clients side data. I have begun working on an open-source data warehousing tool for test data generated by The Grinder. This tool parses the logs generated by each agent, and feeds all the information into a database. Once it has been centralized there, all sorts of post-run analysis becomes possible. I have two goals for this project:

• Allow for simple analysis of Grinder runs: generate summary tables, graph server-side perf data such as CPU and disk activity, graph client side data such as response times and transactions per second.
• Allow for long-term storage of test data. This will enable comparing performance of the server across many different builds, and longer-term, across many different server versions.

Current Situation Here is what is implemented today:

• The database -- tables
• The database -- sql scripts to generate TPS and response time graphs
• The log parser and feeder classes which run on each agent
• The grapher classes

This is an example of a transactions per second graph generated by the current code. In this scale run, the load was increasing over time (using the scheduling mechanism discussed at the end of my previous blog entry.) Here is an example of a response times graph generated by the current code: Future - Short Term In the short term, this basic functionality must be completed.

• SQL code to generate reports summarizing a given scale run. Presenting information on response times, pass/fail rates, etc. in a tabular format.
• Create feeders and graphers for server-side performance data, such as produced by perfmon, vmstat, sar, prstat, etc.

Future - Medium Term After the immediate needs are met, a good way of browsing the test results in the database is needed.

• Storing the graphs themselves in the database rather than on the filesystem.
• Implementing a Web UI for browsing scale test info in the database

Future - Long Term Currently, running a large-scale test with many agents is a cumbersome process. Each agent process must be manually started and connected to the console. In situations with, for example, 30 agent machines, this is problematic. Shell scripts exist today that can ssh in to remote agents and start them up, but they require Linux and are not a generic, cross-platform solution.

• Extending the web UI to provide additional configuration management.
• Extending the web UI to enable remote agent startup in a cross-platform manner, and the ability to drive a Grinder test in an AJAX environment via the console API.