Monitoring with Riemann & riemann-dash
This blog post is a work in progress
Riemann is a monitoring system written in Clojure. It can aggregate streams of data, called events, and it can filter, combine and over limited time windows integrate data, and it can also send notifications over e-mail and SMS.
Riemann is scripted in clojure, which is a Lisp-like language that runs on the Java virtual machine (JVM). Riemann is used to monitor the health and status of services such as Linux machines, databases, web servers. It is used e.g. by The Guardian newspaper, see guardian/riemann-config.
Riemann is not primarily a logger, although inside of Riemann you can define a stream that can be sent to logging.
Riemann-dash
There is an ecosystem surrounding the Riemann server, consisting of data sources, clients and a rather nice-looking and easily configurable web GUI for monitoring called riemann-dash, although I currently think I have some problems with memory leaks while running it in Google Chrome.
Config language of Riemann-dash
Riemann-dash has a configuration language, formally documented here:
reimann/Query.g at master · jdmaturen/reimann
(That is an old version of Riemann but is has the language definition as of now)
In order to select what to show in riemann-dash, you put the field you want to match against. Like so:
service = "cpu"
...if you want to monitor the CPU from the data generated by riemann-health.
The output will then be of the metric field which seems to be the default and maybe the only choice. Not sure how to change what field to display the value of.
Pattern matching is done with the "=~" operator, and "%" is the wildcard. "|" can be used to give alternative values. Haven't used pipe yet but I believe that alternative values are grouped inside a a pair of brackets: "(val1|val2). See monitoring-setup/dashboard.json at master · algernon/monitoring-setup
Here is a working example of how to get metrics from Supermann, a plugin to Supervisord that sends data to Riemann:
service =~ "process:%:cpu:percent"
"Rows" and "Columns" decides if the data will be presented row-wise or column-wise per host.
How to save settings from riemann-dash
You need to have a JSON file to save to. You can just add the path to a config file when starting riemann-dash.
riemann-dash ./settings/config.rb
But what should that config file contain? On Ubuntu 15.04 there is an example config.rb file in
/var/lib/gems/2.1.0/gems/riemann-dash-0.2.12/examples
and it would nice to just copy that, but that does not work as a whole. But the local config file is only part of the config. You can just include the changes from the default that you want.
Make a copy of config.rb and delete all lines except "config.store[:ws_config]". Write something like this:
config.store[:ws_config] = "/path/writable/by/riemann-dash/config/dir/config.json"
Riemann-dash will save its settings to that JSON file. Which file should be in a directory writable by the user that runs riemann-dash of course. I'd advise to version that config file with e.g. git, since it is easy to overwrite config changes that you actually wanted to keep.