RSB Performance Monitoring

h1. Dashboard Pipeline Setup

{{>toc}}

h2. Deployment Decisions

There are currently two supported backends for storing timeseries data which can be visualized by the grafana dashboard:

# "InfluxDB":https://influxdb.com/: More modern timeseries database with well-structured query language

# "Graphite":https://graphite.readthedocs.org/en/latest/: Well-known monitoring database and web-frontend. Less structured query language and harder to deploy

While InfluxDB provides the much more modern experience and is easier to administer, graphite currently provides better performance with less CPU impact and better control of automatic retention. Depending on your needs, decide on one of the solutions.

h2. Installation of Required Software

# Install rsb, rst

# Install rsb-performance-monitor

# Install rsb-performance-monitor-graphite-adapter (git in this project)

# Install a timeseries database:

#* InfluxDB: version 0.8! as DEB archive or citk

#* Graphite: See https://graphite.readthedocs.org/en/latest/install.html or citk

# Install grafana version 2 (just extract the pre-built archive somewhere, or citk)

h2. Backend Configuration

For both backend solutions you need to decide on a retention policy, that means how long data will be kept in the database.  This decides on the storage requirements of your database files and the possibility to scroll back in time in the dashboard.

h3. InfluxDB

# Configure InfluxDB as desired using the example config files and the instructions found here: https://influxdb.com/docs/v0.8/introduction/installation.html. A config file which uses just a local computer and /tmp for data is attached. Please note that in this file the @assets@ line in the @admin@ section needs to be fixed so that it point to the installation of the required files for the admin web interface.

# Start the database server, e.g. using

  <pre>

$prefix/opt/influxdb/influxdb -config=$prefix/etc/vdemo_scripts/data/influxdb-config-tmp.toml

</pre>

# Using the web interface (http://host:8083 per default), create a database for the monitoring data. The retention for the proposed default shard space should be set to the desired time you plan to keep data available in the database (E.g. 7d). Duration should probably be set to 1d.

h3. Graphite

Unfortunately, graphite configuration is a bit harder than for InfluxDB. Basically, you need to follow the instructions given here: https://graphite.readthedocs.org/en/latest/install.html#initial-configuration. The steps are:

# Configure the carbon daemon, which accepts new data via HTTP and pushes them into the timeseries database.

## Create a @carbon.conf@ file and adapt as required. The comments about all data directories decending from @STORAGE_DIR@ are misleading. This is only true for the defaults. In case you want to move your data, you need to set all of these directories explicitly. An exemplary configuration is attached: attachment:carbon.conf. The file needs to be placed in the @conf@ directory of your installation root.

## Create a @storage-schemas.conf@ file which defines the retention policies. The highest resolution needs to be >= the intended reporting speed of data. Otherwise you will end up with null values in the database. An example is attached: attachment: storage-schemas.conf. The file needs to be placed in the @conf@ directory of your installation root.

h2. Startup

# Start InfluxDB, e.g. with

  <pre>$prefix/opt/influxdb/influxdb -config=$prefix/etc/vdemo_scripts/data/influxdb-config-tmp.toml</pre>

# Ensure that a database exists inside the running DB. In this case, we will call it @data@

  <pre>curl -X POST 'http://localhost:8086/db?u=root&p=root' -d '{"name": "data"}'</pre>

# Start some rsb-performace-monitor instances, assuming that monitoring results will be published on @/monitoring@ and @/host@

# Start rsb-performance-monitor-graphite-adapter with the InfluxDB backend

  <pre>$prefix/bin/rsbperfmon-graphite -d -b influxdb -a</pre>

  Assumes that InfluxDB runs on localhost with port 8086 and root:root as credentials. If not, provide the required arguments. @-d@ pre-calculates derivatives of some values for better visualization purposes later on, @-a@ aggregates all subprocesses into the results of a single monitored process since most of the time only the cumulative results of a functional component are interesting.

# Start a webserver to server the static grafana files, e.g.:

  <pre> cd $prefix/opt/grafana && python2 -m SimpleHTTPServer 8020</pre>

# Optionally, start an elasticsearch server

  <pre>$prefix/opt/elasticsearch/bin/elasticsearch -Des.path.data=$HOME/.elasticsearch</pre>

  Chose an appropriate data path

h2. Usage

# Open the grafana webpage at http://localhost:8020

# Configure the dashboard as needed

# Save the dashboard, either to elasticsearch or to your browser

h2. Notes

* Use meaningful credentials instead of root:root