RSB Performance Monitoring

h1. Dashboard Pipeline Setup

{{>toc}}

h2. Deployment Decisions

There are currently two supported backends for storing timeseries data which can be visualized by the grafana dashboard:

# "InfluxDB":https://influxdb.com/: More modern timeseries database with well-structured query language

# "Graphite":https://graphite.readthedocs.org/en/latest/: Well-known monitoring database and web-frontend. Less structured query language and harder to deploy

While InfluxDB provides the much more modern experience and is easier to administer, graphite currently provides better performance with less CPU impact and better control of automatic retention. Depending on your needs, decide on one of the solutions.

h2. Installation of Required Software

# Install rsb, rst

# Install rsb-performance-monitor

# Install rsb-performance-monitor-graphite-adapter (git in this project)

# Install a timeseries database:

#* InfluxDB: version 0.8! as DEB archive or citk

#* Graphite: See https://graphite.readthedocs.org/en/latest/install.html or citk

# Install grafana version 2 (just extract the pre-built archive somewhere, or citk)

h2. Backend Configuration

For both backend solutions you need to decide on a retention policy, that means how long data will be kept in the database.  This decides on the storage requirements of your database files and the possibility to scroll back in time in the dashboard.

h3. InfluxDB

# Configure InfluxDB as desired using the example config files and the instructions found here: https://influxdb.com/docs/v0.8/introduction/installation.html. A config file which uses just a local computer and /tmp for data is attached. Please note that in this file the @assets@ line in the @admin@ section needs to be fixed so that it point to the installation of the required files for the admin web interface.

# Start the database server, e.g. using

  <pre>

$prefix/opt/influxdb/influxdb -config=$prefix/etc/vdemo_scripts/data/influxdb-config-tmp.toml

</pre>

# Using the web interface (http://host:8083 per default), create a database for the monitoring data. The retention for the proposed default shard space should be set to the desired time you plan to keep data available in the database (E.g. 7d). Duration should probably be set to 1d.

h2. Grafana Configuration

Create an ini file to configure the grafana server. An example is attached: attachment:grafana-config.ini. You can place this file anywhere.

h3. Graphite

Unfortunately, graphite configuration is a bit harder than for InfluxDB. Basically, you need to follow the instructions given here: https://graphite.readthedocs.org/en/latest/install.html#initial-configuration. The steps are:

# Configure the carbon daemon, which accepts new data via HTTP and pushes them into the timeseries database.

## Create a @carbon.conf@ file and adapt as required. The comments about all data directories decending from @STORAGE_DIR@ are misleading. This is only true for the defaults. In case you want to move your data, you need to set all of these directories explicitly. An exemplary configuration is attached: attachment:carbon.conf. The file needs to be placed in the @conf@ directory of your installation root.

## Create a @storage-schemas.conf@ file which defines the retention policies. The highest resolution needs to be >= the intended reporting speed of data. Otherwise you will end up with null values in the database. An example is attached: attachment:"storage-schemas.conf". The file needs to be placed in the @conf@ directory of your installation root.

# Configure the web application

## Create a @local_settings.py@. The @STORAGE_DIR@ variable needs to match the value that has been used for carbon. Otherwise, the webapp won't find the database. An example is attached: attachment:local_settings.py. This file needs to be placed in the @webapp/graphite@ directory of your installation. In case you do not want to set up a custom webserver, it is important to set @DEBUG = True@.

## Collect static webapp files to be served by the wbeserver:

<pre>

export PYTHONPATH=$GRAPHITE_ROOT/webapp:$PYTHONPATH

django-admin.py collectstatic --noinput --settings=graphite.settings

</pre>

## Create a configuration database for django:

<pre>

PYTHONPATH=$GRAPHITE_ROOT/webapp django-admin.py syncdb --settings=graphite.settings

</pre>

h2. Startup

# Start the backend:

#* Influxdb: as described before

#* Graphite:

#*# Start carbon: @$GRAPHITE_ROOT/bin/carbon-cache.py --nodaemon@

#*# Start the webapp: @$GRAPHITE_ROOT/bin/run-graphite-devel-server.py --port=XXXX@

## Start some @rsb-process-monitor@ and @rsb-node-monitor@ instances, assuming that monitoring results will be published on @/monitoring@ and @/host@

# Start @rsbperfmon-db-adapter -a -d -b {graphite|influxdb}@ and pass additional arguments required to match your backend configuration.

# Start the grafana dashboard server: @$GRAFANA_PREFIX/bin/grafana-server -config=your-config.ini -homepath=$GRAFANA_PREFIX@

h2. Initial Grafana Online Configuration

At first start, you need to perform a few steps in grafana:

# Open the grafana webpage at http://localhost:8020

# On the left side, log in as admin (default: admin:admin)

# In the left menu, select "Data Sources" -> "Add new"

# Add a data source matching your selected backend. For Graphite, the given port is the one used to start the devel server.

# Go to "Dashboards"

# In the "Home" dropdown menu at the top, select import and import the appropriate dashboards provided in a git repository in this project.

h2. Usage

# Open the grafana webpage at http://localhost:8020