Dashboard Pipeline Setup¶

• Dashboard Pipeline Setup
  • Deployment Decisions
  • Installation of Required Software
  • Backend Configuration
    • InfluxDB
  • Grafana Configuration
    • Graphite
  • Startup
  • Initial Grafana Online Configuration
  • Usage

Deployment Decisions¶

There are currently two supported backends for storing timeseries data which can be visualized by the grafana dashboard:

1. InfluxDB: More modern timeseries database with well-structured query language
2. Graphite: Well-known monitoring database and web-frontend. Less structured query language and harder to deploy

While InfluxDB provides the much more modern experience and is easier to administer, graphite currently provides better performance with less CPU impact and better control of automatic retention. Depending on your needs, decide on one of the solutions.

Installation of Required Software¶

1. Install rsb, rst
2. Install rsb-performance-monitor (main git in this project)
3. Install rsb-host-monitor (host-monitor git in this project)
4. Install rsbperfmon-db-adapter (graphite-adapter git in this project)
5. Install a timeseries database:
   • InfluxDB: version 0.8! as DEB archive or citk
   • Graphite: See https://graphite.readthedocs.org/en/latest/install.html or citk
6. Install grafana version 2 (just extract the pre-built archive somewhere, or citk)

Backend Configuration¶

For both backend solutions you need to decide on a retention policy, that means how long data will be kept in the database. This decides on the storage requirements of your database files and the possibility to scroll back in time in the dashboard.

InfluxDB¶

1. Configure InfluxDB as desired using the example config files and the instructions found here: https://influxdb.com/docs/v0.8/introduction/installation.html. A config file which uses just a local computer and /tmp for data is attached. Please note that in this file the assets line in the admin section needs to be fixed so that it point to the installation of the required files for the admin web interface.
2. Start the database server, e.g. using
   

   $prefix/opt/influxdb/influxdb -config=$prefix/etc/vdemo_scripts/data/influxdb-config-tmp.toml
   

3. Using the web interface (http://host:8083 per default), create a database for the monitoring data. The retention for the proposed default shard space should be set to the desired time you plan to keep data available in the database (E.g. 7d). Duration should probably be set to 1d.

Grafana Configuration¶

Create an ini file to configure the grafana server. An example is attached: grafana-config.ini. You can place this file anywhere.

Graphite¶

Unfortunately, graphite configuration is a bit harder than for InfluxDB. Basically, you need to follow the instructions given here: https://graphite.readthedocs.org/en/latest/install.html#initial-configuration. The steps are:

1. Configure the carbon daemon, which accepts new data via HTTP and pushes them into the timeseries database.
   1. Create a carbon.conf file and adapt as required. The comments about all data directories decending from STORAGE_DIR are misleading. This is only true for the defaults. In case you want to move your data, you need to set all of these directories explicitly. An exemplary configuration is attached: carbon.conf. The file needs to be placed in the conf directory of your installation root.
   2. Create a storage-schemas.conf file which defines the retention policies. The highest resolution needs to be >= the intended reporting speed of data. Otherwise you will end up with null values in the database. An example is attached: storage-schemas.conf. The file needs to be placed in the conf directory of your installation root.
2. Configure the web application
   1. Create a local_settings.py. The STORAGE_DIR variable needs to match the value that has been used for carbon. Otherwise, the webapp won't find the database. An example is attached: local_settings.py. This file needs to be placed in the webapp/graphite directory of your installation. In case you do not want to set up a custom webserver, it is important to set DEBUG = True.
   2. Collect static webapp files to be served by the wbeserver:
      

      export PYTHONPATH=$GRAPHITE_ROOT/webapp:$PYTHONPATH
      django-admin.py collectstatic --noinput --settings=graphite.settings
      

   3. Create a configuration database for django:
      

      PYTHONPATH=$GRAPHITE_ROOT/webapp django-admin.py syncdb --settings=graphite.settings
      

Startup¶

1. Start the backend:
   • Influxdb: as described before
   • Graphite:
     1. Start carbon: $GRAPHITE_ROOT/bin/carbon-cache.py --nodaemon
     2. Start the webapp: $GRAPHITE_ROOT/bin/run-graphite-devel-server.py --port=XXXX
2. Start some rsb-process-monitor and rsb-node-monitor instances, assuming that monitoring results will be published on /monitoring and /host. When using Graphite it is important that the cycle time of these programs matches the specified first retention policy for the carbon daemon.
3. Start rsbperfmon-db-adapter -a -d -b {graphite|influxdb} and pass additional arguments required to match your backend configuration.
4. Start the grafana dashboard server: $GRAFANA_PREFIX/bin/grafana-server -config=your-config.ini -homepath=$GRAFANA_PREFIX

Initial Grafana Online Configuration¶

At first start, you need to perform a few steps in grafana:

1. Open the grafana webpage at http://localhost:8020
2. On the left side, log in as admin (default: admin:admin)
3. In the left menu, select "Data Sources" -> "Add new"
4. Add a data source matching your selected backend. For Graphite, the given port is the one used to start the devel server. The influxdb datasource must be called "influxdb" and the graphite one "graphite".
5. Go to "Dashboards"
6. In the "Home" dropdown menu at the top, select import and import the appropriate dashboards provided in a git repository in this project.

Usage¶

1. Open the grafana webpage at http://localhost:8020