Skip to content

Installation

Apache Superset

It was already listed as a prerequisite, but chances are it's not yet installed on your platform. It is used for the last and most remarkable stage: dashboards.

geOrchestra maintains a dedicated doc on how to deploy and configure Apache Superset, please refer to it for deployment.

Note

As already stated, you can choose to use another tool for the vizualization, but you will have to configure the dashboards by yourself.

TimescaleDB

This is a database solution, targeting massive time-series data. It is actually an extension over PostgreSQL. Since we already deal with PostgreSQL in geOrchestra, this makes TimescaleDB a very good match for storing and managing our analytics data.

Choosing the proper version

You will need the community version of TimescaleDB.

TigerData are distributing timescaleDB for free use under 2 different licenses. They're explaining why in this blog post.
While it would make sense to use the OSS version, it lacks so many features that it wouldn't be of much use.
Using the community version is legit in our use-case, it falls under the "Value-added services" category of their licensing legal terms. which we strongly suggest you read to check for yourself and for your specific use-case.

TimescaleDB provides documentation about how to self-host an instance: https://docs.timescale.com/self-hosted/latest/. Or you can also opt for a SAAS database (paid option of course).

Like plain PostgreSQL, it is possible to deploy a light database service or a full-fledged HA service, on-premise or as a SAAS, depending on your choices and needs. This will be up to you to decide how you want to host it.

You will need the following postgresql timescaledb-related extensions:

  • timescaledb

The docker compose and helm chart will come with an optional built-in light installation of TimescaleDB (no HA, no QoS warranty).

Note

We recommend you to use a dedicated DB instance for this, not the same as for the (geOrchestra) application database: the expectations, configuration, backup requirements and processes will be quite different. You don't want, for instance, your application database to increase by several Gb/year.

OpenTelemetry support

OpenTelemetry is an open standard for observability data. It is used in this module to collect the logs from your gateway/security proxy service.

Installation implies:

  • configure the Gateway/security proxy to use the OpenTelemetry java agent and expose the enhanced access logs ( see Configuring access logs)
  • run an OpenTelemetry-capable service, Vector, in charge of collecting the logs and shipping them to the database.

Vector can be installed:

  • as a single binary
  • run as a systemd service
  • deployed using docker (already configured in the provided docker-compose.yml)
  • deployed using kubernetes (already included in the geOrchestra Helm chart as an optional feature)

TODO: implement docker-compose and feature in helm chart

Vector comes with a yaml configuration file. For the docker and k8s setups, a default config file will be provided.

A sample configuration file is provided in the vector configuration page.

Analytics CLI (python task)

The analytics CLI is a python package that can be used:

  • to process the logs collected by Vector and stored in the TimescaleDB base
  • to process (mostly legacy) file-based log records (CLF-like)

It is the last step of the logs processing, before getting the dashboards. In most cases you want it to be run on a regular basis, which means make it run as a cron task.

Run the python code in commandline

If you want to run this in the commandline, your best option is to create a virtualenv and install the package by following the the CLI's instructions.

While on the virtualenv, check the package works: analytics-cli --help.

Then see the configuration section to get an overview of your options.

Run it on a schedule (cron task)

For classic setup and for docker compose setup, your best bet is probably to configure it as a cron task. There is plenty of documentation in the wild about this, it will not be covered here.

For Kubernetes setup, the helm chart should have configured it as a cronjob, so you should be good.

TODO: consider https://github.com/christopher-besch/docker_cron for the docker compose ?