At OVHcloud Metrics, we love open source! Our goal is to provide all of our users with a full experience. We rely on the Warp10 time series database which enables us to build open source tools for our users benefit. Let’s take a look at some in this blogpost.
Storage tool
Our Infrastructure is based on the open source time series database: Warp10. This database includes two versions: a stand-alone one and a distributed one. The distributed one relies on distributed tools such as Apache Kafka, Apache Hadoop and Apache HBase.
Unsurprisingly, our team makes its own contributions to the Warp10 platform. Due to our unique requirements, we even contribute to the underlying open source database HBase!
Metrics data ingest
As a matter of fact, we were the first to get stuck in on the ingest process! We often build adapted tools to collect and push monitoring data on Warp10 – this is how noderig came to life. Noderig is an adapted tool that is able to collect a simple core of metrics from any server or any virtual machine. It is also possible to send metrics safely to a backend. Beamium, a Rust tool, is able to push the Noderig metrics to one or several Warp 10 backend(s).
What if I want to collect my own custom metrics? First, you’ll need to expose them following the ‘Prometheus model’. Beamium is then able to scrap applications based on their configuration file and forward all the data to the configured Warp 10 backend(s)!
If you are looking to monitor specific applications using the Influx Telegraf agent (in order to expose the Metrics you require) we have also contributed the Warp10 Telegraf connector, which was recently merged!
This looks great so far, but what if I usually push Graphite, Prometheus, Influx or OpenTSDB metrics; how can I simply migrate to Warp10? Our answer is Catalyst: a proxy layer that is able to parse Metrics in the related formats, and convert them to Warp10 native.
Catalyst
Catalyst is a Go HTTP proxy used to parse multiple Open Source TimeSeries database writes. At the moment, it supports multiple Open Source TimeSeries database writes; such as OpenTSDB, PromQL, Prometheus-remote_write, Influx and Graphite. Catalyst runs a HTTP server that listens to a specific path; starting with the protocol time series name and then the native query one. For example, in order to collect influx data, you simply send a request to influxdb/write. Catalyst will natively parsed the influx data and convert it to Warp 10 ingest native format.
Metrics queries
Data collection is an important first step, but we have also considered how existing query Monitoring protocols could be used on top of Warp10. This has led us to implement TSL. TSL was discussed at length during the Paris Time Series Meetup as well as in this blog post.
Now let’s take a user that is using Telegraf and pushing data to Warp10 with Catalyst. They will wish to use the native Influx Grafana dashboard, but how? And what about users that automate queries with the OpenTSDB query protocol? Our answer was to develop a proxy: Erlenmeyer.
Erlenmeyer
Erlenmeyer is a Go HTTP proxy that enables users to query Warp 10 based on Open Source query protocol. At the moment, it supports multiple Open Source TimeSeries formats; such as PromQL, Prometheus remote-read, InfluxQL, OpenTSDB or Graphite. Erlenmeyer runs a HTTP server that listens to a specific path; starting with the protocol time series name and then the native query one. For example, to run a promQL query, the user sends a request to prometheus/api/v0/query. Erlenmeyer will natively parsed the promQL request and then build a native WarpScript request that any Warp10 backend can support.
To be continued
At first, Erlenmeyer and Catalyst represented a quick implementation of native protocols, aimed to help internal teams migrate, while still utilising a familiar tool. We have now integrated a lot of the native functionality of each protocol, and feel they are ready for sharing. It’s time to make them available to the Warp10 community, so we can receive feedback and continue to work hard in supporting open source protocols. You can find us in OVHcloud Metrics gitter room!
Other Warp10 users may require unimplemented protocol. They will be able to use Erlenmeyer and Catalyst to support them on their own Warp10 backend.
Welcome Erlenmeyer and Catalyst – Metrics Open Source projects!
Software Developer on the OVHcloud Metrics Data Platform and data lover!