But how has it evolved since then? Is TSL ready to query other time series databases? What about TSL states on the Warp10 eco-system?

TSL to query many time series databases

We wanted TSL to be usable in front of multiple time series databases. That’s why we also released a PromQL query generator.

One year later, we now know this wasn’t the way to go. Based on what we learned, the TSL-Adaptor project was open sourced, as a proof of concept for how TSL can be used to query a non-Warp 10 database. Put simply, TSL-Adaptor allows TSL to query an InfluxDB.

What is TSL-Adaptor?

TSL-Adaptor is a Quarkus JAVA REST API that can be used to query a backend. TSL-Adaptor parses the TSL query, identifies the fetch operation, and loads natively raw data from the backend. It then computes the TSL operations on the data, before returning the result to the user. The main goal of TSL-Adaptor is to make TSL available on top of other TSDBs.

In concrete terms, we are running a JAVA REST API that integrates the WarpScript library in its runtime. TSL is then used to compile the query into a valid WarpScript one. This is fully transparent, and only deals with TSL queries on the user’s side.

To load raw data from the InfluxDB, we created a WarpScript extension. This extension integrates an abstract class LOADSOURCERAW that needs to be implemented to create an TSL-Adaptor data source. This requires only two methods: find and fetch. Find gathers all series selectors matching a query (class names, tags or labels), while fetchactually retrieves the raw data within a time span.

Query Influx with TSL-Adaptor

To get started, simply run an InfluxDB locally on the 8086 port. Then, let’s start an influx Telegraf agent and record Telegraf data on the local influx instance.

Next, make sure you have locally installed TSL-Adaptor and updated its config with the path to a tsl.so library.

To specify a custom influx address or databases, update the TSL-Adaptor configuration file accordingly.

You can start TSL-Adaptor with the following example command:

java -XX:TieredStopAtLevel=1 -Xverify:none -Djava.util.logging.manager=org.jboss.logmanager.LogManager -jar build/tsl-adaptor-0.0.1-SNAPSHOT-runner.jar 

And that’s it! You can now query your influx database with TSL and TSL-Adaptor.

Let’s start with the retrieval of the time series relating to the disk measurements.

curl --request POST \
  --url http://u:p@0.0.0.0:8080/api/v0/tsl \
  --data 'select("disk")'

Now let’s use the TSL analytics power!

First, we would like to retrieve only the data containing a mode set to rw. 

curl --request POST \
  --url http://u:p@0.0.0.0:8080/api/v0/tsl \
  --data 'select("disk").where("mode=rw")'

We would like to retrieve the maximum value at every five-minute interval, for the last 20 minutes. The TSL query will therefore be:

curl --request POST \
  --url http://u:p@0.0.0.0:8080/api/v0/tsl \
  --data 'select("disk").where("mode=rw").last(20m).sampleBy(5m,max)'

Now it’s your turn to have some fun with TSL and InfluxDB. You can find details of all the implemented functions in the TSL documentation. Enjoy exploring!

What’s new on TSL with Warp10?

We originally built TSL as a GO proxy in front of Warp10. TSL has now integrated the Warp 10 ecosystem as a Warp10 extension, or as a WASM library! We have also added some new native TSL functions to make the language even richer!

TSL as WarpScript function

To make TSL work as a Warp10 function, you need to have the tsl.so library available locally. This library can be found in TSL github repository. We have also made a TSL WarpScript extension available from WarpFleet, the extension repository of the Warp 10 community.

To set up TSL extension on your Warp 10, simply download the JAR indicated in WarpFleet. You can then configure the extension in the Warp 10 configuration file:

warpscript.extensions = io.ovh.metrics.warp10.script.ext.tsl.TSLWarpScriptExtension
warpscript.tsl.libso.path = <PATH_TO_THE_tsl.so_FILE>

Once you reboot Warp 10, you are ready to go. You can test if it’s working by running the following query:

// You will need to put here a valid Warp10 token when computing a TSL select statement
// '<A_VALID_WARP_10_TOKEN>' 

// A valid TOKEN isn't needed on the create series statement in this example
// You can simply put an empty string
''

// Native TSL create series statement
 <'
    create(series('test').setValues("now", [-5m, 2], [0, 1])) 
'>
TSL

With the WarpScript TSL function, you can use native WarpScript variables in your script, as shown in the example below:

// Set a Warp10 variable

NOW 'test' STORE

'' 

// A Warp10 variable can be reused in TSL script as a native TSL variable
 <'
    create(series('test').setValues("now", [-5m, 2], [0, 1])).add(test)
'>
TSL

TSL WASM

To expand TSL’s potential uses, we have also exported it as a Wasm library, so you can use it directly in a browser! The Wasm version of the library parses TSL queries locally and generates the WarpScript. The result can then be used to query a Warp 10 backend. You will find more details on the TSL github.

TSL’s new features

As TSL has grown in popularity, we have detected and fixed a few bugs, and also added some additional native functions to accommodate new use cases.

We added the setLabelFromNamemethod, to set a new label to a series, based on its name. This label can be the exact series name, or the result of a regular expression.

We also completed the sortmethod, to allow users to sort their series set based on series meta data (i.e. selector, name or labels).

Finally, we added a filterWithoutLabels, to filter a series set and remove any series that do not contain specific labels.

Thanks for reading! I hope you will give TSL a try, as I would love to hear your feedback.

Paris Time Series meetup

We are delighted to be hosting, at the OVHcloud office in Paris, soon the third Paris Time Series meetup, organised by Nicolas Steinmetz. During this meetup, we will be speaking about TSL, as well as listening to an introduction of the Redis Times Series platform.

If you are available, we will be happy to meet you there!

TSL stands for Time Series Language. In simple terms, TSL is an abstracted way of generating queries for different TSDB backends, in the form of an HTTP proxy. It currently supports Warp 10’s WarpScript and  Prometheus’ PromQL query languages, but we aim to extend the support to other major TSDBs.

To provide some context around why we created TSL, it began with a review of some of the TSDB query languages supported on the OVH Metrics Data Platform. When implementing them, we learned the good, the bad and the ugly of each one. In the end, we decided to build TSL to simplify the querying on our platform, before open-sourcing it to use it on any TSDB solution. 

So why did we decide to invest our time in developing such a proxy? Well, let me tell you the story of the OVH Metrics protocol!

From OpenTSDB…

The first aim of our platform is to be able to support the OVH infrastructure and application monitoring. When this project started, a lot of people were using OpenTSDB, and were familiar with its query syntax. OpenTSDB is a scalable database for Time Series. The OpenTSDB query syntax is easy to read, as you send a JSON document describing the request. The document below will load allsys.cpu.0 metrics of thetestdatacentre, summing them between thestart and end dates:

{
    "start": 1356998400,
    "end": 1356998460,
    "queries": [
        {
            "aggregator": "sum",
            "metric": "sys.cpu.0",
            "tags": {
                "host": "*",
                "dc": "test"
            }
        }
    ]
}

This enables the quick retrieval of specific data, in a specific time range. At OVH, this was used for graphs purpose, in conjunction with Grafana, and helped us to spot potential issues in real time, as well as investigate past events. OpenTSDB integrates simple queries, where you can define your own sampling and deal with counter data, as well as filtered and aggregated raw data.

OpenTSDB was the first protocol supported by the Metrics team, and is still widely used today. Internal statistics shows that 30-40% of our traffic is based on OpenTSDB queries. A lot of internal use cases can still be entirely resolved with this protocol, and the queries are easy to write and understand.

For example, a query with OpenTSDB to get the max value of theusage_systemfor thecpu0 to 9, sampled for a 2-minute span by their values’ average, looks like this:

{
    "start": 1535797890,
    "end": 1535818770,
    "queries":  [{
        "metric":"cpu.usage_system",
        "aggregator":"max",
        "downsample":"2m-avg",
        "filters": [{
            "type":"regexp",
            "tagk":"cpu",
            "filter":"cpu[0–9]+",
            "groupBy":false
            }]
        }]
}

However, OpenTSDB quickly shows its limitations, and some specific uses cases can’t be resolved with it. For example, you can’t apply any operations directly on the back-end. You have to load the data on an external tool and use it to apply any analytics.

One of the main areas where OpenTSDB (version 2.3) is lacking  is multiple Time Series set operators, which allow actions like a divide series. Those operators can be a useful way to compute the individual query time per request, when you have (for example) a set of total time spend in requests and a set of total requests count series. That’s one of the reasons why the OVH Metrics Data Platform supports other protocols.

… to PromQL

The second protocol we worked on was PromQL, the query language of the Prometheus Time Series database. When we made that choice in 2015, Prometheus was gaining some traction, and it still has an impressive adoption rate. But if Prometheus is a success, it isn’t for it’s query language, PromQL. This language never took off internally, although it has started to gain some adoption recently, mainly due to the arrival of people that worked with Prometheus in their previous companies. Internally, PromQL queries represent about 1-2% of our daily traffic. The main reasons are that a lot of simple use cases can be solved quickly and with more control of the raw data with OpenTSDB queries, while a lot of more complex use cases cannot be solved with PromQL. A similar request to the one defined in OpenTSDB would be:

api/v1/query_range?
query=max(cpu. usage_system{cpu=~"cpu[0–9]%2B"})
start=1535797890&
end=1535818770&
step=2m

With PromQL, you lose control of how you sample the data, as the only operator is last. This means that  if (for example) you want to downsample your series with a 5-minute duration, you are only able to keep the last value of each 5-minute series span. In contrast, all competitors include a range of operators. For example, with OpenTSDB, you can choose between several operators, including average, count, standard deviation, first, last, percentiles, minimal, maximal or summing all values inside your defined span.

In the end, a lot of people choose to use a much more complex method: WarpScript, which is powered by the Warp10 Analytics Engine we use behind the scenes.

Our internal adoption of WarpScript

WarpScript is the current Time Series language of Warp 10(R), our underlying backend. WarpScript will help for any complex Time Series use case, and solves numerous real-world problems, as you have full control of all your operations. You have dedicated frameworks of functions to sample raw data and fill missing values. You also have frameworks to apply operations on single-value or window operations. You can apply operations on multiple Time Series sets, and have dedicated functions to manipulate Time Series times, statistics, etc.

It works with a Reverse Polish Notation (like a good, old-fashioned HP48, for those who’ve got one!), and simple uses cases can be easy to express. But when it comes to analytics, while it certainly solves problems, it’s still complex to learn. In particular, Time Series use cases are complex and require a thinking model, so WarpScript helped to solve a lot of hard ones.

This is why it’s still the main query used at OVH on the OVH Metrics platform, with nearly 60% of internal queries making use of it. The same request that that we just computed in OpenTSDB and PromQL would be as follows in WarpScript:

[ "token" "cpu.average" { "cpu" "~cpu[0–9]+" } NOW 2 h ] FETCH
[ SWAP bucketizer.mean 0 2 m 0 ] BUCKETIZE
[ SWAP [ "host" ] reducer.max ] REDUCE

A lot of users find it hard to learn WarpScript at first, but after solving their initial issues with some (sometimes a lot of) support, it becomes the first step of their Time Series adventure. Later, they figure out some new ideas about how they can gain knowledge from their metrics. They then come back with many demands and questions about their daily issues, some of which can be solved quickly, with their own knowledge and experience.

What we learned from WarpScript is that it’s a fantastic tool with which to build analytics for our Metrics data. We pushed many complex use cases with advanced signal-processing algorithms like LTTB, Outliers or Patterns detections, and Kernel Smoothing, where it proved to be a  real enabler. However, it proved quite expensive to support for basic requirements, and feedback indicated the syntax and overall complexity were big concerns.

A WarpScript can involve dozens (or even hundreds) of lines, and a successful execution is often an accomplishment, with the special feeling that comes from having made full use of one’s brainpower. In fact, an inside joke amongst our team is being born able to write a WarpScript in a single day, or to earn a WarpScript Pro Gamer badge! That’s why we’ve distributed Metrics t-shirts to users that have achieved significant successes with the Metrics Data Platform.

We liked the WarpScript semantic, but we wanted it to have a significant impact on a broader range of use cases. This is why we started to write TSL with few simple goals:

• Offer a clear Time Series analytics semantic
• Simplify the writing and making it developer-friendly
• Support data flow queries and ease debugging for complex queries
• Don’t try and be the ultimate tool box. Keep it simple.

We know that users will probably have to switch back to WarpScript every so often. However, we hope that using TSL will simplify their learning curve. TSL is simply a new step in the Time Series adventure!

The path to TSL

TSL is the result of three years of Time Series analytics support, and offers a functional Time Series Language. The aim of TSL is to build a Time Series data flow as code.

With TSL, native methods, such as select and where, exist to choose which metrics to work on. Then, as Time Series data is time-related, we have to use a time selector method on the selected meta. The two available methods are from and last. The vast majority of the other TSL methods take Time Series sets as input and provide Time Series sets as the result. For example, you have methods that only select values above a specific threshold, compute rate, and so on. We have also included specific operations to apply to multiple subsets of Time Series sets, as additions or multiplications.

Finally, for a more readable language, you can define variables to store Time Series queries and reuse them in your script any time you wish. For now, we support only a few native types, such as Numbers, Strings, Time durations, Lists, and Time Series (of course!).

Finally, the same query used throughout this article will be as follows in TSL:

select("cpu.usage_system")
.where("cpu~cpu[0–9]+")
.last(12h)
.sampleBy(2m,mean)
.groupBy(max)

You can also write more complex queries. For example, we condensed our WarpScript hands-on, designed to detect exoplanets from NASA raw data, into a single TSL query:

sample = select('sap.flux')
 .where('KEPLERID=6541920')
 .from("2009–05–02T00:56:10.000000Z", to="2013–05–11T12:02:06.000000Z")
 .timesplit(6h,100,"record")
 .filterByLabels('record~[2–5]')
 .sampleBy(2h, min, false, "none")

trend = sample.window(mean, 5, 5)

sub(sample,trend)
 .on('KEPLERID','record')
 .lessThan(-20.0)

So what did we do here? First we instantiated a sample variable in which we loaded the ‘sap.flux’ raw data of one star, the 6541920. We then cleaned the series, using the timesplit function (to split the star series when there is a hole in the data with a length greater than 6h), keeping only four records. Finally, we sampled the result, keeping the minimal value of each 2-hour bucket.

We then used this result to compute the series trend, using a moving average of 10 hours.

To conclude, the query returns only the points less than 20 from the result of the subtraction of the trend and the sample series.

TSL is Open Source

Even if our first community of users was mostly inside OVH, we’re pretty confident that TSL can be used to solve a lot of Time Series use cases.

We are currently beta testing TSL on our OVH Metrics public platform. Furthermore, TSL is open-sourced on Github, so you can also test it on your own platforms.

We would love to get your feedback or comments on TSL, or Time Series in general. We’re available on the OVH Metrics gitter, and you can find out more about TSL  in our Beta features documentation.