In the Observability team, we encounter these snippets as queries within the Time Series Database (TSDB), to express continuous queries that are responsible for automating different use cases like: deletes, rollups or any business logic that needs to manipulate Time Series data.

We already introduced TSL in a previous blog post, which demonstrated how our customers use the available OVH Metrics protocols, like Graphite, OpenTSDB, PromQL and WarpScript™, but when it comes to manipulating, or even creating new data,  you don’t have a lot of options, although you can use WarpScript™ or TSL as scripting language instead of a query one.

In most cases, this business logic requires building an application, which is more time-consuming than expressing the logic as a query targeting a TSDB. Building the base application code is the first step, followed by the CI/CD (or any delivery process), and setting up its monitoring. However, managing hundreds of little apps like these will add an organic cost, due to the need to maintain them along with the underlying infrastructure.

We wanted to ensure these valuable tasks did not stack up on the heads of few developers, who would then need to carry the responsibilities of data ownership and computing resources, so we wondered how we could automate things, without relying on the team to setup the compute jobs each time someone needed something.

We wanted a solution that would focus on the business logic, without needing to run an entire app. This way, someone wanting to generate a JSON file with a daily data report (for example) would only need to express the corresponding query.

You shall not FaaS!

Scheduling jobs is an old, familiar routine. Be it bash cron jobs, runners, or specialised schedulers, when it comes to wrapping a snippet of code and making it run periodically, there is a name for it: FaaS.

FaaS was born with a simple goal in mind: reduce development time. We could have found an open source implementation to evaluate (e.g. OpenFaas), but most of these relied upon a managed container stack. Having one container per query would be very costly, plus warming up a container to execute the function and then freezing it would have been very counterproductive.

This would have required more scheduling and automation than we wanted for our end-goal, would have lead to suboptimal performance, and would have introduced a new requirement for cluster capacity management. There is also a build time required to deploy a new function in a container, which is consequently not free.

#def <Loops>

That was when we decided to build “Loops”: an application platform where you can push the code you want to run. That’s all. The goal is to push a function (literally!) rather than a module, like all current FaaS solutions do:

function dailyReport(event) {
    return Promise.resolve('Today, everything is fine !')
}

You can then execute it manually, with either an HTTP call or a Cron-like scheduler.
These both aspects are necessary, since you might (for example) have a monthly report, but one day will require an additional one, 15 days after the last report. Loops will make it easy to manually generate your new report, in addition to the monthly one.

There were some necessary constraints when we began building Loops:

• This platform must be able to easily scale, to support OVH’s production load
• It must be highly available
• It must be language-agnostic, because some of us prefer Python, and others JavaScript
• It must be reliable
• The scheduling part mustn’t be correlated with the execution one (μService culture)
• It must be secure and isolated, so anybody can push obscure code on the platform

Loops implementation

We choose to build our first version on V8. We chose JavaScript as the first language, because it’s easy to learn, and asynchronous data flows are easily managed using Promises. Also, it fits very well with a FaaS, since Javascript functions are highly expressive. We built it around the new NodeJS VM module, which allows you to execute code in a dedicated V8 context.

A  V8 context is like an object (JSON), isolated from your execution. In context, you can find native functions and objects. However, if you craft a new V8 context, you will see that some variables or functions are not natively available (setTimeout(), setInterval() or Buffer, for example). If you want to use these, you will have to inject them into your new context. The last important thing to remember is that when you have your new context, you can easily execute a JavaScript script under string form on it.

Contexts fulfil the most important part of our original list of requirements: isolation. Each V8 context is isolated, so it cannot talk to another context. This means a global variable defined in one context is not available in a different one. You will have to build a bridge between them if you want this to be the case.

We didn’t want to execute scripts with eval(), since a call to this function allows you to execute JS code on the main shared context, with the code calling it. You can then access to the same objects, constants, variables, etc. This security issue was a deal breaker for the new platform.

Now we know how to execute our scripts, let’s implement some management for them. To be stateless, each Loops worker instance (i.e. a JavaScript engine able to run code in a VM context) must have the last version of each Loop (a loop is a script to execute). This means that when a user pushes a new Loop, we have to sync it on each Loops worker. This model fits well with the pub/sub paradigm, and since we already use Kafka as a pub/sub infrastructure, it was just a matter of creating a dedicated topic and consuming it from the workers. In this case, publication involves an API where a user submits their Loops, which produce a Kafka event containing the function body. As each worker has its own Kafka consumer group, they all receive the same messages.

Workers subscribe to Loops updates as Kafka consumers and maintain a Loop store, which is an embedded key (the Loop hash)/Value (the function’s current revision). In the API part, Loop hashes are used as URL parameters to identify which Loop to execute. Once called, a Loop is retrieved from the map, then injected in a V8 context, executed, and dropped. This hot code reload mechanism ensures that each Loop can be executed on every worker. We can also leverage our load balancers’ capabilities to distribute the load on the workers. This simple distribution model avoids complex scheduling and eases the maintainability of the overall infrastructure.

In order to be reboot-proof, we make use of Kafka’s very handy log compaction feature. Log compaction allows Kafka to keep the last version of each keyed message. When a user creates a new Loop, it will be given a unique ID, which is used as a Kafka message key. When a user updates a Loop, this new message will be forwarded to all consumers, but since the key already exists, only the last revision will be kept by Kafka. When a worker restarts, it will consume all messages to rebuild its internal KV, so the previous state will be restored. Kafka is used here as a persistent store.

Loops runtimes

Even if the underlying engine is able to run native Javascript, as stated above, we wanted it to run more idiomatic Time Series queries like TSL or WarpScript™. To achieve this, we created a Loops Runtime abstraction that wraps not only Javascript, but also TSL and WarpScript™ queries into Javascript code. Users have to declare a Loop with it’s runtime, after which it’s just a matter of wrappers working. For example, executing a WarpScript™ Loop involves taking the plain WarpScript™ and sending it through a node-request HTTP call.

Loops feedback

Executing code safely is a start, but when it comes to executing arbitrary code, it’s also useful to get some feedback on the execution state. Was it successful or not? Is there an error in the function? If a Loop is in a failure state, the user should be notified straight away.

This leads us to one special condition: a user’s scripts must be able to tell if everything is OK or not.  There are two ways to do that in the underlying JavaScript engine: callbacks and Promises.
We choose to go with Promises which offers a better asynchronous management. Every Loop returns a Promise at the end of the script. A rejected promise will produce an HTTP 500 error status, while a resolved one will produce an HTTP 200 status.

Loops scheduling

When publishing Loops, you can declare several triggers, in a similar way to Cron. Each trigger will perform an HTTP call to your Loop, with optional parameters.

Based on this semantic, to generate multiple reports, we can register a single function that would be scheduled with different contexts, defined by various parameters (region, rate, etc.). See the example below:

functions:
  warp_apps_by_cells:
    handler: apps-by-cells.mc2
    runtime: ws
    timeout: 30
    environment:
    events:
      - agra:
          rate: R/2018-01-01T00:00:00Z/PT5M/ET1M
          params:
            cell: ovh-a-gra
      - abhs:
          rate: R/2018-01-01T00:00:00Z/PT5M/ET1M
          params:
            cell: ovh-a-bhs

The scheduling is based on Metronome, which is an open-source event scheduler, with a specific focus on scheduling rather than execution. It’s a perfect fit for Loops, since Loops handle the execution, while relying on Metronome to drive execution calls.

Loops pipelines

A Loops project can have several Loops. One of our customers’ common use cases was having was to use Loops as a data platform, in a data flow fashion. Data flow is a way to describe a pipeline of execution steps. In a Loops context, there is a global `Loop` object, which allows the script to execute another Loop with this name. You can then chain Loop executions that will act as step functions.

Pain points: scaling a NodeJS application

Loops workers are NodeJS applications. Most of NodeJS developers know that NodeJS uses an mono-threaded event loop. If you don’t take care of the threading model of your nodeJS  application, you would likely suffer for a lack of performance, since only one host thread will be used.

NodeJS also has a cluster module available, which allows an app to use multiple threads. That’s why in a Loops worker, we start with an N-1 thread for handling API calls, where N is the total number of threads available, which leaves one dedicated to the master thread.

The master thread is in charge of consuming Kafka topics and maintaining the Loops store, while the worker thread starts an API server. For every requested Loop execution, it asks the master for the script content, and executes it in a dedicated thread.

With this setup, one NodeJS application with one Kafka consumer is started per server, which make it very easy to scale out the infrastructure, just by adding additional servers or cloud Instances.

Conclusion

In this post, we previewed Loops, a scalable, metrics-oriented FaaS with native JavaScript support, and extended WarpScript™ and TSL support.

We still have a few things to enhance, like ES5-style dependency imports and metrics previews for our customers’ Loops projects. We also plan to add more runtimes, especially WASM, which would allow many other languages that can target it, like Go, Rust or Python, to suit most developer preferences.

The Loops platform was part of a requirement to build higher-level features around OVH Observability products. It’s a first step towards offering more automated services, like metrics rollups, aggregation pipelines, or logs-to-metrics extractors.

This tool was built part of the Observability products suite with a higher abstraction level in mind, but you might also want direct access to the API, in order to implement your own automated logic for your metrics. Would you be interested in such a feature? Visit our Gitter channel to discuss it with us!

In the Prescience team, we have used Kubernetes for more than a year now. Our cluster includes 40 nodes, running on top of PCI. We continuously run about 800 pods, and generate a lot of metrics as a result.

Today, we’ll look at how we handle these metrics to monitor our Kubernetes Cluster, and (equally importantly!) how to do this with your own cluster.

OVH Metrics

Like any infrastructure, you need to monitor your Kubernetes Cluster. You need to know exactly how your nodes, cluster and applications behave once they have been deployed in order to provide reliable services to your customers. To do this with our own Cluster, we use OVH Observability.

OVH Observability is backend-agnostic, so we can push metrics with one format and query with another one. It can handle:

• Graphite
• InfluxDB
• Metrics2.0
• OpentTSDB
• Prometheus
• Warp10

It also incorporates a managed Grafana, in order to display metrics and create monitoring dashboards.

Monitoring Nodes

The first thing to monitor is the health of nodes. Everything else starts from there.

In order to monitor your nodes, we will use Noderig and Beamium, as described here. We will also use Kubernetes DaemonSets to start the process on all our nodes.

So let’s start creating a namespace…

kubectl create namespace metrics

Next, create a secret with the write token metrics, which you can find in the OVH Control Panel.

kubectl create secret generic w10-credentials --from-literal=METRICS_TOKEN=your-token -n metrics

Copy metrics.yml into a file and apply the configuration with kubectl

# This will configure Beamium to scrap noderig
# And push metrics to warp 10
# We also add the HOSTNAME to the labels of the metrics pushed
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: beamium-config
  namespace: metrics
data:
  config.yaml: |
    scrapers:
      nodering:
        url: http://0.0.0.0:9100/metrics
        period: 30000
        format: sensision
        labels:
          app: nodering

    sinks:
      warp:
        url: https://warp10.gra1.metrics.ovh.net/api/v0/update
        token: $METRICS_TOKEN

    labels:
      host: $HOSTNAME

    parameters:
      log-file: /dev/stdout
---
# This is a custom collector that report the uptime of the node
apiVersion: v1
kind: ConfigMap
metadata:
  name: noderig-collector
  namespace: metrics
data:
  uptime.sh: |
    #!/bin/sh
    echo 'os.uptime' `date +%s%N | cut -b1-10` `awk '{print $1}' /proc/uptime`
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: metrics-daemon
  namespace: metrics
spec:
  selector:
    matchLabels:
      name: metrics-daemon
  template:
    metadata:
      labels:
        name: metrics-daemon
    spec:
      terminationGracePeriodSeconds: 10
      hostNetwork: true
      volumes:
      - name: config
        configMap:
          name: beamium-config
      - name: noderig-collector
        configMap:
          name: noderig-collector
          defaultMode: 0777
      - name: beamium-persistence
        emptyDir:{}
      containers:
      - image: ovhcom/beamium:latest
        imagePullPolicy: Always
        name: beamium
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: TEMPLATE_CONFIG
          value: /config/config.yaml
        envFrom:
        - secretRef:
            name: w10-credentials
            optional: false
        resources:
          limits:
            cpu: "0.05"
            memory: 128Mi
          requests:
            cpu: "0.01"
            memory: 128Mi
        workingDir: /beamium
        volumeMounts:
        - mountPath: /config
          name: config
        - mountPath: /beamium
          name: beamium-persistence
      - image: ovhcom/noderig:latest
        imagePullPolicy: Always
        name: noderig
        args: ["-c", "/collectors", "--net", "3"]
        volumeMounts:
        - mountPath: /collectors/60/uptime.sh
          name: noderig-collector
          subPath: uptime.sh
        resources:
          limits:
            cpu: "0.05"
            memory: 128Mi
          requests:
            cpu: "0.01"
            memory: 128Mi

Don’t hesitate to change the collector levels if you need more information.

Then apply the configuration with kubectl…

$ kubectl apply -f metrics.yml
# Then, just wait a minutes for the pods to start
$ kubectl get all -n metrics
NAME                       READY   STATUS    RESTARTS   AGE
pod/metrics-daemon-2j6jh   2/2     Running   0          5m15s
pod/metrics-daemon-t6frh   2/2     Running   0          5m14s

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE AGE
daemonset.apps/metrics-daemon 40        40        40      40           40          122d

You can import our dashboard in to your Grafana from here, and view some metrics about your nodes straight away.

Kube Metrics

As the OVH Kube is a managed service, you don’t need to monitor the apiserver, etcd, or controlplane. The OVH Kubernetes team takes care of this. So we will focus on cAdvisor metrics and Kube state metrics

The most mature solution for dynamically scraping metrics inside the Kube (for now) is Prometheus.

In the next Beamium release, we should be able to reproduce the features of the Prometheus scraper.

To install the Prometheus server, you need to install Helm on the cluster…

kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller \
    --clusterrole cluster-admin \
    --serviceaccount=kube-system:tiller
helm init --service-account tiller

You then need to create the following two files: prometheus.yml and values.yml.

# Based on https://github.com/prometheus/prometheus/blob/release-2.2/documentation/examples/prometheus-kubernetes.yml
serverFiles:
  prometheus.yml:
    remote_write:
    - url: "https://prometheus.gra1.metrics.ovh.net/remote_write"
      remote_timeout: 120s
      bearer_token: $TOKEN
      write_relabel_configs:
      # Filter metrics to keep
      - action: keep
        source_labels: [__name__]
        regex: "eagle.*|\
            kube_node_info.*|\
            kube_node_spec_taint.*|\
            container_start_time_seconds|\
            container_last_seen|\
            container_cpu_usage_seconds_total|\
            container_fs_io_time_seconds_total|\
            container_fs_write_seconds_total|\
            container_fs_usage_bytes|\
            container_fs_limit_bytes|\
            container_memory_working_set_bytes|\
            container_memory_rss|\
            container_memory_usage_bytes|\
            container_network_receive_bytes_total|\
            container_network_transmit_bytes_total|\
            machine_memory_bytes|\
            machine_cpu_cores"

    scrape_configs:
    # Scrape config for Kubelet cAdvisor.
    - job_name: 'kubernetes-cadvisor'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      
      relabel_configs:
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        
      metric_relabel_configs:
      # Only keep systemd important services like docker|containerd|kubelet and kubepods,
      # We also want machine_cpu_cores that don't have id, so we need to add the name of the metric in order to be matched
      # The string will concat id with name and the separator is a ;
      # `/;container_cpu_usage_seconds_total` OK
      # `/system.slice;container_cpu_usage_seconds_total` OK
      # `/system.slice/minion.service;container_cpu_usage_seconds_total` NOK, Useless
      # `/kubepods/besteffort/e2514ad43202;container_cpu_usage_seconds_total` Best Effort POD OK
      # `/kubepods/burstable/e2514ad43202;container_cpu_usage_seconds_total` Burstable POD OK
      # `/kubepods/e2514ad43202;container_cpu_usage_seconds_total` Guaranteed POD OK
      # `/docker/pod104329ff;container_cpu_usage_seconds_total` OK, Container that run on docker but not managed by kube
      # `;machine_cpu_cores` OK, there is no id on these metrics, but we want to keep them also
      - source_labels: [id,__name__]
        regex: "^((/(system.slice(/(docker|containerd|kubelet).service)?|(kubepods|docker).*)?);.*|;(machine_cpu_cores|machine_memory_bytes))$"
        action: keep
      # Remove Useless parents keys like `/kubepods/burstable` or `/docker`
      - source_labels: [id]
        regex: "(/kubepods/burstable|/kubepods/besteffort|/kubepods|/docker)"
        action: drop
        # cAdvisor give metrics per container and sometimes it sum up per pod
        # As we already have the child, we will sum up ourselves, so we drop metrics for the POD and keep containers metrics
        # Metrics for the POD don't have container_name, so we drop if we have just the pod_name
      - source_labels: [container_name,pod_name]
        regex: ";(.+)"
        action: drop
    
    # Scrape config for service endpoints.
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
      
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    # Example scrape config for pods
    #
    # The relabeling allows the actual pod scrape endpoint to be configured via the
    # following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the
    # pod's declared ports (default is a port-free target if none are declared).
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod

      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod_name
      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: host
      - action: labeldrop
        regex: (pod_template_generation|job|release|controller_revision_hash|workload_user_cattle_io_workloadselector|pod_template_hash)

alertmanager:
  enabled: false
pushgateway:
  enabled: false
nodeExporter:
  enabled: false
server:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: traefik
      ingress.kubernetes.io/auth-type: basic
      ingress.kubernetes.io/auth-secret: basic-auth
    hosts:
    - prometheus.domain.com
  image:
    tag: v2.7.1
  persistentVolume:
    enabled: false

Don’t forget to replace your token!

The Prometheus scraper is quite powerful. You can relabel your time series, keep a few that match your regex, etc. This config removes a lot of useless metrics, so don’t hesitate to tweak it if you want to see more cAdvisor metrics (for example).

 Install it with Helm…

helm install stable/prometheus \
    --namespace=metrics \
    --name=metrics \
    --values=values/values.yaml \
    --values=values/prometheus.yaml

Add add a basic-auth secret…

$ htpasswd -c auth foo
New password: <bar>
New password:
Re-type new password:
Adding password for user foo
$ kubectl create secret generic basic-auth --from-file=auth -n metrics
secret "basic-auth" created

You can can access the Prometheus server interface through prometheus.domain.com.

You will see all the metrics for your Cluster, although only the one you have filtered will be pushed to OVH Metrics.

The Prometheus interfaces is a good way to explore your metrics, as it’s quite straightforward to display and monitor your infrastructure. You can find our dashboard here.

Resources Metrics

As @Martin Schneppenheim said in this post, in order to correctly manage a Kubernetes Cluster, you also need to monitor pod resources.

We will install Kube Eagle, which will fetch and expose some metrics about CPU and RAM requests and limits, so they can be fetched by the Prometheus server you just installed.

Create a file named eagle.yml.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app: kube-eagle
  name: kube-eagle
  namespace: kube-eagle
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - pods
  verbs:
  - get
  - list
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  labels:
    app: kube-eagle
  name: kube-eagle
  namespace: kube-eagle
subjects:
- kind: ServiceAccount
  name: kube-eagle
  namespace: kube-eagle
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-eagle
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: kube-eagle
  labels:
    app: kube-eagle
  name: kube-eagle
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kube-eagle
  name: kube-eagle
  labels:
    app: kube-eagle
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-eagle
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
      labels:
        app: kube-eagle
    spec:
      serviceAccountName: kube-eagle
      containers:
      - name: kube-eagle
        image: "quay.io/google-cloud-tools/kube-eagle:1.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: PORT
          value: "8080"
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: http
        readinessProbe:
          httpGet:
            path: /health
            port: http

$ kubectl create namespace kube-eagle
$ kubectl apply -f eagle.yml

Next, add import this Grafana dashboard (it’s the same dashboard as Kube Eagle, but ported to Warp10).

You now have an easy way of monitoring your pod resources in the Cluster!

Custom Metrics

How does Prometheus know that it needs to scrape kube-eagle? If you looks at the metadata of the eagle.yml, you’ll see that:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080" # The port where to find the metrics
  prometheus.io/path: "/metrics" # The path where to find the metrics

Theses annotations will trigger the Prometheus auto-discovery process (described in prometheus.yml line 114).

This means you can easily add these annotations to pods or services that contain a Prometheus exporter, and then forward these metrics to OVH Observability. You can find a non-exhaustive list of Prometheus exporters here.

Volumetrics Analysis

As you saw on the prometheus.yml , we’ve tried to filter a lot of useless metrics. For example, with cAdvisor on a fresh cluster, with only three real production pods, and with the whole kube-system and Prometheus namespace, have about 2,600 metrics per node. With a smart cleaning approach, you can reduce this to 126 series.

Here’s a table to show the approximate number of metrics you will generate, based on the number of nodes (N) and the number of production pods (P) you have:

For example, if you run three nodes with 60 Pods, you will generate 264 * 3 + 32 * 60 ~= 2,700 metrics

NB: A pod has a unique name, so if you redeploy a deployment, you will create 32 new metrics each time.

^{(1) Noderig metrics: os.mem / os.cpu / os.disk.fs / os.load1 / os.net.dropped (in/out) / os.net.errs (in/out) / os.net.packets (in/out) / os.net.bytes (in/out)/ os.uptime}

^{(2) cAdvisor nodes metrics: machine_memory_bytes / machine_cpu_cores}

^{(3) Kube state nodes metrics: kube_node_info}

^{(4) Kube Eagle nodes metrics: eagle_node_resource_allocatable_cpu_cores / eagle_node_resource_allocatable_memory_bytes / eagle_node_resource_limits_cpu_cores / eagle_node_resource_limits_memory_bytes / eagle_node_resource_requests_cpu_cores / eagle_node_resource_requests_memory_bytes / eagle_node_resource_usage_cpu_cores / eagle_node_resource_usage_memory_bytes}

^{(5) With our filters, we will monitor around five system.slices} 

^{(6)  Metrics are reported per container. A pod is a set of containers (a minimum of two): your container + the pause container for the network. So we can consider (2* 10 + 6) for the number of metrics per pod. 10 metrics from the cAdvisor and six for the network (see below) and for system.slice we will have 10 + 6, because it’s treated as one container.}

^{(7) cAdvisor will provide these metrics for each container: container_start_time_seconds / container_last_seen / container_cpu_usage_seconds_total / container_fs_io_time_seconds_total / container_fs_write_seconds_total / container_fs_usage_bytes / container_fs_limit_bytes / container_memory_working_set_bytes / container_memory_rss / container_memory_usage_bytes}

^{(8) cAdvisor will provide these metrics for each interface: container_network_receive_bytes_total * per interface / container_network_transmit_bytes_total * per interface}

^{(9) kube-dns / beamium-noderig-metrics / kube-proxy / canal / metrics-server} 

^{(10) Kube Eagle pods metrics: eagle_pod_container_resource_limits_cpu_cores / eagle_pod_container_resource_limits_memory_bytes / eagle_pod_container_resource_requests_cpu_cores / eagle_pod_container_resource_requests_memory_bytes / eagle_pod_container_resource_usage_cpu_cores / eagle_pod_container_resource_usage_memory_bytes}

Conclusion

As you can see, monitoring your Kubernetes Cluster with OVH Observability is easy. You don’t need to worry about how and where to store your metrics, leaving you free to focus on leveraging your Kubernetes Cluster to handle your business workloads effectively, like we have in the Machine Learning Services Team.

The next step will be to add an alerting system, to notify you when your nodes are down (for example). For this, you can use the free OVH Alert Monitoring tool.

Stay in touch

For any questions, feel free to join the Observability Gitter or Kubernetes Gitter!
Follow us on Twitter: @OVH

This data can be classified in two ways: host or application monitoring. Host monitoring is mostly based on hardware counters (CPU, memory, network, disk…) while application monitoring is based on the service and its scalability (requests, processing, business logic…).

We provide this service for internal teams, who enjoy the same experience as our customers. Basically, our Observability service is SaaS with a compatibility layer (supporting InfluxDB, OpenTSDB, Warp10, Prometheus, and Graphite) that allows it to integrate with most of the existing solutions out there. This way, a team that is used to a particular tool, or have already deployed a monitoring solution, won’t need to invest much time or effort when migrating to a fully managed and scalable service: they just pick a token, use the right endpoint, and they’re done. Besides, our compatibility layer offers a choice: you can push your data with OpenTSDB, then query it in either PromQL or WarpScript. Combining protocols in this way results in a unique open-source interoperability that delivers more value, with no restrictions created by a solution’s query capabilities.

Scollector, Snap, Telegraf, Graphite, Collectd…

Drawing on this experience, we collectively tried most of the collection tools, but we always arrived at the same conclusion: we were witnessing metrics bleeding. Each tool focused on scraping every reachable bit of data, which is great if you are a graph addict, but can be counterproductive from an operational point-of-view,  if you have to monitor thousands of hosts. While it’s possible to filter them, teams still need to understand the whole metrics set in order to know what needs to be filtered.

At OVH, we use laser-cut collections of metrics. Each host has a specific template (web server, database, automation…) that exports a set amount of metrics, which can be used for health diagnostics and monitoring application performance.

This finely-grained management leads to greater understanding for operational teams, since they know what’s available and can progressively add metrics to manage their own services.

Beamium & Noderig — The Perfect Fit

Our requirements were rather simple:
— Scalable: Monitor one node in the same way as we’d monitor thousands
— Laser-cut: Only collect the metrics that are relevant
— Reliable: We want metrics to be available even in the worst conditions
— Simple: Multiple plug-and-play components, instead of intricate ones
— Efficient: We believe in impact-free metrics collection

The first solution was Beamium

Beamium handles two aspects of the monitoring process: application data scrapping and metrics forwarding.

Application data is collected is the well-known and widely-used Prometheus format. We chose Prometheus as the community was growing rapidly at the time, and many instrumentation libraries were available for it. There are two key concepts in Beamium: Sources and Sinks.

The Sources, where Beamium will scrape data, are just Prometheus HTTP endpoints. This means it’s as simple as supplying the HTTP endpoint, and eventually adding a few parameters. This data will be routed to Sinks, which allows us to filter them during the routing process between a Source and a Sink. Sinks are Warp 10(R) endpoints, where we can push the data.

Once scraped, metrics are first stored on disk, before being routed to a Sink. The Disk Fail-Over (DFO) mechanism allows for network or remote failure recovery . This way, locally we retain the Prometheus pull logic, but decentralized, and we reverse it to push to feed the platform which has many advantages:

• support for a transactional logic over the metrics platform
• recovery from network partitioning or platform unavailability
• dual writes with data consistency (as there’s otherwise no guarantee that two Prometheus instances would scrape the same data at the same timestamp)

We have many different customers, some of whom use the Time Series store behind the Observability product to manage their product consumption or transactional changes over licensing. These use cases can’t be handled with Prometheus instances, which are better suited to metrics-based monitoring.

The second was Noderig

During conversations with some of our customers, we came to the conclusion that the existing tools needed a certain level of expertise if they were to be used at scale. For example, a team with a 20k node cluster with Scollector would end up with more than 10 million metrics, just for the nodes… In fact, depending on the hardware configuration, Scollector would generate between 350 and 1,000 metrics from a single node.

That’s the reason behind Noderig. We wanted it to be as simple to use as the node-exporter from Prometheus, but with more finely-grained metrics production as the default.

Noderig collects OS metrics (CPU, memory, disk, and network) using a simple level semantic. This allows you to collect the right amount of metrics for any kind of host, which is particularly suitable for containerized environments.

We made it compatible with Scollector’s custom collectors to ease the migration process, and allow for extensibility. External collectors are simple executables that act as providers for data that is collected by Noderig, as with any other metrics.

The collected metrics are available through a simple rest endpoint, allowing you to see your metrics in real-time, and easily integrate them with Beamium.

Does it work?

Beamium and Noderig are extensively used at OVH, and support the monitoring of very large infrastructures. At the time of writing, we collect and store hundreds of millions of metrics using these tools. So they certainly seem to work!

In fact, we’re currently working on the 2.0 release, which will be a rework, incorporating autodiscovery and hot reload.

Stay in touch

For any questions, feel free to join our Gitter!
Follow us on Twitter: @OVH