For years, the Kubernetes Ingress API, and the popular Ingress NGINX controller (ingress-nginx), have been the default way to expose applications running inside a Kubernetes cluster.

But the ecosystem is changing: the Kubernetes SIG network has announced the retirement of Ingress NGINX in March 2026.

After March 2026 the Ingress NGINX will no longer get new features, new releases, security patches and bug fixes.

Furthermore, the Kubernetes project recommends using Gateway instead of Ingress.

The Ingress API has already been frozen, which means it is no longer being developed, and will have no further changes or updates made to it. The Kubernetes project has no plans to remove Ingress from Kubernetes.

While OVHcloud Managed Kubernetes Service (MKS) does not yet provide a native GatewayClass, you can already benefit from Gateway API capabilities today by deploying your own controller 💪 .

Also, until Gateway API becomes fully integrated with OpenStack providers, there is an intermediate option: using a modern, actively maintained Ingress controller other than ingress-nginx.

The limitations of the current Ingress controller model

The traditional Kubernetes Ingress model was intentionally simple: define an Ingress, install an Ingress Controller, and let it configure a single proxy (usually Nginx) to route traffic.

This design works, but it comes with limitations:

– Single Monolithic “Entry Point”: All HTTP routing for the entire cluster goes through one shared proxy. It adds complexity, configuration conflicts and scaling challenges.
– Protocol limitations: only HTTP and HTTPS.Support for gRPC, HTTP/2, TCP, UDP or TLS passthrough is inconsistent and controller-specific.
– Heavy Reliance on Annotations: Advanced features (timeouts, rewrites, header handling…) rely on custom annotations.
– Strong 3rd parties and cloud Load Balancers support: Every Ingress controllers (3rd parties providers) come with their specialized annotations.

Finally, as mentioned, the most used Ingress controller, Ingress NGINX, will be retired in March 2026.

A Transitional Solution: Using a Modern Ingress Controller (Traefik, Contour, HAProxy…)

Before moving to the Gateway API, as a transitional solution, OVHcloud MKS users can simply replace Ingress Nginx with a modern, actively maintained Ingress controller.

This allows you to:

– keep using your existing Ingress manifests
– keep the same architecture: Service type LoadBalancer → OVHcloud Public Cloud Load Balancer → Ingress Controller
– avoid relying on unsupported or deprecated components
– gain features (better gRPC support, built‑in dashboards, improved L7 behaviour…)

Popular alternatives:

Traefik:
– Very easy to deploy
– Excellent support for HTTP/2, gRPC, WebSockets
– Built‑in dashboard
– Supports both Ingress and Gateway API
– Actively maintained
– Seamless migration from NGINX Ingress Controller to Traefik with NGINX annotation compatibility

Contour (Envoy):
– Envoy-based Ingress Controller
– Excellent performance
– Good stepping‑stone toward Gateway API

HAProxy Ingress:
– Extremely performant
– Enterprise-grade L7 routing
– Optional Gateway API support

NGINX Gateway Fabric (NGF):
– The successor to Ingress NGINX
– Built directly around Gateway API
– Still maturing but a strong long‑term candidate

If you are interested, you can read the more exhaustive list of Ingress controllers.

Installing an Alternative Ingress Controller on OVHcloud MKS

We will show you how to install Traefik, as an alternative Ingress controller and use it to spawn a single OVHcloud Public Cloud Load Balancer (based on OpenStack Octavia).

Install Traefik:

helm repo add traefik https://traefik.github.io/charts
helm repo update

helm install traefik traefik/traefik --namespace traefik --create-namespace --set service.type=LoadBalancer

This automatically triggers:
– the OpenStack CCM (used by OVHcloud)
– the creation of an OVHcloud Public Cloud Load Balancer
– exposure of Traefik through a public IP

After several seconds, the Load Balancer will be active.

Check that Traefik is running:

$ kubectl get all -n traefik
NAME                           READY   STATUS    RESTARTS   AGE
pod/traefik-6777c5db85-pddd6   1/1     Running   0          31s

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/traefik   LoadBalancer   10.3.129.188   <pending>     80:30267/TCP,443:30417/TCP   31s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/traefik   1/1     1            1           31s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/traefik-6777c5db85   1         1         1       31s

Then in order to use it, create an ingress.yaml file with the following content:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "traefik"  # Specifies Traefik as the ingress controller
spec:
  rules:
    - host: my-app.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-service
                port:
                  number: 80

And apply it in your cluster:

kubectl apply -f ingress.yaml

Using this type of alternative provides a fully supported, modern Ingress Controller while you prepare a long‑term transition to the Gateway API.

Gateway API: A modern, flexible networking model

The Gateway API is the next-generation Kubernetes networking specification. It introduces clearer roles and more flexible architectures.

Gateway API splits responsibilities across:
– GatewayClass: defines the type of gateway and which controller manages it
– Gateway: the actual entry point (e.g., a Load Balancer)
– Routes: routing rules, protocol-specific (HTTPRoute, TLSRoute, GRPCRoute, TCPRoute…)

Gateway API supports:
– HTTP(S)
– HTTP/2
– gRPC
– TCP
– TLS passthrough
…in a consistent and portable way.

Unlike Ingress, Gateway API is explicitly designed to allow providers like OVHcloud, AWS, GCP, Azure to:
– provision Load Balancers (LB)
– manage listeners
– expose multiple ports
– integrate with their LB features
This paves the way for native OVHcloud GatewayClass support.

How does it work today on OVHcloud MKS?

OVHcloud MKS relies on the OpenStack Cloud Controller Manager (CCM) to provision OVHcloud Public Cloud Load Balancers in response to a Service of type LoadBalancer.

Since MKS does not yet include a native GatewayClass, you can use Gateway API today as follows:

1. You deploy an existing Gateway Controller (Envoy Gateway, Traefik, Contour/Envoy…) and its GatewayClass.
2. The controller deploys a Data Plane proxy inside the cluster.
3. To expose that proxy, you still have to create a Service of type LoadBalancer (and your app of course).
4. The CCM provisions an OVHcloud Public Cloud Load Balancer and forwards traffic to your proxy.

Thanks to that, you will have a fully functional Gateway API. The workflow is very similar to that which is required for using NGINX Ingress controller.

Using the Gateway API on OVHcloud MKS today

You can already use the Gateway API by deploying your preferred controller.

Here’s an example using Envoy Gateway, one of the most future-proof options.

Install Gateway API CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/latest/download/standard-install.yaml

Deploy Envoy Gateway:

helm install eg oci://docker.io/envoyproxy/gateway-helm -n envoy-gateway-system --create-namespace

You should have a result like this:

$ helm install eg oci://docker.io/envoyproxy/gateway-helm -n envoy-gateway-system --create-namespace

Pulled: docker.io/envoyproxy/gateway-helm:1.6.0
Digest: sha256:5c55e7844ae8cff3152ca00330234ef61b1f9fa3d466f50db2c63a279f1cd1df
NAME: eg
LAST DEPLOYED: Mon Dec  1 16:27:07 2025
NAMESPACE: envoy-gateway-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
**************************************************************************
*** PLEASE BE PATIENT: Envoy Gateway may take a few minutes to install ***
**************************************************************************

Envoy Gateway is an open source project for managing Envoy Proxy as a standalone or Kubernetes-based application gateway.

Thank you for installing Envoy Gateway! 🎉

Your release is named: eg. 🎉

Your release is in namespace: envoy-gateway-system. 🎉

To learn more about the release, try:

  $ helm status eg -n envoy-gateway-system
  $ helm get all eg -n envoy-gateway-system

To have a quickstart of Envoy Gateway, please refer to https://gateway.envoyproxy.io/latest/tasks/quickstart.

To get more details, please visit https://gateway.envoyproxy.io and https://github.com/envoyproxy/gateway.

Check the Envoy gateway is running:

$ kubectl get po -n envoy-gateway-system
NAME                            READY   STATUS    RESTARTS   AGE
envoy-gateway-9cbbc577c-5h5qw   1/1     Running   0          16m

As a quickstart, you can install directly the GatewayClass, Gateway, HTTPRoute and an example app:

kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default

This command deploys a GatewayClass, a Gateway, a HTTPRoute and an app deployed in a deployment and exposed through a service:

gatewayclass.gateway.networking.k8s.io/eg created
gateway.gateway.networking.k8s.io/eg created
serviceaccount/backend created
service/backend created
deployment.apps/backend created
httproute.gateway.networking.k8s.io/backend created

As you can see, a GatewayClass have been deployed:

$ kubectl get gatewayclass -o yaml | kubectl neat
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: GatewayClass
  metadata:
    name: eg
  spec:
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
kind: List
metadata:
  resourceVersion: ""

Note that a GatewayClass is a cluster-wide resource so you don’t have to specify any namespace.

A Gateway have been deployed also:

$ kubectl get gateway -o yaml -n default | kubectl neat
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: Gateway
  metadata:
    name: eg
    namespace: default
  spec:
    gatewayClassName: eg
    listeners:
    - allowedRoutes:
        namespaces:
          from: Same
      name: http
      port: 80
      protocol: HTTP
kind: List
metadata:
  resourceVersion: ""

A HTTPRoute also:

$ kubectl get httproute -o yaml -n default | kubectl neat
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata:
    name: backend
    namespace: default
  spec:
    hostnames:
    - www.example.com
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: backend
        port: 3000
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /
kind: List
metadata:
  resourceVersion: ""

In order to retrieve the external IP (of the external Load Balancer), you just have to get information about the Gateway and export it in an environment variable:

$ kubectl get gateway eg
NAME   CLASS   ADDRESS        PROGRAMMED   AGE
eg     eg      xx.xxx.xx.xxx   True        18m

$ export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

$ echo $GATEWAY_HOST
xx.xxx.xx.xxx

And finally, a backend service have been deployed with its deployment:

$ kubectl get pod,svc -l app=backend -n default
NAME                           READY   STATUS    RESTARTS   AGE
pod/backend-765694d47f-zr6hh   1/1     Running   0          21m

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/backend   ClusterIP   10.3.114.179   <none>        3000/TCP   21m

In order to create your own Gateway and *Route resources, don’t hesitate to take a look at the Gateway API website.

Conclusion

Two migration paths are currently available for OVHcloud MKS users:

• Short-term: switch to a modern Ingress Controller (Traefik, Contour, HAProxy, NGF…). It provides full support for current Ingress usage, without requiring API changes.
• Long-term: adopt the Gateway API. Gateway API brings multi‑protocol support, clearer separation of roles, and is the strategic direction of Kubernetes networking.

Which approach and which tool should you choose? Well, it’s up to you, depending on your use cases, your teams, your needs… 🙂

As we have seen in this blog post, OVHcloud MKS users can begin adopting these technologies today, safely and incrementally.

This ecosystem is evolving quickly, so stay tuned to find out about the coming release of a pre-installed official GatewayClass (based on OpenStack Octavia) 💪.

For the past few months, Docker has been announcing the implementation of new pull rate limits for the Docker Hub. The most significant change is the 10 pulls-per-hour limit, per IP address, for unauthenticated users that can quickly lead to a “You have reached your pull rate limit” error message.

Even if these changes have been implemented and rollbacked as of April 1, 2025, at OVHcloud, we are aware that these upcoming changes could impact your deployments and daily work.

In this blog post, you will find several solutions and best practices that can help you reduce Docker pull commands and avoid hitting Docker Hub’s pull rate limit.

Use OVHcloud Managed Private Registry and activate the proxy cache

OVHcloud Managed Private Registry (MPR) is a container image registry, based on CNCF project Harbor. It allows you to store and manage Docker (or OCI-compliant) container images and artifacts in a private, secure, and scalable environment, hosted in OVHcloud’s infrastructure.

MPR provides a proxy cache feature that helps you mirror and cache images from external registries, like Docker Hub, Github Container Registry, Quay, JFrog Artifactory Registry, etc. External registries can be private or public. This improves performance and reduces rate limits imposed by external registries 💪.

Configure proxy cache in OVHcloud Managed Private Registry

If you don’t have deployed a MPR yet, you can deploy it through the OVHcloud Control Panel, the OVHcloud Terraform provider, the OVHcloud Pulumi provider and even the API. Follow the guide according to your needs.

First, log in the Harbor user interface on your private registry, follow the guide if you needed to.

⚠️ In order to activate the proxy cache, you need to log in the Harbor UI with an administrator account.

Registry endpoint creation

In the left sidebar, click on Registries (inside the Administration section).

Then click on the New endpoint button.

Select Docker Hub in the provider list, enter a name (“Docker Hub” for example), fill your Docker Hub login in Access ID field and fill your Docker Hub password in Access Secret field.

⚠️ Note that we strongly recommend using a Docker account (even a free one) to avoid rate limits, for unanthenticated users, when pulling images. Without authentication, Docker Hub enforces strict pull limits, which may cause failures when pulling frequently used images.

Click on the Test connection button to test if your login and password are correct.

Now click on the OK button in order to create the new endpoint.

The Docker Hub endpoint is created 🎉

Proxy cache project creation

In the left sidebar, click on Projects, then click on the New project button.

Enter a project name (“docker-hub” for example), enable the Proxy Cache, click on the Docker Hub endpoint in the list and click on the OK button.

ℹ️ Note that a project is private by default, so you have to click on the Public checkbox if you want to change the visibilty of a project.

⚠️ The name of a proxy cache project should not contains dot(s), indeed it can causes issues with external tools like Kaniko.

Your proxy cache project have been created 🎉

⚠️ A proxy cache project works similarly to a normal Harbor project, except that you are not able to push images to a proxy cache project.

Now when you want to pull a Docker images hosted in the Docker Hub you proxy cached, instead of pulling directly from the Docker Hub, you need to configure your docker/podman pull commands and Kubernetes pod manifests to pull images from the OVHcloud Managed Private Registry:

$ docker pull xxxxxxxx.c1.de1.container-registry.ovh.net/docker-hub/ovhcom/ovh-platform-hello:latest
latest: Pulling from docker-hub/ovhcom/ovh-platform-hello
1f3e46996e29: Pull complete 
6aa905c35cc0: Pull complete 
Digest: sha256:fddb76f0eb92d95b3721bfa0ea87350c5d39ea262e90cd30d66f429bb40c8b07
Status: Downloaded newer image for xxxxxxxx.c1.de1.container-registry.ovh.net/docker-hub/ovhcom/ovh-platform-hello:latest
xxxxxxxx.c1.de1.container-registry.ovh.net/docker-hub/ovhcom/ovh-platform-hello:latest

Disable the AlwaysPullImages admission plugin on your MKS cluster

By default, the AlwaysPullImages Kubernetes admission plugin is enabled in your OVHcloud Managed Kubernetes (MKS) cluster.

⚠️ When it is enabled, this forces the imagePullPolicy of a container to be set to Always, no matter how it is specified when creating the resource.

This is useful in a multitenant cluster so that users can be assured that their private images can only be used by those who have the credentials to pull them. Without this admission controller, once an image has been pulled to a node, any pod from any user can use it by knowing the image’s name (assuming the Pod is scheduled onto the right node), without any authorization check against the image.

But, it can cause a lot of pull requests to the Docker Hub and you can reach the rate limits.

So a solution can be to deactivate the AlwaysPullImages admission plugin in your MKS cluster.

In this blog post, we will deactivate it in the OVHcloud Control Panel.

Enable/Disable MKS admission plugins

Log in the OVHcloud Control Panel. In the left sidebar, click on the Managed Kubernetes Service and then click on the wanted MKS cluster.

In the Cluster information section, scroll down and click on Enable/disable plugin. A popup will appear.

Then click on Disable for the Always Pull Images plugin and click on the Save button.

⚠️ Any changes on the Admission plugins require a redeployment of the MKS cluster API server (without data loss) so the API server can be potentially not available during the redeployment.

Conclusion

To learn more about how to use and configure OVHcloud private registries and OVHcloud MKS clusters, don’t hesitate to follow our guides.

In this blog post we will see what is MRS and how to create several different Kubernetes clusters through the Rancher UI.

Managed Rancher Service

Managed Rancher Services (MRS), in General Availability since September 2024, is based on Rancher, an open-source container management platform, that simplifies the deployment and management of Kubernetes clusters. Managed Rancher Service by OVHcloud provides a powerful platform for orchestrating Kubernetes clusters seamlessly.

With the Managed Rancher Service it becomes easy to manage and create multiple Kubernetes clusters on any platform and location including:

• Hosted Kubernetes provider (e.g. OVHcloud MKS, AWS EKS, GCP GKE, etc).
• Infrastructure Provider – Public Cloud or Private Cloud (vSphere, Nutanix, etc).
• Bare-metal servers, cloud hosted or on premise.
• Virtual machines, cloud hosted or on premise

You can also import your existing Kubernetes clusters and then manage them for a multi-cloud purpose:

Find more information on our dedicated Managed Rancher Services page.

How To

Through this blog post we will show how to create several Kubernetes clusters:

• a Managed Kubernetes (MKS) cluster (Hosted Kubernetes provider)
• a Kubernetes cluster running on OVHcloud Public Cloud Compute Instances (PCI) (Infrastructure Provider)
• a K3s Kubernetes cluster using existing nodes (Custom driver)

Create an OVHcloud Managed Kubernetes (MKS) cluster

In this part of this blog post, we will create a MKS cluster with 3 nodes based on b3-8 flavor:

Log in to your Managed Rancher Service UI and then click on Create button.

To create an MKS cluster, use the hosted Kubernetes provider way and click on the OVHcloud MKS driver.

First, enter an MKS cluster name, for example my-rancher-mks-cluster:

At this step, you can optionally configure Member Roles or Labels & Annotations, follow our guide to know more.

For the Account Configuration, you need to provide your OVHcloud API credentials (Application Key, Application Secret and Consumer Key). If you don’t have OVHcloud API credentials, you can follow our guide on how to Generate your OVHcloud API keys.

Also provide your Public Cloud project ID. The project ID is where your Managed Kubernetes Service (MKS) cluster will be deployed. You can follow the guide on How to create your first Project or if already existing, you can copy/paste it from the OVHcloud Control Panel or API.

And finally select the OVHcloud API endpoint, depending on your location: ovh-eu, ovh-ca or ovh-us.

For the Cluster Configuration, you need to select the Region where your cluster will be deployed. Then, select the Kubernetes Version. Then, select Update Policy information. If you want further information, refer to the Managed Kubernetes Update Policies guide.

For the Network Configuration, in the Private Network ID field, select an existing OVHcloud Public Cloud private network or choose None if you want to create a cluster with nodes using only public interfaces.

For the NodePools Configuration, for every NodePool you want to:

• Enter the Name of the NodePool. The name must be unique inside a same MKS cluster.
• Choose an OVHcloud instance Flavor used by this NodePool.
• Enable or disable the Autoscaling.
• Enter the number of nodes you want, it’s the Size of your NodePool. If the autoscaling is enabled, then choose the minimum and maximum number of nodes.
• Enable the Monthly Billing (Hourly billing by default).
• Click on the Add Node Pool button to add the node pool in the list below.

Click on the Finish & Create Cluster button.

Your MKS cluster is provisioning, the creation will take around 3-4 minutes for the cluster creation and 3-4 minutes for the node pool with 3 nodes and the Rancher agent deployed into them.

Create a Kubernetes cluster based on OVHcloud Public Cloud Compute Instances 

In this part of this blog post, we will create a Kubernetes cluster with 3 nodes based on b3-16 flavor for etcd & control-plane & 2 nodes based on b3-8 flavor for workers.

In the Rancher UI, you have to first create OVHcloud Public Cloud credentials.

Then, go back to the Rancher UI Home and click on the Create button.

This time, you will create a Kubernetes cluster running in Compute Instances, so you have to provision new nodes and create a cluster using RKE2/K3s through the Infrastructure provider and specifically the OVHcloud Public Cloud driver:

Select the OVHcloud Public Cloud credential created earlier in this blog post:

Then, define the cluster name, my-rancher-k8s-pci for example.

In the Machine Pools section you will configure your cluster. When you configure a machine pool in Rancher, there are three roles that can be assigned to nodes: etcd, Control Plane and Worker.

Note:
In Rancher when you configure a node, there are three roles that can be assigned to nodes: etcd, controlplane and worker.

There are some good practices:

• At least 3 machines/nodes with the role etcd are needed to survive a loss of 1 node and have a minimum high availability configuration for etcd. 3 etcd nodes are generally sufficient for smaller and medium clusters, and 5 etcd nodes for large clusters.
• At least 2 machines/nodes with the role Control Plane for master component high availability.
• You can set both the etcd and Control Plane roles for one instance.
• The Worker role should not be used or added to nodes with the etcd or Control Plane role.
• At least 2 machines/nodes with the Worker role for workload rescheduling upon node failure.

For each of the machine pools, you have to:

• Define the pool name (node-pool-1 for example for the first machine pool).
• Define machine count (3 for example for the first machine pool).
• Select roles (check etcd and Control Plane for the first machine pool)/
• Choose the region (GRA11 for example for the first machine pool). If you want to check the availability of specific products that you plan to use alongside Kubernetes, you can refer to the Availability of Public Cloud Product page.
• Choose the flavor (b3-16 for example). You can refer to the OVHcloud Flavor list.
• Choose the image for the Operating System (OS) used for your machines/nodes. Please refer to Rancher Operating Systems and Container Runtime Requirements.
• Choose a Key Pair (optional). It’s the SSH Key Pair that will be used to access your nodes. Please refer to this guide on how to create a SSH KeyPair and add it to your Public Cloud project. If you leave this field empty, a new keypair will be generated automatically.
• Choose the Security Group that will be applied to created instances. You can leave the field empty.
• Choose the Availability Zone (only nova is supported at the moment).
• Choose the Floating IP Pools (only Ext-Net is supported at the moment).
• Choose the Networks. You need to choose a private network (with a gateway). The compute instances will be created in this private network.

At the bottom of the Machine Pools section, click on the + button to add the second machine pool with 2 workers machines/nodes and the same configuration.

As you can see, we can choose another flavor type for worker machines/nodes.

In the Cluster Configuration section, choose the Kubernetes version. You need to choose between RKE2 and K3s Kubernetes Operating System (OS). For a production environment, we recommend choosing RKE2.

You need also to choose the container network (CNI), we decided to choose CIlium for this blog post but you can select calico or canal instead depending on your needs.

Select the Container Network, choose if you want to activate a Project Network isolation and the System Services tooling you want to install in your cluster.

Follow the RKE2 cluster configuration reference for the Cluster Configuration.

In the Member Roles tab, you can add members for users that need to access the cluster. After creating the cluster, you can also add members.

Finally, click the Create button to create your Kubernetes cluster with OVHcloud PCI driver.

The cluster creation can take several minutes (depending on the OS and on the number of nodes you want).

Create a Kubernetes cluster with existing nodes

Another possibility through MRS is to create a Kubernetes cluster based on existing nodes. You can bring your own nodes and create a Kubernetes cluster running on them 🙂

For that the pre-requisite is to have existing machines (virtual or physical) accessible through SSH.

In the Rancher UI create on the Create button, scroll down and select the Custom driver:

Fill in a cluster name (custom-kube-cluster for example) and choose the Kubernetes version You can choose between K3s and RKE2. For production needs we recommend RKE2. And choose the container network (calico by default).

Click on the different tabs to configure your cluster depending on your needs and then click on the Create button.

As we already said in the previous chapter, in Rancher, there are three roles that can be assigned to nodes: etcd, Control Plane and Worker.

For the configuration of our etcd + Control Plane nodes, check only the etcd and Control Plane Nodes Roles:

SSH to your machines/nodes you created for etcd and Control Plane and copy/paste the registration command.

ssh xxxxx@xxx.xxx.xxx.xxx

curl -fL https://xxxxxx.xxxx.rancher.ovh.net/system-agent-install.sh | sudo  sh -s - --server https://xxxxxx.xxxx.rancher.ovh.net --label 'cattle.io/os=linux' --token z2r458coqudhfilgdsifgdsqilgfqsdigfidsufgoisdnvzj --etcd --controlplane

For the configuration of our Worker nodes, uncheck the checkboxes and check only the Worker checkbox:

SSH to your machines/nodes you created for etcd and controlpane and copy/paste the registration command.

ssh xxxxx@xxx.xxx.xxx.xxx

curl -fL https://xxxxxx.xxxx.rancher.ovh.net/system-agent-install.sh | sudo  sh -s - --server https://xxxxxx.xxxx.rancher.ovh.net --label 'cattle.io/os=linux' --token z2r458coqudhfilgdsifgdsqilgfqsdigfidsufgoisdnvzj --worker

After executing these commands to the machines/nodes, wait until the cluster is in Active state in the Rancher UI.

Conclusion

Managed Rancher Service can help you to create, import and manage your new and existing Kubernetes clusters with a centralised interface. You saw in this blog posts three ways to create a Kubernetes clusters but we encourage you to test the other possibilities and explore the Rancher UI.

Want to go further?

Visit our technical guides and how to about OVHcloud Managed Rancher Service.

The challenge behind a fully managed service

Readers unfamiliar with our orchestration service and/or GPUs may be surprised that we did not yet offer this integration in general availability. This lies in the fact that our team is focused on providing a totally managed experience, including patching the OS (Operating System) and Kubelet of each Node each time it is required. To achieve this goal, we have built and maintained a single hardened image for the dozens of flavors, in each of the 10+ regions.
Based on the experience of selected beta users, we found that this approach doesn’t always work for use cases that require a very specific NVIDIA driver configuration. Working with our technical partners at NVIDIA, we found a solution to leverage GPUs is a simple way that allows fine tuning such as the CUDA configuration for example.

NVIDIA to the rescue

This Keep-It-Simple-Stupid (KISS) solution relies on the great work of NVIDIA building and maintaining an official NVIDIA GPU operator. The Apache 2.0 licensed software uses the operator framework within Kubernetes. It does this to automate the management of all NVIDIA software components needed to use GPUs, such as NVIDIA drivers, Kubernetes device plugin for GPUs, and others.

We ensured it was compliant with our fully maintained Operating System (OS), based on a recent Ubuntu LTS version. After testing it, we documented how to use it on our Managed Kubernetes Service. We appreciate that this solution leverages an open source software that you can use on any compatible NVIDIA hardware. This allows you to guarantee consistent behavior in hybrid or multicloud scenarios, aligned with our SMART motto.

Here is an illustration describing the shared responsibility model of the stack:

All our OVHcloud Public customers can now leverage the feature, adding a GPU node pool to any of their existing or new clusters. This can be done in the regions where both Kubernetes and T1 or T2 instances are available: GRA5, GRA7 and GRA9 (France), DE1 (Germany) (available in the upcoming weeks) and BHS5 (Canada) at the date this blog post is published.
Note that GPUs worker nodes are compatible with all features released, including vRack technology and cluster autoscaling for example.

Having Kubernetes clusters with GPU options means deploying typical AI/ML applications, such as Kubeflow, MLFlow, JupyterHub, NVIDIA NGC is easy and flexible. Do not hesitate to discuss this feature with other Kubernetes users on our Gitter Channel. You may also have a look to our fully managed AI Notebook or AI training services for even simpler out-of-the box experience and per-minute pricing!

Kubernetes is in constant evolution and amelioration, every new version brings a lot of new feature and fixes. The 1.19 is not the exception:

Finally, we have arrived with Kubernetes 1.19, the second release for 2020, and by far the longest release cycle lasting 20 weeks in total. It consists of 34 enhancements: 10 enhancements are moving to stable, 15 enhancements in beta, and 9 enhancements in alpha.

So what’s new in Kubernetes 1.19?

You can find the whole change list in the official release notes, and a more casual reading version on the Kubernetes blog, but in this post we want to have a look to some of the (in our highly subjective point of view) major themes in this release:

Ingress graduates to General Availability

Ingress is a Kubernetes API object that manages external access to the services in a cluster, typically HTTP. It may provide load balancing, SSL termination and name-based virtual hosting.

Ingress is nowadays a central feature of Kubernetes, widely used in production. And surprisingly enough, it was still officially a beta. By graduating it to GA, the Kubernetes community acknowledges its importance and its de facto standard status. And of course, it opens the way to working on an Ingress v2, or some extensions, to give it even more features.

To get an overview of what’s Ingress and how does it compares (and interacts with) other ways to get external traffic into your Kubernetes cluster, you can read our Getting external traffic into Kubernetes – ClusterIp, NodePort, LoadBalancer, and Ingress post.

An you can of course find more informations about Ingress at the official documentation.

Avoiding permanent beta API Versions

Ingress isn’t the only API that have been in beta status for ages, in fact it’s only one of the numerous examples of a semi-permanent beta status for new APIs, where widely used features remain as beta release after release even if they are considered production-ready by most users.

From Kubernetes v1.20 and onwards, SIG community have decided of a new policy is to avoid features staying in beta for a long time: when a new feature’s API reaches beta, that starts a countdown. The beta-quality API now has three releases (about nine calendar months) to either:

• reach general availability (GA), and deprecate the beta, or
• have a new beta version (and deprecate the previous beta).

More informations in this Kubernetes blog post.

Increase Kubernetes support window to one year

Until this release 1.19, minor versions of Kubernetes had a support window of nine months, from Kubernetes 1.19 onwards, the support window for minor releases increases to one year.

As a Managed Kubernetes provider, we are particularly happy of this change, as our observations fully align with the working group (WG) Long Term Support (LTS) showing that a significant proportion of Kubernetes users didn’t upgrade in the 9-month support period.

Other APIs graduate to GA, and some new betas

Some other Kubernetes APIs that been upgraded to general availability, and the old API endpoints have been deprecated:

• apiextensions.k8s.io/v1beta1 -> apiextensions.k8s.io/v1
• apiregistration.k8s.io/v1beta1 -> apiregistration.k8s.io/v1
• authentication.k8s.io/v1beta1 -> authentication.k8s.io
• authorization.k8s.io/v1beta1 -> authorization.k8s.io/v1
• coordination.k8s.io/v1beta1 -> coordination.k8s.io/v1

Some APIs also get a new beta version:

• autoscaling/v2beta1 -> autoscaling/v2beta2

Better Logging

Dealing with logs in Kubernetes has always been a tricky task. As there was no uniform structure neither for log messages in the Kubernetes control plane nor for the reference to Kubernetes objects in the logs, automating the log management relied in ad-hoc solutions and in fine on dreaded regular expressions… In consequence, building analytical solutions using those logs, was not only complicated to develop but also hard to maintain.

Version 1.19, the  SIG-instrumentation begins the migration towards a new structured log paradigm. The migration is ongoing, not all the logs are structured yet, so you still should have to handle the unstructured log messages. But the promise is that when the migration will be done, most common logs will be more easily queryable, with standardized log messages and references to Kubernetes objects.

You can get more information in the corresponding KEP (Kubernetes Enhancement Proposal) and in the system logs section of the documentation.

Did you know ?

Deprecated detection

In Kubernetes 1.19, SIG API Machinery has implemented deprecation warning on Kubernetes APIs. From this version onwards, making an API request to a deprecated REST API endpoint returns a Warning header in the response and a deprecation annotation on the audit event associated to the API call, and some metrics.

The warning includes details about the release in which the API will no longer be available, and the replacement API version. The idea is to inform both the end-user and the cluster administrator when deprecated APIs are used in the cluster.

To help you to upgrade those deprecated APIs, kubectl may warn you when you’re using them. the resource is deprecated. Simply use the –raw or --warnings-as-errors flag in your kubectl calls:

~$ kubectl get --raw /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions > /dev/null
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition

More information on this Kubernetes blog post.

Ecosystem

The Certified Kubernetes Security Specialist (CKS) coming in November! CKS focuses on cluster & system hardening, minimizing microservice vulnerabilities and the security of the supply chain.

Community

Kubernetes.dev, a Kubernetes contributor focused website has been launched. It brings the contributor documentation, resources and project event information into one central location.

Some useful links

https://kubernetes.io/docs/setup/release/notes/

https://www.kubernetes.dev/

https://www.cncf.io/blog/2020/07/15/certified-kubernetes-security-specialist-cks-coming-in-november/

With approximately 100,000 weekly users of the mybinder.org public deployment and 3,000 unique git repositories hosting Binder badges, the need for more resources and computing time was felt.

Today, we are thrilled to announce that OVH is now part of the world-wide federation of BinderHubs powering mybinder.org. All traffic to mybinder.org is now split between two BinderHubs – one run by the Binder team, and another run on OVH infrastructure.

So for those who don’t already know mybinder.org, here’s a summary.

What is Jupyter?

Jupyter is an awesome open-source project that allows users to create, visualise and edit interactive notebooks. It supports a lot of popular programming languages such as Python, R and Scala as well as some presentation standards such as markdown, code snippet, charts visualisation…

Example of a local Jupyter Notebook reading a notebook inside the OVH GitHub repository prescience client.

The main use case is the ability to share your work with tons of people,  who can try, use and edit the work directly from their web browser.

Many researchers and professors are now able to work remotely on the same projects, without any infrastructure or environment issues. It’s a major improvement for communities.

Here is for example a notebook (Github project) allowing you to use Machine Learning,  from dataset ingestion to classification:

Example of a Machine Learning Jupyter Notebook

What is JupyterHub?

JupyterHub is an even more awesome open-source project bringing the multi-user feature for Jupyter notebooks. With several pluggable authentication mechanisms (ex: PAM, OAuth), it allows Jupyter notebooks to be spawned on the fly from a centralised infrastructure. Users can then easily share their notebooks and access rights with each other. That makes JupyterHub perfect for companies, classrooms and research labs.

What is BinderHub?

Finally, BinderHub is the cherry on the cake: it allows users to turn any Git repository (such as GitHub) into a collection of interactive Jupyter notebooks with only one click.

Landing page of the binder project

The Binder instance deployed by OVH can be accessed here.

• Just choose a publicly accessible git repository (better if it already contains some Jupyter notebooks).
• Copy the URL of a chosen repository into the correct binder field.
• Click the launch button.
• If it is the first time that binder sees the repository you provide, you will see compilation logs appear. Your repository is being analysed and prepared for the start of a related Jupyter notebook.
• Once the compilation is complete you will be automatically redirected to your dedicated instance.
• You can then start interacting and hacking inside the notebook.
• On the initial binder page you will see a link to share your repository with others.

How does it work?

Tools used by BinderHub

BinderHub connects several services together to provide on-the-fly creation and registry of Docker images. It uses the following tools:

• A cloud provider such as OVH.
• Kubernetes to manage resources on the cloud
• Helm to configure and control Kubernetes.
• Docker to use containers that standardise computing environments.
• A BinderHub UI that users can access to specify Git repos they want built.
• BinderHub to generate Docker images using the URL of a Git repository.
• A Docker registry that hosts container images.
• JupyterHub to deploy temporary containers for users.

What happens when a user clicks a Binder link?

After a user clicks a Binder link, the following chain of events happens:

1. BinderHub resolves the link to the repository.
2. BinderHub determines whether a Docker image already exists for the repository at the latest reference (git commit hash, branch, or tag).
3. If the image doesn’t exist, BinderHub creates a build pod that uses repo2docker to:
   • Fetch the repository associated with the link.
   • Build a Docker container image containing the environment specified in configuration files in the repository.
   • Push that image to a Docker registry, and send the registry information to the BinderHub for future reference.
4. BinderHub sends the Docker image registry to JupyterHub.
5. JupyterHub creates a Kubernetes pod for the user that serves the built Docker image for the repository.
6. JupyterHub monitors the user’s pod for activity, and destroys it after a short period of inactivity.

A diagram of the BinderHub architecture

How we deployed it?

Powered by OVH Kubernetes

One great thing about the Binder project is that it is completely cloud agnostic, you just need a Kubernetes cluster to deploy on.

Kubernetes is one of the best choices to make when it comes to scalability on a micro-services architecture stack. The managed Kubernetes solution is powered by OVH’s Public Cloud instances. With OVH Load Balancers and integrated additional disks, you can host all types of workloads, with total reversibility.

To this end, we used 2 services in the OVH Public Cloud:

• A Kubernetes Cluster today consuming 6 nodes of C2-15 VM instances (it will grow in the future)
• A Docker Registry

We also ordered a specific domain name so that our binder stack could be publicly accessible from anywhere.

Installation of HELM on our new cluster

Once the automatic installation of our Kubernetes cluster was complete we downloaded the administration YAML file allowing us to manage our cluster and to launch kubectl commands on it.

kubectl is the official and most popular tool used to administrate Kubernetes cluster. More information about how to install it can be found here.

The automatic deployment of the full Binder stack is already prepared in the form of Helm package. Helm is a package manager for kubernetes and it needs a client part (helm) and a server part (tiller) to work.

All information about installing helm and tille can be found here.

Configuration of our Helm deployment

With tiller installed on our cluster, everything was ready to automate the deployment of binder in our OVH infrastructure.

The configuration of the helm deployment is pretty straightforward and all the steps have been described by the Binder team here.

Integration into the binderhub CD/CI process

The binder team already had a travis workflow existing for the automation of their test and deployment processes. Everything is transparent and they expose all their configurations (except secrets) on their GitHub project. We just had to integrate with their current workflow and push our specific configuration on their repository.

We then waited for their next launch of their Travis workflow and it worked.

From this moment onward, the ovh stack for binder was running and accessible by anyone from everywhere at this adress: https://ovh.mybinder.org/.

What comes next?

OVH will continue engaging with the data open-source community, and keep building a strong relationship with the Jupyter foundation and more generally the python community.

This first collaborative experience with such a data-driven open-source organisation helped us to establish the best team organisation and management to ensure that both OVH and the community achieve their goals in the best way possible

Working with open source is very different from the industry as it requires a different mindset: very human-centric, where everyone has different objectives, priorities, timeline and points of view that should all be considered.

Special Thanks

We are grateful to the Binder, Jupyter, and QuantStack team for their help, the OVH K8s team for the OVH Managed Kubernetes and OVH Managed Private Registry, and the OVH MLS team for the support. You rock, people!

Hey, Horacio, that Kubernetes thing is rather cool, but what I would have loved to see is a Functions-as-a-Service platform. Most of my apps could be easily done with a database and several serverless functions!

It wasn’t the first time I’d got that question…

Being, above all, a web developer, I can definitely relate. Kubernetes is a wonderful product – you can install complicated web architectures with a click – but what about the database + some functions model?

Well, you can also do it with Kubernetes!

That’s the beauty of the rich Kubernetes ecosystem: you can find projects to address many different use cases, from game servers with Agones to FaaS platforms…

There is an Helm chart for that!

Saying “You can do it with Kubernetes!” is almost the new “There is an app for that!”, but it doesn’t help a lot of people who are looking for solutions. As the question had come up several times, we decided to prepare a small tutorial on how to deploy and use a FaaS platform on OVH Managed Kubernetes.

We began by testing several FaaS platform on our Kubernetes. Our objective was to find the following solution:

• Easy to deploy (ideally with a simple Helm chart)
• Manageable with both an UI and a CLI, because different customers have different needs
• Auto-scalable, in both the upscaling and downscaling senses
• Supported by comprehensive documentation

We tested lots of platforms, like Kubeless, OpenWhisk, OpenFaaS and Fission, and I must say that all of them performed quite well. In the end though, the one that scored the best in terms of our objectives was OpenFaaS, so we decided to use it as the reference for this blog post.

OpenFaaS – a Kubernetes-native FaaS platform

OpenFaaS is an open-source framework for building serverless functions with Docker and Kubernetes. The project is already mature, popular and active, with more than 14k stars on GitHub, hundreds of contributors, and lots of users (both corporate and private).

OpenFaaS is very simple to deploy, using a Helm chart (including an operator for CRDs, i.e. kubectl get functions). It has both a CLI and a UI, manages auto-scaling effectively, and its documentation is really comprehensive (with a Slack channel to discuss it, as a nice bonus!).

Technically, OpenFaaS is composed of several functional blocks:

• The Function Watchdog. A tiny golang HTTP server that transforms any Docker image into a serverless function
• The API Gateway, which provides an external route into functions and collects metrics
• The UI Portal, which creates and invokes functions
• The CLI (essentially a REST client for the API Gateway), which can deploy any container as a function

Functions can be written in many languages (although I mainly used JavaScript, Go and Python for testing purposes), using handy templates or a simple Dockerfile.

Deploying OpenFaaS on OVH Managed Kubernetes

There are several ways to install OpenFaaS on a Kubernetes cluster. In this post we’re looking at the easiest one: installing with Helm.

If you need information on how to install and use Helm on your OVH Managed Kubernetes cluster, you can follow our tutorial.

The official Helm chart for OpenFaas is available on the faas-netes repository.

Adding the OpenFaaS Helm chart

The OpenFaaS Helm chart isn’t available in Helm’s standard stable repository, so you’ll need to add their repository to your Helm installation:

helm repo add openfaas https://openfaas.github.io/faas-netes/
helm repo update

Creating the namespaces

OpenFaaS guidelines recommend creating two namespaces, one for OpenFaaS core services and one for the functions:

kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml

Generating secrets

A FaaS platform that’s open to the internet seems like a bad idea. That’s why we are generating secrets, to enable authentication on the gateway:

# generate a random password
PASSWORD=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)

kubectl -n openfaas create secret generic basic-auth \
    --from-literal=basic-auth-user=admin \
    --from-literal=basic-auth-password="$PASSWORD"

Note: you will need this password later in the tutorial (to access the UI portal, for example). You can view it at any point in the terminal session with echo $PASSWORD.

Deploying the Helm chart

The Helm chart can be deployed in three modes: LoadBalancer, NodePort and Ingress. For our purposes, the simplest way is simply using our external Load Balancer, so we will deploy it in LoadBalancer, with the --set serviceType=LoadBalancer option.

If you want to better understand the difference between these three modes, you can read our Getting external traffic into Kubernetes – ClusterIp, NodePort, LoadBalancer, and Ingress blog post.

Deploy the Helm chart as follows:

helm upgrade openfaas --install openfaas/openfaas \
    --namespace openfaas  \
    --set basic_auth=true \
    --set functionNamespace=openfaas-fn \
    --set serviceType=LoadBalancer

As suggested in the install message, you can verify that OpenFaaS has started by running:

kubectl --namespace=openfaas get deployments -l "release=openfaas, app=openfaas"

If it’s working, you should see a list of the available OpenFaaS deployment objects:

$ kubectl --namespace=openfaas get deployments -l "release=openfaas, app=openfaas"
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
alertmanager   1         1         1            1           33s
faas-idler     1         1         1            1           33s
gateway        1         1         1            1           33s
nats           1         1         1            1           33s
prometheus     1         1         1            1           33s
queue-worker   1         1         1            1           33s

Install the FaaS CLI and log in to the API Gateway

The easiest way to interact with your new OpenFaaS platform is by installing faas-cli, the command line client for OpenFaaS on a Linux or Mac (or in a WSL linux terminal in Windows):

curl -sL https://cli.openfaas.com | sh

You can now use the CLI to log in to the gateway. The CLI will need the public URL of the OpenFaaS LoadBalancer, which you can get via kubectl:

kubectl get svc -n openfaas gateway-external -o wide

Export the URL to an OPENFAAS_URL variable:

export OPENFAAS_URL=[THE_URL_OF_YOUR_LOADBALANCER]:[THE_EXTERNAL_PORT]

Note: you will need this URL later on the tutorial, for example to access the UI portal. You can see it at any moment in the terminal session by doing echo $OPENFAAS_URL.

And connect to the gateway:

echo -n $PASSWORD | ./faas-cli login -g $OPENFAAS_URL -u admin --password-stdin

Now your’re connected to the gateway, and you can send commands to the OpenFaaS platform.

By default, there is no function installed on your OpenFaaS platform, as you can verify with the faas-cli list command.

In my own deployment (URLs and IP changed for this example), the preceding operations gave:

$ kubectl get svc -n openfaas gateway-external -o wide
 NAME               TYPE           CLUSTER-IP    EXTERNAL-IP                        PORT(S)          AGE     SELECTOR
 gateway-external   LoadBalancer   10.3.xxx.yyy   xxxrt657xx.lb.c1.gra.k8s.ovh.net   8080:30012/TCP   9m10s   app=gateway

 $ export OPENFAAS_URL=xxxrt657xx.lb.c1.gra.k8s.ovh.net:8080

 $ echo -n $PASSWORD | ./faas-cli login -g $OPENFAAS_URL -u admin --password-stdin
 Calling the OpenFaaS server to validate the credentials...
 WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
 credentials saved for admin http://xxxrt657xx.lb.c1.gra.k8s.ovh.net:8080

$ ./faas-cli version
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|
CLI:
 commit:  b42d0703b6136cac7b0d06fa2b212c468b0cff92
 version: 0.8.11
Gateway
 uri:     http://xxxrt657xx.lb.c1.gra.k8s.ovh.net:8080
 version: 0.13.0
 sha:     fa93655d90d1518b04e7cfca7d7548d7d133a34e
 commit:  Update test for metrics server
Provider
 name:          faas-netes
 orchestration: kubernetes
 version:       0.7.5 
 sha:           4d3671bae8993cf3fde2da9845818a668a009617

$ ./faas-cli list Function Invocations Replicas 

Deploying and invoking functions

You can easily deploy functions on your OpenFaaS platform using the CLI, with this command: faas-cli up.

Let’s try out some sample functions from the OpenFaaS repository:

./faas-cli deploy -f https://raw.githubusercontent.com/openfaas/faas/master/stack.yml

Running a faas-cli list command now will show the deployed functions:

$ ./faas-cli list
Function                          Invocations     Replicas
base64                            0               1    
echoit                            0               1    
hubstats                          0               1    
markdown                          0               1    
nodeinfo                          0               1    
wordcount                         0               1    

As an example, let’s invoke wordcount (a function that takes the syntax of the unix wc command, giving us the number of lines, words and characters of the input data):

echo 'I love when a plan comes together' | ./faas-cli invoke wordcount

$ echo 'I love when a plan comes together' | ./faas-cli invoke wordcount
       1         7        34

Invoking a function without the CLI

You can use the faas-cli describe command to get the public URL of your function, and then call it directly with your favorite HTTP library (or the good old curl):

$ ./faas-cli describe wordcount
Name:                wordcount
Status:              Ready
Replicas:            1
Available replicas:  1
Invocations:         1
Image:               functions/alpine:latest
Function process:    
URL:                 http://xxxxx657xx.lb.c1.gra.k8s.ovh.net:8080/function/wordcount
Async URL:           http://xxxxx657xx.lb.c1.gra.k8s.ovh.net:8080/async-function/wordcount
Labels:              faas_function : wordcount
Annotations:         prometheus.io.scrape : false

$ curl -X POST --data-binary "I love when a plan comes together" "http://xxxxx657xx.lb.c1.gra.k8s.ovh.net:8080/function/wordcount"
       0         7        33

Containers everywhere…

The most attractive part of a FaaS platform is being able to deploy your own functions.
In OpenFaaS, you can write your these function in many languages, not just the usual suspects (JavaScript, Python, Go etc.). This is because in OpenFaaS, you can deploy basically any container as a function, although this does mean you need to package your functions as containers in order to deploy them.

That also means that in order to create your own functions, you need to have Docker installed on your workstation, and you will need to push the images in a Docker registry (either the official one or a private one).

If you need a private registry, you can install one on your OVH Managed Kubernetes cluster. For this tutorial we are choosing to deploy our image on the official Docker registry.

Writing our first function

For our first example, we are going to create and deploy a hello word function in JavaScript, using NodeJS. Let’s begin by creating and scaffolding the function folder:

mkdir hello-js-project
cd hello-js-project
../faas-cli new hello-js --lang node

The CLI will download a JS function template from OpenFaaS repository, generate a function description file (hello-js.yml in this case) and a folder for the function source code (hello-js). For NodeJS, you will find a package.json (to declare eventual dependencies to your function, for example) and a handler.js (the function main code) in this folder.

Edit hello-js.yml to set the name of the image you want to upload to the Docker registry:

version: 1.0
provider:
  name: openfaas
  gateway: http://6d6rt657vc.lb.c1.gra.k8s.ovh.net:8080
functions:
  hello-js:
    lang: node
    handler: ./hello-js
    image: ovhplatform/openfaas-hello-js:latest

The function described in the handler.js file is really simple. It exports a function with two parameters: a context where you will receive the request data, and a callback that you will call at the end of your function and where you will pass the response data.

"use strict"

module.exports = (context, callback) => {
    callback(undefined, {status: "done"});
}

Let’s edit it to send back our hello world message:

"use strict"

module.exports = (context, callback) => {
    callback(undefined, {message: 'Hello world'});
}

Now you can build the Docker image and push it to the public Docker registry:

# Build the image
../faas-cli build -f hello-js.yml
# Login at Docker Registry, needed to push the image
docker login     
# Push the image to the registry
../faas-cli push -f hello-js.yml

With the image in the registry, let’s deploy and invoke the function with the OpenFaaS CLI:

# Deploy the function
../faas-cli deploy -f hello-js.yml
# Invoke the function
../faas-cli invoke hello-js

Congratulations! You have just written and deployed your first OpenFaaS function.

Using the OpenFaaS UI Portal

You can test the UI Portal by pointing your browser to your OpenFaaS gateway URL (the one you have set on the $OPENFAAS_URL variable), and entering the admin user and the password you have set on the $PASSWORD variable when prompted to.

In the UI Portal, you will find the list of the deployed functions. For each function, you can find its description, invoke it and see the result.

Where do we go from here?

So you now have a working OpenFaaS platform on your OVH Managed Kubernetes cluster.

To learn more about OpenFaaS, and how you can get the most out of it, please refer to the official OpenFaaS documentation. You can also follow the OpenFaaS workshops for more practical tips and advice.

One of the latests additions to this ecosystem is the Agones project, an open-source, multiplayer, dedicated game-server hosting built on Kubernetes, developed by Google in collaboration with Ubisoft. The project was announced in March, and has already made quite a bit of noise…

In the OVH Platform Team we are fans of both online gaming and Kubernetes, so we told ourselves that we needed to test Agones. And what better way to test it than deploying it on our OVH Managed Kubernetes service, installing a Xonotic game server cluster and playing some old-school deathmatches with collegues?

And of course, we needed to write about it to share the experience…

Why Agones?

Agones (derived from the Greek word agōn, contests held during public festivals or more generally “contest” or “competition at games”) aims to replace the usual proprietary solutions to deploy, scale and manage game servers.

Agones enriches Kubernetes with a Custom Controller and a Custom Resource Definition. With them, you can standardise Kubernetes tooling and APIs to create, scale and manage game server clusters.

Wait, what game servers are you talking about?

Well, Agones’s main focus is online multiplayer games such as FPSs and MOBAs, fast-paced games requiring dedicated, low-latency game servers that synchronize the state of the game between players and serve as a source of truth for gaming situations.

These kinds of games ask for relatively ephemeral dedicated gaming servers, with every match running on a server instance. The servers need to be stateful (they must keep the game status), with the state usually held in memory for the duration of the match.

Latency is a key concern, as the competitive real-time aspects of the games ask for quick responses from the server. That means that the connection from the player device to the game server should be the most direct possible, ideally bypassing any intermediate server such as a load-balancer.

And how do you connect the players to the right server?

Every game publisher used to have their own proprietary solutions, but most on them follow a similar flow, with a matchmaking service that groups players into a match, deals with a cluster manager to provision a dedicated instance of game server and send to the players its IP address and port, to allow them to directly connect to the server and play the game.

Agones and it’s Custom Controller and Custom Resource Definition replaces the complex cluster management infrastructure with a standardised, Kubernetes-based tooling and APIs. The matchmaker services interact with these APIs to spawn new game server pods and get their IP address and ports to the concerned players.

The cherry on the cake

Using Kubernetes for these tasks also gives some nice additional bonus, like being able to deploy the full gaming infrastructure in a developer environnement (or even in a minikube), or easily clone it to deploy in a new data center or cloud region, but also offering a whole platform to host all the additional services needed to build a game: account management, leaderboards, inventory…

And of course, the simplicity of operating Kubernetes-based platforms, especially when they dynamic, heterogeneous and distributed, as most online gaming platforms.

Deploying Agones on OVH Managed Kubernetes

There are several ways to install Agones in a Kubernetes cluster. For our test we chose the easiest one: installing with Helm.

Enabling creation of RBAC resources

The first step to install Agones is to setup a service account with enough permissions to create some special RBAC resource types.

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin --serviceaccount=kube-system:default

Now we have the Cluster Role Binding needed for the installation.

Installing the Agones chart

Now let’s continue by adding Agones repository to Helm’s repository list.

helm repo add agones https://agones.dev/chart/stable

And then installing the stable Agones chart:

helm install --name my-agones --namespace agones-system agones/agones

The installation we have just done isn’t suited for production, as the official install instructions recommend running Agones and the game servers in separate, dedicated pools of nodes. But for the needs of our test, the basic setup is enough.

Confirming Agones started successfully

To verify that Agones is running on our Kubernetes cluster, we can look at the pods in the agones-system namespace:

kubectl get --namespace agones-system pods

If everything is ok, you should see an agones-controller pod with a Running status:

$ kubectl get --namespace agones-system pods
NAME                                 READY   STATUS    RESTARTS   AGE
agones-controller-5f766fc567-xf4vv   1/1     Running   0          5d15h
agones-ping-889c5954d-6kfj4          1/1     Running   0          5d15h
agones-ping-889c5954d-mtp4g          1/1     Running   0          5d15h

You can also see more details using:

kubectl describe --namespace agones-system pods

Looking at the agones-controller description, you should see something like:

$ kubectl describe --namespace agones-system pods
Name:               agones-controller-5f766fc567-xf4vv
Namespace:          agones-system
[...]
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 

Where all the Conditions should have status True.

Deploying a game server

The Agones Hello world is rather boring, a simple UDP echo server, so we decided to skip it and go directly to something more interesting: a Xonotic game server.

Xonotic is an open-source multi-player FPS, and a rather good one, with lots of interesting game modes, maps, weapons and customization options.

Deploying a Xonotic game server over Agones is rather easy:

kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/agones/release-0.9.0/examples/xonotic/gameserver.yaml

The game server deployment can take some moments, so we need to wait until its status is Ready before using it. We can fetch the status with:

kubectl get gameserver

We wait until the fetch gives a Ready status on our game server:

$ kubectl get gameserver
NAME      STATE   ADDRESS         PORT   NODE       AGE
xonotic   Ready   51.83.xxx.yyy   7094   node-zzz   5d

When the game server is ready, we also get the address and the port we should use to connect to our deathmatch game (in my example, 51.83.xxx.yyy:7094).

It’s frag time

So now that we have a server, let’s test it!

We downloaded the Xonotic client for our computers (it runs on Windows, Linux and MacOS, so there is no excuse), and lauched it:

Then we go to the Multiplayer menu and enter the address and port of our game server:

And we are ready to play!

And on the server side?

On the server side, we can spy how things are going for our game server, using kubectl logs. Let’s begin by finding the pod running the game:

kubectl get pods

We see that our game server is running in a pod called xonotic:

$ kubectl get pods 
NAME      READY   STATUS    RESTARTS   AGE
xonotic   2/2     Running   0          5d15h

We can then use kubectl logs on it. In the pod there are two containers, the main xonotic one and a Agones sidecar, so we must specify that we want the logs of the xonotic container:

$ kubectl logs xonotic
Error from server (BadRequest): a container name must be specified for pod xonotic, choose one of: [xonotic agones-gameserver-sidecar]
$ kubectl logs xonotic xonotic
>>> Connecting to Agones with the SDK
>>> Starting health checking
>>> Starting wrapper for Xonotic!
>>> Path to Xonotic server script: /home/xonotic/Xonotic/server_linux.sh 
Game is Xonotic using base gamedir data
gamename for server filtering: Xonotic
Xonotic Linux 22:03:50 Mar 31 2017 - release
Current nice level is below the soft limit - cannot use niceness
Skeletal animation uses SSE code path
execing quake.rc
[...]
Authenticated connection to 109.190.xxx.yyy:42475 has been established: client is v6xt9/GlzxBH+xViJCiSf4E/SCn3Kx47aY3EJ+HOmZo=@Xon//Ks, I am /EpGZ8F@~Xon//Ks
LostInBrittany is connecting...
url_fclose: failure in crypto_uri_postbuf
Receiving player stats failed: -1
LostInBrittany connected
LostInBrittany connected
LostInBrittany is now spectating
[BOT]Eureka connected
[BOT]Hellfire connected
[BOT]Lion connected
[BOT]Scorcher connected
unconnected changed name to [BOT]Eureka
unconnected changed name to [BOT]Hellfire
unconnected changed name to [BOT]Lion
unconnected changed name to [BOT]Scorcher
[BOT]Scorcher picked up Strength
[BOT]Scorcher drew first blood! 
[BOT]Hellfire was gunned down by [BOT]Scorcher's Shotgun
[BOT]Scorcher slapped [BOT]Lion around a bit with a large Shotgun
[BOT]Scorcher was gunned down by [BOT]Eureka's Shotgun, ending their 2 frag spree
[BOT]Scorcher slapped [BOT]Lion around a bit with a large Shotgun
[BOT]Scorcher was shot to death by [BOT]Eureka's Blaster
[BOT]Hellfire slapped [BOT]Eureka around a bit with a large Shotgun, ending their 2 frag spree
[BOT]Eureka slapped [BOT]Scorcher around a bit with a large Shotgun
[BOT]Eureka was gunned down by [BOT]Hellfire's Shotgun
[BOT]Hellfire was shot to death by [BOT]Lion's Blaster, ending their 2 frag spree
[BOT]Scorcher was cooked by [BOT]Lion
[BOT]Eureka turned into hot slag
[...]

Add some friends…

The next step is mostly enjoyable: asking the collegues to connect to the server and doing a true deathmatch like in Quake 2 times.

And now?

We have a working game server, but we have barely uncovered the possibilities of Agones: deploying a fleet (a set of warm GameServers that are available to be allocated from), testing the FleetAutoscaler (to automatically scale up and down a Fleet in response to demand), making some dummy allocator service. In future blog posts we will dive deeper into it, and explore those possibilities.

And in a wider context, we are going to continue our exploratory journey on Agones. The project is still very young, an early alpha, but it shows some impressive perspectives.

In the Prescience team, we have used Kubernetes for more than a year now. Our cluster includes 40 nodes, running on top of PCI. We continuously run about 800 pods, and generate a lot of metrics as a result.

Today, we’ll look at how we handle these metrics to monitor our Kubernetes Cluster, and (equally importantly!) how to do this with your own cluster.

OVH Metrics

Like any infrastructure, you need to monitor your Kubernetes Cluster. You need to know exactly how your nodes, cluster and applications behave once they have been deployed in order to provide reliable services to your customers. To do this with our own Cluster, we use OVH Observability.

OVH Observability is backend-agnostic, so we can push metrics with one format and query with another one. It can handle:

• Graphite
• InfluxDB
• Metrics2.0
• OpentTSDB
• Prometheus
• Warp10

It also incorporates a managed Grafana, in order to display metrics and create monitoring dashboards.

Monitoring Nodes

The first thing to monitor is the health of nodes. Everything else starts from there.

In order to monitor your nodes, we will use Noderig and Beamium, as described here. We will also use Kubernetes DaemonSets to start the process on all our nodes.

So let’s start creating a namespace…

kubectl create namespace metrics

Next, create a secret with the write token metrics, which you can find in the OVH Control Panel.

kubectl create secret generic w10-credentials --from-literal=METRICS_TOKEN=your-token -n metrics

Copy metrics.yml into a file and apply the configuration with kubectl

# This will configure Beamium to scrap noderig
# And push metrics to warp 10
# We also add the HOSTNAME to the labels of the metrics pushed
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: beamium-config
  namespace: metrics
data:
  config.yaml: |
    scrapers:
      nodering:
        url: http://0.0.0.0:9100/metrics
        period: 30000
        format: sensision
        labels:
          app: nodering

    sinks:
      warp:
        url: https://warp10.gra1.metrics.ovh.net/api/v0/update
        token: $METRICS_TOKEN

    labels:
      host: $HOSTNAME

    parameters:
      log-file: /dev/stdout
---
# This is a custom collector that report the uptime of the node
apiVersion: v1
kind: ConfigMap
metadata:
  name: noderig-collector
  namespace: metrics
data:
  uptime.sh: |
    #!/bin/sh
    echo 'os.uptime' `date +%s%N | cut -b1-10` `awk '{print $1}' /proc/uptime`
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: metrics-daemon
  namespace: metrics
spec:
  selector:
    matchLabels:
      name: metrics-daemon
  template:
    metadata:
      labels:
        name: metrics-daemon
    spec:
      terminationGracePeriodSeconds: 10
      hostNetwork: true
      volumes:
      - name: config
        configMap:
          name: beamium-config
      - name: noderig-collector
        configMap:
          name: noderig-collector
          defaultMode: 0777
      - name: beamium-persistence
        emptyDir:{}
      containers:
      - image: ovhcom/beamium:latest
        imagePullPolicy: Always
        name: beamium
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: TEMPLATE_CONFIG
          value: /config/config.yaml
        envFrom:
        - secretRef:
            name: w10-credentials
            optional: false
        resources:
          limits:
            cpu: "0.05"
            memory: 128Mi
          requests:
            cpu: "0.01"
            memory: 128Mi
        workingDir: /beamium
        volumeMounts:
        - mountPath: /config
          name: config
        - mountPath: /beamium
          name: beamium-persistence
      - image: ovhcom/noderig:latest
        imagePullPolicy: Always
        name: noderig
        args: ["-c", "/collectors", "--net", "3"]
        volumeMounts:
        - mountPath: /collectors/60/uptime.sh
          name: noderig-collector
          subPath: uptime.sh
        resources:
          limits:
            cpu: "0.05"
            memory: 128Mi
          requests:
            cpu: "0.01"
            memory: 128Mi

Don’t hesitate to change the collector levels if you need more information.

Then apply the configuration with kubectl…

$ kubectl apply -f metrics.yml
# Then, just wait a minutes for the pods to start
$ kubectl get all -n metrics
NAME                       READY   STATUS    RESTARTS   AGE
pod/metrics-daemon-2j6jh   2/2     Running   0          5m15s
pod/metrics-daemon-t6frh   2/2     Running   0          5m14s

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE AGE
daemonset.apps/metrics-daemon 40        40        40      40           40          122d

You can import our dashboard in to your Grafana from here, and view some metrics about your nodes straight away.

Kube Metrics

As the OVH Kube is a managed service, you don’t need to monitor the apiserver, etcd, or controlplane. The OVH Kubernetes team takes care of this. So we will focus on cAdvisor metrics and Kube state metrics

The most mature solution for dynamically scraping metrics inside the Kube (for now) is Prometheus.

In the next Beamium release, we should be able to reproduce the features of the Prometheus scraper.

To install the Prometheus server, you need to install Helm on the cluster…

kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller \
    --clusterrole cluster-admin \
    --serviceaccount=kube-system:tiller
helm init --service-account tiller

You then need to create the following two files: prometheus.yml and values.yml.

# Based on https://github.com/prometheus/prometheus/blob/release-2.2/documentation/examples/prometheus-kubernetes.yml
serverFiles:
  prometheus.yml:
    remote_write:
    - url: "https://prometheus.gra1.metrics.ovh.net/remote_write"
      remote_timeout: 120s
      bearer_token: $TOKEN
      write_relabel_configs:
      # Filter metrics to keep
      - action: keep
        source_labels: [__name__]
        regex: "eagle.*|\
            kube_node_info.*|\
            kube_node_spec_taint.*|\
            container_start_time_seconds|\
            container_last_seen|\
            container_cpu_usage_seconds_total|\
            container_fs_io_time_seconds_total|\
            container_fs_write_seconds_total|\
            container_fs_usage_bytes|\
            container_fs_limit_bytes|\
            container_memory_working_set_bytes|\
            container_memory_rss|\
            container_memory_usage_bytes|\
            container_network_receive_bytes_total|\
            container_network_transmit_bytes_total|\
            machine_memory_bytes|\
            machine_cpu_cores"

    scrape_configs:
    # Scrape config for Kubelet cAdvisor.
    - job_name: 'kubernetes-cadvisor'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      
      relabel_configs:
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        
      metric_relabel_configs:
      # Only keep systemd important services like docker|containerd|kubelet and kubepods,
      # We also want machine_cpu_cores that don't have id, so we need to add the name of the metric in order to be matched
      # The string will concat id with name and the separator is a ;
      # `/;container_cpu_usage_seconds_total` OK
      # `/system.slice;container_cpu_usage_seconds_total` OK
      # `/system.slice/minion.service;container_cpu_usage_seconds_total` NOK, Useless
      # `/kubepods/besteffort/e2514ad43202;container_cpu_usage_seconds_total` Best Effort POD OK
      # `/kubepods/burstable/e2514ad43202;container_cpu_usage_seconds_total` Burstable POD OK
      # `/kubepods/e2514ad43202;container_cpu_usage_seconds_total` Guaranteed POD OK
      # `/docker/pod104329ff;container_cpu_usage_seconds_total` OK, Container that run on docker but not managed by kube
      # `;machine_cpu_cores` OK, there is no id on these metrics, but we want to keep them also
      - source_labels: [id,__name__]
        regex: "^((/(system.slice(/(docker|containerd|kubelet).service)?|(kubepods|docker).*)?);.*|;(machine_cpu_cores|machine_memory_bytes))$"
        action: keep
      # Remove Useless parents keys like `/kubepods/burstable` or `/docker`
      - source_labels: [id]
        regex: "(/kubepods/burstable|/kubepods/besteffort|/kubepods|/docker)"
        action: drop
        # cAdvisor give metrics per container and sometimes it sum up per pod
        # As we already have the child, we will sum up ourselves, so we drop metrics for the POD and keep containers metrics
        # Metrics for the POD don't have container_name, so we drop if we have just the pod_name
      - source_labels: [container_name,pod_name]
        regex: ";(.+)"
        action: drop
    
    # Scrape config for service endpoints.
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints
      
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    # Example scrape config for pods
    #
    # The relabeling allows the actual pod scrape endpoint to be configured via the
    # following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the
    # pod's declared ports (default is a port-free target if none are declared).
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod

      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod_name
      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: host
      - action: labeldrop
        regex: (pod_template_generation|job|release|controller_revision_hash|workload_user_cattle_io_workloadselector|pod_template_hash)

alertmanager:
  enabled: false
pushgateway:
  enabled: false
nodeExporter:
  enabled: false
server:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: traefik
      ingress.kubernetes.io/auth-type: basic
      ingress.kubernetes.io/auth-secret: basic-auth
    hosts:
    - prometheus.domain.com
  image:
    tag: v2.7.1
  persistentVolume:
    enabled: false

Don’t forget to replace your token!

The Prometheus scraper is quite powerful. You can relabel your time series, keep a few that match your regex, etc. This config removes a lot of useless metrics, so don’t hesitate to tweak it if you want to see more cAdvisor metrics (for example).

 Install it with Helm…

helm install stable/prometheus \
    --namespace=metrics \
    --name=metrics \
    --values=values/values.yaml \
    --values=values/prometheus.yaml

Add add a basic-auth secret…

$ htpasswd -c auth foo
New password: <bar>
New password:
Re-type new password:
Adding password for user foo
$ kubectl create secret generic basic-auth --from-file=auth -n metrics
secret "basic-auth" created

You can can access the Prometheus server interface through prometheus.domain.com.

You will see all the metrics for your Cluster, although only the one you have filtered will be pushed to OVH Metrics.

The Prometheus interfaces is a good way to explore your metrics, as it’s quite straightforward to display and monitor your infrastructure. You can find our dashboard here.

Resources Metrics

As @Martin Schneppenheim said in this post, in order to correctly manage a Kubernetes Cluster, you also need to monitor pod resources.

We will install Kube Eagle, which will fetch and expose some metrics about CPU and RAM requests and limits, so they can be fetched by the Prometheus server you just installed.

Create a file named eagle.yml.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app: kube-eagle
  name: kube-eagle
  namespace: kube-eagle
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - pods
  verbs:
  - get
  - list
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  labels:
    app: kube-eagle
  name: kube-eagle
  namespace: kube-eagle
subjects:
- kind: ServiceAccount
  name: kube-eagle
  namespace: kube-eagle
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-eagle
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: kube-eagle
  labels:
    app: kube-eagle
  name: kube-eagle
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kube-eagle
  name: kube-eagle
  labels:
    app: kube-eagle
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-eagle
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
      labels:
        app: kube-eagle
    spec:
      serviceAccountName: kube-eagle
      containers:
      - name: kube-eagle
        image: "quay.io/google-cloud-tools/kube-eagle:1.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: PORT
          value: "8080"
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: http
        readinessProbe:
          httpGet:
            path: /health
            port: http

$ kubectl create namespace kube-eagle
$ kubectl apply -f eagle.yml

Next, add import this Grafana dashboard (it’s the same dashboard as Kube Eagle, but ported to Warp10).

You now have an easy way of monitoring your pod resources in the Cluster!

Custom Metrics

How does Prometheus know that it needs to scrape kube-eagle? If you looks at the metadata of the eagle.yml, you’ll see that:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080" # The port where to find the metrics
  prometheus.io/path: "/metrics" # The path where to find the metrics

Theses annotations will trigger the Prometheus auto-discovery process (described in prometheus.yml line 114).

This means you can easily add these annotations to pods or services that contain a Prometheus exporter, and then forward these metrics to OVH Observability. You can find a non-exhaustive list of Prometheus exporters here.

Volumetrics Analysis

As you saw on the prometheus.yml , we’ve tried to filter a lot of useless metrics. For example, with cAdvisor on a fresh cluster, with only three real production pods, and with the whole kube-system and Prometheus namespace, have about 2,600 metrics per node. With a smart cleaning approach, you can reduce this to 126 series.

Here’s a table to show the approximate number of metrics you will generate, based on the number of nodes (N) and the number of production pods (P) you have:

For example, if you run three nodes with 60 Pods, you will generate 264 * 3 + 32 * 60 ~= 2,700 metrics

NB: A pod has a unique name, so if you redeploy a deployment, you will create 32 new metrics each time.

^{(1) Noderig metrics: os.mem / os.cpu / os.disk.fs / os.load1 / os.net.dropped (in/out) / os.net.errs (in/out) / os.net.packets (in/out) / os.net.bytes (in/out)/ os.uptime}

^{(2) cAdvisor nodes metrics: machine_memory_bytes / machine_cpu_cores}

^{(3) Kube state nodes metrics: kube_node_info}

^{(4) Kube Eagle nodes metrics: eagle_node_resource_allocatable_cpu_cores / eagle_node_resource_allocatable_memory_bytes / eagle_node_resource_limits_cpu_cores / eagle_node_resource_limits_memory_bytes / eagle_node_resource_requests_cpu_cores / eagle_node_resource_requests_memory_bytes / eagle_node_resource_usage_cpu_cores / eagle_node_resource_usage_memory_bytes}

^{(5) With our filters, we will monitor around five system.slices} 

^{(6)  Metrics are reported per container. A pod is a set of containers (a minimum of two): your container + the pause container for the network. So we can consider (2* 10 + 6) for the number of metrics per pod. 10 metrics from the cAdvisor and six for the network (see below) and for system.slice we will have 10 + 6, because it’s treated as one container.}

^{(7) cAdvisor will provide these metrics for each container: container_start_time_seconds / container_last_seen / container_cpu_usage_seconds_total / container_fs_io_time_seconds_total / container_fs_write_seconds_total / container_fs_usage_bytes / container_fs_limit_bytes / container_memory_working_set_bytes / container_memory_rss / container_memory_usage_bytes}

^{(8) cAdvisor will provide these metrics for each interface: container_network_receive_bytes_total * per interface / container_network_transmit_bytes_total * per interface}

^{(9) kube-dns / beamium-noderig-metrics / kube-proxy / canal / metrics-server} 

^{(10) Kube Eagle pods metrics: eagle_pod_container_resource_limits_cpu_cores / eagle_pod_container_resource_limits_memory_bytes / eagle_pod_container_resource_requests_cpu_cores / eagle_pod_container_resource_requests_memory_bytes / eagle_pod_container_resource_usage_cpu_cores / eagle_pod_container_resource_usage_memory_bytes}

Conclusion

As you can see, monitoring your Kubernetes Cluster with OVH Observability is easy. You don’t need to worry about how and where to store your metrics, leaving you free to focus on leveraging your Kubernetes Cluster to handle your business workloads effectively, like we have in the Machine Learning Services Team.

The next step will be to add an alerting system, to notify you when your nodes are down (for example). For this, you can use the free OVH Alert Monitoring tool.

Stay in touch

For any questions, feel free to join the Observability Gitter or Kubernetes Gitter!
Follow us on Twitter: @OVH

In the next few posts, I am going to tell you some stories about this beta phase. We’ll be taking a look at feedback from some of our beta testers, technical insights, and some fun anecdotes about the development of this new service.

Today, we’ll start with one of the most frequent questions I got during the early days of the beta: How do I route external traffic into my Kubernetes service? The question came up a lot as our customers began to explore Kubernetes, and when I tried to answer it, I realised that part of the problem was the sheer number of possible answers, and the concepts needed to understand them.

Related to that question was a feature request: most users wanted a load balancing tool. As the beta phase is all about confirming the stability of the product and validating the feature set prioritisation, we were able to quickly confirm LoadBalanceras a key feature of our first commercial release.

To try to better answer the external traffic question, and to make the adoption of LoadBalancereasier, we wrote a tutorial and added some drawings, which got nice feedback. This helped people to understand the concept underlaying the routing of external traffic on Kubernetes.

This blog post is an expanded version of this tutorial. We hope that you will find it useful!

Some concepts:  ClusterIP,  NodePort,  Ingress and  LoadBalancer

When you begin to use Kubernetes for real-world applications, one of the first questions to ask is how to get external traffic into your cluster. The official documentation offers a comprehensive (but rather dry) explanation of this topic, but here we are going to explain it in a more practical, need-to-know way.

There are several ways to route external traffic into your cluster:

• Using Kubernetes proxy and ClusterIP: The default Kubernetes ServiceType is ClusterIp, which exposes the Service on a cluster-internal IP. To reach the ClusterIp from an external source, you can open a Kubernetes proxy between the external source and the cluster. This is usually only used for development.

• Exposing services as NodePort: Declaring a Service as NodePortexposes it on each Node’s IP at a static port (referred to as the NodePort). You can then access the Service from outside the cluster by requesting <NodeIp>:<NodePort>. This can also be used for production, albeit with some limitations.

• Exposing services as LoadBalancer: Declaring a Service as LoadBalancer exposes it externally, using a cloud provider’s load balancer solution. The cloud provider will provision a load balancer for the Service, and map it to its automatically assigned NodePort. This is the most widely used method in production environments.

Using Kubernetes proxy and ClusterIP

The default Kubernetes ServiceType is ClusterIp, which exposes the Service on a cluster-internal IP. To reach the ClusterIp from an external computer, you can open a Kubernetes proxy between the external computer and the cluster.

You can use kubectl to create such a proxy. When the proxy is up, you’re directly connected to the cluster, and you can use the internal IP (ClusterIp) for thatService.

Exposing services as NodePort

Declaring a service as NodePort exposes the Service on each Node’s IP at the NodePort (a fixed port for that Service, in the default range of 30000-32767). You can then access the Service from outside the cluster by requesting <NodeIp>:<NodePort>. Every service you deploy as NodePort will be exposed in its own port, on every Node.

It’s rather cumbersome to use NodePortfor Servicesthat are in production. As you are using non-standard ports, you often need to set-up an external load balancer that listens to the standard ports and redirects the traffic to the <NodeIp>:<NodePort>.

Exposing services as LoadBalancer

Declaring a service of type LoadBalancer exposes it externally using a cloud provider’s load balancer. The cloud provider will provision a load balancer for the Service, and map it to its automatically assigned NodePort. How the traffic from that external load balancer is routed to the Service pods depends on the cluster provider.

The LoadBalancer is the best option for a production environment, with two caveats:

• Every Service that you deploy as LoadBalancer will get it’s own IP.
• The LoadBalancer is usually billed based on the number of exposed services, which can be expensive.

We are currently offering the OVH Managed Kubernetes LoadBalancer service as a free preview, until the end of summer 2019.

What about Ingress?

According to the official documentation, an Ingress is an API object that manages external access to the services in a cluster (typically HTTP). So what’s the difference between this and LoadBalancer or NodePort?

Ingress isn’t a type of Service, but rather an object that acts as a reverse proxy and single entry-point to your cluster that routes the request to different services. The most basic Ingress is the NGINX Ingress Controller, where the NGINX takes on the role of reverse proxy, while also functioning as SSL.

Ingress is exposed to the outside of the cluster via ClusterIP and Kubernetes proxy, NodePort, or LoadBalancer, and routes incoming traffic according to the configured rules.

The main advantage of using an Ingress behind a LoadBalancer is the cost: you can have lots of services behind a single LoadBalancer.

Which one should I use?

Well, that’s the one million dollar question, and one which will probably elicit a different response depending on who you ask!

You could go 100% LoadBalancer, getting an individual LoadBalancer for each service. Conceptually, it’s simple: every service is independent, with no extra configuration needed. The downside is the price (you will be paying for one LoadBalancer per service), and also the difficulty of managing lots of different IPs.

You could also use only one LoadBalancer and an Ingress behind it. All your services would be under the same IP, each one in a different path. It’s a cheaper approach, as you only pay for one LoadBalancer, but if your services don’t have a logical relationship, it can quickly become chaotic.

If you want my personal opinion, I would try to use a combination of the two…

An approach I like is having a LoadBalancer for every related set of services, and then routing to those services using an Ingressbehind the  LoadBalancer. For example, let’s say you have two different microservice-based APIs, each one with around 10 services. I would put one LoadBalancer in front of one Ingress for each API, the LoadBalancerbeing the single public entry-point, and theIngress routing traffic to the API’s different services.

But if your architecture is quite complex (especially if you’re using microservices), you will soon find that manually managing everything with LoadBalancer and Ingress is  rather  cumbersome. If that’s the case, the answer could be to delegate those tasks to a service mesh…

What’s a service mesh?

You may have heard of Istio or Linkerd, and how they make it easier to build microservice architectures on Kubernetes, adding nifty perks like A/B testing, canary releases, rate limiting, access control, and end-to-end authentication.

Istio, Linkerd, and similar tools are service meshes, which allow you to build networks of microservices and define their interactions, while simultaneously adding some high-value features that make the setup and operation of microservice-based architectures easier.

There’s a lot to talk about when it comes to using service meshes on Kubernetes, but as they say, that’s a story for another time…