Several months ago, we released the Beta version of the OVHcloud Secret Manager and we guided you how to manage your secrets thanks to the existing External Secret Operator (ESO) Hashicorp Vault provider.

As our Secret Manager is now in General Availability, our teams worked on the development of an OVHcloud ESO Provider now available in the ESO v2.3.0 new release 🎉.

In this blog post, you will learn how to create a new secret in the OVHcloud Secret Manager and how to manage it within your Kubernetes clusters through the OVHcloud ESO provider.

External Secrets Operator (ESO)

The External Secrets Operator (ESO), a CNCF sanbox project since 2022, is a Kubernetes operator that integrates external secret management systems.

The operator reads the information from an external APIs and automatically injects the values into a Kubernetes Secret. If the secret changes in the external API, the operator updates the secret in the Kubernetes cluster.

The ESO connects to an external Secret Manager, such as OVHcloud, Vault, AWS, or GCP, via a provider configured in a (Cluster)SecretStore. An ExternalSecret resource then specifies which secrets to retrieve. ESO fetches those values and creates a corresponding Kubernetes Secret within the cluster.

For more details, read the ESO official documentation.

Prerequisites

To be able to use the ESO OVHcloud provider, you need to follow some prerequisites:

• Have an OVHcloud account
• Created an OKMS domain (“305db938-331f-454d-83a7-3a0a29291661” for example in this blog post)
• Created an IAM local user (“secretmanager-305db938-331f-454d-83a7-3a0a29291661” for example in this blog post)
• Installed the OVHcloud CLI
• Have a Kubernetes cluster

The ESO OVH provider supports both token and mTLS authentication. In this blog post, we will use the token authentication mode. Please follow the OVHcloud ESO provider guide if you wish to use mTLS authentication mode.

Generate a PAT token (For token authentication only)

The ESO (Cluster)SecretStore needs the permission to fetch secrets from Secret Manager.

If you want to use token autentication, you’ll need a token (PAT). You can use the ovhcloud CLI to do that:

PAT_TOKEN=$(ovhcloud iam user token create <iam-local-user-name> --name pat-<iam-local-user-name> --description "PAT secret manager for domain <okms-id>" -o json  | jq .details.token |  tr -d '"')

echo $PAT_TOKEN
<your-token>

You should have a result like this:

$ PAT_TOKEN=$(ovhcloud iam user token create secretmanager-305db938-331f-454d-83a7-3a0a29291661 --name pat-secretmanager-305db938-331f-454d-83a7-3a0a29291661 --description "PAT secret manager for domain 305db938-331f-454d-83a7-3a0a29291661" -o json  | jq .details.token |  tr -d '"')
2026/04/07 14:07:45 Final parameters:
{
 "description": "PAT secret manager for domain 305db938-331f-454d-83a7-3a0a29291661",
 "name": "pat-secretmanager-305db938-331f-454d-83a7-3a0a29291661"
}

$ echo $PAT_TOKEN
eyJhbGciOiJFZERTQSIsImtpZCI6IjgzMkFGNUE5ODg3MzFCMDNGM0EzMTRFMDJFRUJFRjBGNDE5MUY0Q0YiLCJraW5kIjoicGF0IiwidHlwIjoiSldUIn0.eyJ0b2tlbiI6InBBSFh1WE5JdVNHYVpmV3F2OUFzVmJrU3UwR2UySTJrdFU0OGdTZkwyZ1k9In0.-VDbiUf4vNm1KB9qSv7i4sGMCvxs_EuZFAETB-eaOFf3IX8-9m7akN800--ASgXy55_DDFHdy4Z5uSq8lww-Bw

Encode the PAT token in base 64 and save it in an environment variable:

export PAT_TOKEN_B64=$(echo -n $PAT_TOKEN | base64)
echo $PAT_TOKEN_B64

Retrieve and save the KMS information

List the OKMS domains:

$ ovhcloud okms list
┌──────────────────────────────────────┬─────────────┐
│                  id                  │   region    │
├──────────────────────────────────────┼─────────────┤
│ 305db938-331f-454d-83a7-3a0a29291661 │ eu-west-par │
│ xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx │ eu-west-par │
└──────────────────────────────────────┴─────────────┘

Save the KMS endpoint and the OKMS ID in two environment variables. For example:

export OKMS_ID="305db938-331f-454d-83a7-3a0a29291661"
export KMS_ENDPOINT=$(ovhcloud okms get 305db938-331f-454d-83a7-3a0a29291661 -o json | jq .restEndpoint | xargs)

Create a secret in the Secret Manager

In the OVHcloud Control Panel (UI), go to ‘Secret Manager’ section and click on the Create a secret button.

Then in order to create a secret ‘prod/eu-west-par/dockerconfigjson’, in the Europe region (France – Paris) eu-west-par, choose this region:

Then, choose the OKMS domain and create”prod/eu-west-par/dockerconfigjson” in the path and fill the content:

Finally, click on the Create button to finalise the creation of the new secret.

Install or update the ESO

If you’d never installed ESO in your Kubernetes cluster, you can install it via Helm:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

If you already installed it, now you should update it in order to use this new provider:

helm upgrade external-secrets external-secrets/external-secrets -n external-secrets

⚠️ In order to use the OVHcloud provider, you need to have a running instance of ESO equals to version 2.3.0 or more.

$ helm list -n external-secrets

NAME            	NAMESPACE       	REVISION	UPDATED                              	STATUS  	CHART                 	APP VERSION
external-secrets	external-secrets	1       	2026-04-13 13:56:29.071329 +0200 CEST	deployed	external-secrets-2.3.0	v2.3.0

Let’s deploy a Secret in Kubernetes using the ESO provider!

Deploy a ClusterSecretStore to connect ESO to Secret Manager

Set up a ClusterSecretStore to manage synchronization with Secret Manager.
It will use the OVHcloud provider with token authorization mode, and the OKMS endpoint as the backend.

Create a clustersecretstore.yaml.template file with the content below:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: secret-store-ovh
spec:
  provider:
    ovh:
      server: "$KMS_ENDPOINT" # for example: "https://eu-west-rbx.okms.ovh.net"
      okmsid: "$OKMS_ID" # for example: "734b9b45-8b1a-469c-b140-b10bd6540017"
      auth:
        token:
          tokenSecretRef:
            name: ovh-token
            namespace: external-secrets
            key: token
---
apiVersion: v1
kind: Secret
metadata:
  name: ovh-token
  namespace: external-secrets
data:
  token: $PAT_TOKEN_B64

Generate the clustersecretstore.yaml file from the environment variables you defined:

envsubst < clustersecretstore.yaml.template > clustersecretstore.yaml

You should obtain a file filled with the OVHcloud KMS information:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: secret-store-ovh
spec:
  provider:
    ovh:
      server: "https://eu-west-par.okms.ovh.net" # for example: "https://eu-west-rbx.okms.ovh.net"
      okmsid: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # for example: "734b9b45-8b1a-469c-b140-b10bd6540017"
      auth:
        token:
          tokenSecretRef:
            name: ovh-token
            namespace: external-secrets
            key: token
---
apiVersion: v1
kind: Secret
metadata:
  name: ovh-token
  namespace: external-secrets
data:
  token: ZXlK...UJ3

Apply it in your Kubernetes cluster:

kubectl apply -f clustersecretstore.yaml

Check:

$ kubectl get clustersecretstore.external-secrets.io/secret-store-ovh

NAME               AGE   STATUS   CAPABILITIES   READY
secret-store-ovh   7s    Valid    ReadWrite      True

Create an ExternalSecret

Create an externalsecret.yaml file with the content below:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: docker-config-secret
  namespace: external-secrets
spec:
  refreshInterval: 30m
  secretStoreRef:
    name: secret-store-ovh
    kind: ClusterSecretStore
  target:
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: "{{ .mysecret | toString }}"
    name: ovhregistrycred
    creationPolicy: Owner
  data:
  - secretKey: ovhregistrycred
    remoteRef:
      key: prod/eu-west-par/dockerconfigjson

Apply it:

$ kubectl apply -f externalsecret.yaml

externalsecret.external-secrets.io/docker-config-secret created

Check:

$ kubectl get externalsecret.external-secrets.io/docker-config-secret -n external-secrets 

NAME                   STORETYPE            STORE              REFRESH INTERVAL   STATUS         READY   LAST SYNC
docker-config-secret   ClusterSecretStore   secret-store-ovh   30m                SecretSynced   True    4s

After applying this command, it will create a Kubernetes Secret object.

$ kubectl get secret ovhregistrycred -n external-secrets

NAME              TYPE                             DATA   AGE
ovhregistrycred   kubernetes.io/dockerconfigjson   1      49s

The Kubernetes Secret have been created 🎉

We created a Secret directly from the key, but the OVHcloud ESO provider allows you to fetch the original secret from different parameters (fetch the whole secret, fetch nested values, fetch multiple secrets…), according to your needs.

Conclusion

In this blog, we’ve explained how to create secrets in the OVHcloud Secret Manager and then integrate them directly in your Kubernetes clusters using the new ESO OVHcloud provider.

With this brand new OVHcloud provider, you will have a smoother integration between the Secret Manager and your Kubernetes clusters with ESO.

Our team are working on several other integrations, so stay tuned, and please share your thoughts with us!

Before an application go to production, it passes through several stages: source code, build, packaging and distribution. But Malicious code – such as a compromised dependency, breached CI pipeline, or modified package in a registry – can be introduced at any point in the development cycle, potentially impacting thousands of projects

This is precisely where Software Supply Chain Security (SSCS) comes in: to protect not just the code itself, but also how it’s built, delivered, and utilised.

Attacks like SolarWinds and Log4Shell aren’t isolated incidents, but rather subtle indicators that have escalated in severity.

This blog post explores recommended solutions and best practices for OVHcloud Managed Private Registry (MPR), an OCI-compliant artifact registry, to help you enhance your Software Supply Chain Security.

Generate a Software Bill Of Materials (SBOM)

SBOMs provides a list of all the ingredients (OS, libraries, code) and anything that composes the images that will run on your Kubernetes cluster. 

From that list, you can find out more about the image, its vulnerabilities, and licenses.

Generate an SBOM manually

To manually generate an SBOM from your image, click the ‘GENERATE SBOM’ button:

Within seconds, the SBOM column for your image will display “Queued”, then change to “Generating”, and a “SBOM details” link will appear.

Click the ‘SBOM details’ link to view the SBOM:

Your application’s SBOM is generated by Trivy in SPDX format. This item is then listed as an accessory for your image in the registry.

Click the ‘sbom.harbor’ accessory type for more details:

Generate an SBOM automatically

Manually generating an SBOM is a good practice, but automating the process is even better. The private registry can automatically generates the SBOM for you once an image is pushed to the desired project.

Click the project your image is part of, navigate to the ‘Configuration’ tab, then tick the SBOM generation checkbox:

Vulnerabilities scanning

We recommend running vulnerability scans on the images to confirm that:

• the images provided are free of any known vulnerabilities (CVEs);
• security patches are well integrated before deployment;
• the images used in production comply with security and compliance policies.

There are several vulnerability scanners available, like Trivy, Docker Scout, and Grype.

The OVHcloud Managed Private Registry uses Trivy as its default vulnerability scanner, but you can add more scanners if needed. Go to the Administration panel, click ‘Interrogation Services’, then navigate to the ‘Scanners’ tab:

Scan your image manually

To manually run a vulnerability scan on your image, go to your project and click the SCAN VULNERABILITIES button:

Within a few seconds, a scan will run and reveal any vulnerabilities detected in your image.

Click your image to take a look at the CVEs list:

Scan your image automatically

To automatically scan images on push, click the project your image is part of, then the ‘Configuration’ tab, and tick the ‘Vulnerabilities scanning’ checkbox:

Schedule vulnerability scans

Another way to stay informed is by configuring your vulnerability scanner to run scans every day. Go in the Administration panel, click ‘Interrogation Services’, then the ‘Vulnerability’ tab:

You can choose to schedule the scan Hourly, Daily, Weekly or you can customize when the scan will be triggered.

Scheduled scans ensure that existing images are regularly/periodically analyzed for newly discovered vulnerabilities (CVEs).

Prevent vulnerable images from running

You can also configure a project to prevent vulnerable images from being pulled. In order to do that, check the Prevent vulnerable images from running checkbox.

Select the severity level of vulnerabilities to prevent images from running, from None to Critical.

With this configuration, images cannot be pulled if their level is equal to or higher than the selected level of severity.

Exploitable vulnerabilities

When a scanner found vulnerabilities for your images, it is not necessary that they are exploitable in your application/in your image.

In this example, my application is build with golang 1.25-alpine, but Trivy found several CVEs that are only exploitable in golang 1.19.1 or less.

In order to remove/skip the “false positive”, a solution exists.

VEX (Vulnerability Exploitability eXchange) is a standard “format” to state whether a vulnerability is exploitable or not in a specific context.

You can generate a VEX file with vexctl or govulncheck tools.

Example:

# With vexctl
$ VULN_ID="CVE-2022-27664"
$ PRODUCT="pkg:golang/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd"
$ vexctl create --file vex.json --author 'Aurélie Vache' --product "pkg:oci/demo@sha256:$HASH?repository_url=$REGISTRY/$HARBOR_PROJECT/demo" --vuln "$VULN_ID" --status 'not_affected' --justification 'vulnerable_code_not_present' --impact-statement "HTTP/2 vulnerability $VULN_ID is not exploitable because the image is compiled with Go 1.20, which contains the patched library."

# With govulncheck (for Go apps)
$ govulncheck -format openvex ./... > ../demo.vex.json

For the moment, OVHcloud MPR (managed Harbor) does not support VEX files (and the OpenVEX format) but it is planned in the future.

💡But the good news is that you can configure a CVEs whitelist with the list of not exploitable CVEs to ignore them during vulnerability scanning:

You can optionally uncheck the Never expires checkbox and use the calendar selector to set an expiry date for the allowlist.

Sign your images

It’s recommended to sign your images to ensure they haven’t been modified and originate from your pipeline (CI/CD).

Signing your images is crucial for protecting them against compromised registries and unauthorised image replacements.

Without a signature, there’s no guarantee the deployed image is the one you originally built!

You can sign your images with Sigstore Cosign or Notation tools:

$ export HARBOR_PROJECT=supply-chain
$ export IMAGE=xxxxxx.c1.de1.container-registry.ovh.net/$HARBOR_PROJECT/demo
$ export HASH=$(skopeo inspect docker://${IMAGE}:latest | jq -r .Digest | sed "s/^sha256://")

# Sign with Cosign
## Generate a private and a public key
$ cosign generate-key-pair
## Sign the image with the OCI 1.1 Referrers API
$ cosign sign -y --key cosign.key $IMAGE@sha256:$HASH 

# Sign with Notation
## Generate a RSA key & a self-signed X.509 test certificate
$ notation cert generate-test --default "test"

## Sign the image with the OCI 1.1 Refferrers API
$ export NOTATION_EXPERIMENTAL=1 ; notation sign -d --allow-referrers-api ${IMAGE}@sha256:${HASH}

You can use Cosign or Notation to sign your images, OVHcloud MPR supports both.

Your signature will appear beside your image as an accessory, plus a green checkmark ✅ in your column:

⚠️ Keep in mind, MPR (Harbor) doesn’t support signatures generated by Cosign v3 (the signature will upload and appear as an accessory, but the mark will stay red instead of turning green). This bug should be fixed in Harbor 2.15 💪.

Signing your OCI artifacts and linking them to your images is recommended, and you can do this using Cosign:

$ cosign attest -y --predicate sbom.spdx.json --key cosign.key $IMAGE@sha256:$HASH

They will be uploaded to the OVHcloud private registry and listed as accessories.

Ensure only verified images are pushed to your registry’s projects

To allow only verified/signed images to be deployed on a project, click the project your image is part of, navigate to the ‘Configuration’ tab, and tick the Cosign and/or Notation checkbox:

When checked, the registry will only allow verified images to be pulled from the project. Verified images are determined by Cosign or Notation, depending on the policy you have checked. Note that if you have both Cosign and Notation policies enforced, then images will need to be signed by both Cosign and Notation to be pulled.

Tag immutability

By default, tags are mutables, it means that you can push an image demo with the tag 1.0.0, do a modification in the code and push again to this same tag.

It could be useful to fix a bug but in term of security a mutable tag does not guarantee that the image you’ve built and pushed for the 1.0.0 version is the same image that exists now in the registry.

Moreover, on Harbor (so on OVHcloud MPR), due to limitations in the upstream OCI Distribution specification, the registry does not enforce a strict link between a tag and an image digest.

As a result, a tag can be reassigned to a different artifact. And it causes a side effect on the registry, this causes the tag to migrate across the artifacts and every artifact that has its tag taken away becomes tagless.

To prevent this situation, you can configure tag immutability rules. Tag immutability guarantees that an immutable tagged artifact cannot be deleted, and also cannot be altered in any way such as through re-pushing, re-tagging, or replication from another target registry.

To do that, click on your project and on the Policy tab and select TAG IMMUTABILITY:

And then click the ADD RULE button.

Fill the repositories and tags list according to your needs.

Example:

⚠️ You can add a maximum of 15 immutability rules per project.

To wrap thing up

Software supply chain security is super important these days. Everything is changing quickly – the concept, standards, and tools. So, leveraging useful tools like OVHcloud MPR and knowing how to set them up can boost your Software Supply Chain Security efforts.

To learn more about how to use and configure OVHcloud private registries, don’t hesitate to follow our guides.

For years, the Kubernetes Ingress API, and the popular Ingress NGINX controller (ingress-nginx), have been the default way to expose applications running inside a Kubernetes cluster.

But the ecosystem is changing: the Kubernetes SIG network has announced the retirement of Ingress NGINX in March 2026.

After March 2026 the Ingress NGINX will no longer get new features, new releases, security patches and bug fixes.

Furthermore, the Kubernetes project recommends using Gateway instead of Ingress.

The Ingress API has already been frozen, which means it is no longer being developed, and will have no further changes or updates made to it. The Kubernetes project has no plans to remove Ingress from Kubernetes.

While OVHcloud Managed Kubernetes Service (MKS) does not yet provide a native GatewayClass, you can already benefit from Gateway API capabilities today by deploying your own controller 💪 .

Also, until Gateway API becomes fully integrated with OpenStack providers, there is an intermediate option: using a modern, actively maintained Ingress controller other than ingress-nginx.

The limitations of the current Ingress controller model

The traditional Kubernetes Ingress model was intentionally simple: define an Ingress, install an Ingress Controller, and let it configure a single proxy (usually Nginx) to route traffic.

This design works, but it comes with limitations:

– Single Monolithic “Entry Point”: All HTTP routing for the entire cluster goes through one shared proxy. It adds complexity, configuration conflicts and scaling challenges.
– Protocol limitations: only HTTP and HTTPS.Support for gRPC, HTTP/2, TCP, UDP or TLS passthrough is inconsistent and controller-specific.
– Heavy Reliance on Annotations: Advanced features (timeouts, rewrites, header handling…) rely on custom annotations.
– Strong 3rd parties and cloud Load Balancers support: Every Ingress controllers (3rd parties providers) come with their specialized annotations.

Finally, as mentioned, the most used Ingress controller, Ingress NGINX, will be retired in March 2026.

A Transitional Solution: Using a Modern Ingress Controller (Traefik, Contour, HAProxy…)

Before moving to the Gateway API, as a transitional solution, OVHcloud MKS users can simply replace Ingress Nginx with a modern, actively maintained Ingress controller.

This allows you to:

– keep using your existing Ingress manifests
– keep the same architecture: Service type LoadBalancer → OVHcloud Public Cloud Load Balancer → Ingress Controller
– avoid relying on unsupported or deprecated components
– gain features (better gRPC support, built‑in dashboards, improved L7 behaviour…)

Popular alternatives:

Traefik:
– Very easy to deploy
– Excellent support for HTTP/2, gRPC, WebSockets
– Built‑in dashboard
– Supports both Ingress and Gateway API
– Actively maintained
– Seamless migration from NGINX Ingress Controller to Traefik with NGINX annotation compatibility

Contour (Envoy):
– Envoy-based Ingress Controller
– Excellent performance
– Good stepping‑stone toward Gateway API

HAProxy Ingress:
– Extremely performant
– Enterprise-grade L7 routing
– Optional Gateway API support

NGINX Gateway Fabric (NGF):
– The successor to Ingress NGINX
– Built directly around Gateway API
– Still maturing but a strong long‑term candidate

If you are interested, you can read the more exhaustive list of Ingress controllers.

Installing an Alternative Ingress Controller on OVHcloud MKS

We will show you how to install Traefik, as an alternative Ingress controller and use it to spawn a single OVHcloud Public Cloud Load Balancer (based on OpenStack Octavia).

Install Traefik:

helm repo add traefik https://traefik.github.io/charts
helm repo update

helm install traefik traefik/traefik --namespace traefik --create-namespace --set service.type=LoadBalancer

This automatically triggers:
– the OpenStack CCM (used by OVHcloud)
– the creation of an OVHcloud Public Cloud Load Balancer
– exposure of Traefik through a public IP

After several seconds, the Load Balancer will be active.

Check that Traefik is running:

$ kubectl get all -n traefik
NAME                           READY   STATUS    RESTARTS   AGE
pod/traefik-6777c5db85-pddd6   1/1     Running   0          31s

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/traefik   LoadBalancer   10.3.129.188   <pending>     80:30267/TCP,443:30417/TCP   31s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/traefik   1/1     1            1           31s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/traefik-6777c5db85   1         1         1       31s

Then in order to use it, create an ingress.yaml file with the following content:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "traefik"  # Specifies Traefik as the ingress controller
spec:
  rules:
    - host: my-app.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-service
                port:
                  number: 80

And apply it in your cluster:

kubectl apply -f ingress.yaml

Using this type of alternative provides a fully supported, modern Ingress Controller while you prepare a long‑term transition to the Gateway API.

Gateway API: A modern, flexible networking model

The Gateway API is the next-generation Kubernetes networking specification. It introduces clearer roles and more flexible architectures.

Gateway API splits responsibilities across:
– GatewayClass: defines the type of gateway and which controller manages it
– Gateway: the actual entry point (e.g., a Load Balancer)
– Routes: routing rules, protocol-specific (HTTPRoute, TLSRoute, GRPCRoute, TCPRoute…)

Gateway API supports:
– HTTP(S)
– HTTP/2
– gRPC
– TCP
– TLS passthrough
…in a consistent and portable way.

Unlike Ingress, Gateway API is explicitly designed to allow providers like OVHcloud, AWS, GCP, Azure to:
– provision Load Balancers (LB)
– manage listeners
– expose multiple ports
– integrate with their LB features
This paves the way for native OVHcloud GatewayClass support.

How does it work today on OVHcloud MKS?

OVHcloud MKS relies on the OpenStack Cloud Controller Manager (CCM) to provision OVHcloud Public Cloud Load Balancers in response to a Service of type LoadBalancer.

Since MKS does not yet include a native GatewayClass, you can use Gateway API today as follows:

1. You deploy an existing Gateway Controller (Envoy Gateway, Traefik, Contour/Envoy…) and its GatewayClass.
2. The controller deploys a Data Plane proxy inside the cluster.
3. To expose that proxy, you still have to create a Service of type LoadBalancer (and your app of course).
4. The CCM provisions an OVHcloud Public Cloud Load Balancer and forwards traffic to your proxy.

Thanks to that, you will have a fully functional Gateway API. The workflow is very similar to that which is required for using NGINX Ingress controller.

Using the Gateway API on OVHcloud MKS today

You can already use the Gateway API by deploying your preferred controller.

Here’s an example using Envoy Gateway, one of the most future-proof options.

Install Gateway API CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/latest/download/standard-install.yaml

Deploy Envoy Gateway:

helm install eg oci://docker.io/envoyproxy/gateway-helm -n envoy-gateway-system --create-namespace

You should have a result like this:

$ helm install eg oci://docker.io/envoyproxy/gateway-helm -n envoy-gateway-system --create-namespace

Pulled: docker.io/envoyproxy/gateway-helm:1.6.0
Digest: sha256:5c55e7844ae8cff3152ca00330234ef61b1f9fa3d466f50db2c63a279f1cd1df
NAME: eg
LAST DEPLOYED: Mon Dec  1 16:27:07 2025
NAMESPACE: envoy-gateway-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
**************************************************************************
*** PLEASE BE PATIENT: Envoy Gateway may take a few minutes to install ***
**************************************************************************

Envoy Gateway is an open source project for managing Envoy Proxy as a standalone or Kubernetes-based application gateway.

Thank you for installing Envoy Gateway! 🎉

Your release is named: eg. 🎉

Your release is in namespace: envoy-gateway-system. 🎉

To learn more about the release, try:

  $ helm status eg -n envoy-gateway-system
  $ helm get all eg -n envoy-gateway-system

To have a quickstart of Envoy Gateway, please refer to https://gateway.envoyproxy.io/latest/tasks/quickstart.

To get more details, please visit https://gateway.envoyproxy.io and https://github.com/envoyproxy/gateway.

Check the Envoy gateway is running:

$ kubectl get po -n envoy-gateway-system
NAME                            READY   STATUS    RESTARTS   AGE
envoy-gateway-9cbbc577c-5h5qw   1/1     Running   0          16m

As a quickstart, you can install directly the GatewayClass, Gateway, HTTPRoute and an example app:

kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default

This command deploys a GatewayClass, a Gateway, a HTTPRoute and an app deployed in a deployment and exposed through a service:

gatewayclass.gateway.networking.k8s.io/eg created
gateway.gateway.networking.k8s.io/eg created
serviceaccount/backend created
service/backend created
deployment.apps/backend created
httproute.gateway.networking.k8s.io/backend created

As you can see, a GatewayClass have been deployed:

$ kubectl get gatewayclass -o yaml | kubectl neat
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: GatewayClass
  metadata:
    name: eg
  spec:
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
kind: List
metadata:
  resourceVersion: ""

Note that a GatewayClass is a cluster-wide resource so you don’t have to specify any namespace.

A Gateway have been deployed also:

$ kubectl get gateway -o yaml -n default | kubectl neat
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: Gateway
  metadata:
    name: eg
    namespace: default
  spec:
    gatewayClassName: eg
    listeners:
    - allowedRoutes:
        namespaces:
          from: Same
      name: http
      port: 80
      protocol: HTTP
kind: List
metadata:
  resourceVersion: ""

A HTTPRoute also:

$ kubectl get httproute -o yaml -n default | kubectl neat
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata:
    name: backend
    namespace: default
  spec:
    hostnames:
    - www.example.com
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: backend
        port: 3000
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /
kind: List
metadata:
  resourceVersion: ""

In order to retrieve the external IP (of the external Load Balancer), you just have to get information about the Gateway and export it in an environment variable:

$ kubectl get gateway eg
NAME   CLASS   ADDRESS        PROGRAMMED   AGE
eg     eg      xx.xxx.xx.xxx   True        18m

$ export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

$ echo $GATEWAY_HOST
xx.xxx.xx.xxx

And finally, a backend service have been deployed with its deployment:

$ kubectl get pod,svc -l app=backend -n default
NAME                           READY   STATUS    RESTARTS   AGE
pod/backend-765694d47f-zr6hh   1/1     Running   0          21m

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/backend   ClusterIP   10.3.114.179   <none>        3000/TCP   21m

In order to create your own Gateway and *Route resources, don’t hesitate to take a look at the Gateway API website.

Conclusion

Two migration paths are currently available for OVHcloud MKS users:

• Short-term: switch to a modern Ingress Controller (Traefik, Contour, HAProxy, NGF…). It provides full support for current Ingress usage, without requiring API changes.
• Long-term: adopt the Gateway API. Gateway API brings multi‑protocol support, clearer separation of roles, and is the strategic direction of Kubernetes networking.

Which approach and which tool should you choose? Well, it’s up to you, depending on your use cases, your teams, your needs… 🙂

As we have seen in this blog post, OVHcloud MKS users can begin adopting these technologies today, safely and incrementally.

This ecosystem is evolving quickly, so stay tuned to find out about the coming release of a pre-installed official GatewayClass (based on OpenStack Octavia) 💪.

Secrets resources in Kubernetes help us keep sensitive information like logins, passwords, tokens, credentials and certificates secure. But just a heads up: Secrets in Kubernetes are base64 encoded, not encrypted so anyone can read and decode them if they know how.

The good news is that OVHcloud has just launched the Secret Manager Beta, which you can use within your Kubernetes clusters via the External Secrets Operator (ESO) 🎉.

External Secrets Operator

The External Secrets Operator (ESO) extends Kubernetes with Custom Resource Definitions (CRDs) ) that define where secrets are and how to sync them.

The controller retrieves secrets from an external API and creates Kubernetes Secrets. If the secret changes in the external API, the controller updates the secret in the Kubernetes cluster.

Basically, the ESO can connect to an external Secret Manager like OVHcloud, Vault, AWS, or GCP using a (Cluster)SecretStore, and an ExternalSecret to figure out which Secret it needs to fetch. It then creates a Secret in the Kubernetes cluster with the fetched secret’s value.

Plus, it can sync secrets across all the namespaces in your Kubernetes cluster (I love this feature ❤️):

You can use External Secrets with different Providers, including AWS Secrets Manager, HashiCorp Vault, Google Secret Manager. In this blog I’ll show you how to create a secret in the new OVHcloud Secret Manager using Hashicorp Vault.

For more details, read the ESO official documentation.

Let’s jump in!

Create an IAM local user

To fetch secrets in Secret Manager, you’ll need an IAM user with the right permissions. You can either set it up or use an existing one.

In the OVHcloud Control Panel (UI), go to ‘Identity and Access Management’, then ‘Identities’.

Click the ‘Add user’ button to create an IAM local user and complete the fields as shown below:

Quick note, I’ve named the user ‘secretmanager-’ followed by the ID of the OKMS domain I want to use.

The user needs to be an ADMIN, or, ideally, have the following policies:

okms:apikms:secret/create
okms:apikms:secret/version/getData
okms:apiovh:secret/get

Get the Personal Access Token (PAT)

The ESO ClusterSecretStore needs the permission to fetch secrets from Secret Manager, so you’ll need a token (PAT).

You can access it via our API, which you’ll find here: https://eu.api.ovh.com/console/?section=%2Fme&branch=v1#post-/me/identity/user/-user-/token

Path parameters

user: secretmanager-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx

Request body:

{
  "description": "PAT secretmanager-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx",
  "name": "pat-secretmanager-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"
}

You should obtain a response like this:

{
  "creation": "2025-11-07T14:02:56.679157188Z",
  "description": "PAT secretmanager-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx",
  "expiresAt": null,
  "lastUsed": null,
  "name": "pat-secretmanager-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx",
  "token": "eyJhbGciOiJ...punpVAg"
}

Save the token value, because you’ll need it in a bit.

Create a secret in the Secret Manager

Here’s how to create a secret with OVHcloud MPR credentials for use in Kubernetes cluster(s).

In the OVHcloud Control Panel (UI), go to ‘Secret Manager’, then create a secret ‘prod/va1/dockerconfigjson’ in the Europe region (France – Paris) eu-west-par:

You’ll need to activate the region if you’re selecting it for the first time:

Select an OKMS domain:

Enter the path and value of your secret. For example:

Your secret is all set!

Install External Secrets Operators on your cluster

Deploy external secret through Helm:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

Install from the chart repository:

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

Your result should look something like this:

$ helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

NAME: external-secrets
LAST DEPLOYED: Mon Nov 24 17:08:58 2025
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
external-secrets has been deployed successfully in namespace external-secrets!

In order to begin using ExternalSecrets, you will need to set up a SecretStore
or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).

More information on the different types of SecretStores and how to configure them
can be found in our Github: https://github.com/external-secrets/external-secrets

This command will install the External Secrets Operator in your cluster.

Check ESO is running:

$ kubectl get all -n external-secrets
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/external-secrets-6b9f8ff5d4-jwd6g                   1/1     Running   0          25m
pod/external-secrets-cert-controller-7bf8fd894c-d24xb   1/1     Running   0          25m
pod/external-secrets-webhook-df488ddff-2xv4t            1/1     Running   0          25m

NAME                               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/external-secrets-webhook   ClusterIP   10.3.106.32   <none>        443/TCP   25m

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/external-secrets                   1/1     1            1           25m
deployment.apps/external-secrets-cert-controller   1/1     1            1           25m
deployment.apps/external-secrets-webhook           1/1     1            1           25m

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/external-secrets-6b9f8ff5d4                   1         1         1       25m
replicaset.apps/external-secrets-cert-controller-7bf8fd894c   1         1         1       25m
replicaset.apps/external-secrets-webhook-df488ddff            1         1         1       25m

Create a Secret contains the PAT

Encode the PAT in base64:

$ echo -n "<token>" | base64

ZXlKaG...wVkFn

Create a secret with it inside a secret.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: ovhcloud-vault-token
  namespace: external-secrets
data:
  token: ZXlKaG...wVkFn

Apply the resource in your cluster:

kubectl apply -f secret.yaml

Check that the secret have been created:

$ kubectl get secret ovhcloud-vault-token -n external-secrets
NAME                   TYPE     DATA   AGE
ovhcloud-vault-token   Opaque   1      5m

Deploy a ClusterSecretStore to connect ESO to Secret Manager

Set up a ClusterSecretStore to manage synchronisation with Secret Manager.
It will use the HashiCorp Vault provider with token auth, and the OKMS endpoint as the backend.

Create a clustersecretstore.yaml file with the content below:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-secret-store
spec:
  provider:
      vault:
        server: "https://eu-west-par.okms.ovh.net/api/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # OKMS endpoint, fill with the correct region and your okms_id
        path: "secret"
        version: "v2"
        auth:
            tokenSecretRef:
              name: ovhcloud-vault-token # The k8s secret that contain your PAT
              key: token

Keep in mind, in our example, we’ve selected the “eu-west-par” region. You can enter a different server URL, depending on your desired region.

Apply it:

kubectl apply -f clustersecretstore.yaml

Check:

$ kubectl get clustersecretstore.external-secrets.io/vault-secret-store
NAME                 AGE   STATUS   CAPABILITIES   READY
vault-secret-store   2m   Valid    ReadWrite      True

Create an ExternalSecret

Create an externalsecret.yaml file with the content below:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: docker-config-secret
  namespace: external-secrets
spec:
  refreshInterval: 30m
  secretStoreRef:
    name: vault-secret-store
    kind: ClusterSecretStore
  target:
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: "{{ .mysecret | toString }}"
    name: ovhregistrycred
    creationPolicy: Owner
  data:
  - secretKey: mysecret
    remoteRef:
      key: prod/va1/dockerconfigjson

Apply it:

$ kubectl apply -f externalsecret.yaml
externalsecret.external-secrets.io/docker-config-secret created

Check:

$ kubectl get externalsecret.external-secrets.io/docker-config-secret -n external-secrets
NAME                   STORETYPE            STORE                REFRESH INTERVAL   STATUS         READY
docker-config-secret   ClusterSecretStore   vault-secret-store   30m0s              SecretSynced   True

After applying this command, it will create a Kubernetes Secret object.

$ kubectl get secret -n external-secrets
NAME                                     TYPE                             DATA   AGE
...
ovhregistrycred                          kubernetes.io/dockerconfigjson   1      17d
...

As you can see, the Secret is ready, and you can now use it as an imagePullSecret in your Pods!

Conclusion

In this blog, we’ve explained how to create secrets in the new OVHcloud Secret Manager and integrate them directly in your Kubernetes clusters using the ESO Vault provider.

And here’s some great news: our teams are working on an OVHcloud External Secret Operator, set to go live in the coming months, which you can use 🎉.

Stay tuned and share your thoughts!

Since this summer, it’s possible to create encrypted OVHcloud Block Storage with OMK (OVHcloud managed key) in RBX, SBG, Paris & BHS regions. More regions will come in the coming months 💪.

And the good news is that you can use encrypted Block Storage using Persistent Volumes in your OVHcloud Managed Kubernetes Service (MKS) clusters 🎉.

In this post, we’ll show you how to encrypt persistent volumes on an OVHcloud Managed Kubernetes (MKS) cluster using a csi-cinder-high-speed-gen2-luks Storage Class. Leveraging LUKS-based encryption at the storage layer, you’ll learn how to protect your data at rest without sacrificing the performance of NVMe-backed volumes.

We’ll guide you step by step: defining the Storage Class, creating a Persistent Volume Claim (PVC), and deploying a Pod that mounts the encrypted volume.

This practical walkthrough is designed for developers and platform engineers looking to secure their Kubernetes workloads on OVHcloud in a straightforward way.

How to

You will create a Persistent Volume Claim (PVC), linked to a Storage Class, that will automatically create a Persistent Volume (PV) that will automatically create an associated encrypted Public Cloud Block Storage volume.
Then you will create a Pod attached to the PVC.

Let’s create an encrypted Persistent Volume in our OVHcloud MKS cluster

Prerequisite: Have an OVHcloud MKS cluster.

First, create a csi-cinder-high-speed-gen2-luks.yaml file with the following content:

💡 Note that if you deploy in on a MKS 1AZ cluster (instead of my 3AZ MKS cluster), you should define the volumeBindingMode to Immediate instead.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cinder-high-speed-gen2-luks
allowVolumeExpansion: true
parameters:
  fsType: ext4
  type: high-speed-gen2-luks
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer 

This StorageClass is using the same configuration as existing csi-cinder-high-speed-gen2 but with the high-speed-gen2-luks type.

So the result will be the usage of SSD disks with NVMe interfaces encrypted with LUKS (Linux Unified Key Setup) which is a standard on-disk format for hard disk encryption.

Apply the manifest file:

kubectl apply -f csi-cinder-high-speed-gen2-luks.yaml

⚠️ You can’t modify the volumeBindingMode value for an existing Storage Class, you have to delete it and create a new one.

List the Storage Classes in the cluster:

$ kubectl get sc
NAME                              PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-cinder-high-speed (default)   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   33d
csi-cinder-high-speed-gen-2       cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   33d
csi-cinder-high-speed-gen2-luks   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   4s

Create a pvc-luks.yaml file with the following content:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-luks
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: csi-cinder-high-speed-gen2-luks

Create a new namespace and apply the manifest file into it:

kubectl create ns test-pvc-luks
kubectl apply -f pvc-luks.yaml -n test-pvc-luks

Check the status of our newly created PVC:

$ kubectl get pvc -n test-pvc-luks
NAME       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                      VOLUMEATTRIBUTESCLASS   AGE
pvc-luks   Pending                                      csi-cinder-high-speed-gen2-luks   <unset>                 3s


$ kubectl describe pvc pvc-luks -n test-pvc-luks
Name:          pvc-luks
Namespace:     test-pvc-luks
StorageClass:  csi-cinder-high-speed-gen2-luks
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                From                         Message
  ----    ------                ----               ----                         -------
  Normal  WaitForFirstConsumer  10s (x2 over 10s)  persistentvolume-controller  waiting for first consumer to be created before binding
$ kubectl describe pvc pvc-luks
Name:          pvc-luks
Namespace:     test-pvc-luks
StorageClass:  csi-cinder-high-speed-gen2-luks
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                From                         Message
  ----    ------                ----               ----                         -------
  Normal  WaitForFirstConsumer  10s (x2 over 10s)  persistentvolume-controller  waiting for first consumer to be created before binding

As you can see, your PVC have been creating, with the luks Storage Class, and is Pending to be Bound, until the creation of a Pod with a volume (because of the WaitForFirstConsumer value):

Create a pod.yaml file with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-encrypted-volume
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - mountPath: "/usr/share/nginx/html"
      name: encrypted-volume
  volumes:
  - name: encrypted-volume
    persistentVolumeClaim:
      claimName: pvc-luks

Create a new namespace and apply the manifest file into it:

kubectl apply -f pod.yaml -n test-pvc-luks

The PVC should now be Bound and a new PV should be created:

$ kubectl get pvc -n test-pvc-luks
NAME       STATUS   VOLUME                                                                     CAPACITY   ACCESS MODES   STORAGECLASS                      VOLUMEATTRIBUTESCLASS   AGE
pvc-luks   Bound    ovh-managed-kubernetes-siti343p-pvc-3a3b1d2e-ebdf-41a2-8f8f-4ee6984b6149   10Gi       RWO            csi-cinder-high-speed-gen2-luks   <unset>                 3m27s

$ kubectl get pv -n test-pvc-luks
NAME                                                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS                      VOLUMEATTRIBUTESCLASS   REASON   AGE
ovh-managed-kubernetes-siti343p-pvc-3a3b1d2e-ebdf-41a2-8f8f-4ee6984b6149   10Gi       RWO            Delete           Bound    test-pvc-luks/pvc-luks   csi-cinder-high-speed-gen2-luks   <unset>                          32s

First the Pod should be in ContainerCreating state (waiting the creation and the attachment of the volume) and after few seconds it will be Running:

$ kubectl get pod pod-with-encrypted-volume -n test-pvc-luks
NAME                        READY   STATUS              RESTARTS   AGE
pod-with-encrypted-volume   0/1     ContainerCreating   0          44s

# Wait a little...

$ kubectl get pod pod-with-encrypted-volume -n test-pvc-luks
NAME                        READY   STATUS    RESTARTS   AGE
pod-with-encrypted-volume   1/1     Running   0          2m10s

The Pod is now created with an attached volume:

$ kubectl describe pod pod-with-encrypted-volume -n test-pvc-luks
Name:             pod-with-encrypted-volume
Namespace:        test-pvc-luks
Priority:         0
Service Account:  default
Node:             my-pool-zone-c-h5xjf-7n7kt/192.168.142.174
Start Time:       Tue, 19 Aug 2025 10:10:41 +0200
Labels:           <none>
Annotations:      <none>
Status:           Running
IP:               10.240.0.203
IPs:
  IP:  10.240.0.203
Containers:
  nginx:
    Container ID:   containerd://c38c0a0e19970503ad1bfaa0c74b5cc320cb9df08456c7613b9a9a8c908b9190
    Image:          nginx
    Image ID:       docker.io/library/nginx@sha256:33e0bbc7ca9ecf108140af6288c7c9d1ecc77548cbfd3952fd8466a75edefe57
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 19 Aug 2025 10:11:42 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/nginx/html from encrypted-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vbcnk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  encrypted-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc-luks
    ReadOnly:   false
  kube-api-access-vbcnk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Normal   Scheduled               3m48s                  default-scheduler        Successfully assigned test-pvc-luks/pod-with-encrypted-volume to my-pool-zone-c-xxxx-xxxx
  ...
  Normal   SuccessfulAttachVolume  3m8s                   attachdetach-controller  AttachVolume.Attach succeeded for volume "ovh-managed-kubernetes-siti343p-pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  Normal   Pulling                 2m53s                  kubelet                  Pulling image "nginx"
  Normal   Pulled                  2m48s                  kubelet                  Successfully pulled image "nginx" in 5.072s (5.072s including waiting). Image size: 72324501 bytes.
  Normal   Created                 2m48s                  kubelet                  Created container: nginx
  Normal   Started                 2m48s                  kubelet                  Started container nginx

Logging in the OVHcloud Control Panel, you can see that the encrypted volume have been successfully created:

Finally, you can use your volume.

Execute a shell in the Nginx Pod and create an index.html file into it:

$ kubectl exec -it pod-with-encrypted-volume -n test-pvc-luks -- /bin/bash

root@pod-with-encrypted-volume:/# echo "Hello from OVHcloud encrypted Block Storage!" > /usr/share/nginx/html/index.html

And curl the webserver:

root@pod-with-encrypted-volume:/# apt update
root@pod-with-encrypted-volume:/# apt install curl
root@pod-with-encrypted-volume:/# curl http://localhost/
Hello from OVHcloud encrypted Block Storage!

🎉

What’s next?

In this blog post we saw a basic (but concrete) usage of the encrypted Persistent Volume on OVHcloud Kubernetes clusters that just bee released, don’t hesitate to think about it for your sensitive data.

In the coming months, the encrypted Block Storage will be available worldwide. Follow the Encrypted Block Volumes issue on GitHub to stay informed.

And don’t hesitate to take a look to our Cloud Roadmap & Changelog to see the state of all of the coming features in OVHcloud Public Cloud products.

When working on Infrastructure as Code projects, with Terraform or OpenTofu, Terraform States files are created and modified locally in a terraform.tfstate file. It’s a common usage and practice but not convenient when working as a team.

Do you know that you can configure Terraform to store data remotely on OVHcloud S3-compatible Object Storage?

OVHcloud Terraform/OpenTofu provider

To easily provision your infrastructures, OVHcloud provides a Terraform provider which is available in the official Terraform registry.

The provider is synchronized in the OpenTofu registry also:

Read the Infrastructure as Code (IaC) on OVHcloud – part 1: Terraform / OpenTofu blog post to have more information about the provider and IaC on OVHcloud.

Note that in the rest of the blog post we will be using terraform CLI and talking about Terraform, but you can also follow the blog post if you are using OpenTofu and tofu CLI instead 😉.

How to

In this blog post we will handle two projects:

• object-storage-tf: creation of an OVHcloud S3-compatible Object Sorage and an user and necessary policies
• my-app: usage of a backend.tf file that store and get TF states in your newly created S3-compatible bucket

Note that all the following source code are available on the OVHcloud Public Cloud examples GitHub repository.

Prerequisites:

• Install the Terraform CLI
• For non Linux users, install gettext (that included `envsubst` command)

$ brew install gettext

$ brew link --force gettext

• Get the credentials from the OVHCloud Public Cloud project

Let’s create an Object Storage with Terraform

Create a new folder, named object-storage-tf, for example and go into it.

Create a provider.tf file:

terraform {
  required_providers {
    ovh = {
      source  = "ovh/ovh"
    }
    
    random = {
      source  = "hashicorp/random"
      version = "3.6.3"
    }
  }
}

provider "ovh" {
}

The OVHcloud Terraform provider need the endpoint, the secret keys and the Public Cloud ID that needs to be retrieved from your environment variables:

• OVH_ENDPOINT
• OVH_APPLICATION_KEY
• OVH_APPLICATION_SECRET
• OVH_CONSUMER_KEY
• OVH_CLOUD_PROJECT_SERVICE

Then, create a variables.tf.template file with the following content:

variable "service_name" {
  default = "$OVH_CLOUD_PROJECT_SERVICE"
}


variable bucket_name {
  type        = string
}

variable bucket_region {
  type        = string
  default     = "GRA"
}

Replace the value of your OVH_CLOUD_PROJECT_SERVICE environment variable in the variables.tf file (in the service_name variable):

$ envsubst < variables.tf.template > variables.tf

Define the resources you want to create in a new file called s3.tf:

resource "random_string" "bucket_name_suffix" {
  length  = 16
  special = false
  lower   = true
  upper   = false
}

resource "ovh_cloud_project_storage" "s3_bucket" {
  service_name = var.service_name
  region_name = var.bucket_region
  name = "${var.bucket_name}-${random_string.bucket_name_suffix.result}" # the name must be unique within OVHcloud
}

resource "ovh_cloud_project_user" "s3_user" {
  description	= "${var.bucket_name}-${random_string.bucket_name_suffix.result}"
  role_name	= "objectstore_operator"
}

resource "ovh_cloud_project_user_s3_credential" "s3_user_cred" {
  user_id	= ovh_cloud_project_user.s3_user.id
}

resource "ovh_cloud_project_user_s3_policy" "s3_user_policy" {
  service_name = var.service_name
  user_id      = ovh_cloud_project_user.s3_user.id
  policy = jsonencode({
    "Statement": [{
      "Action": ["s3:*"],
      "Effect": "Allow",
      "Resource": ["arn:aws:s3:::${ovh_cloud_project_storage.s3_bucket.name}","arn:aws:s3:::${ovh_cloud_project_storage.s3_bucket.name}/*"],
      "Sid": "AdminContainer"
    }]
  })
}

In this file we defined that we want to create a S3-compatible Object Storage bucket and an user (with its credentials) that will have the rights (policies) to do actions on this bucket.

Define the information that you want to get after the creation of the resources, in an output.tf file:

output "s3_bucket" {
  value = "${ovh_cloud_project_storage.s3_bucket.name}"
}

output "access_key_id" {
    value = ovh_cloud_project_user_s3_credential.s3_user_cred.access_key_id
}

output "secret_access_key" {
    value = ovh_cloud_project_user_s3_credential.s3_user_cred.secret_access_key
    sensitive = true
}

Now we need to initialise Terraform:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/random versions matching "3.6.3"...
- Reusing previous version of ovh/ovh from the dependency lock file
- Installing hashicorp/random v3.6.3...
- Installed hashicorp/random v3.6.3 (signed by HashiCorp)
- Using previously-installed ovh/ovh v2.5.0

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Generate the plan and apply it:

$ terraform apply -var bucket_name=my-bucket

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # ovh_cloud_project_storage.s3_bucket will be created
  + resource "ovh_cloud_project_storage" "s3_bucket" {
      + created_at    = (known after apply)
      + encryption    = (known after apply)
      + limit         = (known after apply)
      + marker        = (known after apply)
      + name          = (known after apply)
      + objects       = (known after apply)
      + objects_count = (known after apply)
      + objects_size  = (known after apply)
      + owner_id      = (known after apply)
      + prefix        = (known after apply)
      + region        = (known after apply)
      + region_name   = "GRA"
      + replication   = (known after apply)
      + service_name  = "xxxxxxxxxxx"
      + versioning    = (known after apply)
      + virtual_host  = (known after apply)
    }

  # ovh_cloud_project_user.s3_user will be created
  + resource "ovh_cloud_project_user" "s3_user" {
      + creation_date = (known after apply)
      + description   = (known after apply)
      + id            = (known after apply)
      + openstack_rc  = (known after apply)
      + password      = (sensitive value)
      + role_name     = "objectstore_operator"
      + roles         = (known after apply)
      + service_name  = "xxxxxxxxxxx"
      + status        = (known after apply)
      + username      = (known after apply)
    }

  # ovh_cloud_project_user_s3_credential.s3_user_cred will be created
  + resource "ovh_cloud_project_user_s3_credential" "s3_user_cred" {
      + access_key_id     = (known after apply)
      + id                = (known after apply)
      + internal_user_id  = (known after apply)
      + secret_access_key = (sensitive value)
      + service_name      = "xxxxxxxxx"
      + user_id           = (known after apply)
    }

  # ovh_cloud_project_user_s3_policy.s3_user_policy will be created
  + resource "ovh_cloud_project_user_s3_policy" "s3_user_policy" {
      + id           = (known after apply)
      + policy       = (known after apply)
      + service_name = "xxxxxxxx"
      + user_id      = (known after apply)
    }

  # random_string.bucket_name_suffix will be created
  + resource "random_string" "bucket_name_suffix" {
      + id          = (known after apply)
      + length      = 16
      + lower       = true
      + min_lower   = 0
      + min_numeric = 0
      + min_special = 0
      + min_upper   = 0
      + number      = true
      + numeric     = true
      + result      = (known after apply)
      + special     = false
      + upper       = false
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + access_key_id     = (known after apply)
  + s3_bucket         = (known after apply)
  + secret_access_key = (sensitive value)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

random_string.bucket_name_suffix: Creating...
random_string.bucket_name_suffix: Creation complete after 0s [id=4qiyj7ywrt2sspfe]
ovh_cloud_project_user.s3_user: Creating...
ovh_cloud_project_storage.s3_bucket: Creating...
ovh_cloud_project_storage.s3_bucket: Creation complete after 1s [name=my-bucket-4qiyj7ywrt2sspfe]
ovh_cloud_project_user.s3_user: Still creating... [10s elapsed]
ovh_cloud_project_user.s3_user: Creation complete after 20s [id=535967]
ovh_cloud_project_user_s3_credential.s3_user_cred: Creating...
ovh_cloud_project_user_s3_policy.s3_user_policy: Creating...
ovh_cloud_project_user_s3_credential.s3_user_cred: Creation complete after 0s [id=5ab69860beb34575acb42c7ba8553884]
ovh_cloud_project_user_s3_policy.s3_user_policy: Creation complete after 0s [id=xxxxxxxxxxx/535967]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

access_key_id = "5ab69860beb34575acb42c7ba8553884"
s3_bucket = "my-bucket-4qiyj7ywrt2sspfe"
secret_access_key = <sensitive>

🎉

Save the s3 user credentials in environment variables (mandatory for the following section):

$ export AWS_ACCESS_KEY_ID=$(terraform output -raw access_key_id)
$ export AWS_SECRET_ACCESS_KEY=$(terraform output -raw secret_access_key)

Let’s configure an OVHcloud S3-compatible Object Storage as Terraform Backend

Create a new folder, named my-app, and go into it.

Create a backend.tf file with the following content:

⚠️ If you have a terraform version before 1.6.0:

terraform {
    backend "s3" {
      bucket = "<my-bucket>"
      key    = "my-app.tfstate"
      region = "gra"
      endpoint = "s3.gra.io.cloud.ovh.net"
      skip_credentials_validation = true
      skip_region_validation      = true
    }
}

⚠️ Since Terraform version 1.6.0:

terraform {
    backend "s3" {
      bucket = "<my-bucket>"
      key    = "my-app.tfstate"
      region = "gra"
      endpoints = {
        s3 = "https://s3.gra.io.cloud.ovh.net/"
      }
      skip_credentials_validation = true
      skip_region_validation      = true
      skip_requesting_account_id  = true
      skip_s3_checksum            = true
    }
}

You can replace <my-bucket> with the newly created bucket or with an existing bucket you created.

Initialise Terraform:

$ terraform init

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding latest version of ovh/ovh...
- Installing ovh/ovh v2.5.0...
- Installed ovh/ovh v2.5.0 (signed by a HashiCorp partner, key ID F56D1A6CBDAAADA5)

...

As you can see, now, terraform is using “s3” backend! 💪

Want to go further?

In this blog post, we created an S3-compatible Object Storage with basic configuration but be aware that you can configure a S3-compatible bucket with encryption, versioning and more:

💡 Terraform States are not encrypted at rest when stored by Terraform so we recommend to enable the encryption the OVHcloud S3-compatible Object Storage bucket 🙂.

Kubernetes 1.33 version has just been released few days/weeks ago.
As this new release contains 64 enhancements (!), it can not be easy to know what are the interesting and useful features and how to use them.

In this blog post, let’s discover one of interesting and useful new feature: “Topology aware routing in multi-zones Kubernetes clusters”.

⚠️ Kubernetes 1.33 should be available on OVHcloud MKS clusters at the end of June/beginning of July but the demo is working also on MKS with Kubernetes 1.32 release 😉.

Topology aware routing

Since Kubernetes 1.33, the topology aware routing and traffic distribution feature is in General Availability (GA).

This feature allows to optimize service traffic in multi-zone clusters and reduce latency and cross-zone data transfer cost.

Topology Aware Routing provides a mechanism to help keep traffic within the zone it originated from.

In a context of multi-zone clusters, it helps reliability, performance, reduce costs or improve network performance.

As OVHcloud just released, in Beta, the launch of their Managed Kubernetes clusters (MKS) on 3 AZ (Availability Zones), it’s the perfect occasion for me to test this brand new Kubernetes feature 🙂.

Demo

Prerequisite: Have a Kubernetes cluster with at least 2 nodes running in 2 different zones.

If you already don’t have one, you can follow this blog post in order to create an OVHcloud MKS cluster with 3 nodes pools, one per AZ.

On my side I set-up a MKS cluster in 3AZ (one per node pool), with 3 nodes per node pool:

$ kubectx kubernetes-admin@multi-zone-mks
Switched to context "kubernetes-admin@multi-zone-mks".

$ kubectl get np
NAME             FLAVOR   AUTOSCALED   MONTHLYBILLED   ANTIAFFINITY   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   MIN   MAX   AGE
my-pool-zone-a   b3-8     false        false           false          3         3         3            3           0     100   20d
my-pool-zone-b   b3-8     false        false           false          3         3         3            3           0     100   20d
my-pool-zone-c   b3-8     false        false           false          3         3         3            3           0     100   20d

$ kubectl get no
NAME                         STATUS   ROLES    AGE   VERSION
my-pool-zone-a-b9ztj-brgpq   Ready    <none>   20d   v1.32.3
my-pool-zone-a-b9ztj-gt5vd   Ready    <none>   20d   v1.32.3
my-pool-zone-a-b9ztj-mss8j   Ready    <none>   20d   v1.32.3
my-pool-zone-b-tr6wf-5wfgz   Ready    <none>   20d   v1.32.3
my-pool-zone-b-tr6wf-ct7fs   Ready    <none>   20d   v1.32.3
my-pool-zone-b-tr6wf-vlkwg   Ready    <none>   20d   v1.32.3
my-pool-zone-c-wgrl6-b2f9s   Ready    <none>   20d   v1.32.3
my-pool-zone-c-wgrl6-lp22l   Ready    <none>   20d   v1.32.3
my-pool-zone-c-wgrl6-slkq5   Ready    <none>   20d   v1.32.3

⚠️ As you saw, the Kubernetes version installed on my cluster is not equals to 1.33, but the ServiceTrafficDistribution feature gate is in Beta and it is activated:

$ kubectl get --raw /metrics | grep kubernetes_feature_enabled | grep Traffic

kubernetes_feature_enabled{name="ServiceTrafficDistribution",stage="BETA"} 1

A visual architecture of my MKS cluster:

⚠️ In MKS Standard clusters, don’t forget to enable the topology aware routing for 3AZ region.

In order to test this feature, in a new namespace, we will deploy:

• a deployment with two pods named receiver-xxx
• a ClusterIP service named svc-prefer-close with the feature enabled
• a Pod named sender

Let’s do that!

Create a deploy.yaml file with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: service-traffic-example
  name: receiver
  namespace: prefer-close
spec:
  replicas: 2
  selector:
    matchLabels:
      app: service-traffic-example
  template:
    metadata:
      labels:
        app: service-traffic-example
    spec:
      containers:
      - image: scraly/hello-pod:1.0.1
        name: receiver
        ports:
        - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName

Create a svc.yaml file with the following content:

apiVersion: v1
kind: Service
metadata:
  name: svc-prefer-close
  namespace: prefer-close
  annotations:
    service.kubernetes.io/topology-mode: auto
spec:
  ports:
    - name: http
      protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: service-traffic-example
  type: ClusterIP
  trafficDistribution: PreferClose

As you can see, this Service has two specific configurations.
First, we added the service.kubernetes.io/topology-mode: auto annotation to enable Topology Aware Routing for a Service.
Then, we configured the trafficDistribution to PreferClose in order to ask Kubernetes to send the traffic, preferably, to a pod that is “closed” to the sender.

Create a new namespace and apply the manifest files:

$ kubectl create ns prefer-close
$ kubectl apply -f deploy.yaml
$ kubectl apply -f svc.yaml

Result:
You should have two running Pods on 2 differents Nodes.

$ kubectl get po -o wide -n prefer-close

NAME                        READY   STATUS              RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
receiver-7cfd89d78d-dhv6z   1/1     Running             0          94s   10.240.4.91   my-pool-zone-c-wgrl6-slkq5   <none>           <none>
receiver-7cfd89d78d-hrxrt   1/1     Running             0          94s   10.240.5.63   my-pool-zone-a-b9ztj-mss8j   <none>           <none>

OK, receiver-xxxxxxxx-dhv6z is running on my-pool-zone-c-xxxx and the other pod is running on my-pool-zone-a-xxxx. There are running on differents Availability Zones.

Now, we can create a Pod sender. it will be scheduled on a Node:

Run it and execute a curl command to test the traffic redirection to the “svc-prefer-close” Service:

$ kubectl run sender -n prefer-close --image=curlimages/curl -it -- sh
If you don't see a command prompt, try pressing enter.
~ $ curl http://svc-prefer-close.prefer-close:8080
Version: 1.0.1
Hostname: receiver-7cfd89d78d-dhv6z
Node: my-pool-zone-c-wgrl6-slkq5

Let’s verify where are our Pods:

$ kubectl get po -n prefer-close -o wide
NAME                        READY   STATUS    RESTARTS     AGE   IP             NODE                         NOMINATED NODE   READINESS GATES
receiver-7cfd89d78d-dhv6z   1/1     Running   0            9d    10.240.4.91    my-pool-zone-c-wgrl6-slkq5   <none>           <none>
receiver-7cfd89d78d-hrxrt   1/1     Running   0            9d    10.240.5.63    my-pool-zone-a-b9ztj-mss8j   <none>           <none>
sender                      1/1     Running   1 (5s ago)   21s   10.240.3.134   my-pool-zone-c-wgrl6-b2f9s   <none>           <none>

Kube-proxy sent the traffic from sender to a receiver-xx Pod on the same Availability Zone 🎉

⚠️ Note that because preferClose means “topologically proximate”, it may vary across implementations and could encompass endpoints within the same node, rack, zone, or even region.

How is it working?

When calculating the endpoints for a Service, the EndpointSlice controller considers the topology (region and zone) of each endpoint and populates the hints field to allocate it to a zone.

Cluster components such as kube-proxy can then consume those hints, and use them to influence how the traffic is routed (favoring topologically closer endpoints).

So, with PreferClose value for trafficDistribution, we ask kube-proxy to redirect traffic to the nearest available endpoints based on the network topology.

That’s why the option is called PreferClose.

What’s next?

In the future you will be able to configure the trafficDistribution field with other values.

Indeed, two new values, more explicit, are currently in Alpha since the Kubernetes 1.33 release: PreferSameZone and PreferSameNode.

Personally I can’t wait to test them 😇.

Want to go further?

Want to learn more on this topic? In the coming days, we will publish a blog post about MKS Premium plan.

Visit our Managed Kubernetes Service (MKS) Premium plan in the OVHcloud Labs website to know more about Premium MKS.

Join the free Beta: https://labs.ovhcloud.com/en/managed-kubernetes-service-mks-premium-plan/

Read the documentation about the new Managed Kubernetes Service (MKS) Premium plan.

Join us on Discord and give us your feedbacks.

This blog post will first explain briefly what is the new MKS Premium plan, for who and which use case, then you will see how to deploy a new MKS cluster in 3 availability zones and how to deploy your workloads with this new architecture of Kubernetes cluster.

What’s inside the Premium MKS?

The 30th of April, we launched, in Beta, our brand new “Premium plan” of our Managed Kubernetes Services (MKS) 🎉

Concretely, with MKS Premium you will have:

💡 For the moment, only Paris is available for the 3AZ region but several new regions will be available in the coming months including Milan.

Behind this new plan, this new version of our MKS offering actually represents a complete overhaul of our platform based on several Cloud Native Open Source projects like Cluster API, Kamaji, ArgoCD and several homemade Kubernetes operators.

For who? For what?

The new MKS Premium plan has been designed for those who wants high availability and scalability of their critical applications.

Thanks to a dedicated and fully managed control plane, resilience across multiple availability zones, dedicated resources for the Kubernetes control plane, and the ability to deploy the data plane across multiple availability zones.

You will be able to design cloud-native applications that are resilient to failures and deploy highly resilient cloud-native applications across our multi-zones region.

You will have the full control on how to deploy your worker node in our new 3AZ region (EU-WEST-PAR).

Deploying your cloud-native applications in our new Paris 3-AZ region also means enjoying the full range of services available:

• Well architected application relying on resilient managed services (MKS + Load Balancer + Gateway + DBaaS + Object Storage …),
• Advanced internal cluster networking with the new Cilium CNI
• Better API server performances and scaling capacity
• And much more to come!

Let’s deploy a MKS Premium cluster in 3 AZ at Paris!

Like the actual Standard MKS, you can deploy MKS on the 3AZ via the Control Panel (OVHcloud UI), the API and also our Infrastructure as Code (IaC) providers (Terraform/OpenTofu, Pulumi…).

In this blog post, we will deploy a new MKS cluster, in a 3AZ region (Paris) with 3 node pools (one per availability zone).

With OVHcloud Control Panel

Log in to the OVHcloud Control Panel, go to the Public Cloud section and select the Public Cloud project concerned.

In the left panel, go in the Containers & Orchestration section, click on Managed Kubernetes Service link and click on the Create a Kubernetes cluster button

Fill the name of the cluster, choose a 3AZ region, click on Paris (EU-WEST-PAR) and select the Premium plan:

Then, select the Kubernetes version and the security policy.

⚠️ Contrary to the Standard MKS, which is public by default, the Premium MKS is private by default so it is mandatory to create a private network, a subnet and a gateway.

Then, create one node pool by Availability Zone, with 3 nodes by node pool, for example:

Confirm the creation of your cluster and wait its creation.

Finally, click on the new created cluster and get the kubeconfig file.

With Terraform

In a previous blog post, we showed you how to deploy a MKs cluster with Terraform/OpenTofu. Please read the post if you are not familiar with Terraform or OpenTofu.

Create a ovh_kube.tf file with the following content:

resource "ovh_cloud_project_network_private" "network" {
  service_name = var.service_name
  vlan_id      = 84
  name         = "terraform_mks_multiaz_private_net"
  regions      = ["EU-WEST-PAR"]
}

resource "ovh_cloud_project_network_private_subnet" "subnet" {
  service_name = ovh_cloud_project_network_private.network.service_name
  network_id   = ovh_cloud_project_network_private.network.id

  # whatever region, for test purpose
  region     = "EU-WEST-PAR"
  start      = "192.168.142.100"
  end        = "192.168.142.200"
  network    = "192.168.142.0/24"
  dhcp       = true
  no_gateway = false
}

resource "ovh_cloud_project_gateway" "gateway" {
  service_name = ovh_cloud_project_network_private.network.service_name
  name       = "gateway"
  model      = "s"
  region     = "EU-WEST-PAR"
  network_id = tolist(ovh_cloud_project_network_private.network.regions_attributes[*].openstackid)[0]
  subnet_id  = ovh_cloud_project_network_private_subnet.subnet.id
}

resource "ovh_cloud_project_kube" "my_multizone_cluster" {
  service_name  = ovh_cloud_project_network_private.network.service_name
  name          = "multi-zone-mks"
  region        = "EU-WEST-PAR"
  plan          = "standard"

  private_network_id = tolist(ovh_cloud_project_network_private.network.regions_attributes[*].openstackid)[0]
  nodes_subnet_id    = ovh_cloud_project_network_private_subnet.subnet.id

  depends_on    = [ ovh_cloud_project_gateway.gateway ] //Gateway is mandatory for multizones cluster
}

resource "ovh_cloud_project_kube_nodepool" "node_pool_multi_zones_a" {
  service_name       = ovh_cloud_project_network_private.network.service_name
  kube_id            = ovh_cloud_project_kube.my_multizone_cluster.id
  name               = "my-pool-zone-a" //Warning: "_" char is not allowed!
  flavor_name        = "b3-8"
  desired_nodes      = 3
  availability_zones = ["eu-west-par-a"] //Currently, only one zone is supported
}

resource "ovh_cloud_project_kube_nodepool" "node_pool_multi_zones_b" {
  service_name       = ovh_cloud_project_network_private.network.service_name
  kube_id            = ovh_cloud_project_kube.my_multizone_cluster.id
  name               = "my-pool-zone-b"
  flavor_name        = "b3-8"
  desired_nodes      = 3
  availability_zones = ["eu-west-par-b"]
}

resource "ovh_cloud_project_kube_nodepool" "node_pool_multi_zones_c" {
  service_name       = ovh_cloud_project_network_private.network.service_name
  kube_id            = ovh_cloud_project_kube.my_multizone_cluster.id
  name               = "my-pool-zone-c"
  flavor_name        = "b3-8"
  desired_nodes      = 3
  availability_zones = ["eu-west-par-c"]
}

output "kubeconfig_file_eu_west_par" {
  value     = ovh_cloud_project_kube.my_multizone_cluster.kubeconfig
  sensitive = true
}

This HCL configuration will create several OVHcloud services:

• a private network
• a subnet
• a gateway (S size)
• a MKS cluster in EU_WEST_PAR region
• one node pool in eu-west-par-a availability zone with 3 nodes
• one node pool in eu-west-par-b availability zone with 3 nodes
• one node pool in eu-west-par-c availability zone with 3 nodes

Apply the configuration:

$ terraform apply

...

ovh_cloud_project_network_private.network: Creating...
ovh_cloud_project_network_private.network: Still creating... [10s elapsed]
ovh_cloud_project_network_private.network: Creation complete after 14s [id=pn-xxxxxxxx_xx]
ovh_cloud_project_network_private_subnet.subnet: Creating...
ovh_cloud_project_network_private_subnet.subnet: Creation complete after 3s [id=c14cbb87-xxxx-xxxx-xxxx-7b9d4940d857]
ovh_cloud_project_gateway.gateway: Creating...
ovh_cloud_project_gateway.gateway: Still creating... [10s elapsed]
ovh_cloud_project_gateway.gateway: Creation complete after 13s [id=7dafdcfe-xxxx-xxxx-xxxx-240df8f93af1]
ovh_cloud_project_kube.my_multizone_cluster: Creating...
ovh_cloud_project_kube.my_multizone_cluster: Still creating... [10s elapsed]
ovh_cloud_project_kube.my_multizone_cluster: Still creating... [20s elapsed]
ovh_cloud_project_kube.my_multizone_cluster: Still creating... [30s elapsed]
...
ovh_cloud_project_kube.my_multizone_cluster: Still creating... [1m40s elapsed]
ovh_cloud_project_kube.my_multizone_cluster: Still creating... [1m50s elapsed]
ovh_cloud_project_kube.my_multizone_cluster: Still creating... [2m0s elapsed]
ovh_cloud_project_kube.my_multizone_cluster: Creation complete after 2m2s [id=0196cd9a-xxxx-xxxx-xxxx-3acbb48d6dda]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Creating...
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Creating...
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Creating...
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Still creating... [10s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Still creating... [10s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Still creating... [10s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Still creating... [20s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Still creating... [20s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Still creating... [20s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Still creating... [30s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Still creating... [30s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Still creating... [30s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Still creating... [40s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Still creating... [40s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Still creating... [40s elapsed]
...
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Still creating... [4m0s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Still creating... [4m0s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Still creating... [4m10s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Still creating... [4m10s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Still creating... [4m10s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Still creating... [4m20s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Still creating... [4m20s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Still creating... [4m20s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_c: Creation complete after 4m24s [id=0196cd9c-xxxx-xxxx-xxxx-8e1925c4c18e]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_b: Creation complete after 4m24s [id=0196cd9c-xxxx-xxxx-xxxx-96a18b9202ff]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Still creating... [4m30s elapsed]
ovh_cloud_project_kube_nodepool.node_pool_multi_zones_a: Creation complete after 4m35s [id=0196cd9c-xxxx-xxxx-xxxx-8a08cdc2e68d]

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

kubeconfig_file_eu_west_par = <sensitive>

Our MKS in 3AZ have been deployed 🎉

To connect into it, retrieve the kubeconfig file locally:

$ terraform output -raw kubeconfig_file_eu_west_par > ~/.kube/multi-zone-mks.yml

Connect and discover your MKS cluster

Initialize or append the KUBECONFIG environment variable with the new kubeconfig files:

export KUBECONFIG=/Users/my-user/.kube/mks.yml:/Users/my-user/.kube/multi-zone-mks.yml

Display the node pools. Our cluster have 3 nodes pools, one per AZ:

$ kubectl get np
NAME             FLAVOR   AUTOSCALED   MONTHLYBILLED   ANTIAFFINITY   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   MIN   MAX   AGE
my-pool-zone-a   b3-8     false        false           false          3         3         3            3           0     100   7h8m
my-pool-zone-b   b3-8     false        false           false          3         3         3            3           0     100   7h8m
my-pool-zone-c   b3-8     false        false           false          3         3         3            3           0     100   7h8m

You can also display the control plane’s pods in order to discover the new components of the MKS Premium:

$ kubectl get po -n kube-system

How To

Deploy pods accross several availability zones

Now, let’s create a Depoyment with 6 pods and ask Kubernetes to deploy them in our 3 AZ (in the three node pools).

To do that, create a nginx-cross-az.yaml file with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-cross-az
  labels:
    app: nginx-cross-az
spec:
  replicas: 6
  selector:
    matchLabels:
      app: nginx-cross-az
  template:
    metadata:
      labels:
        app: nginx-cross-az
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "topology.kubernetes.io/zone"
                operator: In
                values:
                - eu-west-par-a
                - eu-west-par-b
                - eu-west-par-c
      containers:
      - name: nginx
        image: nginx:1.28.0
        ports:
        - containerPort: 80

Thanks to the nodeAffinity feature of Kubernetes, we declare that we want 6 replicas (pods) running in 3 zones: eu-west-par-a, eu-west-par-b, eu-west-par-c.

Create a new namespace and apply the deployment:

$ kubectl create ns hello-app
$ kubectl apply -f nginx-cross-az.yaml -n hello-app

As you can see, 6 pods have been created, and they are running on the nodes located in the 3 AZ.

$ kubectl get po -o wide -l app=nginx-cross-az -n hello-app
NAME                             READY   STATUS    RESTARTS   AGE    IP             NODE                         NOMINATED NODE   READINESS GATES
nginx-cross-az-6ffd957c4-7528p   1/1     Running   0          6s     10.240.2.140   my-pool-zone-b-tr6wf-5wfgz   <none>           <none>
nginx-cross-az-6ffd957c4-96mnh   1/1     Running   0          6s     10.240.3.91    my-pool-zone-c-wgrl6-b2f9s   <none>           <none>
nginx-cross-az-6ffd957c4-b48cv   1/1     Running   0          115m   10.240.6.182   my-pool-zone-c-wgrl6-lp22l   <none>           <none>
nginx-cross-az-6ffd957c4-k7rwf   1/1     Running   0          115m   10.240.1.237   my-pool-zone-b-tr6wf-ct7fs   <none>           <none>
nginx-cross-az-6ffd957c4-pb7zp   1/1     Running   0          115m   10.240.8.195   my-pool-zone-a-b9ztj-gt5vd   <none>           <none>
nginx-cross-az-6ffd957c4-vhhcw   1/1     Running   0          6s     10.240.7.40    my-pool-zone-a-b9ztj-brgpq   <none>           <none>

Deploy pods only in a desired availability zone

You can also choose to deploy a Deployment with 3 replicas, only in the AZ of your choice, only in eu-west-par-a for example.

Create a nginx-one-az.yaml file with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-one-az
  labels:
    app: nginx-one-az
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-one-az
  template:
    metadata:
      labels:
        app: nginx-one-az
    spec:
      nodeSelector:
        topology.kubernetes.io/zone: eu-west-par-a
      containers:
      - name: nginx
        image: nginx:1.28.0
        ports:
        - containerPort: 80

Deploy the manifest file in your cluster:

$ kubectl apply -f nginx-one-az.yaml -n hello-app
deployment.apps/nginx-one-az created

As you can see, our three pods are running in the PAR region only in the zone-a nodes:

$ kubectl get po -o wide -l app=nginx-one-az -n hello-app
NAME                            READY   STATUS    RESTARTS   AGE    IP             NODE                         NOMINATED NODE   READINESS GATES
nginx-one-az-6b5f9bdccc-8vv9l   1/1     Running   0          98s    10.240.7.13    my-pool-zone-a-b9ztj-brgpq   <none>           <none>
nginx-one-az-6b5f9bdccc-ck99s   1/1     Running   0          100s   10.240.5.216   my-pool-zone-a-b9ztj-mss8j   <none>           <none>
nginx-one-az-6b5f9bdccc-tlg4d   1/1     Running   0          96s    10.240.8.221   my-pool-zone-a-b9ztj-gt5vd   <none>           <none>

Want to go further?

Want to learn more on this topic? In the coming days, we will publish a blog post about MKS Premium plan.

Visit our Managed Kubernetes Service (MKS) Premium plan in the OVHcloud Labs website to know more about Premium MKS.

Join the free Beta: https://labs.ovhcloud.com/en/managed-kubernetes-service-mks-premium-plan/

Read the documentation about the new Managed Kubernetes Service (MKS) Premium plan.

Join us on Discord and give us your feedbacks.

For the past few months, Docker has been announcing the implementation of new pull rate limits for the Docker Hub. The most significant change is the 10 pulls-per-hour limit, per IP address, for unauthenticated users that can quickly lead to a “You have reached your pull rate limit” error message.

Even if these changes have been implemented and rollbacked as of April 1, 2025, at OVHcloud, we are aware that these upcoming changes could impact your deployments and daily work.

In this blog post, you will find several solutions and best practices that can help you reduce Docker pull commands and avoid hitting Docker Hub’s pull rate limit.

Use OVHcloud Managed Private Registry and activate the proxy cache

OVHcloud Managed Private Registry (MPR) is a container image registry, based on CNCF project Harbor. It allows you to store and manage Docker (or OCI-compliant) container images and artifacts in a private, secure, and scalable environment, hosted in OVHcloud’s infrastructure.

MPR provides a proxy cache feature that helps you mirror and cache images from external registries, like Docker Hub, Github Container Registry, Quay, JFrog Artifactory Registry, etc. External registries can be private or public. This improves performance and reduces rate limits imposed by external registries 💪.

Configure proxy cache in OVHcloud Managed Private Registry

If you don’t have deployed a MPR yet, you can deploy it through the OVHcloud Control Panel, the OVHcloud Terraform provider, the OVHcloud Pulumi provider and even the API. Follow the guide according to your needs.

First, log in the Harbor user interface on your private registry, follow the guide if you needed to.

⚠️ In order to activate the proxy cache, you need to log in the Harbor UI with an administrator account.

Registry endpoint creation

In the left sidebar, click on Registries (inside the Administration section).

Then click on the New endpoint button.

Select Docker Hub in the provider list, enter a name (“Docker Hub” for example), fill your Docker Hub login in Access ID field and fill your Docker Hub password in Access Secret field.

⚠️ Note that we strongly recommend using a Docker account (even a free one) to avoid rate limits, for unanthenticated users, when pulling images. Without authentication, Docker Hub enforces strict pull limits, which may cause failures when pulling frequently used images.

Click on the Test connection button to test if your login and password are correct.

Now click on the OK button in order to create the new endpoint.

The Docker Hub endpoint is created 🎉

Proxy cache project creation

In the left sidebar, click on Projects, then click on the New project button.

Enter a project name (“docker-hub” for example), enable the Proxy Cache, click on the Docker Hub endpoint in the list and click on the OK button.

ℹ️ Note that a project is private by default, so you have to click on the Public checkbox if you want to change the visibilty of a project.

⚠️ The name of a proxy cache project should not contains dot(s), indeed it can causes issues with external tools like Kaniko.

Your proxy cache project have been created 🎉

⚠️ A proxy cache project works similarly to a normal Harbor project, except that you are not able to push images to a proxy cache project.

Now when you want to pull a Docker images hosted in the Docker Hub you proxy cached, instead of pulling directly from the Docker Hub, you need to configure your docker/podman pull commands and Kubernetes pod manifests to pull images from the OVHcloud Managed Private Registry:

$ docker pull xxxxxxxx.c1.de1.container-registry.ovh.net/docker-hub/ovhcom/ovh-platform-hello:latest
latest: Pulling from docker-hub/ovhcom/ovh-platform-hello
1f3e46996e29: Pull complete 
6aa905c35cc0: Pull complete 
Digest: sha256:fddb76f0eb92d95b3721bfa0ea87350c5d39ea262e90cd30d66f429bb40c8b07
Status: Downloaded newer image for xxxxxxxx.c1.de1.container-registry.ovh.net/docker-hub/ovhcom/ovh-platform-hello:latest
xxxxxxxx.c1.de1.container-registry.ovh.net/docker-hub/ovhcom/ovh-platform-hello:latest

Disable the AlwaysPullImages admission plugin on your MKS cluster

By default, the AlwaysPullImages Kubernetes admission plugin is enabled in your OVHcloud Managed Kubernetes (MKS) cluster.

⚠️ When it is enabled, this forces the imagePullPolicy of a container to be set to Always, no matter how it is specified when creating the resource.

This is useful in a multitenant cluster so that users can be assured that their private images can only be used by those who have the credentials to pull them. Without this admission controller, once an image has been pulled to a node, any pod from any user can use it by knowing the image’s name (assuming the Pod is scheduled onto the right node), without any authorization check against the image.

But, it can cause a lot of pull requests to the Docker Hub and you can reach the rate limits.

So a solution can be to deactivate the AlwaysPullImages admission plugin in your MKS cluster.

In this blog post, we will deactivate it in the OVHcloud Control Panel.

Enable/Disable MKS admission plugins

Log in the OVHcloud Control Panel. In the left sidebar, click on the Managed Kubernetes Service and then click on the wanted MKS cluster.

In the Cluster information section, scroll down and click on Enable/disable plugin. A popup will appear.

Then click on Disable for the Always Pull Images plugin and click on the Save button.

⚠️ Any changes on the Admission plugins require a redeployment of the MKS cluster API server (without data loss) so the API server can be potentially not available during the redeployment.

Conclusion

To learn more about how to use and configure OVHcloud private registries and OVHcloud MKS clusters, don’t hesitate to follow our guides.

Several month ago we discovered Falco, a Cloud Native near real-time threats detection tool, and we saw how to install it on an OVHcloud MKS cluster.

Today we will connect our Falco instance to a MKS cluster in order to retrieve Kubernetes Audit Logs events and watch if everything is OK in our cluster.

Concretely, in this blog post we will:

• deploy an OVHcloud LDP (Logs Data Platform)
• create a data stream into this LDP
• connect an OVHcloud MKS cluster to the data stream (to send Audit Logs into it)
• use the k8saudit-ovh Falco plugin to retrieve in realtime the Audit Logs of a MKS cluster
• test a rule and detect security events based on MKS audit logs activity

Prerequisites

This blog post presupposes that you already have a working OVHcloud Managed Kubernetes (MKS) cluster, and a running instance of Falco.

If it is not the case, follow the Near real-time threats detection with Falco on OVHcloud Managed Kubernetes blog post.

Deploying a Logs Data Platform (LDP)

LDP is the managed platform for collecting, processing, analyzing and storing your logs of the OVHcloud products. To be able to access to our Kubernetes clusters Audit Logs we need to deploy a LDP.

Find more information on our dedicated LDP page.

We can deploy a LDP through the OVHcloud Control Panel and the API. In this blog post, we will deploy it through the Control Panel.

First, you have to log in to the OVHcloud Control Panel, click on the Bare Metal Cloud section located at the top in the header and then click on the Logs Data Platform in the sidebar.

Choose the LDP plan you want: Standard (free) or Enterprise one, depending on your needs.

Select a region (North America or Europe). We will choose “GRA” for this blog post, click on Order button and follow the instructions.

After several minutes your LDP will be created.

Refresh the page, click on the new deployed LDP, then enter a password and click on the Save button.

Creating a Data stream and retrieving the Websocket URL

Our Kubernetes Audit Logs will be stored in a data stream so click on the Data stream tab and then click on the Add data stream button.

Choose a name of the data stream. On my side I like to call it with the name of my MKS cluster following by “-audit-logs” to know easily what it is this data stream for. My MKS cluster’s name is “my-rancher-mks-cluster” so let’s name it “my-rancher-mks-cluster-audit-logs”. Fill the description (mandatory).

The OVHcloud Audit Logs Falco plugin you will use receive the audit logs through Websocket so you need to enable Websocket broadcasting then click on the Save button.

Now, to retrieve the Websocket URL of your data stream, click on the Data stream tab, then click on the … button (located at the right in the line of your data stream), and click on Monitor in real time action.

Finally, click on the Action button and in the Copy Websocket address, then save the LDP Websocket URL somewhere ;-).

Note that the Websocket address have this kind of format: wss://<region>.logs.ovh.com/tail/?tk=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx

Connect a MKS cluster to a LDP data stream

Now we need to send the Kubernetes Audit Logs of our MKS cluster in the data stream.

For that, in the OVHcloud Control Panel, click on the Public Cloud section in the header and then in Managed Kubernetes Service in the sidebar.

Click on your Kubernetes cluster (my-rancher-mks-cluster for example), then in the Logs tab and click on the Subscribe button.

Click on the Add data stream button to visualize in real time the Audit Logs of your cluster. Then select the LDP instance and click on the Subscribe button for the data stream your created:

Retrieve the MKS Audit Logs with Falco

Falco can receive Events, compare them to a set of Rules to determine the actions to perform and generate Alerts to different endpoints.

Thanks to the k8saudit-ovh plugin, Falco can receive a new sort of Events: the Audit Logs of your MKS cluster. These events have also some rules to follow.

Concretely, when an user will execute some kubectl commands in an OVHcloud MKS cluster, Audit Logs will be generated. Falco is listening from them and depending on the configured rules, it will generate some alerts.

Let’s install or update a Falco configuration running in a MKS cluster and use this plugin.

Create a values.yaml file with the following content:

tty: true
kubernetes: false

# Just a Deployment with 1 replica (instead of a Daemonset) to have only one Pod that pulls the MKS Audit Logs from a OVHcloud LDP
controller:
  kind: deployment
  deployment:
    replicas: 1

falco:
  rule_matching: all
  rules_files:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-ovh
      library_path: libk8saudit-ovh.so
      open_params: "<region>.logs.ovh.com/tail/?tk=<ID>" # Replace with your LDP Websocket URL
    - name: json
      library_path: libjson.so
      init_config: ""
  # Plugins that Falco will load. Note: the same plugins are installed by the falcoctl-artifact-install init container.
  load_plugins: [k8saudit-ovh, json]

driver:
  enabled: false
collectors:
  enabled: false

# use falcoctl to install automatically the plugin and the rules
falcoctl:
  artifact:
    install:
      enabled: true
    follow:
      enabled: true
  config:
    indexes:
    - name: falcosecurity
      url: https://falcosecurity.github.io/falcoctl/index.yaml
    artifact:
      allowedTypes:
        - plugin
        - rulesfile
      install:
        resolveDeps: false
        refs: [k8saudit-rules:0, k8saudit-ovh:0.1, json:0]
      follow:
        refs: [k8saudit-rules:0]

This values.yaml file will install Falco with the k8saudit-ovh and the json plugins.

Install the latest version of Falco with helm install command:

$ helm install falco --create-namespace --namespace falco --values=values.yaml falcosecurity/falco

This command will install the latest version of Falco, with the k8saudit-ovh and json plugins, and create a new falco namespace:

$ helm install falco --create-namespace --namespace falco --values=values.yaml falcosecurity/falco

NAME: falco
LAST DEPLOYED: Mon Feb 10 10:15:20 2025
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
No further action should be required.

Or if you already have Falco deployed in a Kubernetes cluster, you can use the helm update command instead:

$ helm upgrade falco --create-namespace --namespace falco --values=values.yaml falcosecurity/falco

You can check if the Falco pods are correctly running:

$ kubectl get pods -n falco

NAME                                      READY   STATUS    RESTARTS   AGE
falco-6b8bc77d8b-v24jr                    2/2     Running   0          96s
falco-falcosidekick-67877d6946-4hmbn      1/1     Running   0          96s
falco-falcosidekick-67877d6946-tpjk6      1/1     Running   0          96s
falco-falcosidekick-ui-78b96fd57d-4wb6q   1/1     Running   0          96s
falco-falcosidekick-ui-78b96fd57d-v7rnm   1/1     Running   0          96s
falco-falcosidekick-ui-redis-0            1/1     Running   0          96s

Wait and execute the command again if the pods are in “Init” or “ContainerCreating” state.

Once the Falco pod is ready, run the following command to see the logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco

You should see logs like that:

$ kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco

Mon Feb 10 09:15:35 2025:    /etc/falco/k8s_audit_rules.yaml | schema validation: ok
Mon Feb 10 09:15:35 2025: Hostname value has been overridden via environment variable to: my-pool-1-node-921b61
Mon Feb 10 09:15:35 2025: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Mon Feb 10 09:15:35 2025: Starting health webserver with threadiness 2, listening on 0.0.0.0:8765
Mon Feb 10 09:15:35 2025: Loaded event sources: syscall, k8s_audit
Mon Feb 10 09:15:35 2025: Enabled event sources: k8s_audit
Mon Feb 10 09:15:35 2025: Opening 'k8s_audit' source with plugin 'k8saudit-ovh'
{"hostname":"my-pool-1-node-921b61","output":"09:15:40.698757000: Warning K8s Operation performed by user not in allowed list of users (user=csi-cinder-controller target=csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/volumeattachments verb=patch uri=/apis/storage.k8s.io/v1/volumeattachments/csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/status resp=200)","output_fields":{"evt.time":1739178940698757000,"ka.response.code":"200","ka.target.name":"csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3","ka.target.resource":"volumeattachments","ka.uri":"/apis/storage.k8s.io/v1/volumeattachments/csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/status","ka.user.name":"csi-cinder-controller","ka.verb":"patch"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:40.698757000Z"}
{"hostname":"my-pool-1-node-921b61","output":"09:15:57.508657000: Warning K8s Operation performed by user not in allowed list of users (user=yacht target=my-pool-1.18051c0a88716868/events verb=patch uri=/api/v1/namespaces/default/events/my-pool-1.18051c0a88716868 resp=403)","output_fields":{"evt.time":1739178957508657000,"ka.response.code":"403","ka.target.name":"my-pool-1.18051c0a88716868","ka.target.resource":"events","ka.uri":"/api/v1/namespaces/default/events/my-pool-1.18051c0a88716868","ka.user.name":"yacht","ka.verb":"patch"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:57.508657000Z"}
{"hostname":"my-pool-1-node-921b61","output":"09:15:57.807013000: Warning K8s Operation performed by user not in allowed list of users (user=yacht target=my-pool-1/nodepools verb=update uri=/apis/kube.cloud.ovh.com/v1alpha1/nodepools/my-pool-1/status resp=200)","output_fields":{"evt.time":1739178957807013000,"ka.response.code":"200","ka.target.name":"my-pool-1","ka.target.resource":"nodepools","ka.uri":"/apis/kube.cloud.ovh.com/v1alpha1/nodepools/my-pool-1/status","ka.user.name":"yacht","ka.verb":"update"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:57.807013000Z"}

The logs confirm that Falco k8saudit-ovh plugin and the k8saudit rules have been loaded correctly 💪.

Testing Falco

In order to test Falco we need to know which rules are installed by default. In our case, as we defined it in the values.yaml file, the k8saudit-ovh plugin follow the k8s_audit_rules.yaml file. You can take a look at them in order to know them.

In this blog post we will test one of well-known default k8s audit rules:

- rule: Attach/Exec Pod
  desc: >
    Detect any attempt to attach/exec to a pod
  condition: kevt_started and pod_subresource and (kcreate or kget) and ka.target.subresource in (exec,attach) and not user_known_exec_pod_activities
  output: Attach/Exec to pod (user=%ka.user.name pod=%ka.target.name resource=%ka.target.resource ns=%ka.target.namespace action=%ka.target.subresource command=%ka.uri.param[command])
  priority: NOTICE
  source: k8s_audit
  tags: [k8s]

This rule is interesting because an event will be generated if/when an user execute commands in a pod.

Let’s test the rule!

In a tab of your terminal, watch the coming logs:

$ kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco -f

In an another tab of your terminal, create a Nginx pod and execute a command into it:

$ kubectl run nginx --image=nginx

$ kubectl exec -it nginx -- cat /etc/shadow

Several seconds later, in the logs you should see this you will see this Attach/Exec to pod logs:

...
{"hostname":"my-pool-1-node-921b61","output":"09:29:46.302906000: Notice Attach/Exec to pod (user=kubernetes-admin pod=nginx-676b6c5bbc-4xc6t resource=pods ns=hello-app action=exec command=cat)","output_fields":{"evt.time":1739179786302906000,"ka.target.name":"nginx-676b6c5bbc-4xc6t","ka.target.namespace":"hello-app","ka.target.resource":"pods","ka.target.subresource":"exec","ka.uri.param[command]":"cat","ka.user.name":"kubernetes-admin"},"priority":"Notice","rule":"Attach/Exec Pod","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:29:46.302906000Z"}
...

🎉

Conclusion

Ensuring the security of Kubernetes clusters is important and in general we have a lot of information in the Audit Logs but we don’t use them so don’t hesitate to use this new plugin.

We installed the new k8saudit-ovh plugin in an OVHcloud MKS cluster but note that you can deploy it in a Kubernetes cluster in another Cloud provider and even in a Falco instance running locally 💪.

We visualized the logs/the events in the terminal but you can also visualize them in the sidekick UI, create a custom rule and even use Talon to execute some actions.