Hey, Horacio, that Kubernetes thing is rather cool, but what I would have loved to see is a Functions-as-a-Service platform. Most of my apps could be easily done with a database and several serverless functions!

It wasn’t the first time I’d got that question…

Being, above all, a web developer, I can definitely relate. Kubernetes is a wonderful product – you can install complicated web architectures with a click – but what about the database + some functions model?

Well, you can also do it with Kubernetes!

That’s the beauty of the rich Kubernetes ecosystem: you can find projects to address many different use cases, from game servers with Agones to FaaS platforms…

There is an Helm chart for that!

Saying “You can do it with Kubernetes!” is almost the new “There is an app for that!”, but it doesn’t help a lot of people who are looking for solutions. As the question had come up several times, we decided to prepare a small tutorial on how to deploy and use a FaaS platform on OVH Managed Kubernetes.

We began by testing several FaaS platform on our Kubernetes. Our objective was to find the following solution:

• Easy to deploy (ideally with a simple Helm chart)
• Manageable with both an UI and a CLI, because different customers have different needs
• Auto-scalable, in both the upscaling and downscaling senses
• Supported by comprehensive documentation

We tested lots of platforms, like Kubeless, OpenWhisk, OpenFaaS and Fission, and I must say that all of them performed quite well. In the end though, the one that scored the best in terms of our objectives was OpenFaaS, so we decided to use it as the reference for this blog post.

OpenFaaS – a Kubernetes-native FaaS platform

OpenFaaS is an open-source framework for building serverless functions with Docker and Kubernetes. The project is already mature, popular and active, with more than 14k stars on GitHub, hundreds of contributors, and lots of users (both corporate and private).

OpenFaaS is very simple to deploy, using a Helm chart (including an operator for CRDs, i.e. kubectl get functions). It has both a CLI and a UI, manages auto-scaling effectively, and its documentation is really comprehensive (with a Slack channel to discuss it, as a nice bonus!).

Technically, OpenFaaS is composed of several functional blocks:

• The Function Watchdog. A tiny golang HTTP server that transforms any Docker image into a serverless function
• The API Gateway, which provides an external route into functions and collects metrics
• The UI Portal, which creates and invokes functions
• The CLI (essentially a REST client for the API Gateway), which can deploy any container as a function

Functions can be written in many languages (although I mainly used JavaScript, Go and Python for testing purposes), using handy templates or a simple Dockerfile.

Deploying OpenFaaS on OVH Managed Kubernetes

There are several ways to install OpenFaaS on a Kubernetes cluster. In this post we’re looking at the easiest one: installing with Helm.

If you need information on how to install and use Helm on your OVH Managed Kubernetes cluster, you can follow our tutorial.

The official Helm chart for OpenFaas is available on the faas-netes repository.

Adding the OpenFaaS Helm chart

The OpenFaaS Helm chart isn’t available in Helm’s standard stable repository, so you’ll need to add their repository to your Helm installation:

helm repo add openfaas https://openfaas.github.io/faas-netes/
helm repo update

Creating the namespaces

OpenFaaS guidelines recommend creating two namespaces, one for OpenFaaS core services and one for the functions:

kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml

Generating secrets

A FaaS platform that’s open to the internet seems like a bad idea. That’s why we are generating secrets, to enable authentication on the gateway:

# generate a random password
PASSWORD=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)

kubectl -n openfaas create secret generic basic-auth \
    --from-literal=basic-auth-user=admin \
    --from-literal=basic-auth-password="$PASSWORD"

Note: you will need this password later in the tutorial (to access the UI portal, for example). You can view it at any point in the terminal session with echo $PASSWORD.

Deploying the Helm chart

The Helm chart can be deployed in three modes: LoadBalancer, NodePort and Ingress. For our purposes, the simplest way is simply using our external Load Balancer, so we will deploy it in LoadBalancer, with the --set serviceType=LoadBalancer option.

If you want to better understand the difference between these three modes, you can read our Getting external traffic into Kubernetes – ClusterIp, NodePort, LoadBalancer, and Ingress blog post.

Deploy the Helm chart as follows:

helm upgrade openfaas --install openfaas/openfaas \
    --namespace openfaas  \
    --set basic_auth=true \
    --set functionNamespace=openfaas-fn \
    --set serviceType=LoadBalancer

As suggested in the install message, you can verify that OpenFaaS has started by running:

kubectl --namespace=openfaas get deployments -l "release=openfaas, app=openfaas"

If it’s working, you should see a list of the available OpenFaaS deployment objects:

$ kubectl --namespace=openfaas get deployments -l "release=openfaas, app=openfaas"
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
alertmanager   1         1         1            1           33s
faas-idler     1         1         1            1           33s
gateway        1         1         1            1           33s
nats           1         1         1            1           33s
prometheus     1         1         1            1           33s
queue-worker   1         1         1            1           33s

Install the FaaS CLI and log in to the API Gateway

The easiest way to interact with your new OpenFaaS platform is by installing faas-cli, the command line client for OpenFaaS on a Linux or Mac (or in a WSL linux terminal in Windows):

curl -sL https://cli.openfaas.com | sh

You can now use the CLI to log in to the gateway. The CLI will need the public URL of the OpenFaaS LoadBalancer, which you can get via kubectl:

kubectl get svc -n openfaas gateway-external -o wide

Export the URL to an OPENFAAS_URL variable:

export OPENFAAS_URL=[THE_URL_OF_YOUR_LOADBALANCER]:[THE_EXTERNAL_PORT]

Note: you will need this URL later on the tutorial, for example to access the UI portal. You can see it at any moment in the terminal session by doing echo $OPENFAAS_URL.

And connect to the gateway:

echo -n $PASSWORD | ./faas-cli login -g $OPENFAAS_URL -u admin --password-stdin

Now your’re connected to the gateway, and you can send commands to the OpenFaaS platform.

By default, there is no function installed on your OpenFaaS platform, as you can verify with the faas-cli list command.

In my own deployment (URLs and IP changed for this example), the preceding operations gave:

$ kubectl get svc -n openfaas gateway-external -o wide
 NAME               TYPE           CLUSTER-IP    EXTERNAL-IP                        PORT(S)          AGE     SELECTOR
 gateway-external   LoadBalancer   10.3.xxx.yyy   xxxrt657xx.lb.c1.gra.k8s.ovh.net   8080:30012/TCP   9m10s   app=gateway

 $ export OPENFAAS_URL=xxxrt657xx.lb.c1.gra.k8s.ovh.net:8080

 $ echo -n $PASSWORD | ./faas-cli login -g $OPENFAAS_URL -u admin --password-stdin
 Calling the OpenFaaS server to validate the credentials...
 WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
 credentials saved for admin http://xxxrt657xx.lb.c1.gra.k8s.ovh.net:8080

$ ./faas-cli version
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|
CLI:
 commit:  b42d0703b6136cac7b0d06fa2b212c468b0cff92
 version: 0.8.11
Gateway
 uri:     http://xxxrt657xx.lb.c1.gra.k8s.ovh.net:8080
 version: 0.13.0
 sha:     fa93655d90d1518b04e7cfca7d7548d7d133a34e
 commit:  Update test for metrics server
Provider
 name:          faas-netes
 orchestration: kubernetes
 version:       0.7.5 
 sha:           4d3671bae8993cf3fde2da9845818a668a009617

$ ./faas-cli list Function Invocations Replicas 

Deploying and invoking functions

You can easily deploy functions on your OpenFaaS platform using the CLI, with this command: faas-cli up.

Let’s try out some sample functions from the OpenFaaS repository:

./faas-cli deploy -f https://raw.githubusercontent.com/openfaas/faas/master/stack.yml

Running a faas-cli list command now will show the deployed functions:

$ ./faas-cli list
Function                          Invocations     Replicas
base64                            0               1    
echoit                            0               1    
hubstats                          0               1    
markdown                          0               1    
nodeinfo                          0               1    
wordcount                         0               1    

As an example, let’s invoke wordcount (a function that takes the syntax of the unix wc command, giving us the number of lines, words and characters of the input data):

echo 'I love when a plan comes together' | ./faas-cli invoke wordcount

$ echo 'I love when a plan comes together' | ./faas-cli invoke wordcount
       1         7        34

Invoking a function without the CLI

You can use the faas-cli describe command to get the public URL of your function, and then call it directly with your favorite HTTP library (or the good old curl):

$ ./faas-cli describe wordcount
Name:                wordcount
Status:              Ready
Replicas:            1
Available replicas:  1
Invocations:         1
Image:               functions/alpine:latest
Function process:    
URL:                 http://xxxxx657xx.lb.c1.gra.k8s.ovh.net:8080/function/wordcount
Async URL:           http://xxxxx657xx.lb.c1.gra.k8s.ovh.net:8080/async-function/wordcount
Labels:              faas_function : wordcount
Annotations:         prometheus.io.scrape : false

$ curl -X POST --data-binary "I love when a plan comes together" "http://xxxxx657xx.lb.c1.gra.k8s.ovh.net:8080/function/wordcount"
       0         7        33

Containers everywhere…

The most attractive part of a FaaS platform is being able to deploy your own functions.
In OpenFaaS, you can write your these function in many languages, not just the usual suspects (JavaScript, Python, Go etc.). This is because in OpenFaaS, you can deploy basically any container as a function, although this does mean you need to package your functions as containers in order to deploy them.

That also means that in order to create your own functions, you need to have Docker installed on your workstation, and you will need to push the images in a Docker registry (either the official one or a private one).

If you need a private registry, you can install one on your OVH Managed Kubernetes cluster. For this tutorial we are choosing to deploy our image on the official Docker registry.

Writing our first function

For our first example, we are going to create and deploy a hello word function in JavaScript, using NodeJS. Let’s begin by creating and scaffolding the function folder:

mkdir hello-js-project
cd hello-js-project
../faas-cli new hello-js --lang node

The CLI will download a JS function template from OpenFaaS repository, generate a function description file (hello-js.yml in this case) and a folder for the function source code (hello-js). For NodeJS, you will find a package.json (to declare eventual dependencies to your function, for example) and a handler.js (the function main code) in this folder.

Edit hello-js.yml to set the name of the image you want to upload to the Docker registry:

version: 1.0
provider:
  name: openfaas
  gateway: http://6d6rt657vc.lb.c1.gra.k8s.ovh.net:8080
functions:
  hello-js:
    lang: node
    handler: ./hello-js
    image: ovhplatform/openfaas-hello-js:latest

The function described in the handler.js file is really simple. It exports a function with two parameters: a context where you will receive the request data, and a callback that you will call at the end of your function and where you will pass the response data.

"use strict"

module.exports = (context, callback) => {
    callback(undefined, {status: "done"});
}

Let’s edit it to send back our hello world message:

"use strict"

module.exports = (context, callback) => {
    callback(undefined, {message: 'Hello world'});
}

Now you can build the Docker image and push it to the public Docker registry:

# Build the image
../faas-cli build -f hello-js.yml
# Login at Docker Registry, needed to push the image
docker login     
# Push the image to the registry
../faas-cli push -f hello-js.yml

With the image in the registry, let’s deploy and invoke the function with the OpenFaaS CLI:

# Deploy the function
../faas-cli deploy -f hello-js.yml
# Invoke the function
../faas-cli invoke hello-js

Congratulations! You have just written and deployed your first OpenFaaS function.

Using the OpenFaaS UI Portal

You can test the UI Portal by pointing your browser to your OpenFaaS gateway URL (the one you have set on the $OPENFAAS_URL variable), and entering the admin user and the password you have set on the $PASSWORD variable when prompted to.

In the UI Portal, you will find the list of the deployed functions. For each function, you can find its description, invoke it and see the result.

Where do we go from here?

So you now have a working OpenFaaS platform on your OVH Managed Kubernetes cluster.

To learn more about OpenFaaS, and how you can get the most out of it, please refer to the official OpenFaaS documentation. You can also follow the OpenFaaS workshops for more practical tips and advice.

One of the latests additions to this ecosystem is the Agones project, an open-source, multiplayer, dedicated game-server hosting built on Kubernetes, developed by Google in collaboration with Ubisoft. The project was announced in March, and has already made quite a bit of noise…

In the OVH Platform Team we are fans of both online gaming and Kubernetes, so we told ourselves that we needed to test Agones. And what better way to test it than deploying it on our OVH Managed Kubernetes service, installing a Xonotic game server cluster and playing some old-school deathmatches with collegues?

And of course, we needed to write about it to share the experience…

Why Agones?

Agones (derived from the Greek word agōn, contests held during public festivals or more generally “contest” or “competition at games”) aims to replace the usual proprietary solutions to deploy, scale and manage game servers.

Agones enriches Kubernetes with a Custom Controller and a Custom Resource Definition. With them, you can standardise Kubernetes tooling and APIs to create, scale and manage game server clusters.

Wait, what game servers are you talking about?

Well, Agones’s main focus is online multiplayer games such as FPSs and MOBAs, fast-paced games requiring dedicated, low-latency game servers that synchronize the state of the game between players and serve as a source of truth for gaming situations.

These kinds of games ask for relatively ephemeral dedicated gaming servers, with every match running on a server instance. The servers need to be stateful (they must keep the game status), with the state usually held in memory for the duration of the match.

Latency is a key concern, as the competitive real-time aspects of the games ask for quick responses from the server. That means that the connection from the player device to the game server should be the most direct possible, ideally bypassing any intermediate server such as a load-balancer.

And how do you connect the players to the right server?

Every game publisher used to have their own proprietary solutions, but most on them follow a similar flow, with a matchmaking service that groups players into a match, deals with a cluster manager to provision a dedicated instance of game server and send to the players its IP address and port, to allow them to directly connect to the server and play the game.

Agones and it’s Custom Controller and Custom Resource Definition replaces the complex cluster management infrastructure with a standardised, Kubernetes-based tooling and APIs. The matchmaker services interact with these APIs to spawn new game server pods and get their IP address and ports to the concerned players.

The cherry on the cake

Using Kubernetes for these tasks also gives some nice additional bonus, like being able to deploy the full gaming infrastructure in a developer environnement (or even in a minikube), or easily clone it to deploy in a new data center or cloud region, but also offering a whole platform to host all the additional services needed to build a game: account management, leaderboards, inventory…

And of course, the simplicity of operating Kubernetes-based platforms, especially when they dynamic, heterogeneous and distributed, as most online gaming platforms.

Deploying Agones on OVH Managed Kubernetes

There are several ways to install Agones in a Kubernetes cluster. For our test we chose the easiest one: installing with Helm.

Enabling creation of RBAC resources

The first step to install Agones is to setup a service account with enough permissions to create some special RBAC resource types.

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin --serviceaccount=kube-system:default

Now we have the Cluster Role Binding needed for the installation.

Installing the Agones chart

Now let’s continue by adding Agones repository to Helm’s repository list.

helm repo add agones https://agones.dev/chart/stable

And then installing the stable Agones chart:

helm install --name my-agones --namespace agones-system agones/agones

The installation we have just done isn’t suited for production, as the official install instructions recommend running Agones and the game servers in separate, dedicated pools of nodes. But for the needs of our test, the basic setup is enough.

Confirming Agones started successfully

To verify that Agones is running on our Kubernetes cluster, we can look at the pods in the agones-system namespace:

kubectl get --namespace agones-system pods

If everything is ok, you should see an agones-controller pod with a Running status:

$ kubectl get --namespace agones-system pods
NAME                                 READY   STATUS    RESTARTS   AGE
agones-controller-5f766fc567-xf4vv   1/1     Running   0          5d15h
agones-ping-889c5954d-6kfj4          1/1     Running   0          5d15h
agones-ping-889c5954d-mtp4g          1/1     Running   0          5d15h

You can also see more details using:

kubectl describe --namespace agones-system pods

Looking at the agones-controller description, you should see something like:

$ kubectl describe --namespace agones-system pods
Name:               agones-controller-5f766fc567-xf4vv
Namespace:          agones-system
[...]
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 

Where all the Conditions should have status True.

Deploying a game server

The Agones Hello world is rather boring, a simple UDP echo server, so we decided to skip it and go directly to something more interesting: a Xonotic game server.

Xonotic is an open-source multi-player FPS, and a rather good one, with lots of interesting game modes, maps, weapons and customization options.

Deploying a Xonotic game server over Agones is rather easy:

kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/agones/release-0.9.0/examples/xonotic/gameserver.yaml

The game server deployment can take some moments, so we need to wait until its status is Ready before using it. We can fetch the status with:

kubectl get gameserver

We wait until the fetch gives a Ready status on our game server:

$ kubectl get gameserver
NAME      STATE   ADDRESS         PORT   NODE       AGE
xonotic   Ready   51.83.xxx.yyy   7094   node-zzz   5d

When the game server is ready, we also get the address and the port we should use to connect to our deathmatch game (in my example, 51.83.xxx.yyy:7094).

It’s frag time

So now that we have a server, let’s test it!

We downloaded the Xonotic client for our computers (it runs on Windows, Linux and MacOS, so there is no excuse), and lauched it:

Then we go to the Multiplayer menu and enter the address and port of our game server:

And we are ready to play!

And on the server side?

On the server side, we can spy how things are going for our game server, using kubectl logs. Let’s begin by finding the pod running the game:

kubectl get pods

We see that our game server is running in a pod called xonotic:

$ kubectl get pods 
NAME      READY   STATUS    RESTARTS   AGE
xonotic   2/2     Running   0          5d15h

We can then use kubectl logs on it. In the pod there are two containers, the main xonotic one and a Agones sidecar, so we must specify that we want the logs of the xonotic container:

$ kubectl logs xonotic
Error from server (BadRequest): a container name must be specified for pod xonotic, choose one of: [xonotic agones-gameserver-sidecar]
$ kubectl logs xonotic xonotic
>>> Connecting to Agones with the SDK
>>> Starting health checking
>>> Starting wrapper for Xonotic!
>>> Path to Xonotic server script: /home/xonotic/Xonotic/server_linux.sh 
Game is Xonotic using base gamedir data
gamename for server filtering: Xonotic
Xonotic Linux 22:03:50 Mar 31 2017 - release
Current nice level is below the soft limit - cannot use niceness
Skeletal animation uses SSE code path
execing quake.rc
[...]
Authenticated connection to 109.190.xxx.yyy:42475 has been established: client is v6xt9/GlzxBH+xViJCiSf4E/SCn3Kx47aY3EJ+HOmZo=@Xon//Ks, I am /EpGZ8F@~Xon//Ks
LostInBrittany is connecting...
url_fclose: failure in crypto_uri_postbuf
Receiving player stats failed: -1
LostInBrittany connected
LostInBrittany connected
LostInBrittany is now spectating
[BOT]Eureka connected
[BOT]Hellfire connected
[BOT]Lion connected
[BOT]Scorcher connected
unconnected changed name to [BOT]Eureka
unconnected changed name to [BOT]Hellfire
unconnected changed name to [BOT]Lion
unconnected changed name to [BOT]Scorcher
[BOT]Scorcher picked up Strength
[BOT]Scorcher drew first blood! 
[BOT]Hellfire was gunned down by [BOT]Scorcher's Shotgun
[BOT]Scorcher slapped [BOT]Lion around a bit with a large Shotgun
[BOT]Scorcher was gunned down by [BOT]Eureka's Shotgun, ending their 2 frag spree
[BOT]Scorcher slapped [BOT]Lion around a bit with a large Shotgun
[BOT]Scorcher was shot to death by [BOT]Eureka's Blaster
[BOT]Hellfire slapped [BOT]Eureka around a bit with a large Shotgun, ending their 2 frag spree
[BOT]Eureka slapped [BOT]Scorcher around a bit with a large Shotgun
[BOT]Eureka was gunned down by [BOT]Hellfire's Shotgun
[BOT]Hellfire was shot to death by [BOT]Lion's Blaster, ending their 2 frag spree
[BOT]Scorcher was cooked by [BOT]Lion
[BOT]Eureka turned into hot slag
[...]

Add some friends…

The next step is mostly enjoyable: asking the collegues to connect to the server and doing a true deathmatch like in Quake 2 times.

And now?

We have a working game server, but we have barely uncovered the possibilities of Agones: deploying a fleet (a set of warm GameServers that are available to be allocated from), testing the FleetAutoscaler (to automatically scale up and down a Fleet in response to demand), making some dummy allocator service. In future blog posts we will dive deeper into it, and explore those possibilities.

And in a wider context, we are going to continue our exploratory journey on Agones. The project is still very young, an early alpha, but it shows some impressive perspectives.

In the next few posts, I am going to tell you some stories about this beta phase. We’ll be taking a look at feedback from some of our beta testers, technical insights, and some fun anecdotes about the development of this new service.

Today, we’ll start with one of the most frequent questions I got during the early days of the beta: How do I route external traffic into my Kubernetes service? The question came up a lot as our customers began to explore Kubernetes, and when I tried to answer it, I realised that part of the problem was the sheer number of possible answers, and the concepts needed to understand them.

Related to that question was a feature request: most users wanted a load balancing tool. As the beta phase is all about confirming the stability of the product and validating the feature set prioritisation, we were able to quickly confirm LoadBalanceras a key feature of our first commercial release.

To try to better answer the external traffic question, and to make the adoption of LoadBalancereasier, we wrote a tutorial and added some drawings, which got nice feedback. This helped people to understand the concept underlaying the routing of external traffic on Kubernetes.

This blog post is an expanded version of this tutorial. We hope that you will find it useful!

Some concepts:  ClusterIP,  NodePort,  Ingress and  LoadBalancer

When you begin to use Kubernetes for real-world applications, one of the first questions to ask is how to get external traffic into your cluster. The official documentation offers a comprehensive (but rather dry) explanation of this topic, but here we are going to explain it in a more practical, need-to-know way.

There are several ways to route external traffic into your cluster:

• Using Kubernetes proxy and ClusterIP: The default Kubernetes ServiceType is ClusterIp, which exposes the Service on a cluster-internal IP. To reach the ClusterIp from an external source, you can open a Kubernetes proxy between the external source and the cluster. This is usually only used for development.

• Exposing services as NodePort: Declaring a Service as NodePortexposes it on each Node’s IP at a static port (referred to as the NodePort). You can then access the Service from outside the cluster by requesting <NodeIp>:<NodePort>. This can also be used for production, albeit with some limitations.

• Exposing services as LoadBalancer: Declaring a Service as LoadBalancer exposes it externally, using a cloud provider’s load balancer solution. The cloud provider will provision a load balancer for the Service, and map it to its automatically assigned NodePort. This is the most widely used method in production environments.

Using Kubernetes proxy and ClusterIP

The default Kubernetes ServiceType is ClusterIp, which exposes the Service on a cluster-internal IP. To reach the ClusterIp from an external computer, you can open a Kubernetes proxy between the external computer and the cluster.

You can use kubectl to create such a proxy. When the proxy is up, you’re directly connected to the cluster, and you can use the internal IP (ClusterIp) for thatService.

Exposing services as NodePort

Declaring a service as NodePort exposes the Service on each Node’s IP at the NodePort (a fixed port for that Service, in the default range of 30000-32767). You can then access the Service from outside the cluster by requesting <NodeIp>:<NodePort>. Every service you deploy as NodePort will be exposed in its own port, on every Node.

It’s rather cumbersome to use NodePortfor Servicesthat are in production. As you are using non-standard ports, you often need to set-up an external load balancer that listens to the standard ports and redirects the traffic to the <NodeIp>:<NodePort>.

Exposing services as LoadBalancer

Declaring a service of type LoadBalancer exposes it externally using a cloud provider’s load balancer. The cloud provider will provision a load balancer for the Service, and map it to its automatically assigned NodePort. How the traffic from that external load balancer is routed to the Service pods depends on the cluster provider.

The LoadBalancer is the best option for a production environment, with two caveats:

• Every Service that you deploy as LoadBalancer will get it’s own IP.
• The LoadBalancer is usually billed based on the number of exposed services, which can be expensive.

We are currently offering the OVH Managed Kubernetes LoadBalancer service as a free preview, until the end of summer 2019.

What about Ingress?

According to the official documentation, an Ingress is an API object that manages external access to the services in a cluster (typically HTTP). So what’s the difference between this and LoadBalancer or NodePort?

Ingress isn’t a type of Service, but rather an object that acts as a reverse proxy and single entry-point to your cluster that routes the request to different services. The most basic Ingress is the NGINX Ingress Controller, where the NGINX takes on the role of reverse proxy, while also functioning as SSL.

Ingress is exposed to the outside of the cluster via ClusterIP and Kubernetes proxy, NodePort, or LoadBalancer, and routes incoming traffic according to the configured rules.

The main advantage of using an Ingress behind a LoadBalancer is the cost: you can have lots of services behind a single LoadBalancer.

Which one should I use?

Well, that’s the one million dollar question, and one which will probably elicit a different response depending on who you ask!

You could go 100% LoadBalancer, getting an individual LoadBalancer for each service. Conceptually, it’s simple: every service is independent, with no extra configuration needed. The downside is the price (you will be paying for one LoadBalancer per service), and also the difficulty of managing lots of different IPs.

You could also use only one LoadBalancer and an Ingress behind it. All your services would be under the same IP, each one in a different path. It’s a cheaper approach, as you only pay for one LoadBalancer, but if your services don’t have a logical relationship, it can quickly become chaotic.

If you want my personal opinion, I would try to use a combination of the two…

An approach I like is having a LoadBalancer for every related set of services, and then routing to those services using an Ingressbehind the  LoadBalancer. For example, let’s say you have two different microservice-based APIs, each one with around 10 services. I would put one LoadBalancer in front of one Ingress for each API, the LoadBalancerbeing the single public entry-point, and theIngress routing traffic to the API’s different services.

But if your architecture is quite complex (especially if you’re using microservices), you will soon find that manually managing everything with LoadBalancer and Ingress is  rather  cumbersome. If that’s the case, the answer could be to delegate those tasks to a service mesh…

What’s a service mesh?

You may have heard of Istio or Linkerd, and how they make it easier to build microservice architectures on Kubernetes, adding nifty perks like A/B testing, canary releases, rate limiting, access control, and end-to-end authentication.

Istio, Linkerd, and similar tools are service meshes, which allow you to build networks of microservices and define their interactions, while simultaneously adding some high-value features that make the setup and operation of microservice-based architectures easier.

There’s a lot to talk about when it comes to using service meshes on Kubernetes, but as they say, that’s a story for another time…

Using Kubernetes is a great experience, operating it in production is way less simple. And building a managed Kubernetes platform is even worse…

In November 2018 we released the beta versi on of our Managed Kubernetes service. It was the outcome of a journey that took us from Kubernetes users to build a fully managed Kubernetes service, become a certified Kubernetes platform and learn lots of things about building, operating and taming Kubernetes at scale…

As the beta is running now, and the lasts issues are being worked out for the final release, we take some time to share some of the lessons we have learnt, the technological choices that we have taken and the tooling we have built in the process.

In today’s post we will introduce our Managed Kubernetes, explaining why do we built it. In next posts we will look deeper at some aspects of the architecture, like scaling the ETCD or how we start customers Kubernetes masters inside our master Kubernetes’ worker nodes…

And of course, if you want to know more about our Managed Kubernetes, if you would like to see a post about a particular matter, don’t hesitate to leave a comment!

The Kubernetes journey 

The first time you play with Minikube if often astonishing. No more worrying about managing the instances, no need to monitor if the containers are running, you stop an instance and Kubernetes re-creates the containers in another instance… It’s a kind of magic!

Then, as a new believer, you tell yourself that you should try to build a true cluster, and deploy some bigger apps on it. You create some VM, you learn to use kubeadm and some time later you have spawned a fresh Kubernetes cluster to deploy your apps on. The magic is still there, but you begin to feel that, like in most tales, magic comes with a price…

Putting Kubernetes in production?

And when you try to deploy your first production Kubernetes cluster on-premises, on a hypervisor or bare-metal platform, you discover than the price could be a bit steep… 

Deploying the Kubernetes cluster is only the beginning, in order to considerer it prod ready, you also need to ensure that: 

• The installation process is automatisable and repeatable
• The upgrade/rollback process is safe 
• A recovery procedure exists, is documented and tested
• Performance is predictable and consistent, specially when using persistent volumes
• The cluster is operable, with enough traces, metrics and logs to detect and debug failures and problems
• The service is secure and high-available

Our answer to this operational complexity

Well, if you though deploying your new Kubernetes cluster was going to give you this whole NoOps thing, it seems you were wrong. Keeping the magic metaphor, learning to master magic takes a long time and it’s not without risk…

So, as many powerful technologies, Kubernetes apparent simplicity and versatility in the Dev side comes with a high complexity in the Ops side. No wonder that most users looks at the managed Kubernetes front when they need to upgrade from proof-of-concept to production.

At OVH, as a user-focused company, we wanted to answer that demand by creating our managed Kubernetes solution, fully based on open source, without vendor-locking, fully compatible with any pure Kubernetes solution. Our objective was to give our users a fully managed turnkey Kubernetes cluster, ready to use, without the hassle of installation or operation.

On the shoulders of giants…

So we wanted to build a managed Kubernetes solution, but how to do it? The first step was simple: we needed to be sure that the underlying infrastructure was rock solid, so we decided to base it on our own, OpenStack based, Public Cloud offer.

Building our platform over a mature, high available, standards based product as OVH Public Cloud, allowed us to concentrate our efforts in the real problem we had in hand: creating a highly scalable, easy to operate, CNCF certified, managed Kubernetes service.

What’s next?

In the next posts in the series, we are going to dive into the architecture of OVH Managed Kubernetes service, detailing some of our technological choices, explaining why we took them and how we made it work.

We will begin with one of our boldest decisions: running Kubernetes over Kubernetes, or as we like to call it, the Kubinception.