Back

Hosting an open-source LLM: how to choose the right option


Hosting an open-source LLM: how to choose the right option

So, you've chosen your open-weight model. That's half the work done. However, you still have one decision to make that will impact your costs, your latency and your data control: where and how to run it. To host an LLM, you have three main options. This guide compares them and gives you five practical criteria you can use to make your choice.

Three ways to expose a model

The difference between the options mainly lie in what you manage yourself:

  • On-demand inference: access open models via API, already optimised for standard use cases;
  • Managed container orchestration platform: a container management service like Kubernetes lets you scale your models and provide APIs to your users in just a few hours;
  • Serverless AI: run containerised workloads on GPU or CPU without configuring the infrastructure, while reducing unnecessary spending on GPUs.

 

The further down this list you go, the more control you get and the more operational responsibilities you take on. In absolute terms, no one option is better than another: each one suits a certain project profile.

On-demand inference for quickly deploying intelligent services

With AI Endpoints, you can access a catalogue of open-weight models via a simple API call. The interface is compatible with OpenAI-compatible APIs: if your code already calls this type of API, switching provider often amounts to editing a URL and a key.

This option is a good choice if:

  • You are prototyping or launching a first use case;
  • You are embedding intelligent features into your digital product;
  • A model from the catalogue meets your need.

Its limitation is the trade-off for its simplicity. You use the models proposed, in their configuration. If you have fine-tuned a model using your data or if you need a model that isn't found in the catalogue, you will need to move on to the next option.

Serverless AI to roll out your own models

AI Deploy is a container platform designed for AI models. You provide a Docker image, for example the one for the open-source vLLM inference server, you choose your GPU or CPU and the platform takes care of the rest.

This means that you still get to choose the model, its version and its service parameters. The tutorial dedicated to the deployment of Mistral Small 24B shows that it only takes a single command to launch a model of this size on two L40S GPUs.

This option is a good choice if:

  • You have models trained specifically for your tasks, using your data;
  • You need to control the performance of your models, their isolation and your data;
  • You want to have your own API without having to optimise your inference platform yourself.

To find out more about this deployment model, the How to deploy LLMs with vLLM guide details each step.

orchestration platform for your scaling needs

If your team already operates Kubernetes, deploying your models on a Managed Kubernetes Service allows you to treat them like any other workload. You add GPU nodes to your cluster, deploy vLLM and connect your existing monitoring.

This is the option that offers the most flexibility: high availability, detailed scaling, co-location of the model with your applications and data, along with API server, permission and access management, and resource isolation. It is also the option that requires the most skills. Your team is responsible for updating GPU drivers and adjusting autoscaling.

This option is a good choice if:

  • You have skills in managing container platforms, Kubernetes being the best known;
  • You need significant capacity, stable performance, resource management, and complete isolation;
  • You want a single architecture for your applications and your models.

Hosting an LLM: five criteria to help you decide

To make your choice, ask yourself these five questions in order.

1. Is your token consumption irregular? You're adding intelligence to your applications without yet being able to estimate their footprint, you need multiple model providers on demand, or you specialise your agents by task: pay-as-you-go billing for on-demand inference is then an appropriate choice. Sustained and predictable consumption can make dedicated resources more attractive.

2. What latency are you aiming for? Latency has a direct impact on your users' experience. If you need stable response times for a large number of simultaneous requests, dedicated resources are preferable.

3. Is your model standard? If your use cases don't require a model trained on your datasets, ready-to-use models can absolutely be a competitive option.

4. What internal skills do you have? Configuring and maintaining your own inference platform can prove complex, even with data science skills. If you opt for this solution for contractual or sectoral reasons, get in touch with trade experts.

5. What is your business model? Most on-demand services are paid for by usage, per unit consumed. If your usage remains limited, the time saved on the production of a platform and its maintenance can tip the balance towards a solution entirely managed by a provider.

Criteria

On-demand inference

Serverless AI

orchestration platform

Start-up

Instant

Fast

Longer

Choice of model

Catalogue

Flexible

Flexible

Operation burden

None

Low

High

Token consumption

Irregular

Regular

Sustained, multi-model

Skills required

Low

Moderate

High

 AI Endpoints is operated from the Gravelines datacentre in France, on infrastructure subject to European law. For serverless AI and the orchestration platform, the location of your data depends on the region you choose during deployment: select a European region if your data needs to stay in Europe.

From lab to production, scale through our various stacks

A lot of projects follow the same trajectory. They start with the on-demand inference to validate the use case, switch to serverless AI resources when the model becomes specialised, then scale up on a Kubernetes-type orchestration platform when AI becomes a central component of the application architecture.

In all three cases, your API contracts remain the same: the OpenAI-compatible API is used on our managed AI Endpoints and AI Deploy services (for example, with vLLM). You therefore don't need to rewrite your API routes each time you scale up, and you retain the configuration of your agentic chain by keeping the same open-weight models.

To host an LLM today without committing to significant infrastructure costs, start by testing AI Endpoints for your use case, then scale your architecture in line with your traffic.


Share on: