IPMI was created in a cooperative partnership between Intel, Dell, Hewlett Packard, and NEC. Since its creation, it has become an industry standard as an important hardware solution that allows Server Admins to monitor hardware status, log server data, and allow access to the server without having physical access of the server. By accessing a server through IPMI you are granted access to the system’s BIOS, having this access allows you to install or reinstall your own operating system, fix any network misconfigurations, or re enable SSH or RDP access using KVM (Keyboard Video Mouse) access to a server.

By utilizing OVHcloud® infrastructure you will be able to use IPMI and have access to your server’s BIOS. This enables you to be an effective server administrator and troubleshoot any issues you may have with your server as well as install any operating system compatible with your server’s components.

At OVHcloud it’s important to us that our customers have the freedom and flexibility to innovate solutions to any challenge or problem they see before them; utilizing IPMI is one way we can give our customers such freedom.

To learn more about how to access IPMI from your OVHcloud Manager and how to install an operating system utilizing IPMI please consult the following guides that take you step by step through the process: Getting Started with IPMI, How to Install an OS with IPMI.

How to continuously monitor the health of all OVH servers, without any impact on their performance, and no intrusion on the operating systems running on them – this was the issue to address. The end goal of this data collection is to allow us to detect and forecast potential hardware failure, in order to improve the quality of service delivered to our customers.

We began by splitting the problem into four general steps:

• • Data collection
  • Data storage
  • Data analytics
  • Visualisation/actions

Data collection

How did we collect massive amounts of server health data, in a non-intrusive way, within short time intervals?

Which data to collect?

On modern servers, a BMC (Board Management Controller) allows us to control the firmware updates, reboots, etc.. This controller is independent of the system running on the server. In addition, the BMC gives us access to sensors for all the motherboard components through an I2C bus. The protocol used to communicate with the BMC is the IPMI protocol, which accessible via LAN (RMCP).

What is IPMI?

• Intelligent Platform Management Interface.
• Management and monitoring capabilities independently of the host’s OS.
• Led by INTEL, first published in 1998.
• Supported by more than 200 computer system vendors such as Cisco, DELL, HP, Intel, SuperMicro…

Why use IPMI?

• Access to hardware sensors (cpu temp, memory temp, chassis status, power, etc.).
• No dependency on the OS (i.e. an agentless solution)
• IPMI functions accessible after OS/system failure
• Restricted access to IPMI functionalities via user privileges

Multi-source data collection

We needed a scalable and responsive multi-source data collection tool to grab the IPMI data of about 400k servers at fixed intervals.

We decided to build our IPMI data collector on an Akka framework. Akka is a open-source toolkit and runtime, simplifying the construction of concurrent and distributed applications on the JVM.

The Akka framework defines an abstraction built above thread called ‘actor’. This actor is an entity that handles messages. This abstraction eases the creation of multi-thread applications, so there’s no need to fight against deadlock. By selecting the dispatcher policy for a group of actors, you can fine-tune your application to be fully reactive and adaptable to the load. This way, we were able to design an efficient data collector that could adapt to the load, as we intended to grab each sensor value every minute.

In addition, the cluster architecture provided by the framework allowed us to handle all the servers in a datacentre with a single cluster. The cluster architecture also helped us to design a resilient system, so if a node of the cluster crashes or becomes too slow, it will automatically restart. The servers monitored by the failing node are then handled by the remaining, valid nodes of the cluster.

With the cluster architecture, we implemented a quorum feature, to take down the whole cluster if the minimal number of started nodes is not reached. With this feature, we can easily solve the split-brain problem, as if the connection is broken between nodes, the cluster will be split into two entities, and the one that does not reached the quorum will be automatically shut down.

A REST API is defined to communicate with the data collector in two ways:

• To send the configurations
• To get information on the monitored servers

A cluster node is running on one JVM, and we are able to launch one or more nodes on a dedicated server. Each dedicated server used in the cluster is put in an OVH VRACK. An IPMI gateway pool is used to access the BMC of each server, with the communication between the gateway and the IPMI data collector secured by IPSEC connections.

Data storage

Of course, we use the OVH Metrics service for data storage! Before storing the data, the IPMI data collector unifies the metrics, by qualifying each sensor. The final metric name is defined by the entity the sensor belongs to and the base unit of the value. This will ease the post-treatment processes and data visualisation/comparison.

Each datacentre IPMI collector pushes its data to a Metrics live cache server with a limited persistence time. All important information is persisted in the OVH Metrics server.

Data analytics

We store ours metrics in warp10. Warp 10 comes with a Time series scripting language: WarpScript which wakes the analytics powerful to easily manipulate and post-process (on the server side) our collected data.

We have defined three levels of analysis to monitor the health of the servers:

• • A simple threshold-per-server metric.
  • By using OVH metric loops service, we aggregate data per rack and per room and calculate a mean. We set a threshold for this mean, this permits to detect racks or room common failure in the cooling or power supply system.
  • The OVH MLS service performs some anomaly detections on the racks and rooms by forecasting the possible evolution of metrics, depending on past values. If the metrics value is outside of this template, an anomaly is raised.

Visualisation/actions

All the alerts generated by the data analysis are pushed under TAT, which is an OVH tool we use to handle the alerting flow.

Grafana is used to monitored the metrics. We have dashboards to visualise the metrics and the aggregations for each rack and room, the detected anomalies, and the evolution of the opened alerts.