OVHcloud’s IaaS (Infraestructure as a Service) services allow us to hire dedicated servers (bare metal) for rent that we can have at our disposal in a matter of minutes. There is a wide range of options (rise, advance, storage, scale, high quality, etc.) from which we can choose according to our needs.

In this article we will see how we can use these dedicated servers to create Proxmox VE Ceph Clusters with the same functionalities that we would have if we used proprietary servers in our facilities (on premises). In addition, we will see how OVHcloud services allow us to create a very complete DRP (Disaster Recovery Plan) to achieve maximum resilience for our data

OVHCloud – Dedicated Servers

Ceph Proxmox VE Cluster

A cluster of Proxmox VE servers combined with a Ceph distributed storage system allows you to create a highly available, load-balanced, horizontally scalable, hyperconverged virtualization infrastructure with ease.

Let’s first look at some concepts to fully understand what a Proxmox VE Ceph Cluster is.

What is a cluster?

A cluster in computing refers to a group of interconnected computers or nodes that function together as if they were a single entity. Clusters are used to improve the availability, performance, and scalability of applications and services. There are different types of clusters, but in general they share the objective of providing greater processing capacity and redundancy.

What is Ceph?

Ceph is a distributed storage system designed to provide object, block, and file storage in a single unified cluster. . Proxmox can use Ceph as a virtual machines storege.

What is a Proxmox VE Ceph cluster?

It is three or more servers forming part of a Proxmox cluster and using Ceph as a distributed storage system, all managed from the Proxmox web interface, thanks to which we achieve a hyperconverged virtualization infrastructure.

What is hyperconvergence?

A hyperconverged virtualization infrastructure is an integrated system that combines compute, storage, and networking in a single environment. This simplifies management, improves efficiency, and enables easy scalability, making it easy to create and manage virtual machines in a single cluster.

Proxmox VE Ceph Cluster

By means of the OVHcloud dedicated server service we can create a Proxmox VE Ceph cluster with three or more nodes in the same way that we would do if we acquired our own servers to create the cluster in our facilities, but with the versatility and advantages that come with using rental servers instead of proprietary hardware, among which we can mention:

• Abstraction of the hardware layer since any breakdown will be solved by OVHcloud and, if necessary, they will replace the damaged parts or even the entire server.
• We forget about hardware obsolescence. After a few years we will be able to add new servers with the latest technologies to our cluster and eliminate the old ones in a completely transparent way for the user of the virtualization environment, without any interruption of service.
• Easily add more nodes to our virtualization cluster to increase your computing power. In a matter of minutes we can have new servers that we can add to our cluster.

The large number of dedicated server options that OVHcloud makes available to its clients allows us to create Proxmox VE Ceph Clusters for practically any client. We can use different CPUs depending on our needs, large amounts of RAM, large NVMe disks, dedicated 25Gbps networks for Ceph communication, the vRACK service to connect our servers, dedicated IP ranges, etc. All this allows us to cover practically the needs of any client.

OVHCloud – A2-Scale Servers

DRP (Disaster Recovery Plan)

A DRP (Disaster Recovery Plan) is crucial to maintaining business operations in the event of disasters, guaranteeing the continuity and protection of essential data. It is very important to have a good DRP to ensure the resilience of the data.

What is a Disaster Recovery Plan?

Disaster Recovery Plan in IT refers to a set of strategies, policies and procedures that an organization implements to restore its critical systems and data after a catastrophic event or disaster that causes significant disruptions to operations. normal.que cause interrupciones significativas en las operaciones normales.

These events may include:

• Natural disasters: Such as earthquakes, floods, storms, etc.
• Man-made disasters: Such as cyber attacks, infrastructure failures, acts of vandalism, etc.

The goal of the Disaster Recovery Plan is to minimize downtime and ensure business continuity, allowing the organization to recover quickly after a disaster.

DRP (Disaster Recovery Plan)

Below we will list several options, taking into account that our DRP may have some of these or even a combination of all of them, depending on the level of data resilience desired. Option 1: Proxmox VE Ceph Cluster distributed in OVHcloud 3-AZ region Our partner OVHcloud has a service called 3-AZ region thanks to which we can distribute a Proxmox VE Ceph Cluster made up of dedicated servers (bare metal) between three different data centers separated by a few tens of kilometers. These data centers that make up the 3-AZ region are interconnected through redundant fibers with minimal latency, thanks to which we can set up a Proxmox VE Ceph Cluster distributed within said region. This provides us with great resilience of the data against incidents located in one of the data centers since our virtualization service will not be affected by it.

OVHcloud – 3-AZ Region

Option 2: Proxmox Backup Server with frequent replication Make use of a Proxmox Backup Server (PBS) or by means of our PBS Online service in a data center (even a different country or continent) where you have a backup copy of all virtual machines, maintaining a history of various versions over time depending on the space available for copies. For those virtual machines that are more critical, you can even make more frequent copies (for example, every hour) so that if you have to activate the DRP, the data loss is as little as possible. This option can be implemented using the PBS Online service, the OVHcloud IaaS service to rent a dedicated server on which to install the PBS or any other cloud service or even own facilities in which to locate the PBS.

Option 3: Ceph Replicación Create a Proxmox VE Ceph Cluster identical to the production one in a data center geographically separated from the main cluster and activate Ceph replication between both clusters. This is the option with the least data loss if we compare it with option 2, but much more expensive since we have to have a cluster equal to the main cluster that we will only activate in case of disaster. This option can be implemented using the OVHcloud IaaS service, given that they have data centers distributed in several countries and continents. Therefore, it would be viable to host the main cluster in a data center and the replication cluster located in another country. It can also be implemented using different cloud providers and also on the customer’s premises if he has geographically separated data centers.

Conclusion

As we have seen throughout this article, we can create a hyperconverged virtualization infrastructure with great data resilience using the Proxmox VE hypervisor and dedicated OVHCloud servers. A Proxmox VE Ceph Cluster with three or more nodes located in a 3-AZ region of OVHcloud, in combination with a PBS (Proxmox Backup Server) hosted in a different data center and country is a highly available solution, with high scalability and with great data resilience. If we also add an identical cluster in another data center outside the 3-AZ region with real-time Ceph replication, we can have a DRP (Disaster Recovery Plan) that allows rapid disaster recovery.

Orchestration tools like Kubernetes have significantly changed the way applications are developed and deployed. They provide developers with the opportunity to focus fully on creating software, which has enabled containers to revolutionize software development. Today, containers have become key elements for hosting and managing applications, and they continue to gain popularity in the world of technology. Their use has allowed companies to gain efficiency and flexibility in deploying their applications, thus improving their competitiveness in the market.

Kubernetes has become the go-to platform for container orchestration, with more and more organizations using it for their production environments. Kubernetes provides many benefits, but it also introduces new challenges and risks. Indeed, as with any technology, Kubernetes is not immune to disasters, data loss, downtime, or infrastructure failures. Therefore, it is crucial to consider the importance of backup and disaster recovery planning (DRP) for Kubernetes clusters too.

To start with, we’ll explain why backup and DRP are critical for Kubernetes clusters, like any other type of clusters, how to implement a backup and DRP strategy, and some best practices for maintaining a healthy Kubernetes environment. Then, we’ll explore how infrastructure as code (IAC) can facilitate DRP, and why Velero Backup is an ideal tool to protect Kubernetes cluster data.

Why Backup and DRP are Critical for Kubernetes Clusters

Kubernetes clusters are complex systems that consist of multiple components and layers, such as nodes, pods, services, and etc. These components are distributed across different hosts and can be subject to failures or errors, whether caused by hardware or software issues, human errors, or even cyber-attacks. When a component fails, it can result in data loss, downtime, or degraded performance.

To avoid these risks and maintain business continuity, it’s essential to have a backup and DRP plan for your Kubernetes clusters. Here are some reasons why:

1. Data protection: Kubernetes stores critical data, such as application configurations, persistent volumes, and stateful sets, in etcd. Backups ensure that your critical data is protected in case of a failure or outage.
2. Disaster recovery: In case of a disaster or outage, a DRP plan helps you recover your data and applications quickly and minimize downtime.
3. Compliance: Many industries and regulations ( GDPR ) require businesses to have a DRP plan to protect sensitive data and maintain business continuity.
4. Risk management: A backup and DRP plan can help you identify potential risks and vulnerabilities in your Kubernetes environment and proactively mitigate them.

How to Implement a Backup and DRP Strategy for Kubernetes Clusters

Implementing a backup and DRP strategy for Kubernetes clusters involves several steps, including:

1. Define your backup and DRP requirements: Identify your critical data and applications, recovery time objectives (RTOs), recovery point objectives (RPOs), and compliance requirements.
2. Choose a backup and DRP solution: There are many backup and DRP solutions available for Kubernetes, such as Velero, Kubernetes Disaster Recovery (KDR), Kasten and Stash. Choose a solution that meets your requirements and integrates with your OVHcloud environment (i.e. using OVHcloud S3 object storage services ensuring full interoperability and resilience), existing tools and workflows.
3. Configure and test your backup and DRP solution: Set up your backup and DRP solution and test it thoroughly to ensure it meets your RTOs and RPOs. Regularly validate and update your backup and DRP plan as your Kubernetes environment changes.
4. Document and train your team: Make sure your backup and DRP plan are clear and familiar to all, and keep your team trained on how to execute it in case of a disaster or outage. Regularly review and update your plan to reflect changes in your Kubernetes environment or compliance requirements.

Infrastructure as Code (IaC) to Facilitate DRP

Infrastructure as code (IAC) is a methodology that uses code to automate the provisioning, configuration, and management of infrastructure.

With IAC, you can define your Kubernetes environment as code, using tools such as Terraform, Ansible, or Helm. This code can be version controlled, tested, and audited, making it easier to maintain, replicate, and recover your Kubernetes environment in case of a disaster or outage.

IAC can facilitate DRP by providing a consistent, repeatable, and scalable way to deploy and configure your Kubernetes environment. With IAC, you can easily rebuild your Kubernetes environment from scratch, which is especially important in case of a catastrophic failure.

In the context of a Disaster Recovery Plan (DRP), GitOps can be a particularly useful method for managing Kubernetes infrastructure. Thanks to its consistent, reproducible, and auditable approach, GitOps allows for effective control over changes and deployments in the Kubernetes environment. Additionally, by using continuous deployment practices, GitOps makes it easier to monitor and update the Kubernetes environment, which can be crucial for maintaining service availability in the event of an incident.

In case of a disaster or outage, GitOps allows for the quick recovery of the previous state of the infrastructure by using Git’s version control features. Indeed, all changes made to the infrastructure are versioned and documented, making it easier to restore the previous state. Moreover, the consistency of the infrastructure ensured by GitOps reduces the risk of human error and facilitates collaboration between operational and development teams. Overall, GitOps can be a key element in ensuring business continuity within a DRP.

Advantages of Velero Backup to Protect Kubernetes Cluster Data

Velero is an open-source tool that provides backup and disaster recovery capabilities for Kubernetes clusters. Velero offers several advantages for protecting Kubernetes cluster data, including:

1. Automated backups: Velero allows you to schedule automated backups of your Kubernetes resources and their associated volumes. This means that you don’t have to manually create and manage backups, saving you time and reducing the risk of human error.
2. Customizable backup policies: Velero offers flexible backup policies that enable you to customize your backups to meet your specific needs. For example, you can choose to back up only certain namespaces, labels, or annotations.
3. Incremental backups: Velero performs incremental backups, which means that only the changes made since the last backup are saved. This reduces the backup window and saves storage space.
4. Disaster recovery: Velero provides an easy way to restore your Kubernetes resources and their associated volumes in case of a disaster or outage. You can restore your resources to the same cluster or to a different one, making it easier to switch between different environments.
5. Support for multiple cloud providers: Velero supports several cloud providers, allowing multi-cloud strategy and making it easier to use the same tool for backup and disaster recovery across different environments.
6. Integration with IAC tools: Velero integrates with infrastructure as code (IAC) tools, such as Kubernetes manifests and Helm charts. This means that you can include backup and disaster recovery configurations in your IAC templates, making it easier to manage your backups alongside your infrastructure.

To ensure optimal use of Velero, it is recommended to enable Velero metrics to monitor the status of backups. Using the open-source monitoring tools Prometheus and Grafana, you can create a dedicated Kubernetes backup dashboard to display key metrics such as backup size, backup duration, number of successful and failed backups. These metrics provide useful information about the progress and success of backups, allowing for early detection of any potential issues. In addition, to improve visibility and resolution of backup issues, it is recommended to implement Loki and Promtail to retrieve logs and trigger alerts in case of warnings or errors on backups. This proactive approach helps resolve problems faster and improve the quality of backups. By following these best practices, Velero users can ensure that their backups are reliable and available when needed.

In a nutshell, Velero offers a reliable and flexible way to protect your Kubernetes cluster data, automate backups, and streamline disaster recovery.

Conclusion

Backup and DRP are critical for Kubernetes environments to ensure data protection, disaster recovery, compliance, and risk management. IAC can facilitate DRP by providing a consistent, repeatable, and scalable way to deploy and configure your Kubernetes clusters.

As highlighted above, Velero Backup is for sure a reliable and flexible solution to automate backups and streamline disaster recovery.

But there are various solutions available on the market to protect Kubernetes cluster data, and OVHcloud Managed Kubernetes offer (CNCF software compliant) is compatible with all of them by design – you can find out more in the dedicated content library available here.

If you want to get expert advice and support on this topic, we invite you to reach out to our partner LecPac-Consulting using the following contact details or to our Professional Services team.

Email info@lecpac-consulting.com
Telephone 06 13 02 81 71
Website https://lecpac-consulting.com/contact/

Currently we have two available solutions for Zerto replication:

• replication between 2 OVHcloud Hosted Private Cloud PCC
• replication between your VMware platform on premise and a OVHcloud Hosted Private Cloud PCC. (note that a maximum of 120ms latency is recommended between the 2 sites)

The choice is made in the OVHcloud manager when you enable the option. The Zerto licence required for the on-premises platform is automatically provided by OVHcloud.

In both scenarios, all the replicated data is sent to the other site, there is no local replication.

Zerto replication can only be setup one to one (not « one to many » or « many to one » or « multi-hops »)

As of Oct 5th 2021, the option is available from 30€HT/VM/mois

Architecture 

Let’s dive further in our Zerto implementation and how it works

Upon activation in the OVHcloud manager, a Zerto Virtual Manager (ZVM) is deployed inside OVHcloud own infrastructure, and a Virtual Replication Appliance (VRA) on each host of the customer PCC (using only vCPU and vRAM, storage is provided by OVHcloud and doesn’t consume PCC storage). The ZVM is the Zerto UI and the control plane for the solution. The VRA are the « data movers ».

A dedicated VPN is also created for the VRA data traffic as early Zerto versions didn’t encrypt in-flight data.

In case of a on premise to OVHcloud setup, the customer needs to deploy the ZVM, VRA and VPN on his own platform.

Zerto replication is bi-directional, so we can protect both sites with each other, providing they have enough space to host the replicas, and enough CPU/RAM to restart the VM’s.

To start replicating VM’s, we have to set-up at least one Virtual Protection Group (VPG). This is an important step because :

• a VM can only be part of 1 VPG
• only a running VM can be added to a VPG
• all VM in the same VPG will have the same recovery checkpoint, allowing for a consistent recovery/restart. This is very useful for a multi-tiered application.

Zerto uses Continuous Data Protection (CDP) mechanism and journalisation. All writes to a VM are kept in a journal, and all VM in a VPG are kept consistent by regular checkpoints. This enables very low Recovery Point Objectives (RPO), and high granularity for recovery.

The journal can log changes up to 30 days. Depending on the workload this can represent a LOT of data, you have to make sure your datastores can handle the load and have enough capacity.

An important aspect of Zerto on OVHcloud is that the customer is responsible for configuring the VPG (and the ZVM, VRA and VPN for the on-premises side) and ensuring replication is running without error and triggering a failover and/or failback. We do not monitor the replication processes as we do not have access to them (and to your data).

Day to day

Initial setup and failover are covered in our Setting up Zerto Virtual Replication for your DRP and Using Zerto between OVHcloud and a third party platform guides.

We manage the ZVM, ZRA and VPN tunnel on OVHcloud infrastructure, and you manage the VPG, the replication options and the replication reports. And this is achieved by accessing the Zerto UI at your personal URL: https://zerto.pcc-xxx-xxx-xxx-xxx.ovh.xxx/

You also have to manage the ZVM, ZRA and your end of the VPN tunnel, if you replicate to/from your own datacentre to/from OVHcloud PCC.

But Zerto is not only a solution enabling to fail-over and restart production on a distant site. It is also a mean to restore locally VM’s that have been damaged/corrupted/deleted by using the target replica.

A single VM or an entire VPG can be restored this way, the time required for the restoration depending on CPU/RAM/network available considering the data is on the remote site.

Sources

• Zerto-Architecture-Guide.pdf
• Zerto @OVHcloud.com