In this post, we will take a look at how to activate Federation on your Private Cloud solution, and the advantages of doing so.

What for?

By default, access rights to a vCenter in the Private Cloud are managed directly by that vCenter. Users are created locally (localos or SSO domain) and all access-based control mechanisms (RBACs) are managed by the SSO service. Enabling Federation will delegate user management to a Microsoft Active Directory (AD). As a result, the vCenter server will interact with the domain controller to ensure that the user trying to connect is who they claim to be. The vCenter retains role and privilege management for the objects it manages. Once Federation is configured, it is possible to associate the AD’s users with vCenter roles, so that they can access and/or manage certain objects within the infrastructure (virtual machines, networks, folders, etc.).

One of the main applications of this will be to facilitate vCenter access for administrators by reducing the number of accounts needed to maintain the various elements of the infrastructure. In addition, it will be possible to extend and unify the password management policy between the Active Directory and the vCenter Private Cloud.

The fact that Federation is manageable via OVH’s API makes it possible to automate the configuration, and also to ensure it is kept in operational condition. Finally, it is very simple to add checks in any monitoring tool (Nagios, Zabbix, Sensu, etc.) to monitor the state of the Federation and the rights assigned to users.

Here is an example of a simple PowerShell script that will periodically check if the Federation configuration is in the desired state:

Architecture and prerequisites

Since the vCenter will have to communicate with the domain controllers, the first step will be to allow flows between these elements. There are several ways to achieve this goal, such as combining OVHCloud Connect with a Private Gateway. A study of all the different possibilities would require a whole article, so we advise you to contact OVH or one of our partners to assist you in choosing the most suitable architecture. The following diagram gives you a simplified overview of what this might look like:

Once interconnected, you will need to make sure you have gathered the following information before starting the configuration process:

• Your OVH credentials (nic and password)
• The name of your Private Cloud (in the form pcc-X-X-X-X)
• The required information about the Active Directory infrastructure, namely:
  • Active Directory domain’s short and long name (for example, contoso and contoso.com)
  • IP address of the domain controller
  • AD account’s username and password, with sufficient rights to consult the directory
  • Location of groups and users in the AD hierarchy as “base DN” (example: OU = Users, DC = contoso, DC = com). It should be noted that even though the group information is mandatory, it is not currently possible to use it to manage authentication
  • List of Active Directory users that you want to bind to the vCenter. It will be necessary to provide user names in the form username@FQDN.domain (for example, federation@contoso.com)

Please note that it’s not currently possible to have multiple users with the same shortname, regardless of whether they are managed locally or by an Active Directory.

Activation and configuration

Once you have gathered all the necessary information, it will be possible to activate and configure Federation. The operation will take place in three steps:

1. Activation of the link between the Active Directory and the Private Cloud
2. Binding of one or more AD users to the Private Cloud
3. Assignment of rights to users

The configuration is only accessible via OVH’s API for now, but it should be possible in the medium-term to do it via the OVH Control Panel. The API offers all the necessary options to activate, configure, or even delete your Private Cloud’s Federation:

Enabling the connection between the AD and the Private Cloud

Go to the API’s explorer site and authenticate with your OVH credentials. If you do not already have it, get the name (also called serviceName in the API) of your Private Cloud, as it will be mandatory for all other steps of the configuration. You can access this information by performing a GET on the /dedicatedCloud URI:

Enable Federation by providing all the information about your Active Directory via a POST on the /dedicatedCloud/{serviceName}/federation/activeDirectory URI. All requested information is mandatory:

The activation of the Federation will take a while, and take place in the background. You can follow the progress of the operation via the OVH Control Panel:

Once completed, you can retrieve the Federation ID by sending a GET request to the /dedicatedCloud/{serviceName}/federation/activeDirectory URI:

Binding one or more AD users

Now that your AD is declared in the vCenter Private Cloud, we will be able to bind the Active Directory’s users to it. Note that even if your users are binded, they won’t have any vCenter roles associated with them, so they will not be able to log in.

To bind an user, you will need to send a POST request to the /dedicatedCloud/{serviceName}/federation/activeDirectory/{activeDirectory}/grantActiveDirectoryUser URI by specifying the full username:

Make sure that the user is present in the search OU you declared when associating your AD with the vCenter. Once again, you can validate that the import task is done via the API, or via the Control Panel:

You should also receive an email stating that the user has been imported.

Assigning access rights

The last step will be to give the users access rights to the different objects in the virtual infrastructure. This step does not differ from the usual way of managing Private Cloud’s users rights. It can be done through the API or the OVH Control Panel.

You should now be able to log in to your vCenter with your AD’s users and start managing your Private Cloud!

In this post, we have seen how to activate the Federation option, and the benefits it brings to users of the OVH Private Cloud. In a future post, we will talk about another new feature: Granular Rights. So stay tuned on the OVH blog!

With OVH Private Cloud, we take away from you this complexity. We’re managing this time-costing and stressful aspect to allow you to concentrate in your business and your production.

But, this doesn’t mean it’s not a challenge for us.

Upgrading hundreds of vSphere 5.5 to 6.0

vSphere is the lead product of the Private Cloud offer, it’s part of the SDDC suite provided by VMware. vSphere is a software making the user able to manage his hosts, storage, network … Through a client, and he can create clusters with these assets for hosting a production reliable, stable, and highly available.

Since September 2018, vSphere (vCenter, ESXi …) version 5.5 is end-of-support by VMware. Owner of the security and stability of the Private Cloud infrastructures, we started update processes for all the vCenter.

We had around of 850 vCenter in 5.5 version in production, which represents a significant work to update everything, if it’s done manually. But at OVH, we have a common leitmotiv : automate all human actions  for effectiveness, and avoid errors.

That’s how we managed to update 850 vCenter from 5.5 to 6.0 version, in 4 weeks. In other words, more than 210 vCenter per week, 30 vCenter per day, with a team of 10 people following this maintenance in background, without creating any impact on customers production.

Our dev team has designed and created a set of scripts (which we call internally “robot”) to automate vCenter upgrades years ago. This robot has evolved a lot since the beginning of the Private Cloud product, and follows us from 4.1 to 6.5 version, which is a work in progress.

We encountered lots of issues while setting up the automated actions, like database corruption, services not integrated in Single Sign-On (it was very hard to manage it in 5.0 and 5.1 version), but also thumbprint which wasn’t updated for all services, very hard to troubleshoot, and to reproduce it. We even had some operating systems which blocked upgrade softwares, making everything brutally stopped.

Our operations teams worked a lot with VMware support team, to find workarounds to issues encountered, and automate all them with dev team. This led to VMware KBs creation to notify customers about issues we faced off, recognized as bugs by VMware. The teams spent a lot of nights to ensure a minimal impact of vSphere availability for customers.

Upgrading the upgrader: a new version of the robot

All of these issues convince us to act on two things. First, push a new version of the upgrade robot, creating less errors, having a faster execution from customer point of view, more reliable and trustful. Second, we abandoned the default upgrade process, using VMware software upgrade, for a solution where we start from a freshly installed vCenter updated stack, on an updated virtual machine, and then we reconnect every components (database, NSX …) to this new vCenter.

This greatly improved our service stability, as we ensure we have a new healthy and updated basis for the vCenter. All this has drastically reduced the number of interventions of our SREs on Private Cloud infrastructures.

If we sum up our actions: we verify that the service is working before doing anything, then we prepare all our saves and snapshots to prepare the upgrade. Once it’s done, we deploy our automation to launch the upgrade. Every step integrate an automatic verification to be sure all actions have been done.

We created this upgrade robot in an orchestrator robot, which, according to parameters entered, will create upgrade tasks to each Private Cloud concerned by the maintenance, and will schedule it at automatic dates, within a minimum of 72 hours of consideration for the customer, but also the number of upgrades launched by hour, and critical periods (such as Black Friday, or Winter Sales). Customers can reschedule their upgrades by using the Manager in the Operations part, to run the maintenance in a better time for their production.

Our SRE teams are following the robots, and ensure that the maintenances are running as expected, at the time scheduled.

To sum up, we went from a need to automate a vCenter upgrade operation which should take at least a minimum of 12 hours per vCenter to be done, to a first version of an automation which permit to accomplish this operation in 4 hours, but with an error rate too high (20%) because of recurrent bugs, which had to be fixed manually by SREs. Now, the second version is solid, reliable and stable, avoiding known issues, and creating only rare and unique issues which will be fixed in the automation in a curated pass.

What’s next?

In the next months, other maintenances will follow, host upgrades from 5.5 to 6.0 version, upgrades of our Veeam Backup option from 8.0 to 9.5 version, upgrade of our Zerto option from 5.0 to 5.5, and a lot of other upgrades of our internal machines to ensure the PCI-DSS audit routine.

We will keep the same transparency and communication, while we are listening your feedbacks, and improving our maintenance system.