Ransomware targeting VMware ESXi

A wave of attacks is currently targetting ESXi servers. No OVHcloud managed service are impacted by this attack however, since a lot of customers are using this operating system on their own servers, we provide this post as a reference in support to help them in their remediation.

These attacks are detected globally. According to experts from the ecosystem as well as authorities, the malware is probably using CVE-2021-21974 as compromission vector. Investigation are still ongoing to confirm those assumptions.

Our technical teams are working to identify the detailed characteristics of the attack all the while coordinating with our peers from other CERTs and security teams.

Update 07/02/2023

We continue our investigations and to provide support to our customers.
We prioritize our efforts:

to identify our impacted customers on our networks to provide the most accurate and appropriate information to help them to recover from the attack.
to identify potentially vulnerable customers to ensure they mitigate the risks appropriately as soon as possible in the case of on an other wave of similar attack.

Several security researchers may have found a link between the Babuk Ransomware source code leaked in September 2021. The encryption cipher (Sosemanuk) is used in the both cases but the code structure seems to be slightly different.

In addition to the recovery procedure described earlier, we noted that the encryption process is only impacting a small amount of data within the file. Depending of your VM OS and file system type, you might be able to recover data with data revery tools, at least partially. Be carefull, this tools might have irreversible action on the file so, We recommend to copy the VM files on an other location to protect the data before trying any recovery operation.

We are referencing a list of companies that can assist you to recover your data and reconstruct your systems. The list of companies will be available at OVHcloud support.

We also remind to our customers acting as Data Controller that they might have legal requirements to notify autorities in case of security incident. Ensure you declared the incident to the appropriate autorities within the right timeframe.
You will find below the Data Protection Autorities procedures for databreach violation for mainly impacted countries and CERT websites as well. Check with your legal department or counsel to ensure you notify the right organisation according to your status.

For PII data controllers:

CERT:

Additionnal references:
https://www.bleepingcomputer.com/news/security/
https://www.bleepingcomputer.com/forums/t/782193/esxi-ransomware-help-and-support-topic-esxiargs-args-extension/massive-esxiargs-ransomware-attack-targets-vmware-esxi-servers-worldwide/
https://blogs.vmware.com/security/2023/02/83330.html
https://members.loria.fr/MMinier/static/papers/sosemanuk_08.pdf

Update on 05/02/2023

We continue to work on the technical analysis in coordination with authorities and security community to determine IOCs and understand how the malware is behaving after the initial compromission.

So far we identified the following behavior:

The compromission vector is confirmed to use a OpenSLP vulnerability that might be CVE-2021-21974 (still to be confirmed). The logs actually show the user dcui as involved in the compromission process.
Encryption is using a public key deployed by the malware in /tmp/public.pem
The encryption process is specifically targeting virtual machines files (“.vmdk”, “.vmx”, “.vmxf”, “.vmsd”, “.vmsn”, “.vswp”, “.vmss”, “.nvram”,”*.vmem”)
The malware tries to shutdown virtual machines by killing the VMX process to unlock the files. This function is not systematically working as expected resulting in files remaining locked.
The malware creates argsfile to store arguments passed to the encrypt binary (number of MB to skip, number of MB in encryption block, file size)
No data exfiltration occurred.

In some cases, encryption of files may partially fail, allowing to recover data. Enes Sönmez (@enes_dev), a turkish security researcher has documented the procedure for recovery of VMDK files. The procedure is described on his blog (https://enes.dev/). We tested this procedure as well as many security experts with success on several impacted servers. The success rate is about 2/3. Be aware that following this procedure requires strong skills on ESXi environnements. Use it at your own risk and seek the help of experts to assist.

In the previous version of the post, we made the assumption the attack was linked to the Nevada Ransomware which was a mistake. No material can lead us to attribute this attack to any group. Attribution is never easy and we leave security researchers to make their own conclusions.

ESXi OS can only be installed on bare metal servers. We launched several initiatives to identify vulnerable servers, based on our automation logs to detect ESXI installation by our customers.
We have limited means of action since we have no logical access to our customer servers. For identified bare metal hosts:

We sent emails on Friday’s afternoon to warn customer of the risk and provide them information on to mitigate the risk
We blocked the OpenSLP port (427) between internet and the servers with ESXI installed. Customer can deactivate the filtering rule in their management interface if the use of port 427 is required for whatever reason.

We launched scan to identify compromised hosts, by testing the presence of the web page and/or the ssh banner specifying the host has been compromised to notify impacted customers.

Our support team is fully mobilized to help our customers to protect their systems and to help them to recover if they are impacted by the attack.

Additionnal references:
https://www.cert.ssi.gouv.fr/alerte/CERTFR-2023-ALE-015/
https://enes.dev/
https://straightblast.medium.com/my-poc-walkthrough-for-cve-2021-21974-a266bcad14b9

First response action items on 03/02/2023

The attack is primarily targetting ESXi servers in version before 7.0 U3i, apparently through the OpenSLP port (427).

To check your version of ESXi, please refer to your server page in your customer interface to identify wich version has been deployed on the server or to the ESXi interface on the system itself.

So far, we can identify the following recommendations regarding our services:

For Bare Metal customer using ESX-i we strongly recommend in emergency :

to deactivate the OpenSLP service on the server or to restrict access to only trusted IP addresses (https://kb.vmware.com/s/article/76372)
to upgrade you ESXi on the latest security patch

In a second time, ensure:

your data are backed up (on immutable storage?)
only necessary services are active and filtered with ACL to only trusted IP adresse
monitor your system for any abnormal behaviour.

Our clients using VMware Private Cloud are not impacted. By design, the SSL gateway prevent this typology of attack by blocking the external access to this port (OpenSLP 427).

For our Public Cloud customers, there is no dependency to ESXi so no risk are identified.

No other product among OVHcloud’s portfolio is threatened by this ransomware campaign.

We will update this blog post with any information that could help to reduce the risk associated with this threat.

Additionnal references:

Julien Levrard

CISO OVHcloud | + posts