On December 10th, a group of security researchers published a security notice regarding a vulnerability in Log4j, referenced as CVE-2021-44228. This vulnerability has been named “Log4Shell” since it leverages remote code execution through a log entry on a targeted system which enables the attacker to gain remote access to the system.

Log4j is an open-source project that delivers a library used in Java environments to manage logging activities. Log4j is not the default logging library of Java, but since it provides developers a set of interesting logging features accessible directly in the code, the library is widely used in production systems and embedded in a very large number of off-the-shelve software solutions. Unless you are a Java developer you might never have heard about it but it is safe to bet that we will discover within the next few months that this library is used in a lot of unsuspected places.

The vulnerability is linked to JNDI (Java Naming and Directory Interface). JNDI allows to implement within the logs a variable that can be used to request an external resource, accessible over the network. While it might be convenient for a lot of use cases to enrich logs with external information, this feature also allows to execute code locally with the same rights than the application calling it. The attack protocol widely observed consists in injecting a variable within the logs which includes an URI to a java object served by a LDAP server. If you control the LDAP server, you may execute arbitrary code on the target. Log4j vulnerable versions are 2.0.X to 2.14.x.

As of today, we observed that the vulnerability is used to deploy malware used to mine crypto-currencies. However, since the exploitation is simple, we have all reasons to believe the vulnerability is used to leverage other types of attacks. There are many realistic scenarios you should consider, including ransomware deployment, exfiltration of data or pivot to other systems.

Update 22/12: 2 new vulnerabilities have been identify. Those vulnerabilities are also impacting the initial patchs (2.15.0 and 2.16.0):

• CVE-2021-45105 : Risk of Denial of Service (DOS)
• CVE-2021-45046 : Risk of information leak and remote code execution in some environments and local code execution in all environments

The right versions of Log4J to use are 2.17.0 (java 8), 2.12.3 (Java 7) and 2.3.1 (Java 6). Those 3 versions are correcting the 3 vulnerabilities.

How to identify vulnerable systems?

Identifying your vulnerable systems will be a complicated task, and you should structure your investigation approach to lower the risks in a prioritized way.

Publicly exposed systems check

You should start immediately to investigate all systems with a public IP address. Any vulnerable systems directly exposed to internet is at high risk. Even if your application is not written in Java, it might use some Java components using Log4j. Consider every behavior that might generate a log entry in your applications. Focus first on everything that is logged pre-authentication, like HTTP requests or login form. When done, dive into the post-authentication behaviors especially on systems where you cannot trust every user.

In-house developed applications crawling

When you have access to the source code, look in your code repository to identify precisely where log4j library is included in the code. By doing it you might even identify what types of logs are used and what are the attack entry points. If your organization is managing a very large number of projects in Java, it might be a good idea to look for the string “import org.apache.log4j“  in your code repository. You will see very quickly all the applications using the library. It will help you to narrow the search in the emergency phase.

Operating systems level check

Having a vulnerable version of log4j installed on your system does not mean your server is vulnerable since it must be used by a legitimate application to trigger the vulnerability. Log4j is included in most Linux distribution package management system, so if your SysAdmins are not following a strict hardening policy and a very strict software control on systems, you might have the package installed. Checking the presence and the version of log4j on all your systems is a reasonable check to do. The library can be installed manually or can be packaged within an application, so check for any jar file including “log4j” in the file name on your systems, it may help you to measure how this library is used and identify some systems you need to closely work on.

Open-Source and off-the-shelve software

Checking the security announcement pages of the software editors should be next on your list. Community and security researchers are also publishing web pages to track applications packaging a vulnerable version of Log4j. It’s neither perfect nor exhaustive but it will help you spot Log4j presence on your systems. As you may expect, the information from editors is not always sufficiently detailed to know exactly if you are at risk, but it will help you to identify some area for further investigations.

Other systems

Finally, for all the other pieces of software or devices with no active community to track the issue, you unfortunately have no easy way to determine if the systems is vulnerable. Any black box system might use a vulnerable version of Log4j without any way for you to find out. This is a scary perspective for a lot of security incidents to come.

Damage control recommendations

As you may start to understand, Log4shell will be a burden for all IT people in the coming days, months, and years. It’s quite probable that some security researchers will identify new vulnerabilities of the same kind and new attack protocols soon. So, we recommend working on safety hygiene to lower the risks and you shall consider the following best practices:

• Patch, patch, patch and re-check that you patched everything
• Maintain an up-to-date inventory of all software assets
• Implement a security watch to monitor new attack vectors and editor’s announcements
• Segregate your systems to narrow the impacted scope in case of successful attack
• Run your applications under unprivileged user accounts when possible
• Block any egress connection from your servers to external untrusted resources
• Avoid any unnecessary secrets and credentials on production servers
• Block IP and domain names that are known to host payloads for Log4Shell attack
• Implement WAF and filtering to drop all requests including attack pattern (starting with the string “jndi” is a good start)
• Monitor the logs of your systems to detect any abnormal behavior and egress connections
• Backup your systems and perform regular recovery tests

Assessing the risks at OVHcloud

At OVHcloud, as soon as the vulnerability has been published, we started to launch investigations to identify vulnerable systems and started to push patch or mitigation. We have been facing 3 different cases:

• The vulnerability is present on our customer systems under their responsibility. We have no means to detect the vulnerability since we do not access our customer systems. We invite our customers to follow the recommendations above as soon as possible.
• The vulnerability is present on a production system used in support for production of the product and service we provide to customers. Those systems are our top priority and we put most of our efforts on those systems.
• The vulnerability is present on internal systems, but not publicly exposed and not directly in support of our services. We prioritize the treatment of those systems taking into account their criticality and information we receive from editors.

We continue to investigate our systems for vulnerable Log4j library. The actual status of our investigations is detailed below and will be updated if necessary.

Product	Investigation results	Update 22/12 CVE-2021-45105 CVE-2021-45046	Recommendation	Status
Dedicated Servers	No vulnerable version of Log4j is used in support of these products.	N/A	We invite our customers to perform a deep review of their servers configurations following the above guidelines.
Storage and Backups	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
Nutanix	Several Nutanix products are impacted by the vulnerability. Platform delivered by OVHcloud might be vulnerable. OVHcloud has no mean to check the platform since we do not have access to the infrastructure once delivered. Mitigation have been implemented in collaboration with customers to restrict exposition at network level.	N/A	We recommend to our customers to follow editors recommendations on their Nutanix infrastructure (See link below) We invite our customers to perform a deep review of their virtual machines configurations following the above guidelines. We also recommend to limit network exposition of vulnerable services.
Logs Data Platform	This platform was using a vulnerable version of Log4j. All vulnerable versions have been patched.	All customers instances have been patched with 2.17.0 version of Log4J	N/A
Hosted Private Cloud powered by VMware	Several VMware products are impacted by the vulnerability. No patch is available, however VMware published workarounds for each impacted products. We are deploying the workarounds over our infrastructure. Those workarounds might have impact on production systems, so we are deploying them according to our change management process. By default, access to administration interfaces is restricted to IP addresses defined by the customer. The exposition of vulnerable systems is consequently very limited.	Workarounds for the new vulnerabilities are being deployed.	We invite our customers to perform a deep review of their virtual machines configurations following the above guidelines. We recommend to our customers to ensure the Access Control List is limited to trusted IP addresses only.
Veeam, Zerto	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
Hosted Private Cloud powered by Anthos	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
Public Cloud Instance, Object storage, Block storage, VPS	No vulnerable version of Log4j is used in support of these products.	N/A	We invite our customers to perform a deep review of their instance’s configurations following the above guidelines.
Containers & Orchestration	No vulnerable version of Log4j is used in support of these products.	N/A	We invite our customers to perform a deep review of their containers configurations following the above guidelines.
Collaborative solutions (Exchange, Email pro, MXplan, Sharepoint)	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
Cloud Databases	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
Web Hosting	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
Data processing	This platform was using a vulnerable version of Log4j. All vulnerable versions have been patched.	All vulnerable versions have been patched with Log4J 2.17.0	N/A
AI Training	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
ML serving	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
VoIP	No vulnerable version of Log4j is used in support of these products.	N/A	N/A
www.ovh.com api.ovh.com	No vulnerable version of Log4j is used in support of these management interfaces.	N/A	N/A
OVHcloud Internal systems	We continue to investigate to identify any use of the vulnerable Log4j library versions and applicable mitigations. Vulnerable systems identified so far have been patched when the patch is available or mitigation is being deployed. A deep review of all software assets is being performed to ensure the vulnerable library is not embedded in the software solutions we use to operate our information system.	Action plan includes the new vulnerabilities. Mitigation implementation are still being deployed as editors are providing patches.	N/A

External references

• https://cve.mitre.org/cgi-bin/cvename.cgi?name=2021-44228

• https://logging.apache.org/log4j/2.x/index.html

• https://greenlock.ghost.io/log4j-ou-log4shell-la-javapocalypse-cve-2021-44228

• https://www.vmware.com/security/advisories/VMSA-2021-0028.html

• https://hosted-private-cloud.status-ovhcloud.com/incidents/y6x1h3bls5rb

• https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592

• https://download.nutanix.com/alerts/Security_Advisory_0023.pdf

• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105

• https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046