Earlier this month, when unknown attackers wiped repositories and left ransom notes for approximately 1,000 users on popular repository services BitBucket, GitHub, and GitLab, the companies had to act quickly to investigate the origins of the attack and help users recover their data.
The security teams at Atlassian, GitHub, and GitLab shared data on the attacks and information they uncovered on the attackers’ activities as part of their investigation and recovery efforts. The coordination worked so well, the companies have decided to formalize the relationship and continue sharing information, the companies said in their joint post-mortem incident report. One of the things the companies will do is regularly search for files in their users’ repositories containing credentials for other services.
The security and support teams of all three companies have taken and continue to take steps to notify, protect, and help affected users recover from these events," the joint report said. "Further, the security teams of all three companies are also collaborating closely to further investigate these events in the interest of the greater Git community.
The post-mortem report, posted on each of the sites (BitBucket, GitHub, and GitLab), provides detailed results of the companies’ joint analysis that arose from their collaborative investigation.
“Incident responders from each of the three companies began collaborating to protect users, share intelligence, and identify the source of the activity,” the report said.
On May 2, some BitBucket, GitHub, and GitLab users discovered their repositories had been wiped and replaced with a ransom note asking for Bitcoin. The fact that users were across all three platforms raised the possibility of a large operation where the attackers had figured out a way to compromise all the providers. After some investigation, the three providers confirmed independently that the attackers had compromised every single one of the ransomed repositories with legitimate credentials. In some cases, the attackers had the username and passwords; in other cases, they had application passwords, API keys, and personal access tokens.
“After getting access to the user accounts, the attackers performed command-line Git commits, which resulted in overwriting the source code in repositories with the ransom note,” the companies wrote in their analysis.
Exposed Secrets
This attack was different from other types of credential stuffing attacks in that the attackers weren’t reusing credentials stolen and exposed in other breaches. Instead, the collection of passwords and keys came from the repositories themselves, as users had saved (mistakenly?) files containing these secrets to the repositories. The fact that users save files containing API keys and passwords to cloud storage services and code repositories is a known—and big—problem.
Service providers regularly scan repositories for potential problems, but lots of secrets are still exposed. GitHub is currently working on a token scanning feature, currently in beta, to notify other service providers if credentials and other secrets for their platforms are published to public GitHub repositories. GitLab added the ability to look for secrets to its Static Analysis Security Testing tool in GitLab 11.9.
The companies worked together to identify the server that was the source of the attacks, and found a file on that server containing credentials for about a third of the victims. The companies reset or revoked the credentials after finding the file. The companies found that the server’s IP address was continuously scanning for publicly exposed .git/config and other environmental files. Other IP addresses from the same hosting provider were also scanning for exposed files.
The attackers were still systematically scanning the repositories for git configuration files mistakenly containing stored credentials on May 10, more than a week after the ransom incidents first came to light.
Recovery, Prevention
The attackers erased the commit history of the remote repository, but that doesn’t mean users can’t get their repositories back. If the user has the latest copy of the repository locally, then force pushing the local copy would restore the repository. If a local copy isn’t available, then the user will need to close the repository and search through the command history to find the last commit.
Developers should enable multi-factor authentication on their accounts, to prevent this kind of unauthorized access. The post-mortem also emphasized being careful with personal access tokens, as they skip multi-factor authentication. These tokens may have read/write access to repositories and should be considered as valuable as passwords. Tokens should be used as environmental variables and not be hardcoded in the source code.
There are several public-to-private information sharing programs where enterprises share information with the government, as well as industry-based information sharing and analysis centers (ISACs) and industry-focused consortiums and alliances. A few years ago, companies were skeptical about sharing data about attacks, and most of the sharing relied on ad hoc relationships between security teams at different organizations. Organizations understand that sharing threat data can identify incidents sooner, and the additional information is critical for a thorough incident response.