This makes it goal for deploying one thing like a trojan that connects again to the attackers after which collects all potential delicate info uncovered by future workflow executions. However what to make use of as a trojan that wouldn’t be detected by antivirus merchandise or whose communications wouldn’t get blocked? The GitHub Actions runner agent itself, or fairly one other occasion of it that’s not linked to the PyTorch group however to a GitHub group managed by the attackers.
“Our ‘Runner on Runner’ (RoR) method makes use of the identical servers for C2 as the present runner, and the one binary we drop is the official GitHub runner agent binary, which is already operating on the system. See ya, EDR and firewall protections,” Stawinski mentioned.
Extracting delicate entry tokens
Up till this step, the attackers managed to get a really stealthy trojan program operating inside a machine that’s a part of the group’s improvement infrastructure and which is used to execute delicate jobs as a part of its CI/CD pipeline. The following step is post-exploitation: attempting to exfiltrate delicate knowledge and pivot to different components of the infrastructure.
Workflows typically embody entry tokens to GitHub itself or different third-party companies. These tokens are required for the roles which can be outlined within the workflow to execute accurately. For instance, the construct agent wants learn privileges to take a look at the repository first and may also want write entry to publish the ensuing binary as a brand new launch or to switch present releases.
These tokens are saved on the filesystem of the runner in varied areas just like the.git configuration file or in atmosphere variables and might clearly be learn by the stealthy “trojan” that runs with root privileges. Some, resembling GITHUB_TOKEN, are ephemeral and solely legitimate throughout the execution of the workflow, however the researchers discovered methods to increase their life. Even when they wouldn’t have discovered these strategies, new workflows with newly generated tokens are executed on a regular basis on a busy repository like PyTorch, so there are many new ones to gather.
“The PyTorch repository used GitHub secrets and techniques to permit the runners to entry delicate programs throughout the automated launch course of,” Stawinski mentioned. “The repository used numerous secrets and techniques, together with a number of units of AWS keys and GitHub Private Entry Tokens (PATs).”
PATs are sometimes over privileged and are a sexy goal for attackers, however on this case they had been used as a part of different workflows that weren’t executing on the compromised self-hosted runner. Nevertheless, the researchers discovered methods to make use of the ephemeral GitHub tokens they had been capable of acquire to put malicious code into workflows that had been executing on different runners and contained these PATs.
“It seems that you may’t use a GITHUB_TOKEN to switch workflow information,” Stawinski mentioned. “Nevertheless, we found a number of inventive…’workarounds’…that may allow you to add malicious code to a workflow utilizing a GITHUB_TOKEN. On this state of affairs, weekly.yml used one other workflow, which used a script exterior the .github/workflows listing. We may add our code to this script in our department. Then, we may set off that workflow on our department, which might execute our malicious code. If this sounds complicated, don’t fear; it additionally confuses most bug bounty packages.”
In different phrases, even when an attacker can’t modify a workflow straight, they may be capable to modify an exterior script that is named by that workflow and get their malicious code in that method. Repositories and CI/CD workflows can get fairly complicated with many interdependencies, so such small oversights are usually not unusual.
Even with out the PATs, the GITHUB_TOKEN alone with write privileges would have been sufficient to poison PyTorch’s releases on GitHub and individually extracted AWS keys may have been used to backdoor PyTorch releases hosted on the group’s AWS account. “There have been different units of AWS keys, GitHub PATs, and varied credentials we may have stolen, however we believed we had a transparent demonstration of impression at this level,” the researchers mentioned. “Given the crucial nature of the vulnerability, we wished to submit the report as quickly as potential earlier than one among PyTorch’s 3,500 contributors determined to make a cope with a international adversary.”
Mitigating threat from CI/CD workflows
There are various classes to be taught from this assault for software program improvement organizations: from the dangers related to operating self-hosted GitHub Actions runners in default configurations to the dangers of getting workflows that execute scripts from exterior the workflows listing to dangers related to overprivileged entry tokens and bonafide purposes repurposed as trojans — different researchers did this earlier than with Amazon’s AWS System Supervisor agent and with Google’s SSO and gadget administration resolution for WIndows.
“Securing and defending the runners is the duty of finish customers, not GitHub, which is why GitHub recommends in opposition to utilizing self-hosted runners on public repositories,” Stawinski mentioned. “Apparently, not everybody listens to GitHub, together with GitHub.”
Nevertheless, if self-hosted runners are crucial, organizations ought to on the very least contemplate altering the default setting of “Require approval for first-time contributors” to “Require approval for all exterior collaborators.” It’s additionally a good suggestion to make self-hosted runners ephemeral and to execute workflows from fork PRs solely on GitHub-hosted runners.
This isn’t the primary time when insecure use of GitHub Actions options has generated software program supply-chain security dangers. Different CI/CD companies and platforms have additionally had their very own vulnerabilities and insecure default configurations. “The problems surrounding these assault paths are usually not distinctive to PyTorch,” the researchers mentioned. “They’re not distinctive to ML repositories and even to GitHub. We’ve repeatedly demonstrated provide chain weaknesses by exploiting CI/CD vulnerabilities on this planet’s most superior technological organizations throughout a number of CI/CD platforms, and people are solely a small subset of the larger assault floor.”
