The attack chain behind the v1.107.0 LiteLLM release
On October 21, our release pipeline published a compromised litellm==1.107.0 to PyPI and to our container registries. The incident post covers the affected artifacts and what to do if you ran them. We have completed our…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
On October 21, our release pipeline published a compromised litellm==1.107.0 to PyPI and to our container registries. The incident post covers the affected artifacts and what to do if you ran them. We have completed our investigation into the actions taken by the threat actor, and can confidently assert the timeline and sequence of events that led to the publication of the malicious v1.107.0 release.
We posted regular status updates during our investigation yesterday reflecting the scenario to the best of our knowledge as the investigation proceeded. The investigation was a process in which we continually learned additional information, meaning some of our assumptions and beliefs were later proven to be inaccurate or incomplete. This post should be considered the canonical source of truth where the status updates and the narrative below disagree.
On May 27, 2026, a configuration file was committed to the public LiteLLM-Labs/lite-harness repository which inadvertently contained the master key to a deployment of the LiteLLM Agent Platform, an open source tool we use for orchestrating agents. The error was quickly identified, the commit force-pushed over, and we identified the key as requiring revocation. However, key revocation did not occur.
LiteLLM Agent Platform was an experimental project for self-hosted cloud agents to file PRs to GitHub repositories. In order to create PRs, the system was configured with a GitHub authentication token. The token in question was not associated with a service account and was not scoped down. The key used was a GitHub CLI OAuth token associated with our GitHub organization owner's account.
On October 10th, the threat actor used this key to gain access to our LiteLLM Agent Platform and extracted this GitHub CLI OAuth token. From this point onwards, the threat actor had owner-level access to all repositories in our GitHub. GitHub CLI tokens are post-2fa and not subject to GitHub's "super-user" 2fa reauth flows associated with sensitive organization or repository actions.
Using the access provided by this token, the threat actor began to perform additional credential exfiltration actions. They authored comits into branches on internal repositories to execute workflows designed to exfiltrate any tokens and secrets configured to be available to CI in our repositories. In so doing, they acquired similar GitHub CLI Oauth tokens associated with two engineers with maintainer privileges.
They also acquired tokens to several other internal systems. Reviewing the actor's activity on these other systems shows attempts to identify and exfiltrate additional keys and tokens, with a primary focus on upstream inference API tokens.
While we have traced out all tokens we believe the threat actor could have accessed, we are revoking every token associated with every internal system and vendor platform. This incident has provided a strong illustration of the risks associated with failing to rotate a potentially exposed key.
Having access to multiple GitHub identities, the threat actor was now in a position to be able to make changes to our repositories without being overly noisy, concealing changes within proper-seeming proposals and approvals. The threat actor began by attempting to subvert our project releaser repository to have it pull the package from a different (malicious) repository. They were unable to get approval to merge due to heavily restricted approvals on the repository. As such, they opted to for the noisier approach of using the org owner's token to bypass and disable restrictions on the repository, pushing the change.
In typical operation, building and publishing a release requires being kicked off by an authorized engineer and being approved by a special account locked to a physical device in our office. This restriction was relaxed earlier this week to accommodate the temporary push for more frequent releases and bugfix backports, allowing any two engineers to trigger and approve a release. As such, the threat actor used one of the tokens to trigger the release (after having repointed the deployer to their malicious repository), and another to approve it.
It is worth noting, however, that armed with a token belonging to an organization owner, the threat actor could very well have relaxed this restriction (or removed it entirely) themselves. However, we believe they used the separate tokens in order to evade detection by way of noisy audit logs.
The malicious releases were published to our public registries at around 4:40 AM PDT (11:40 UTC). We became aware of the malicious releases shortly before 6 AM (13:00 UTC) and immediately began the process to remove them. The PyPi releases were frozen by 6:23 AM PDT (13:23 UTC). The Docker images took us a bit longer as we worked to wake the inviduals with access to our Docker registry, and were removed by 7:20 AM PDT (14:20 UTC).
In response, at the beginning of our investigation, we took the following temporary mitigation approaches:
As investigations began to shed light on the scope and details of the attack, we were able to begin laying out a roadmap necessary for secure account and process reinstatement.
At this point, we are confident in our ability to safely issue a release by EOD today, and can finally provide our users with a fix for removing the eMudhra certificate authority from our trust store. We understand it has been frustrating that we have been slow to issue a release to remediate this high-profile issue. We will only issue releases when we can be confident in their safety.
We know many of you want to hear what we’ve learned and what we’re changing. We’ll share that next week. Incidents like this create pressure to announce sweeping changes immediately, but we’d rather take a few days to understand where the real failures were, and why we didn’t detect this sooner, than commit to fixes that address the wrong problems. In the meantime, we didn’t want to hold back the investigation itself.
Our first post suspected the tokens were taken from the logs of an internal LiteLLM deployment. The request logs show the first token came from a different service, described in step 1.
From 69.48.229.143, the attacker called the Agent Platform API with the master key.
With the owner's token, the attacker turned a repository's own CI into the exfiltration channel.
The GitHub audit log records a hash of the token behind each action, and the hash of GH_TOKEN matches the token that approved the release in step 5.
Its token hash matches the token that deleted the release runs in step 6.
Our releases are built by workflows in the private BerriAI/project-releaser repository. From 199.91.221.39, the attacker changed where those workflows take their source from.
Commit 20535d8f exists only on the attacker's fork, but it opens at github.com/BerriAI/litellm/commit/20535d8f because GitHub serves a fork's commits through the repository it was forked from.
Both deletions removed the workflow logs from the repository.
With the owner's token, again by overriding branch protection, the attacker removed the change from the visible history.
main then showed neither the change nor the revert. Both commits are still retrievable by SHA, and the organization audit log recorded each override.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。