Daily WTF!

How a Melting Mac Caught LiteLLM Hackers


Listen Later

On March 24, 2026, the popular Python library LiteLLM—which provides a unified interface for various Large Language Models—was the target of a sophisticated supply chain attack. Malicious code was injected into versions 1.82.7 and 1.82.8 of the package on PyPI. The attack is attributed to a threat actor known as TeamPCP, who gained access to LiteLLM’s publishing pipeline by first compromising Trivy, an open-source security scanner used in LiteLLM's CI/CD process.

The compromise is particularly significant due to LiteLLM's wide adoption, with roughly 3.4 million daily downloads (nearly 97 million monthly).

The Attack Vector and Payload

The attackers used a poisoned Trivy GitHub Action to exfiltrate a PyPI publishing token, allowing them to upload malicious versions directly to PyPI while bypassing GitHub code reviews. The malware featured a three-stage payload designed for extensive data theft and persistence:

  • Credential Harvesting: The script systematically collected SSH private keys, cloud credentials (AWS, GCP, Azure), Kubernetes service tokens, API keys, and cryptocurrency wallet seed phrases.
  • Encryption and Exfiltration: Stolen data was encrypted via AES-256 and exfiltrated to a lookalike domain, models.litellm.cloud, which was registered just one day before the attack.
  • Persistence and Lateral Movement: The malware installed a persistent backdoor as a systemd service named "System Telemetry Service". In Kubernetes environments, it attempted to deploy privileged pods to every node to mount the host filesystem and spread the infection further.

Discovery via "Fork Bomb"

The attack was discovered when researcher Callum McMahon noticed his machine became unresponsive due to an unintended "fork bomb". A bug in the malware's .pth file—which triggers on every Python interpreter startup—caused it to recursively spawn new subprocesses until system RAM was exhausted. Experts noted that if the attackers had better programming skills, the compromise might have gone undetected for weeks.

Immediate Remediation Steps

LiteLLM has since removed the poisoned versions from PyPI and rotated all maintainer credentials. Users are advised to take the following actions:

  1. Check Version: Run pip show litellm to ensure you are not on 1.82.7 or 1.82.8. Version 1.82.6 is the last known safe release.
  2. Rotate All Secrets: If you installed the affected versions, treat all credentials on that system (SSH keys, AWS/GCP keys, API tokens, database passwords) as compromised and rotate them immediately.
  3. Check for Persistence: Inspect your system for malicious files such as ~/.config/sysmon/sysmon.py or Kubernetes pods named node-setup-* in the kube-system namespace.
  4. Pin Dependencies: As a long-term defense, developers should pin dependencies to verified versions and audit CI/CD pipelines for unpinned tools
...more
View all episodesView all episodes
Download on the App Store

Daily WTF!By Don Ramón