[Important] Emergency Security Response and Re-Login Request Following Compromise of Python Package "litellm"

The Python package “litellm” was found to be compromised, with malicious code embedded that steals credentials.

This is an extremely serious incident: the malicious code can execute simply by having the package installed. To ensure the security of the entire cluster, PFCP has taken emergency security measures.

We apologize for the inconvenience. Please review the following information and take the necessary action (re-login) as described below.

Security measures taken:

As a result of this action, all existing credentials have been invalidated. You will need to re-login before using kubectl going forward.

Impact:

How to re-login:

Please follow the steps in the documentation below to log in again.

https://docs.pfcomputing.com/cluster/connect-to-cluster.html

Background (overview of the litellm compromise):

Versions 1.82.7 and 1.82.8 of “litellm” contained code that exfiltrated sensitive information—including environment variables and Kubernetes configuration (~/.kube/config)—to an external server.

The malicious code executes simply by starting a Python process. Because Kubernetes tokens may have been stolen without your knowledge, PFCP performed a cluster-wide reset.

Action requested of users:

  1. Immediately check whether the affected version is installed in your PFCP execution environment and on your local PC.
    • Check command: pip list | grep litellm
  2. Treat all secrets (SSH keys, cloud service API keys, etc.) accessible from any environment where this package was installed as potentially compromised, and revoke and reissue them.

Detailed information (GitHub Issue):

https://github.com/BerriAI/litellm/issues/24512

We apologize for the inconvenience and appreciate your understanding and cooperation as we take these necessary steps to maintain security.