AI models with malicious intent found in the Hugging Face repository, executing code

Researchers from JFrog identified in the repository Hugging Face malicious machine learning models, the installation of which can lead to the execution of attacker code to gain control of the user's system. The problem is caused by the fact that some model distribution formats allow embedding of executable code, for example, models using the “pickle” format may include serialized objects in Python, as well as code executed when a file is loaded, and Tensorflow Keras models can perform code via Lambda Layer.

To prevent the spread of such malicious models, Hugging Face uses serialized code substitution scanning, but the identified malicious models indicate that the existing checks can be bypassed. In total, about 100 potentially malicious models were identified, 95% of which are intended for use with the PyTorch framework, and 5% with Tensorflow. The most common malicious changes include capturing an object, organizing an external login (reverse shell), launching applications, and writing to a file.

It is noted that judging by the actions taken, most of the identified malicious models were created by security researchers trying to receive a reward for discovering vulnerabilities and methods for bypassing Hugging Face protection (for example, instead of an actual attack, such models try to launch a calculator or send a network request with information about the success of the attack). At the same time, there are also instances that launch a reverse shell to connect the attacker to the system.

For example, the models “baller423/goober2” and “star23/baller13” are aimed at attacking systems that load a model file into PyTorch using the torch.load() function. To organize code execution, the “__reduce__” method from the pickle module is used, allowing insert arbitrary Python code into the deserialization process performed when loading the model.

Thanks for reading: