Raphael Friedmann
← Understanding AI

When a downloaded model is safe to run

A model file looks like a set of numbers, and some formats are exactly that. Others run code the moment you load them. Which formats are which, what has already gone wrong, and the checks I'd make before a model touches a work machine.

When someone on the team wants to try a new model, the conversation is usually about size, speed and licence. Almost nobody asks the question I’d ask about any other file from the internet: will this run code on my machine? The intuition is that a model is just weights, a few gigabytes of numbers, and numbers can’t hurt you.

For some formats that intuition is right. For others it is wrong, and the difference is a file extension most people never look at. This lesson is about telling them apart, and about the cheap habits that make the question mostly go away.

Where code can run#

There are three ways a “model” can do more than hand over numbers.

Pickle checkpoints. The classic PyTorch formats (.bin, .pt, .pth, .ckpt) are pickle underneath. A pickle can name any Python function to call while it is being loaded, so a model file can open a reverse shell before you have typed a single prompt. This has already happened: in February 2024 JFrog reported around 100 models on Hugging Face that executed code on load, some of them connecting back to a real IP address. PyTorch 2.6 made weights_only=True the default for torch.load, which loads only tensors and a short list of safe types. Before 2.6 even that flag could be bypassed to run code (CVE-2025-32434), so use 2.6 or later. That helps, but old code, old versions and every tutorial that says “just add weights_only=False to fix the error” still open the door.

trust_remote_code=True. Some models on Hugging Face ship their own Python files for architectures the transformers library doesn’t know yet. Setting this flag tells the library to download and run that code. The documentation says it plainly: only for repositories you trust and whose code you have read. It is not a format problem at all, it is code execution by design.

The loader itself. Formats like GGUF are pure data, but some C code still has to parse them, and parsers have bugs. In January 2024 Databricks reported five heap overflows in the GGUF parsing of the ggml library that llama.cpp is built on (CVE-2024-25664 to CVE-2024-25668), patched in llama.cpp within a week. A few months later a different hole turned up one layer higher: GGUF files carry a chat template, and llama-cpp-python rendered it with an unsandboxed Jinja2 engine, so a crafted template could run code (CVE-2024-34359, versions 0.2.30 to 0.2.71, fixed in 0.2.72). Data formats are safe only as long as the program reading them is current.

Model file formats, from "just data" to "runs code by design"
safetensorsGGUF.bin .pt .ckpttrust_remote_codetensors only,no code pathdata, but theparser must beup to datepickle: coderuns on loadPython from therepo runs onpurposemore trust needed in whoever made the file
The left two are data formats: the risk is a bug in the program that reads them, fixed by updating it. The right two execute code by construction: the only defence is trusting the author, or not loading the file outside a sandbox.

Scanners help, they don’t decide#

Hugging Face scans uploaded pickle files for dangerous imports and flags what it finds. That catches simple attacks, but not more sophisticated ones: in February 2025 ReversingLabs found two models (they called the technique nullifAI) that slipped past the scanner by packing a PyTorch file with 7z instead of ZIP and deliberately breaking the pickle stream after the payload. The scanner refused to analyse a broken file; Python’s loader still ran the instructions up to the break, payload included. Hugging Face removed the models within a day and updated the scanner, but the lesson stands: a green badge means “nothing known was found”, not “safe”.

Check who uploaded it#

Anyone can create an account and name a repository whatever they like. A copy of a popular model under an organisation name one letter off, or a “quantised” re-upload by an account with no history, is the easiest supply-chain attack there is. The checks are simple and fast:

  • Start from the publisher, not from search. Follow the link from the vendor’s own site, blog post or model card to the organisation page, and download from there.
  • Prefer safetensors or GGUF. If the only option is a pickle checkpoint, ask why. Most current releases ship safetensors.
  • Pin what you tested. A repository can change after you looked at it. Download by commit (the revision argument in transformers, or a fixed tag in your tooling) and keep the SHA256 that Hugging Face shows for every large file.
  • Read before trust_remote_code. If a model needs it, someone reads the Python files in that exact revision, or the model waits until the library supports it natively.
  • Keep the runtime current. llama.cpp, Ollama, vLLM and similar tools are parsers for files from the internet. Patch them like you patch a browser.

Defence in depth, because a check can fail#

Even with all of the above, I run new models the way I’d run any unfamiliar service: in a container, as a non-root user, with the model directory mounted read-only, with no access to credentials, home directories or the corporate share, and with no outbound network unless it is needed. Keep the container runtime and the NVIDIA Container Toolkit patched as well: a flaw there (CVE-2025-23266) let a crafted image take over the host. A reverse shell that cannot reach anything is a log entry, not an incident. The step-by-step setup in this course works the same way inside a container; it just takes one more line.

Risky

  • Grabbing the top search hit from an unknown account
  • Loading a .bin or .pt because "it's just weights"
  • Adding weights_only=False to silence an error
  • trust_remote_code=True without reading the code
  • Running the model as your own user, next to your SSH keys

Safe

  • Download via the publisher's own link
  • safetensors or GGUF, with a current runtime
  • Pinned revision and recorded SHA256
  • Container, read-only weights, no secrets, no network
  • Licence checked in the same step

Takeaway#

The safe path has become the default path. Big releases ship safetensors and GGUF, PyTorch refuses arbitrary pickles out of the box, and the known parser holes are patched. The risk that remains lives in the shortcuts: the unverified re-upload, the flag added to make an error go away, the runtime nobody updated since spring. Treat a model file like any other download that ends up executing on your hardware. It’s the same discipline as prompt injection, one layer further down: assume something hostile will eventually get loaded, and make sure it finds nothing worth taking when it does.