File Formats: GGUF, Safetensors, and MLX

The 30-second version: A model file is its weights (99%+ of the bytes) plus config, tokenizer, and metadata โ€” packaged in a *format*. Three formats matter for local inference: GGUF (single file, quantized, the standard for llama.cpp/Ollama/LM Studio/Strata), safetensors (sharded, full-precision or FP8, the standard for vLLM and the PyTorch world), and MLX (Apple Silicon's unified-memory format). The rule that decides everything: your backend picks the format, not the other way around.

What's inside a model file

Before comparing containers, know what they hold. Every model file packages four things:

The format is the container for these four. The same model in three formats is three different download โ€” and only one of them will run on your backend.

The three formats that matter

GGUF โ€” the all-rounder

One file, every quant, every consumer backend. The default choice for local LLMs.

GGUF is the unified file format born out of llama.cpp (the successor of the older GGML format). Weights, config, tokenizer, and metadata all live in a single file โ€” no stitching, no index, no surprises. It carries the entire GGUF quantization ladder, from Q8_0 down through Q4_K_M, IQ3_S, and I-Nano, which is why it is the format of choice for bandwidth-constrained and VRAM-constrained machines.

The Matrix is a good snapshot of the ecosystem: 44 of 50 community submissions run GGUF โ€” through llama.cpp, llama-swap, Ollama, and Strata. Its main limitation is that it is a quants-first format: the FP8 weight ecosystem lives in safetensors, not GGUF. So if a model is only published as native FP8, GGUF is not where you'll find it.

Safetensors โ€” the PyTorch/vLLM standard

Sharded, full-precision or FP8. What training tools output and what vLLM serves.

Safetensors is the tensor container of the PyTorch world โ€” "safe" because, unlike old pickle files, it executes no code when loaded. Big models arrive as multiple shards (more below) with an index file mapping each tensor to its shard. This is the format of full-precision (FP16/BF16) and FP8 checkpoints, and the home of the W4A16-style quants (GPTQ, AWQ, AutoRound) that vLLM and TensorRT-LLM serve.

Both of the Matrix's vLLM submissions โ€” MadGoat's and digitalscream's Swift-Qwen3.8-27B-Uncensored-FP8 on 2ร— R9700 โ€” run safetensors. If you fine-tune, you will also produce safetensors: it is the standard training output.

MLX โ€” the Apple Silicon native

The unified-memory format for Macs. No MLX submissions in the Matrix today โ€” the ecosystem's biggest data gap.

MLX is Apple's array framework and its native model format, built around the fact that a Mac's CPU and GPU share one pool of memory. It carries its own 4-bit and 8-bit quant sets, and it is what you run when you use MLX (or Unsloth Desktop on Metal) on Apple Silicon. Models are commonly published as safetensors and converted to MLX for Mac use.

Honest caveat: the Matrix has no MLX submissions yet โ€” the community data skews NVIDIA and AMD. If you run a Mac, your experience is underrepresented here, which is exactly why an MLX submission from you would be valuable.

GGUF Safetensors MLX
File layout Single file Multiple shards + index Single set (Apple layout)
Quant support Full GGUF ladder (Q8โ†’I-Nano) FP16/BF16/FP8, GPTQ/AWQ/AutoRound MLX 4-bit / 8-bit
Backends that read it llama.cpp, Ollama, LM Studio, llama-swap, Strata vLLM, TensorRT-LLM, PyTorch fine-tuning MLX, Unsloth Desktop (Metal)
Best for Consumer GPUs, tight VRAM Serving, FP8, training Apple Silicon

Which format do I download?

The decision is made for you by the backend you run:

Your backend Download this
llama.cpp / llama-swap GGUF
Ollama GGUF (Ollama pulls them for you)
LM Studio GGUF
Strata GGUF
vLLM / TensorRT-LLM Safetensors (FP8, AWQ, GPTQ, โ€ฆ)
MLX / Mac MLX
Fine-tuning Safetensors

If you are not sure which backend to run, that is a Backends question โ€” answer it first, and the format follows. Never download a format "just in case": a 27B model is 15โ€“26 GB per format, and they are not interchangeable.

Shards: why big models come in boxes

A 70B model in FP16 is ~140 GB. File systems, download managers, and disk quotas all choke on single files that big โ€” so the safetensors world splits models into shards: model-00001-of-00008.safetensors through โ€ฆ-00008โ€ฆ, each around 16 GB, plus a small index JSON that tells the loader which tensor lives in which shard. You download them all; the loader stitches them in memory transparently.

GGUF sidesteps the problem entirely with a single file โ€” one of the practical reasons it won the consumer space. (A 70B at Q4_K_M is a single ~40 GB file; at I-Nano-class bits it shrinks further. The per-size math is the same VRAM formula from the Quantization Basics guide โ€” it works for disk space too.)

Can I convert between formats?

Yes โ€” in one direction. The rule is: you can quantize down from full precision; you cannot un-quantize up.

So the practical flow is: find the model's safetensors (or its MLX conversion), then produce the GGUF quant your backend wants. Converting is cheap; downloading three formats is not.

How this fits the Matrix

The Matrix's quant column is really format + level combined: "GGUF Q4_K_M", "FP8", "GGUF I-Nano". Now you can read each row's packaging โ€” and you know why the data skews GGUF (44 of 50 submissions) with the FP8/safetensors rows concentrated in the vLLM setups. When you submit your own, name the format and the level together, and the next person can pick the right file for their backend.