Local GGUF Model Guide
Local models let you run AI chat without sending prompts to a cloud model provider. Tavern Studio supports local GGUF workflows for users with suitable hardware.
Who this is for
- Users with capable CPUs, GPUs, or dedicated local model setups.
- Users who prefer offline or local-first roleplay.
- Users troubleshooting out-of-memory errors or slow generation.
What you will learn
- What GGUF is.
- How local models differ from cloud APIs.
- How to import or download a local model.
- How GPU Layers and related settings affect performance.
- How to troubleshoot crashes.
GGUF and hardware expectations
GGUF is a common format for local LLM inference. Quantized models reduce memory requirements, but larger models still need enough RAM or VRAM.
Step 1: Download or import a GGUF model
Use the model management area to download or import a .gguf file. Choose a model size your hardware can handle.
Step 2: Start the local model
Open local model settings and configure:
- Backend type.
- Context size.
- GPU Layers.
- CPU threads.
- Chat format when required.
Troubleshooting
Why does the app crash or show “Out of Memory”?
Lower GPU Layers, use a smaller quantization, or choose a smaller model.
Why is generation very slow?
Try a smaller model, adjust threads, use GPU acceleration if available, or reduce context size.
Why does import fail?
The file may be incomplete, corrupted, or not a valid GGUF model.
Next steps
- Interface Overview: understand where local model settings appear.
- Configure Your First API: use cloud APIs when local hardware is not enough.
- FAQ: review more troubleshooting questions.