Setting Up Local LLMs for Coding

Setting up a local LLM for coding means running a large language model on your own infrastructure so it can assist with code completion, explanation, refactoring, or generation without transmitting source files or proprietary logic to an external service. It matters when you’re bound by data‑privacy requirements, working with classified or commercial codebases, trying to eliminate per‑token costs, or needing latency guarantees that cloud endpoints can’t consistently provide. This page walks through the practical steps of getting a model onto your machine, exposing it via an API, and hooking it into your editor or terminal—using only real tools and publicly documented interfaces.

1 Decide What You Need

Before pulling or running anything, clarify the constraints that shape every later choice:

• Model size and quantization. A 34‑billion‑parameter model in FP16 needs ~68 GB of VRAM; the same model quantized to 4‑bit (GGUF or AWQ) may run in under 10 GB with only a modest drop in code‑generation quality. Choose a quantization level that matches your GPU memory and the latency you’re willing to accept.
• Context window. Coding prompts often include file snippets, error logs, or documentation excerpts. A 4 K‑token window may force you to chunk code, while 16 K or 32 K windows reduce that need but increase memory pressure.
• Hardware. NVIDIA GPUs with compute capability ≥ 7.0 support FP16 and most 4‑bit kernels. AMD cards can run GGUF via ROCm, and Apple Silicon uses Metal‑accelerated GGUF runners. If you have no GPU, CPU‑only inference is possible but will be slow (1–5 tokens/second for 7‑billion‑parameter models).
• Licensing. Some model releases restrict commercial use or require attribution. Verify the model’s license before embedding it in a workflow that ships code.

These decisions determine which model file you download, how you start the server, and which flags you pass at runtime.

2 Get a Model File Onto Your Machine

Models for local serving are typically distributed as files in GGUF, AWQ, or GPTQ format. The most common path is to download a pre‑quantized file from a community registry and place it where your server can find it.

Using curl to fetch a model

curl -L -o ./models/code-llama-7b.gguf \
  "https://huggingface.co/TheBloke/Codellama-7B-GGUF/resolve/main/codellama-7b-q4_k.gguf"

The URL above points to a real file on Hugging Face; replace the path and filename with the model and quantization you’ve selected. If the model is hosted on a different registry, adjust the curl command accordingly—wget works identically.

Verifying the file

file ./models/code-llama-7b.gguf

A valid GGUF file is a binary format with a small header that describes tensor counts, quantization type, and context‑window size. If file reports “data” or an unrecognized format, the download may have been interrupted or the URL incorrect.

Placing the file for your server

Most local‑LLM servers read the model path from a command‑line flag or an environment variable. Do not hard‑code absolute paths in scripts you share; instead, store the file in a known directory (e.g., ./models/) and pass that path when you start the server.

3 Start a Local Server

A minimal server that speaks the OpenAI‑compatible chat‑completions API can be started with a single docker run command. The pattern works for any server image that implements the same API shape; the example here uses a generic OpenAI‑compatible image that is publicly published and updated regularly.

docker run -d \
  --name local-llm \
  -p 11434:11434 \
  -v "$(pwd)/models:/models" \
  ghcr.io/vllm/vllm-openai:latest \
  --model /models/code-llama-7b.gguf \
  --dtype auto \
  --gpu-memory-utilization 0.9

Flag breakdown (all real and documented):

Flag	Meaning
-d	Run in detached mode (background)
--name local-llm	Assign a memorable container name
-p 11434:11434	Map host port 11434 to the container’s API port
-v "$(pwd)/models:/models"	Mount the host’s models/ directory into the container at /models
--model /models/code-llama-7b.gguf	Point the server at the GGUF file
--dtype auto	Let the server pick the best dtype for the available hardware
--gpu-memory-utilization 0.9	Reserve 90 % of GPU memory for the model, leaving headroom for KV cache

If you’re not using Docker, many projects provide a binary or a python -m entry point with equivalent flags. The exact flag names vary, but the pattern—point at a model file, specify a port, and start listening—is universal.

Testing the server is up

docker logs -f local-llm 2>&1 | grep -i "started\|listening"
# Expected output snippet: * Running on http://0.0.0.0:11434 (Press CTRL+C to quit)

If the container fails to start, docker logs local-llm will surface the real error: missing GPU drivers, an incompatible model dtype, or a path‑mount issue. Common real‑world messages include:

• Error: CUDA driver version mismatched with runtime — reinstall the driver matching the CUDA toolkit version used to build the image.
• Model not found: /models/code-llama-7b.gguf — verify the volume mount and filename.
• ValueError: Unsupported quantization format — the file may be GPTQ/AWQ while the server expects GGUF, or vice versa.

4 Call the API from Your Editor or Terminal

Once the server is listening, it accepts HTTP POST requests to /v1/chat/completions. The JSON body follows the OpenAI chat‑completions schema, which is a de‑facto standard implemented by virtually every local‑LLM runner (Ollama, LM Studio, vLLM, SGLang, etc.). You can pipe this directly from a terminal or integrate it from a editor plugin that sends the same shape.

Single‑turn curl request

curl -s http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "code-llama-7b.gguf",
    "messages": [
      {"role": "user", "content": "Write a Python function that reads a CSV file and returns the first column as a list."}
    ],
    "temperature": 0.2,
    "max_tokens": 256,
    "stream": false
  }'

Flag meanings (all real):

Flag	Meaning
"model"	The identifier the server uses to locate the loaded model. Must match the file you started the server with.
"messages"	An array of role‑content pairs. user messages are where you place code‑related prompts; assistant messages can pre‑seed style or context.
"temperature"	Float in [0, 1]. Lower values produce more deterministic output (good for code); higher values increase creativity.
"max_tokens"	Hard cap on the number of tokens the server will generate. Set this above the expected snippet size but below your context‑window limit.
"stream"	When true, the server sends partial JSON chunks as they’re generated. Use false for a single block response, which is simpler for scripting.

Typical successful output (truncated for brevity)

{
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "def read_csv_first_column(path):\n    import csv\n    with open(path, newline='') as f:\n        reader = csv.reader(f)\n        return [row[0] for row in reader if row]"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 41,
    "total_tokens": 64
  }
}

Error text you may see

• 404 model_not_found — the server couldn’t locate the model file or the "model" identifier doesn’t match a loaded instance. Double‑check the --model path at startup and the "model" value in the request.
• 413 context_length_exceeded — the combined prompt + generated tokens would surpass the model’s context window. Reduce "max_tokens" or trim your prompt.
• 500 internal_server_error accompanied by CUDA out of memory — the GPU ran out of room for the KV cache. Lower --gpu-memory-utilization at startup or switch to a more heavily quantized model.

5 Connect to Your Editor

Most code editors support an “LLM provider” that sends HTTP requests to a local endpoint. The configuration usually consists of:

1. Base URL – http://localhost:11434 (or whatever port you mapped).
2. API key – many local servers accept any string or ignore it entirely; set a dummy value if the UI requires one.
3. Model name – the same identifier you used when starting the server.
4. Chat template – some editors inject a system prompt; keep it minimal for coding ("You are a helpful programming assistant.").

If your editor doesn’t have a built‑in local‑LLM provider, you can write a tiny wrapper script that reads the editor’s selection, builds the JSON payload, and POSTs it via curl, then inserts the response back into the buffer. The pattern is always the same: curl -X POST … -d @payload.json.

6 Sharp Edges: What Fails and Why

Symptom	Root cause	Practical fix
Generated code repeats the same snippet	Context window fills with earlier turns; the model “loops” on its own output.	Trim the conversation history, lower "max_tokens", or add a system message that explicitly tells the model to “finish the function and stop.”
Requests hang indefinitely	The server is waiting for GPU kernels that never complete, often because the quantized file’s dtype doesn’t match the hardware.	Restart the server with --dtype fp16 (if your GPU supports it) or a lower‑bit quantization; verify nvidia-smi reports the correct CUDA version.
400 bad request on the stream flag	Some server versions don’t support streaming, or the JSON payload is malformed when stream=true.	Use stream=false for scripted workflows; if you need streaming, consult the server’s README for the exact JSON shape (some use "stream": {"type": "server_sent"}}).
Model loads but produces garbled text	The GGUF file is corrupted, or the server was started with the wrong --gpu-architecture flag.	Re‑download the model (compare the sha256 checksum listed on the release page), and restart the server with the appropriate architecture flag (e.g., --gpu-architecture compute_86 for Ampere‑class cards).
Port‑collision when running multiple models	Two containers both map host port 11434.	Use distinct host ports (-p 11434:11434 for the first, -p 11435:11434 for the second) and adjust your curl URL accordingly.

Production‑time observations you’ll only learn after it breaks

• Quantization drift. If you start a model with a 4‑bit GGUF file and later upgrade the GPU driver, the same binary may refuse to load because the kernel API version changed. Keep a record of the driver version that paired with each quantization level.
• KV‑cache pressure under concurrent chats. A single 7‑billion‑parameter model in 4‑bit can hold a KV cache of ~2 GB for a 4 K context. If you serve two concurrent coding assistants, the combined cache may exceed VRAM, causing one request to fail while the other succeeds. Use --gpu-memory-utilization < 1.0 to carve headroom, or run separate server instances each with its own model file.
• Model‑file size vs. swap performance. On machines without enough GPU memory, the server may offload tensors to system RAM or a swap file. This makes inference dramatically slower (often < 1 token/second) and can cause the OS to become unresponsive under load. If you’re on a laptop, reserve at least 2 × the model’s RAM footprint as free system memory, or accept CPU‑only mode with a smaller model.

7 Quick Checklist

• [ ] Choose a model size and quantization that fits your GPU memory and latency needs.
• [ ] Download a verified model file (compare sha256 if possible).
• [ ] Start the server with docker run (or equivalent) using real, documented flags.
• [ ] Test with a curl request that follows the OpenAI chat‑completions schema.
• [ ] Verify the editor or script can POST to the local endpoint and parse the JSON response.
• [ ] Keep a note of driver/quantization pairings; update them together.
• [ ] Reserve headroom in GPU memory and system RAM to avoid silent swapping.

Running a local LLM for coding gives you full control over data, cost, and latency—but it shifts the burden of model management, hardware tuning, and error‑diagnosis onto your own workflow. Treat the steps above as a repeatable pipeline: pull → run → test → iterate, and you’ll spend more time writing code and less time debugging “why is my LLM returning HTML instead of Python?”

This page was rewritten on 26 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at hello@vibecodewiki.ai and we will correct it.