Local LLM coding agent weekend — lessons learned

vhsven

2026-08-30

Goal
Final daily-driver setup
- Memory at 128k context
VS Code configuration
Model lessons
Tool calling matters more than expected
Copilot has enormous context overhead
max-num-seqs
Vision
vLLM vs Ollama
systemd is the right way to run the main server
Conda was preferable for CUDA management
Hugging Face downloads
Network/access
Biggest lessons

Goal

Run coding agents from VS Code while doing all LLM inference on a Linux workstation:

Threadripper Pro workstation
128 GB RAM
2× RTX 3090 24 GB
GPUs connected over PCIe (PHB), no NVLink
Ubuntu 24.04 HWE
vLLM providing an OpenAI-compatible API over the LAN
VS Code Copilot using a custom model endpoint

Final daily-driver setup

The best-performing model tested was:

ulkaa/Qwen3.8-27B-AWQ-INT4

Working vLLM configuration:

CUDA_VISIBLE_DEVICES=0,1 \
vllm serve ulkaa/Qwen3.8-27B-AWQ-INT4 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 131072 \
  --max-num-seqs 2 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Important: do not use --enforce-eager for this model.

With eager mode enabled, simple generation was around 16 tokens/s.

With CUDA graphs enabled by removing --enforce-eager, it jumped to roughly 66 tokens/s.

CUDA graph memory itself was tiny: about 0.07 GiB/GPU.

Memory at 128k context

At startup vLLM reported approximately:

model/runtime: ~9.2 GiB/GPU
peak activation: ~1.6 GiB/GPU
KV cache: ~9.25 GiB/GPU
CUDA graphs: ~0.07 GiB/GPU
total visible usage: ~19–20 GiB/GPU

KV capacity:

GPU KV cache size: 290,768 tokens
Maximum concurrency at 131,072 tokens: 2.22x

So 128k context with max-num-seqs=2 fits surprisingly comfortably.

A key lesson: increasing --max-model-len does not necessarily increase idle nvidia-smi memory usage. vLLM allocates a KV-cache pool according to its GPU memory budget; max-model-len mostly determines how much of that pool one request is allowed to consume.

VS Code configuration

[
  {
    "name": "Workstation vLLM",
    "vendor": "customendpoint",
    "apiKey": "dummy",
    "apiType": "chat-completions",
    "models": [
      {
        "id": "ulkaa/Qwen3.8-27B-AWQ-INT4",
        "name": "Qwen3.8 27B INT4",
        "url": "http://192.168.0.167:8000/v1/chat/completions",
        "toolCalling": true,
        "vision": true,
        "maxInputTokens": 120000,
        "maxOutputTokens": 8000
      }
    ]
  }
]

Advertising slightly less than the server's hard 131 072-token limit gives Copilot room for output and avoids requests landing exactly on the context ceiling.

Model lessons

Parameter count is a terrible proxy for deployment size

We managed to run GPT-OSS-120B, yet initially struggled with a dense 27B Qwen model.

Why?

GPT-OSS-120B is an MoE model shipped heavily quantized. Although it has ~120B total parameters, only a small fraction are active per token, and the distributed checkpoint is around 60 GB.

The stock Qwen 27B model was dense BF16:

27B × ~2 bytes ≈ 54 GB

before runtime overhead, KV cache, activations, vision components, etc.

48 GB of physical VRAM simply isn't enough.

The useful questions are therefore:

Dense or MoE?
Bits per weight?
Checkpoint size?
Active parameters per token?
KV/state requirements?
CPU offload?
Architecture/kernel support?

Not merely: "How many billion parameters?"

Quantization was the right answer for Qwen

The AWQ INT4 version is dramatically more practical on 2×3090:

fits entirely on GPUs
large KV-cache budget remains
128k context works
no CPU weight streaming
~66 tok/s
vision
structured tool calling

This is much better hardware utilization than forcing a high-precision checkpoint onto insufficient VRAM.

CPU offload works, but PCIe becomes the bottleneck

GPT-OSS-120B ran successfully with roughly 14 GB of CPU offload.

It achieved around 11 tok/s after tuning.

nvidia-smi dmon showed huge sustained PCIe reads — around 8–12 GB/s — while the GPUs were at 100% SM utilization.

The limiting resource wasn't GPU compute; it was moving offloaded weights from RAM to VRAM.

Reducing offload from 20 GB to 14 GB improved generation from roughly 7 → 11 tok/s.

12 GB offload didn't boot.

Conclusion: CPU offload is useful for making otherwise impossible models run, but it is not equivalent to having enough VRAM.

A third 3090 would potentially be a much more meaningful upgrade for those huge models than small software tweaks.

Tool calling matters more than expected

A model merely producing plausible JSON/XML is not sufficient.

Direct curl tests against /v1/chat/completions were invaluable because they separated:

model/vLLM problem

from:

VS Code / Continue problem

Before connecting a new model to an agent, test:

plain completion
reasoning output
OpenAI-style tool_calls
only then the agent/client

For Qwen3.8:

--tool-call-parser qwen3_coder
--reasoning-parser qwen3

produced proper OpenAI tool-call structures.

Copilot has enormous context overhead

VS Code Copilot Agent mode can consume a large fraction of the context window before actual project content appears.

Observed contributors included:

system instructions
tool definitions
conversation
workspace/browser context

At smaller context sizes this caused hard vLLM context errors and frequent compaction.

32k was technically usable but restrictive.

64k worked with no noticeable performance or memory penalty.

128k also works and is a much more sensible target for real agent work.

`max-num-seqs`

--max-num-seqs controls how many sequences vLLM can actively schedule concurrently.

It does not reserve a complete max-length context for each request ahead of time.

For a personal agent server:

1 = extremely conservative
2 = good default
4+ = useful for real parallel subagents/multiple clients

Extra requests normally queue rather than immediately fail when all sequence slots are occupied.

2 is a good fit because Copilot occasionally performs concurrent/auxiliary calls even without intentionally launching subagents.

Vision

The earlier models were text-only:

Qwen2.5-Coder-7B
Qwen3-Coder-Next
GPT-OSS-20B
GPT-OSS-120B

Qwen3.8-27B is multimodal.

Its vision capability means:

image/screenshot → model → text/tool calls

It does not generate images.

For Copilot this allows screenshots/images to be included in prompts, hence:

"vision": true

vLLM vs Ollama

Ollama's major operational advantage is not necessarily inference speed. It is model lifecycle management.

One Ollama API can expose many installed models and load/swap them automatically.

A normal vLLM process serves one loaded model.

That matters when accessing the workstation remotely: manually SSHing in to stop/start model servers is undesirable.

Long-term architecture:

VS Code
      |
      v
stable OpenAI-compatible gateway
      |
      +---- Qwen fast/vision
      |
      +---- GPT-OSS smart/slow
      |
      +---- future models

The gateway/model manager can start the requested vLLM backend and stop or sleep whichever model currently owns the GPUs.

This gives the Ollama-style UX while retaining vLLM's performance and tuning control.

systemd is the right way to run the main server

The Qwen server now runs as a system service and starts at boot, not user login.

Important service environment:

User=user
WorkingDirectory=/home/user

Environment="HOME=/home/user"
Environment="PATH=/home/user/miniforge3/envs/vllm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="CUDA_HOME=/home/user/miniforge3/envs/vllm"
Environment="CUDA_VISIBLE_DEVICES=0,1"

Then call vLLM directly from the Conda environment:

ExecStart=/home/user/miniforge3/envs/vllm/bin/vllm ...

No conda activate is required.

Conda activation is mostly environment-variable manipulation; systemd can call the environment's executable directly.

Conda was preferable for CUDA management

Using a Conda/Miniforge environment made the CUDA toolkit much less intrusive than installing another system-wide CUDA toolchain.

Current environment ended up with:

CUDA toolkit 13.2
PyTorch built for CUDA 13.2
vLLM 0.28
CUDA_HOME=$CONDA_PREFIX

vLLM/PyTorch pip packages still install many NVIDIA CUDA runtime wheels. This is intentional: Python packages often depend on specific CUDA user-space libraries even when a toolkit is already available.

Hugging Face downloads

hf download reuses completed cached blobs, but interrupted large/Xet-backed files may appear to restart from the beginning rather than resume byte-for-byte.

Don't assume a progress bar restarting at zero means the complete model is being downloaded again.

Useful cache inspection:

du -sh ~/.cache/huggingface/hub/models--OWNER--MODEL
find ~/.cache/huggingface/hub/models--OWNER--MODEL \
  -name '*.incomplete' -ls

Network/access

For LAN testing:

http://192.168.0.167:8000

works.

For access away from home, don't expose vLLM directly through router port forwarding.

The intended setup is Tailscale on the laptop and workstation, then use the workstation's Tailscale 100.x.x.x address as the permanent model endpoint.

This gives an encrypted WireGuard-based path and lets the same coding setup work away from home.

Biggest lessons

Quantization and architecture matter more than parameter count.
Enough VRAM beats CPU offload.
CUDA graphs can transform performance.
Large context is essential for modern coding-agent harnesses.
Direct API tests should precede debugging the client.
Tool-call parser compatibility is a hard requirement for agents.
Ollama solves lifecycle management; vLLM solves high-performance serving.
A gateway/model manager is the natural next piece of infrastructure.
Treat NVIDIA/kernel upgrades conservatively and preserve a known-good kernel.
2× used RTX 3090 remains an absurdly capable local inference setup when the model is chosen appropriately.