Running Qwen3.8-27B on JupyterHub GPUs and Wiring It Into Agent Harnesses (Deep Code CLI + DeepSeek Harness)

If your lab or department gives you JupyterHub access to shared GPUs, you already have everything you need to run your own private, unmetered coding agent — no API budget, no rate limits, no sending your code or data to a third party. This walks through doing exactly that: serving a large open-weight model on shared JupyterHub GPU infrastructure with vLLM, then connecting two different "Claude-Code-like" terminal/web agent harnesses to it. Every fix below was hit for real — this isn't a clean-room tutorial, it's the actual path including the dead ends, because the dead ends are usually the part that actually costs you time.

Table of contents

• Part 1 — Serve the model with vLLM
• Part 2 — Deep Code CLI (terminal harness, runs on the same box)
• Part 3 — DeepSeek Harness (web UI) run locally on your own laptop
• Postscript — a real bug found the same day
• Summary of the recurring lessons

Model: Qwen/Qwen3.8-27B — 27B params, hybrid Gated DeltaNet + Gated Attention architecture, native vision/video input, 262144-token context (extensible to 1M via YaRN), Apache 2.0, tuned for agentic/tool-use workflows.

Environment this was built on: JupyterHub-on-Kubernetes (<your-hub-domain>), 2x NVIDIA RTX PRO 6000 Blackwell (96GB VRAM each), driver CUDA 13.2, no system-level CUDA toolkit installed (driver only), no root access, jupyter-server-proxy enabled. The commands below are identical regardless of which JupyterHub instance you're on — just swap in your own hub domain, base path, and username wherever you see the placeholders. Your proxy path might not have an extra base segment like the <your-hub-base-path> shown below (e.g. it could just be https://<your-hub-domain>/user/<you>/proxy/8000/) — check what your own hub's URL looks like when you're logged into JupyterLab.

A few concepts you'll see used throughout

If you haven't used these before, here's the two-sentence version of each — skip this box if you already know them.

• tmux — a terminal multiplexer. Anything you run in a normal JupyterLab terminal tab dies the moment you close that tab or lose connection; anything run inside a tmux session keeps running in the background even after you disconnect, and you can reattach to check on it later. We use it for anything that needs to stay alive for hours/days (the model server, the tunnel).
• venv (Python virtual environment) — an isolated folder of Python packages, separate from your system/base Python. We use a fresh one for vLLM so its exact package versions don't conflict with anything else on the shared box.
• jupyter-server-proxy / the "proxy" URLs — JupyterHub normally only exposes the Jupyter interface itself to your browser. This extension lets you also reach other things running inside your session (like a web server you started) through a URL like https://<hub>/user/<you>/proxy/<port>/, tunneled through JupyterHub's own authentication. It's how we reach vLLM's docs page and (unsuccessfully, see Part 3) tried to reach web-UI chat tools.
• "Harness" — in this post, just means an agent tool: something that takes a chat model and wraps it with the ability to read/edit files, run shell commands, and generally act autonomously, the way Claude Code does. Deep Code CLI and DeepSeek Harness are both examples.

Part 1 — Serve the model with vLLM

1.1 Environment setup

python3 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip install --upgrade pip
pip install -U vllm
pip install "huggingface_hub[hf_xet]"
export HF_HUB_ENABLE_HF_TRANSFER=1

Pre-download the model (optional but recommended so download issues surface early, ~54GB in bf16):

hf download Qwen/Qwen3.8-27B

(huggingface-cli is deprecated in newer huggingface_hub releases — use hf instead.)

1.2 CUDA toolkit fixes

This box has the NVIDIA driver but no CUDA developer toolkit (nvcc) installed at the OS level — vLLM needs nvcc to JIT-compile custom kernels (FlashInfer, Triton ops for the model's hybrid attention layers). Three separate issues surfaced here, in order:

(a) nvcc not found at all

Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist

nvidia-cuda-nvcc was already a transitive pip dependency, just not on PATH/CUDA_HOME:

export CUDA_HOME=/home/jovyan/vllm-env/lib/python3.12/site-packages/nvidia/cu13
export PATH=$CUDA_HOME/bin:$PATH

(b) Compiler/toolkit version mismatch

CUDA compiler and CUDA toolkit headers are incompatible

nvidia-cuda-nvcc had resolved to 13.3.73 while nvidia-cuda-runtime (the headers) was 13.0.96 — different minor CUDA releases. Pin nvcc down to match:

pip install "nvidia-cuda-nvcc==13.0.88" "nvidia-cuda-crt==13.0.88" "nvidia-nvvm==13.0.88"
rm -rf ~/.cache/flashinfer ~/.cache/vllm/torch_compile_cache

(c) Linker can't find -lcudart

/usr/bin/ld: cannot find -lcudart

Two separate problems: the actual library lived in nvidia/cu13/lib, but the linker was searching nvidia/cu13/lib64 (didn't exist); and only a versioned libcudart.so.13 existed, no unversioned libcudart.so symlink that -lcudart needs.

ln -sf /home/jovyan/vllm-env/lib/python3.12/site-packages/nvidia/cu13/lib /home/jovyan/vllm-env/lib/python3.12/site-packages/nvidia/cu13/lib64
ln -sf libcudart.so.13 /home/jovyan/vllm-env/lib/python3.12/site-packages/nvidia/cu13/lib/libcudart.so
rm -rf ~/.cache/flashinfer

(libcuda.so, the driver's own library, was already fine at /usr/lib/x86_64-linux-gnu/libcuda.so — a standard system path, no fix needed there.)

1.3 Two-GPU deadlock — had to fall back to one GPU

With --tensor-parallel-size 2, the server would get all the way through weight loading, torch.compile, and CUDA graph capture, then hang forever (before ever printing "Uvicorn running"), repeating:

No available shared memory broadcast block found in 60 seconds.

/dev/shm was confirmed not the cause (30GB free). This is an inter-process/NCCL coordination deadlock between the two GPU worker processes — most likely a bug in this vLLM version's handling of this very new hybrid architecture (Gated DeltaNet linear-attention + regular attention) under multi-GPU tensor parallelism. Dropping to --tensor-parallel-size 1 booted and served cleanly — single GPU has more than enough VRAM for a 27B model (~25.7GB of weights), and it leaves the second GPU free.

1.4 CUDA graph capture memory ceiling

graph capture cannot proceed. Please lower max_num_seqs to at most 605 or increase gpu_memory_utilization.

With a 262144-token max context, the memory needed to capture CUDA graphs for all configured batch sizes exceeded what was left after weights + KV cache at 90% GPU utilization. Fix: cap concurrency explicitly (fine for an agent workload, which rarely needs hundreds of simultaneous sessions):

--max-num-seqs 256

1.5 JupyterHub proxy needs --root-path

Accessing the built-in Swagger docs (/docs) through the JupyterHub reverse proxy failed:

Fetch error: Not Found /openapi.json

FastAPI was generating URLs relative to the domain root instead of the proxy's subpath. Fix — tell it what prefix it's served under:

--root-path /<your-hub-base-path>/user/<your-username>/proxy/8000

This only affects generated URLs (docs/openapi), not actual request routing — direct localhost:8000 access keeps working unprefixed.

1.6 Chat template rejects agent-harness message ordering

Once an agent harness (Deep Code CLI, see Part 2) started sending real requests:

HTTP 400: System message must be at the beginning.

Qwen's default chat template strictly enforces that if a system role message exists, it must be at index 0 of the messages array — many agent harnesses don't guarantee this (they inject reminder/context messages with role: system mid-conversation). Fix: swap in a community-patched chat template that relaxes this check:

hf download froggeric/Qwen-Fixed-Chat-Templates chat_template.jinja --local-dir ~/chat-templates

Then add --chat-template /home/jovyan/chat-templates/chat_template.jinja to the launch command.

1.7 Wrong tool-call parser — calls leaked as plain text instead of structured tool_calls

Once a harness actually tried invoking a tool (e.g. "list the files here"), the agent would announce intent and then just stop — no file listing ever appeared, no error either. Testing directly with curl showed why: the model's raw output was

<tool_call>
<function=list_files>
<parameter=path>
.
</parameter>
</function>
</tool_call>

That's an XML-style tool-call format, not the JSON-style format --tool-call-parser hermes expects (<tool_call>{"name": "...", "arguments": {...}}</tool_call>). Since it didn't match, vLLM couldn't populate the structured tool_calls field and just returned the tags as literal text in content — which every harness silently treats as "the model didn't call anything," so it stalls instead of erroring loudly. Fix: use the XML-aware parser instead:

--tool-call-parser qwen3_xml

(vLLM's docs associate qwen3_xml with Qwen3-Coder models specifically, but this newer Qwen3.8-27B evidently emits the same format — worth testing directly with curl and a tools array whenever tool-calling silently does nothing, rather than assuming the harness is broken.)

1.8 Final working launch script

mkdir -p ~/vllm_logs
cat <<'EOF' > ~/run_vllm.sh
#!/bin/bash
source ~/vllm-env/bin/activate
export HF_HUB_ENABLE_HF_TRANSFER=1
export CUDA_HOME=/home/jovyan/vllm-env/lib/python3.12/site-packages/nvidia/cu13
export PATH=$CUDA_HOME/bin:$PATH

while true; do
  vllm serve "Qwen/Qwen3.8-27B" \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 262144 \
    --max-num-seqs 256 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --root-path /<your-hub-base-path>/user/<your-username>/proxy/8000 \
    --chat-template /home/jovyan/chat-templates/chat_template.jinja \
    --host 0.0.0.0 --port 8000 \
    2>&1 | tee -a ~/vllm_logs/server.log

  echo "$(date) vllm exited (code $?), restarting in 10s..." | tee -a ~/vllm_logs/restart.log
  sleep 10
done
EOF
chmod +x ~/run_vllm.sh

Run it inside tmux so it survives closing the browser tab:

tmux new -s qwen-vllm
~/run_vllm.sh
# wait for "Application startup complete", then Ctrl+b, d to detach

Stopping it properly: the while true loop auto-restarts on any exit, so a single Ctrl+C just relaunches it 10s later. To actually stop it:

tmux kill-session -t qwen-vllm

Chasing stray processes: vLLM's worker/engine processes rename themselves (via setproctitle) to things like VLLM::Worker_TP0 — pkill -f "vllm serve" will not match them. Use pkill -9 -f "VLLM::" instead when cleaning up, and always double-check with nvidia-smi + ps aux | grep -i vllm before restarting (starting on top of leftover processes causes Address already in use failures).

1.9 VRAM and boot time notes

• Weights: ~25.7GB (single GPU, TP=1).
• With --gpu-memory-utilization 0.90, the server claims ~86GB total on that GPU (weights + pre-allocated KV cache pool) as soon as it finishes booting — whether or not it's actively serving requests. This is normal vLLM behavior, not a leak; it doesn't release memory while idle. Stop the server to free it.
• Boot time: ~15s to load weights, ~30s for torch.compile (cached after first run in ~/.cache/vllm/torch_compile_cache), a couple minutes for one-time FlashInfer kernel compilation (cached in ~/.cache/flashinfer), plus KV cache profiling / CUDA graph capture every boot (not cached, a few minutes given the large context). First successful boot: ~5-10 min. Later restarts: ~2-5 min.

1.10 Testing the server without installing any harness

Useful for confirming the server itself works before adding a harness into the mix — that way, if something breaks later, you already know whether the problem is the server or the harness.

curl (always set --max-time — a broken config can hang forever otherwise):

curl -s --max-time 60 http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen/Qwen3.8-27B","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":50,"chat_template_kwargs":{"enable_thinking":false}}' | python3 -m json.tool

Swagger UI (built into vLLM, form-based, no install needed) — through the JupyterHub proxy once --root-path is set:

https://<your-hub-domain>/user/<you>/proxy/8000/docs

Expand POST /v1/chat/completions → "Try it out" → edit the JSON body → "Execute".

Browser devtools console (from any tab already logged into your JupyterHub, credentials: 'include' carries your session auth through the proxy):

fetch('https://<your-hub-domain>/user/<you>/proxy/8000/v1/chat/completions', {
  method: 'POST', headers: {'Content-Type': 'application/json'}, credentials: 'include',
  body: JSON.stringify({model: 'Qwen/Qwen3.8-27B', messages: [{role: 'user', content: 'hello'}], max_tokens: 50})
}).then(r => r.json()).then(console.log)

Testing tool-calling specifically (this is what caught the parser bug in 1.7 — always verify this before trusting a harness with real tool-use):

curl -s --max-time 60 http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen/Qwen3.8-27B","messages":[{"role":"user","content":"What is the weather in Nashville?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get current weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]}' | python3 -m json.tool

Check the response has a proper tool_calls array — not raw <tool_call> tags leaking into content.

Part 2 — Deep Code CLI (terminal harness, runs on the same box)

Deep Code CLI is a terminal coding agent (Claude-Code-like) built for DeepSeek-V4, but works with any OpenAI-compatible endpoint. Since it's a plain terminal tool, there's no web UI / reverse-proxy problem at all — just run it in the same JupyterHub session as the vLLM server.

2.1 Install Node.js (needs 22+)

No root access, so use nvm (installs entirely into your home directory):

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm install 22
nvm alias default 22

Note: Node 20 installs and runs npm install, but throws EBADENGINE warnings — the CLI's terminal-rendering library (ink) needs Node ≥22. Go straight to 22 to avoid it.

Gotcha: nvm adds its loader lines to ~/.bashrc, but some JupyterLab terminal configs start non-login shells that don't source it. If a fresh terminal tab says node: command not found, just re-run the two export/source lines above.

2.2 Install and configure

npm install -g @vegamo/deepcode-cli
deepcode --version

mkdir -p ~/.deepcode
cat <<'EOF' > ~/.deepcode/settings.json
{
  "env": {
    "MODEL": "Qwen/Qwen3.8-27B",
    "BASE_URL": "http://localhost:8000/v1",
    "API_KEY": "not-needed"
  }
}
EOF

API_KEY can be any placeholder — the vLLM server isn't enforcing auth (no --api-key flag was passed).

2.3 Run

cd ~/your-project-dir
deepcode

This is what originally hit both bugs above: the chat-template rejection (1.6) on any message, and the tool-call parser mismatch (1.7) the moment it tried to actually use a tool (e.g. "list the files here" — it would announce intent and then silently stop). Once both fixes were applied and the server restarted, it worked cleanly end to end.

Part 3 — DeepSeek Harness (dsh, web UI) run locally on your own laptop

DeepSeek Harness is a different, more involved tool — a plugin-based agent framework with a web UI (not a terminal app), served at 127.0.0.1:3080.

3.1 Why not run it on the JupyterHub box like Deep Code CLI

Two web-UI tools were tried against the JupyterHub reverse-proxy route (https://<hub>/user/<you>/proxy/<port>/) and both broke the same way:

• Open WebUI — a popular self-hosted ChatGPT-style UI. Installed fine, but its SvelteKit frontend bakes static asset paths in at build time with no support for subpath/reverse-proxy deployment (an open, unresolved upstream issue). Every JS/CSS/image request 404'd under the proxy prefix — broken logo, broken everything.

• DeepSeek Harness's own web UI — same class of problem confirmed (no base-path config found anywhere in its docs), and its own README hints it's designed for either a fully local launch or SSH port-forwarding, not a subpath proxy.

Rather than patching either tool (nontrivial — would need a custom jupyter_server_proxy config with response-rewriting), the simpler fix is: run the harness on your own machine, where its web UI opens at 127.0.0.1:3080 with zero proxy involved. The only remaining problem is letting your laptop reach the vLLM API running on the remote GPU box — and a plain REST API doesn't have the asset-path problem a full SPA does.

3.2 Expose just the vLLM API port with a Cloudflare quick tunnel

On the GPU box:

curl -L --output ~/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
chmod +x ~/cloudflared
tmux new -s tunnel
~/cloudflared tunnel --url http://localhost:8000

It prints a random public URL:

https://<random-words>.trycloudflare.com

Security note: this is a public, unauthenticated URL — anyone who gets the link can send inference requests to your GPU. It's not indexed/guessable, but it's not private the way an SSH tunnel is. Fine for a working session; close the tunnel (tmux kill-session -t tunnel) when you're done, don't leave it running unattended. (If you have SSH access to the box, an SSH tunnel — ssh -L 8000:localhost:8000 user@host — is the private alternative and was the original plan here; the Cloudflare tunnel was the fallback since SSH access wasn't available.)

In plain terms, what this tunnel does: your laptop can't normally reach "localhost:8000" on a different computer — that phrase only ever means "this machine." The tunnel gives the GPU box a real public address that Cloudflare forwards straight through to localhost:8000 on it, so your laptop's requests have somewhere real to go.

3.3 Install Node.js and configure DeepSeek Harness — on your own laptop this time

node --version   # confirm you have a recent Node; if not, install via nvm or your OS package manager

mkdir -p ~/.dsh
cat <<'EOF' > ~/.dsh/settings.yaml
llm-pi-ai:
  providers:
    qwen-local:
      apiKeyEnv: VLLM_API_KEY
      api: openai-completions
      baseURL: https://<your-tunnel-url>.trycloudflare.com/v1
      compat:
        supportsDeveloperRole: false
        maxTokensField: max_tokens
      models:
        - id: Qwen/Qwen3.8-27B
EOF

The compat block matters: since Qwen3.8-27B is a reasoning model, DeepSeek Harness's request layer (pi-ai) infers request shape from the endpoint URL and, for an unrecognized host, defaults to OpenAI's newer reasoning-model conventions — sending the system prompt as role: "developer" and capping output with max_completion_tokens. vLLM supports neither. The docs explicitly document this fix (compat.supportsDeveloperRole: false, compat.maxTokensField: max_tokens).

3.4 Launch

export VLLM_API_KEY=not-needed

First attempt crashed with a Node heap out-of-memory error on this very early (0.1.1-rc.2, developer-preview) release:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Fix — bump the heap limit:

export NODE_OPTIONS="--max-old-space-size=8192"

Also run it fully detached (nohup/disown) so it survives independently rather than dying when a shell/tool session ends:

nohup npx @deepseek-ai/dsh web --no-open > ~/dsh.log 2>&1 &
disown

Then open http://127.0.0.1:3080 in your browser — Qwen/Qwen3.8-27B should be selectable in the model picker, and messages route through the tunnel to your GPU box.

Postscript — a real bug found the same day

Once both harnesses were working, a good stress test for any agentic setup is to give it something genuinely ambitious rather than a toy prompt. Asking DeepSeek Harness to "write a 200 page book on Quantum Machine Learning with all the necessary topics, easy to read, simple english" was a good one: it correctly planned the task (style guide → outline → parallel sub-agents writing chapters concurrently → final assembly), and most of it worked — about 21,000 words of real content landed on disk across seven chapter files, plus a style guide, outline, front matter, and appendices.

But it stalled partway through, missing one chapter entirely. The cause: the harness tried to check on a sub-agent's completion using a tool meant for polling background shell jobs instead, got back Error: unknown job <uuid>, and the whole task just stopped there with no retry and no visible error to the user — the model's own reasoning trace even shows it catching its own mistake ("job_output is for background bash jobs, not for sub-agents...") right before going silent.

Since @deepseek-ai/dsh was already on its latest version with no matching GitHub issue, this got filed as a bug report with the exact failing tool call and the concrete evidence of what did/didn't get written: deepseek-harness discussion #4696.

The takeaway for testing your own setup: an easy prompt ("say hello") only confirms the wiring works. An ambitious, multi-step prompt is what actually surfaces whether a brand-new agent harness's orchestration is solid — and on a 0.1.1-rc.2 developer preview, don't be surprised if it isn't yet.

Summary of the recurring lessons

• Container GPU boxes often have the driver but not the CUDA toolkit — nvcc, headers, and runtime libs may need to be pulled in via pip packages, and their versions must match each other exactly.
• setproctitle-renamed processes evade naive pkill -f patterns — always verify with ps aux/nvidia-smi before assuming a cleanup command worked.
• Multi-GPU tensor parallelism can deadlock on brand-new model architectures before general adoption catches up in the serving framework — dropping to single-GPU is a reasonable, low-cost diagnostic and fallback.
• A reverse proxy under a subpath breaks anything that doesn't explicitly support it — API servers (FastAPI/vLLM) often have a --root-path escape hatch; modern SPA web UIs (Open WebUI, DeepSeek Harness) frequently don't, and the practical fix is running the UI locally instead of fighting the proxy.
• Agent harnesses assume OpenAI's exact request conventions, which self-hosted OpenAI-compatible servers don't always match perfectly (message ordering, developer role, max_completion_tokens vs max_tokens) — expect to patch a chat template or set compatibility flags.