tedivm/qwen-27b-docker

Production-ready vLLM deployment wrapper for Qwen3.6-27B (NVFP4) — self-hosted OpenAI-compatible inference

Self-hosted vLLM inference for Qwen3.6-35B-A3B-NVFP4

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

Qwen3.8-27B on 16-32 GB Nvidia GPUs one-click install for Windows / Linux

Deployment kit: Qwen3.8-27B EXL3 3.5bpw target + DFlash2 EXL3 5.0bpw speculative draft — launcher, config, model cards

RouteWeaver: experimental cache-safe speculative inference and CPU/GPU transfer optimization for Qwen 27B on low-VRAM NVIDIA GPUs.

Fastest validated sglang recipe for Qwen3.8-27B (AWQ-INT4 + DSpark) on RTX 3090s — 1×/2×/4× GPU presets, image+video, full benchmarks

All-NVFP4 vLLM + DFlash2 K7 for Blackwell (SM120/SM121): NVFP4 target/draft/KV, 262K context, optional CPU vision sidecar.

Quasar is an inference engine for Qwen3.6-27B-MTP / Qwen3.8-27B on consumer NVIDIA cards as well as Apple Silicon, in the spirit of antirez/ds4