freriadoc/Avalon

Strata fork: Qwen3.8-Flash-Next 125B MoE in NVFP4 on one RTX 20-50 card (12 GB+) and 64 GB of RAM or more. Setup installs our GPTQ quants (huihui abliterated, OrcaRouter) from Hugging Face. CPU experts on AVX-512 or AVX2 (+AVX-VNNI), all experts in RAM from 92 GB, a low-RAM mode below, vision, W4A8 prompts. Windows / CUDA 13.

From-scratch C++/CUDA inference engine for Qwen3-8B

A custom inference engine tuned for Qwen4 architecture. Currently runs Qwen3.8-Flash-Next at 15+ tok/s on 16GB of VRAM and 30 GB of RAM

C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer

LLM inference engine written from scratch in HIP for AMD Strix Halo (gfx1151) and Qwen3.8-Flash-Next - ~50 tokens/s out to 492k context

Qwen3.8-Flash-Next IQ3_S on 12 GB VRAM + 32 GB RAM + NVMe: ~45-50 tok/s decode, 131k context, Strata unify fork

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

Qwen3.8-Flash-Next on Apple Silicon: an MLX engine that reads Strata's GGUF files and speaks its line protocol (unofficial)

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.