# sch-qwen: Local LLM Inference Server (Qwen3.6-35B-A3B)

Modified `llama.cpp` implementation optimized for Qwen models with dynamic quantization.

## Overview

This project provides a high-performance inference server for the **Qwen3.6-35B-A3B-UD-Q2_K_XL** model (Dynamic 2-bit MoE). It is specifically tuned for CPU inference on Windows with AVX2/VNNI support.

## Hardware Requirements

- **CPU:** Intel Core i5-12400F or better (supports AVX2, FMA, VNNI)
- **RAM:** 32GB Recommended (previously 16GB, but 32GB allows for larger context and speculative decoding)
- **Storage:** SSD (Crucial for performance, mmap latency on HDD is prohibitive)

## Software Stack

- **Backend:** Modified `llama.cpp`
- **Quantization:** Ultra-Dynamic (UD) Q2_K_XL
- **KV-Cache:** Q8_0 (Quantized for memory efficiency)

## Quick Start

1. **Clone the repository:**
   ```bash
   git clone --recursive https://github.com/your-username/sch-qwen.git
   cd sch-qwen
   ```

2. **Download the model:**
   Place the GGUF model file in the `models/` directory.
   Expected file: `Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf`

3. **Build llama.cpp:**
   Follow the build instructions in the `llama.cpp/` directory. For Windows/MSVC:
   ```bash
   cd llama.cpp
   mkdir build
   cd build
   cmake .. -A x64
   cmake --build . --config Release
   ```

4. **Run the server:**
   Execute `run_server.bat` to start the OpenAI-compatible API server on port 8080.

## Optimization Parameters

The current configuration is optimized for stability and speed:
- **Context Window:** 16,384 tokens (default, expandable with 32GB RAM)
- **Threads:** 5 for generation, 6 for batch processing
- **Flash Attention:** Enabled
- **KV Cache:** 8-bit quantized

## Scripts

- `run_server.bat`: Starts the API server.
- `benchmark_8k.bat`: Performance validation script.

## Performance (Empirical)

- **Prompt Processing:** ~30 t/s
- **Token Generation:** ~9.8 t/s

## License

MIT (or check `LICENSE` file)
