All projects
Task-Aware Quantization (TAQ)
Aviation deep research
Quantized and fine-tuned 27B–35B dense models for deep research over 1M+ pages of public aviation accident and incident reports, on a single 16 GB VRAM deployment.
What it is
Task-Aware Quantization (TAQ) for aviation deep research: quantized and fine-tuned dense models for deep research over public aviation accident and incident reports.
Problem addressed
Dense models in the 27B–35B parameter range normally need far more GPU memory than a single 16 GB card. The engagement aimed to run them for deep research on a single 16 GB VRAM deployment while keeping accuracy close to the full-precision model.
What has been built or demonstrated
- Quantized and fine-tuned dense models in the 27B–35B parameter range using Q2/Q3.
- Single 16 GB VRAM deployment.
- Deep research over 1M+ pages of public aviation accident and incident reports.
- Reported accuracy within 0.5% of BF16 on ROUGE, SummaC and BERTScore.
- Reported 5× TPS at one-sixth of the model size.
Limitations
- Results as measured on the stated evaluation metrics for this engagement; they may not carry over to other tasks or data.
- Delivered client R&D, not a product for sale.
Planned next steps
- Apply what we learned to our Custom AI Models layer, including on-premise SLMs.
Interested in a pilot or technical collaboration?
We are open to pilot partners. Tell us about the engineering problem you are working on.