I tested with some tasks i usually assign to the qwen 3.8 27b, for example setup a grafana dashboard for a service i just setup. It did it beautifully.
I'll do some more testing to see how it performs. This thing is doing 150 tok / s on my hardware which is 2x A4500 with NVLink.
I had to do some custom kernel to make it work namely INT8 kv cache. The setup i used is documented here:
The best I could do with established techniques and staying at nvfp4 levels of quant, is around 40-45 toks/s on 5070 Ti with 800 toks/s prefill around 128k kv cache... with the kernels matching output byte for byte and basic manual verification of the patched code.
3060 sits around 20-25 toks/s with prefill around 350-400 toks/s i think maybe it can be improved but my 3060 system was ddr4 so not the greatest.
Either way at below 6gb you cant run beyond q4 quants at above 10-20 tok/s with pretty small context size, afaik that gets awfully close to the roofline depending on expert cache hit rate... unless you quant the logits into the ground.
> The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.
What's the point in quantizing a QAT model? Just for social media karma?
/igotthatreference.jpg
----
Gave it a quick browse to verify what my local clanker told me, it seemed to check out
I believe it is Dynamically allocating weights based on fisher information of conversation traces & For some K-quant look up table stuff it can store in an importance matrix to get better compression.
The results are complete nonsense and really don't back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth's UD-IQ1_S: 0.0895 vs 0.0808 on English/chat/code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt'd model on sample inputs (AFAIT).
According to anthropomorphic-clud their results for GSM8K & HumanEval are within the noise regime and vastly over stated.
This repo's README is a mess of different histories as the author bounces from one popular model to the next. It refers to vague things like "the 1.2GB file" that probably made sense in an early vibecoding pass, but are now lost after the repo has been pushed around to different popular models.
The truth is that random GitHub repos with messy READMEs almost never hold a real secret that the mainstream quantizations have missed. I would recommend sticking to mainstream quantizations like the official GGML release, Unsloth's quants, or the more well known community quantizers like bartowski or AesSedai. They typically follow an optimized size-quality tradeoff so you can find the one that fits your needs.
When you see a repo like this where everything is so obviously vibecoded that the README is a rambling mix of projects and claims, you should probably go looking for a more authoritative source. The local LLM communities get spammed with 10 of these types of projects per day but I have yet to see a real breakthrough come from any of the vibecoded ones. The real work is coming from people who study this and have compute to test at scale or work on implementing advanced techniques which require inference engine support, like trellis quants.