Prefill and decode do not have to run the same weights. This repo holds prefillers: NVFP4 transformer blocks that are streamed off the SSD one layer at a time, paired to complement low-bit GGUF for decode.

The point is that the two phases want different things. Decode is memory-bound, has to be on device for fast generation and is accelerated by being small. Prefill is compute-bound, so there’s little point in compressing it below NVFP4 (compute-native on Blackwell) as long as it doesn’t occupy much memory, which we achieve via offloading.

Accuracy and time-to-first-token against device weight size

Auxiliary prefillers recover ~50% of the accuracy that aggressive quantization gives up, at close to no extra memory cost. Moreover, past a few thousand tokens, the offloading streaming disappears under compute, so it is not paid for in latency either

  • pepperfree@sh.itjust.worksOP
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 month ago

    Ternary quant from prism-ml is not that good for complex task and long context. Probably fixable by using higher bit on certain tensor. As far I know for real world usage, at least non uniform 3 bit base is needed.