As an alternative to mounting full /dev/dri, there are symlinks under /dev/dri/by-path/ which you can use. IIUC they are linked by the used PCI slot, hence should be as stable as the physical order of the cards.
I use this pattern in a compose file:
...
devices:
- source: /dev/dri/by-path/pci-XXX-render
target: /dev/dri/renderD128
permissions: rwm
In case you are unaware, friends don’t let friends use ollama. The llama.cpp CLI has become pretty much equally easy to use, plus since you use Docker there’s no need to compile anything.
Edit: typos and missing words
Haha, that sounds like procrastination to me.
Not necessarly, I’d say. We’ve seen many releases with multiple sizes and the 27B class has been around since at least the Gemma3 times.
I hope an instruction tuned will be released as well.
I find the overall parameter size interesting. On 16GB VRAM the whole thing fits at an excellent quantisation without having to offload anything to system RAM and taking hits on PP & TG.
According to the officially reported benchmarks, Ling wins on GPQA Diamond (73.4 vs. 64.6) and BFCLv4 (62.72 vs. 62.3). No other overlaps.
That being said, I’m neither trusting benchmarks alone nor have tested any of the two extensively enough to have a solid impression.
I used to run SFTPGO in a container as a NAS type service. Their default image is based on Debian, but they offer Alpine versions. Password authentication is enabled by default.