This person on huggingface makes excellent space saving models using APEX quantizations. They produced this one to fit on gpus with 16gb of vram or less.
- 1 Post
- 5 Comments
Joined 3 years ago
Cake day: June 16th, 2023
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
How did you configure DeepSeek harness? I’m guessing a flake would be useful.
I read your post and ended up finding this repo: https://github.com/numtide/llm-agents.nix
I’m considering deploying it for testing in a nix container. 🤔
This one is a Mixture of Experts model (40B total weight, 2.5B active during token generation). Inference engines like llama.cpp have flags you can pass to offload the lesser used experts to system RAM.
It’s slower than having the entire model resident in VRAM, but generally pretty usable.
I’ve been using NixOS/nix containers to host some llama.cpp servers and Hermes Agent. Though I wrote a flake to get Project Maya working so I can serve GLM-5.3-Flash and in legit blown away by how well it works. It’s a fork of strata/llama.cpp that focuses on where the relevant experts are loaded and has a neat dashboard to track their use. https://github.com/mw00/project-maya
My server is assembled into an openair mining rig with a Tesla P40, RTX 3080, and a few Tesla V100s. All of which I bought used off eBay.
I purchased my first domain to self host a matrix server that I can access via my phone and I also have tailscale set up so I can ssh into the server via my laptop while traveling
Oh that sounds rad lol. I was considering the possibility of building a setup within nixos using vfio to declare and configure VMs. I like your integration with systemd, that’s clever