• 0 Posts
  • 12 Comments
Joined 3 years ago
cake
Cake day: June 15th, 2023

help-circle




  • That’s a good question and it’s hard to say for sure. There’ve been some advancements this year in offloading, using both VRAM and system RAM together. With MoE models and predictive loading you can get pretty good results that way…sometimes. I’ve never tried it myself, though I really should because my 24GB GPU is a serious bottleneck.

    You could also think about getting a 512GB Mac Studio (coming soon). Not as fast as a GPU, but maybe faster than a GPU/DDR5 mix? And definitely cheaper per GB.

    I realize now that the truly biggest models need way more than $40k of hardware. The biggest Qwen3.8 is 2.4 TRILLION parameters. Yikes.




  • If you’re counting models that can be run locally but are too big to run on consumer hardware, then it’s pretty small. Like, they’re only lagging by a few months. But you you’d need like $40k of hardware to run them effectively.

    If you mean just the models that you can run on consumer hardware, like, say qwen3.8-27b, then it’s more like 1-2 years behind. You can compensate for the lack of raw capability with a good harness, agentic loops, and human effort. As much as AI has a reputation for being low-effort, it actually requires a lot of skill to use very effectively.