• 0 Posts
  • 10 Comments
Joined 3 years ago
Cake day: June 20th, 2023
  • Dran@lemmy.worldtoLocalLLaMA@sh.itjust.works•Good places to keep up to date on LLMs?English
    4·
    1 month ago

    New models

    Aleph-Alpha/Kolibri-1 – 78.1B total / 3.46B active, up to 1M context, Apache 2.0 A new open-weight English-German Mixture-of-Experts reasoning model from the German lab Aleph Alpha, open-sourced Oct 3, 2026 (Germany’s Reunification Day). 78.1B total params, ~3.46B active per token, native context up to 1,048,576 tokens, weights on Hugging Face under Apache 2.0. Trained on ~20T tokens (mostly English ~62.5%, German ~23.9%, code ~13.6%). Notable as one of the largest sovereign open-weight releases from Europe. Why it matters to you: the 3.46B-active MoE makes it a strong candidate to replace GPT-OSS at roughly two-thirds the size for German/English workloads, and the 1M context is a big draw. Commenters note the model is “basiclly worse than Qwen3.6-35B at twice the size” (u/Training_Visual6159) and that it is “comparable to Qwen 3.5 35B” (u/SnooPaintings8639), so treat it as a strong first attempt rather than a clear winner. At ~78B it likely needs a 96 GB VRAM build to run comfortably. As of this digest there were no community quant releases (no GGUF/INT4/NVFP4 quants posted), and a commenter asked whether llama.cpp supports the architecture yet – so do not plan a deployment until a serving path and quants exist. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwl7y6/alephalphakolibri1_hugging_face_78b_parameters/ Tech report: https://aleph-alpha.com/downloads/tech-report.pdf Model: https://huggingface.co/aleph-alpha/Kolibri-1

    Qwen3.8 Flash-Next (~176B total, 6B active) – the recurring constrained-hardware test subject Not new today, but it is the model two of the five featured posts are benchmarking on 16 GB-class hardware. The MoE with 512 experts and a large n-gram/PLE table makes it unusually suited to tiered SSD offloading. See the “Most interesting posts” section for the two 16 GB data points.

    bilibili “Index-Translate” – multilingual translation family on Qwen3.5 (lower signal) A small (15?) post announcing a multilingual translation model family based on Qwen3.5. Translation-focused, so limited relevance to your coding/agentic backend; flagging it only because it is a new Qwen3.5-based release. No quants or serving configs posted. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wxa1wr/bilibili_released_indextranslatea_a_multilingual/

    Most interesting posts

    1. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwocik/two_300b_moe_models_each_on_one_128_gb_mini_pc/ u/Yaniss916 reports two ~300B MoE models, each on ONE 128 GB AMD Strix Halo (Ryzen AI Max+ 395, gfx1151) running a new ExLlamaV3/ROCm engine called Kyojin. Concrete numbers from the post: GLM-5.3-Flash (99.7 GB) hits ~580 tok/s prefill (at 3.5K), 546 at 64K, 26-30 tok/s decode (MTP); MiMo-V2.6-Flash-MOPD (105 GB) hits ~650 tok/s prefill at 4K and up to 44 tok/s decode (speculative, code). Quality: KLD vs official FP8 of 0.151 / 0.0713 and top-1 agreement 89.3% / 92.0% respectively. The GLM pack mixes turboderp’s public 2.05 and 3.05 bpw EXL3 tensors with a custom layer mix. Includes a quickstart command (clone, ./build.sh, hf download, python tools/glm/serve.py --model ./glm-pack -c 131072 --num-draft 2, OpenAI-style API). Reproducible on Strix Halo hardware only – the author explicitly lists as “not measured yet” any GPU other than gfx1151, and the conversion pipeline stays private. Highly relevant as a technique reference (EXL3 layer-mix quant + ROCm) even though the target ha rdware is AMD, not your NVIDIA stack. Resources: Engine https://github.com/Yamz-Labs/kyojin . Weights https://huggingface.co/yamz-labs . Base fork https://github.com/vcruz305/exllamav3-amd

    2. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwwmy1/running_qwen38_flash_next_176b_on_a_16gb_rtx_3080/ u/fuzhongkai runs Qwen3.8 Flash-Next 176B on an RTX 3080 Laptop (16 GB VRAM) + 32 GB RAM + SSD using his open-source engine TensorSharp, which does MoE-aware scheduling across cache, VRAM, system RAM, and SSD. Reported results: ~11.09 tok/s decode vs Strata’s 10.24, but end-to-end 16.54 s vs Strata’s 62.15 s on his test (peak GPU 14.8 GB, OS working set 19.74 GiB). Reproducibility caveat (flagged in comments): the post does not name the quantization, and top comments ask for exactly that (u/DigitalguyCH: “very vague without specifying the quantization”; u/wizard_of_menlo_park: “Quantization?”). u/leonbollerup also says he gets ~80 tok/s decode and ~2200 tok/s prefill on the same card with Strata, which contradicts the post’s headline – so treat the end-to-end speedup as single-source and unverified. This is directly relevant to whether a 24 GB 3090 can serve the 176B Flash-Next via SSD tiering, but verify the quant and benchmark method before trusting it. Resources: https://github.com/zhongkaifu/TensorSharp . Model docs https://github.com/zhongkaifu/TensorSharp/blob/main/docs/models/qwen38-flash-next.md

    3. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwv0fj/i_built_ninfer_4080_for_16gb_class_gpus/ u/roofkid built NInfer 4080 to run ISTA-DASLab Qwen3.8-27B-GSQ at 100k context on an RTX 4080 16 GB, claiming up to 2720 tok/s prefill and 262 tok/s generation. Part of the NInfer 5090/4090/3090 family of from-scratch engines. Reproducibility caveat: the post is a project announcement; the headline numbers (2720 pp / 262 gen) are the author’s max-observed, and a top comment (u/Pyrolistical) notes these custom engines often compromise with a quantized KV cache and that his own roofline analysis of llama.cpp on an R9700 found <5% decode improvement left on the table – so the gap between custom and general engines is hardware/kernel-specific, not universal. Comments are mostly “how do I get this on a 3080/5080/4070 Ti” requests, so portability to other cards is unconfirmed. Relevant to you primarily as another data point in the overfit-engine trend and for the ISTA-DASLab 3-bit GSQ quant on Qwen3.8-27B. Resources: https://github.com/roofkid/ninfer-4080 . Quant https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ . ByteShape Qwen3.8-27B reference https://byteshape.com/blogs/Qwen3.8-27B/

    4. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwu6zj/the_rise_of_overfit_inference_engines/ u/carteakey’s discussion post on the overfit-engine trend (see topics above). 315?, 199 comments. No single reproducible config, but the most substantive community engineering observations of the day: SGLang’s current gaps (u/buttplugs4life4me), the Xeon-Max NUMA speed gap (u/TokenRingAI), and a practitioner who “vibe-coded” his own engine because upstream PRs were ignored (u/wishstudio). Worth reading for the state of general vs. narrow runtimes rather than as a runnable config.

    5. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwl7y6/alephalphakolibri1_hugging_face_78b_parameters/ The Kolibri-1 announcement (see New models above). 510?, 153 comments. Not a reproducible-serving post (no quants yet), but the most-discussed new model of the day and directly relevant to a European/sovereign model pipeline and as a GPT-OSS-sized MoE candidate.

  • Dran@lemmy.worldtoLocalLLaMA@sh.itjust.works•Good places to keep up to date on LLMs?English
    6·
    1 month ago

    if you actually intend to replicate my setup: [qwen3.8:27:nvfp4 --> vllm (concurrency efficiency)] --> [open-webui (api proxy)] --> [hermes-agent --> hermes-webui]

    […] denotes container boundaries.

    The redlib instance also runs in a container on the same host. A skill teaches hermes-agent how to use redlib to open a subreddit and a separate instruction markdown file teaches it how to build the digest I want. a “cron” (hermes cron, not system cron) runs the job once a day, and the final step of the digest instructions is to email me the digest.

    This was today’s, for example:

    LocalLLaMA Daily Digest – 2026-10-04

    Curated for a [REDACTED] LLM-backend sysadmin running vLLM + Open-WebUI on RTX 3090s and RTX PRO 6000 Blackwell. Focus: quantization (INT4/NVFP4/GGUF/EXL3/EXL2), inference serving, VRAM-limited local deployment, and reproducible configs.

    Most frequently discussed topics

    1. The rise of narrow “overfit” inference engines This was the loudest infrastructure thread today. A wave of deliberately non-general runtimes – Strata, Ninfer, DwarfStar, Splash, llamAmpere, gufo, Kyojin, TensorSharp – give up llama.cpp/vLLM’s generality to squeeze maximum throughput out of a handful of models on a single hardware family (often one GPU or the Strix Halo APU). For a vLLM/Open-WebUI operator, the question is whether it is worth maintaining a second, model-specific serving path next to a general one, or whether the headline tok/s justifies the extra deployment and maintenance burden. Sentiment was broadly positive but with real pushback: the top comment (u/TokenRingAI) argues AI hardware is too expensive to run at slow speeds – on his 2x Xeon Max, llama.cpp gets ~7 tok/s and SGLang ~12 on Qwen Flash Next, while a custom NUMA engine he built hits ~67 tg/s and ~900 pp/s – so the gap is large enough to matter. A skeptic (u/darktotheknight) frames it as an unsustainable fragmentation (the “Apache HTTP server” analogy ) that will consolidate. u/buttplugs4life4me makes the strongest case for the generalists: SGLang is already falling apart under model/hardware explosion (no GGUF support for the most common Qwen arch, no mixed non-block quant, MXFP8/MXFP4 unoptimized, no mixed KV cache). https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwu6zj/the_rise_of_overfit_inference_engines/

    2. EXL3 / ExLlamaV3 on AMD Strix Halo (ROCm) for ~300B MoE models Several posts converge on ExLlamaV3 (EXL3) as the memory-efficient path for running 300B-class MoE models on 128 GB of unified-memory hardware. A new ROCm engine (Kyojin) targets Strix Halo specifically and fits two 300B MoE models on one 128 GB machine with low KLD. For your NVIDIA-only stack this is less directly actionable, but the EXL3 layer-mix quantization technique (mixing ~2.05 and 3.05 bpw tensors to balance context vs. KLD) is the technique to watch, and EXL3 weights are portable where the backend is supported. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwocik/two_300b_moe_models_each_on_one_128_gb_mini_pc/

    3. Qwen3.8 Flash-Next and Qwen3.8-27B on constrained hardware (16 GB class) Two separate posts pushed Qwen3.8 Flash-Next 176B and Qwen3.8-27B onto 16 GB-class GPUs + SSD, using tiered scheduling (VRAM/RAM/SSD) and custom quants. Directly relevant if you are probing whether a 3090 (24 GB) or a laptop can serve these models without a multi-GPU rig. https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwwmy1/running_qwen38_flash_next_176b_on_a_16gb_rtx_3080/ https://llm.r.homelab.internal/r/LocalLLaMA/comments/1wwv0fj/i_built_ninfer_4080_for_16gb_class_gpus/

    (splitting because of post character limit, see below for the rest)