• 0 Posts
  • 46 Comments
Joined 2 years ago
cake
Cake day: February 10th, 2025

help-circle
  • I clearly mean that the more people (both sellers and buyers) do this crap, the worse it becomes.

    Lazy people who want to avoid studying in college have been being taken advantage of since the beginning of academia. Adding AI doesn’t change anything, humanity’s knowledge has survived lazy students and fake notes it can survive fabricated notes.












  • This is true, self hosting doesn’t save you money. It is, in fact, more expensive and costs your time.

    This is because modern technology isn’t sold to you as a product, it’s sold to you as a loss leader so that you use their products and let them spy on you so that they can sell that data to advertisers or anybody who can complete an ACH transfer.

    Your options are

    1. Use commercial services and sell your privacy in exchange for cheap consumer electronics which will be obsolete next year either by design or through engineered hardware failure.

    2. Setup the services on your own devices and pay the cost in hardware and free time.

    In one scenario you spend a bit more money, in the other scenario you don’t have privacy. Everyone gets to choose which is more important.



  • It depends on how complex you want to make it.

    The simplest way would be to use a GUI tool like LM Studio, it has the benefit of selecting the model quants that are likely to work on your hardware and having sane defaults for the various parameters that you can work from.

    Typically you’ll want to increase the context window, the default is good for short one-off questions but if you’re going to do RAG or coding tasks you will need a much larger context window. You will typically find this through trial and error, increasing until you hit memory limits or notice a large slowdown on tokens/second.

    You can also have the model respond to an OpenAI-like API interface on a designated port/interface if you want to run non-chat tools through the same model.

    Then, depending on the task that you’re trying to solve, you may want to find an MCP server built for that task which can provide the ability for the AI to interact with tools in a way that’s more controlled (giving an AI raw terminal access on a user’s machine is typically a bad idea).

    If you need something more flexible that you can incorporate into scripts and automation, llama.cpp has been popular for a while. Like Linux distros, everyone is going to have an opinion on the best and whatever you decide on will be the wrong choice. There’s also Ollama and vLLM if you want to run a datacenter.

    Model-wise it completely depends on the task. Do you need a multi-modal model that supports vision, is it going to be coding, is it trained to use tools/MCP servers, does it use reasoning, etc. The Qwen models are open weight and available in many different quants. The version 3.5 models run well on consumer hardware and can use vision, reasoning and tools.

    I’m running it through Llama.cpp but it will likely be available via LM Studio as well.