Every few months I would try to run a local language model on my own hardware. The motivation was always the same: I do not want my prompts going to a vendor I cannot control, I do not want to pay per token for things I am still iterating on, and I am genuinely curious about what the open models can do on the machines I already own. The intention was always real. The follow-through, less so.
Each attempt followed a similar shape. Pick a runtime (llama.cpp, Ollama, text-generation-webui, vLLM, take your pick). Pick a model file, then pick a quantization (Q4_K_M, Q5_K_S, IQ3_XXS, and a dozen other names that imply some difference you should already know). Find a GGUF that downloads anonymously. Skim three blog posts to figure out which flags matter for your hardware. Realize the binary you built a few weeks ago is already two versions out of date. Start over. After running through this loop a few times and writing essentially the same glue script each time, I stopped and wrote it properly.
I turned those repeated scripts into hydra-llm, a small CLI plus an optional KDE Plasma 6 widget. Underneath it is just llama.cpp running in Docker, with a curated catalog of community-quantized GGUF files (Bartowski, lmstudio-community, mradermacher) that download without a Hugging Face account. The CLI does the four things I actually do at the prompt: list-online, download, chat, and api. The Plasma widget adds glanceable status for the people who, like me, have the panel always visible.
Existing options sit at two extremes. Some options hide more of the runtime and catalog choices than I want. That is my preference, not evidence that unnamed competing tools send telemetry or act against their users. Others are too raw: you write the Docker compose, you mount the volume, you pick the port, you tune the flags, you pin the image, you do it again next month. I wanted something in the middle. Real Docker containers I can see in docker ps. One config file I can read end to end. No telemetry. No vendor account.
Docker carries the cross-distro Linux packaging burden for me, which is a tax I am not willing to pay for a side project. llama.cpp moves fast enough that any binary I built last month is already stale, and rebuilding for half a dozen distros is not how I want to spend my evenings. The cost is that the user has to have Docker. For the audience I care about, that is already true.
The first real design decision was what to do about hardware variance. A small quantized model such as Phi-3-mini can fit a modest mini-PC, though speed depends on its CPU and memory. A Llama-3.3-70B at Q4 needs serious memory or a workstation GPU. Ship one flat catalog and the new user picks the biggest model they have heard of, then watches their machine swap itself into a coma.
So hydra-llm doctor looks at CPU, RAM, GPU, and VRAM, classifies the machine into a tier, and list-online filters the catalog accordingly. The tiers are tiny, laptop, halo, workstation, and server. You can override, but the default is the one that does not destroy your day.
The halo tier covers machines with substantial shared memory. The 5 August 2026 README snapshot includes Strix Point, Strix Halo, and some Apple Silicon systems in this category, with Gemma-3-27B Q4 and Qwen-2.5-32B Q4 among its model suggestions. This source was added on 15 October 2026. The tier is a starting point for model selection; actual fit and speed depend on the machine and runtime settings. My ThinkPad article describes a P16s Gen 4 AMD with Ryzen AI 9 HX PRO 370 and Radeon 890M, which is Strix Point. Hydra calling a machine halo does not make it Strix Halo hardware.
The temptation with configuration is always to invent a beautiful nested format. I resisted, mostly. The result is three flat layers, narrowest wins. Personas live at ~/.config/hydra-llm/personas/<name>.md, are markdown with optional YAML front matter, and apply across models. Per-alias system prompts live at ~/.config/hydra-llm/prompts/<alias>.txt and apply to that catalog id unless a persona overrides them. Per-alias sampling parameters live at ~/.config/hydra-llm/params/<alias>.json and cover temperature, top_p, top_k, repeat_penalty, max_tokens, and seed.
Inside the chat REPL, /params shows what is active, /set temperature 0.4 changes one for the current session, /reset clears history but keeps the system prompt, and /thoughts on toggles reasoning output for models that emit it. The whole surface is meant to be discoverable from /help and forgettable when you are not configuring.
I run KDE Plasma. The CLI is fine but I wanted to see at a glance whether a model is up, what it is doing, and how loaded the machine is. The widget has the obvious controls (start, stop, console, logs, configure) and one detail I am embarrassed by how much I like: a HAL-eye indicator that breathes faster as utilization climbs, shows a yellow scanning ring while a container is loading, and turns solid red once at least one model is healthy. The widget ships as a separate package (hydra-llm-plasma); on non-Plasma desktops the CLI works fully and a native UI for GNOME and XFCE is on the roadmap.
The catalog only references community-quantized GGUFs that download without a Hugging Face account. This was deliberate, and it shaped which models made the cut. If you want gated weights (the official meta-llama/* or google/gemma-* repos), set HF_TOKEN in your environment and the CLI passes it through. The CLI never prompts for one, never stores one, and never sends it anywhere except Hugging Face when you have asked it to. There is no telemetry, no analytics, and no auto-update calls. Sessions are saved as JSON in ~/.local/state/hydra-llm/sessions/, and you can delete them whenever.
The friction I had been hitting was never technical. Local LLM stacks have been runnable for two years. The friction lived in the rituals: which file, which flag, which runtime, which quantization. Once those choices are pre-made for the 80% case, the experience is genuinely good.
I also kept the scope small on purpose. hydra-llm does not fine-tune. The initial version described in this January post did not do RAG or embeddings. Those features arrived in v0.2.0 on 7 May 2026, as covered in the RAG article. It picks a model, downloads it, and lets you talk to it. Every time I was tempted to add a feature I asked whether it sat on the path of "I want to chat with a local model right now," and most of the time the honest answer was no. Side projects die from feature creep more often than from bad code.
I now start a local model with one command and spend my time on the model instead of its runtime. That was the job I wanted hydra-llm to do.