Local LLMs and Agentic Engineering

As my homelab has grown, so has the need for more specialized compute. Currently, my desktop workstation has become the primary engine for running local Large Language Models and image generation models. I've been refining a workflow that moves away from cloud-dependent APIs toward a self-hosted, agentic setup. Especially for internal-only documentation.

The Backbone: LM Studio

The core of my inference setup is LM Studio. I run it directly on my desktop workstation.

For most tasks, I run the server on my loopback address (localhost). This is the most secure way to interact with models during active development. However, as I've started running more services in Docker containers and deploying tools on separate machines in my lab, I've found it necessary to expose the LM Studio server to my local network.

By configuring the server to listen on a network interface rather than just 127.0.0.1, I can point various Dockerized applications to my workstation's IP, allowing them to leverage my GPU.

The Interface: Open WebUI and Stability Matrix

While LM Studio is great for managing models, it isn't my primary way of interacting with them:

• Open WebUI: This is my unified interface for all generative tasks. By connecting it directly to the LM Studio server mode, I get a much richer, ChatGPT-like experience that supports features like web-search and chat history management.
• Stability Matrix Integration: I also use Open WebUI as the frontend for image generation. By leveraging Stability Matrix on my desktop, I can trigger image generation tasks through the same interface used for LLM chat, making it easy to switch between text and image creation within a single environment.

Agentic Engineering with Zed and Contagent

The real power of this setup comes when I integrate these models into my actual development environment.

Documentation Updates in Zed

I've integrated the LM Studio API into Zed. When I'm working on codebase changes or documentation updates, I can use the local models to summarize changes, explain complex logic, or even draft new markdown files based on existing context. It feels seamless as there is no latency from external API calls and none of my code ever leaves my local network. With that being said, my hardware does not run AI inference as quick as frontier models.

Codex (via Contagent)

For more advanced, "agentic" tasks, where the AI needs to navigate the filesystem or execute complex engineering workflows, I use Codex. I run this within a Docker container with full access to any mounted volumes. The setup for this agentic environment can be found in my Contagent repository.

This workflow is closer to agentic engineering - the model can participate in the software development lifecycle with much higher autonomy.

The Philosophy: Helping the Agent

My experience experimenting with tools like Cursor led me to a significant realization: capturing the full power of AI is not about the agent helping you; it's about you helping the agent.

In this workflow, my role has shifted toward providing just enough context for the agent to work as intended while remaining aligned with the end-goal. I've found that allowing the agent more autonomy, such as the ability to read files or search a directory, is far more effective than trying to spell out every single step. When you provide the right tools, even distilled models running on consumer-grade hardware are capable of quite a bit.