Running a large language model locally means downloading an open model and executing it on your own hardware, so prompts and outputs stay on your machine. The simplest way to start is Ollama, an MIT-licensed open-source AI tool that runs open models such as Gemma, DeepSeek, Qwen and gpt-oss on your computer, with a CLI, a REST API and Python and JavaScript libraries. This guide walks through the steps and points to other tools in the catalog that fit into a local setup.

What you need before you start

  • A computer or server with enough memory for the model you choose. Larger models need more RAM or GPU memory, so begin with a smaller model.
  • Disk space for the model files, which can be several gigabytes each.
  • A terminal, since most of the workflow uses the command line.

Step 1: Install Ollama

Download Ollama from its project page and install it for your operating system. Ollama is open source under the MIT license, so you can review the code before running it. After installation, confirm that the command-line tool is available by opening a terminal and checking the installed version.

Step 2: Pull a model

Choose an open model and download it. Ollama stores models locally and loads them on demand. Start with a small model to confirm everything works, then try larger ones once you know your hardware can handle them.

Step 3: Run a prompt

Start an interactive session with the model from the terminal and type a question. The first response may be slower while the model loads into memory. When the session works, you have a fully local chat with an LLM.

Step 4: Call the model from code

Ollama exposes a REST API on your machine, and it provides Python and JavaScript libraries. Use these to send prompts from your own scripts or applications. Keep the model name in a configuration value so you can switch models without changing code.

Step 5: Add tools and context

A local model becomes more useful when it can read current documentation and act on tasks. Two categories in the catalog help here:

Documentation for coding agents

context7 is an MIT-licensed MCP server that supplies up-to-date documentation to AI agents. Browse more options on the MCP servers page.

Agent frameworks

hermes-agent, an MIT-licensed agent that the publisher describes as one that grows with you, can be combined with a local model. OpenHands, an MIT-licensed project for AI-driven development, is another option for coding workflows. For a broader set of reusable components, see Agent skills and the AI Agents category.

Reducing token use

rtk is an Apache-2.0 licensed CLI proxy, written as a single Rust binary, that the publisher says reduces LLM token consumption by 60-90% on common development commands. This is useful when your agent runs many shell commands and you want shorter context.

Step 6: Automate browser tasks

If your local agent needs to interact with websites, browser-use is an MIT-licensed open-source Python library for AI browser agents. It also offers a hosted Browser Use Cloud, but the library itself runs on your own system.

Step 7: Learn the underlying concepts

If you want to understand how these systems are built, ai-engineering-from-scratch, an MIT-licensed project, is designed to teach AI engineering by building things step by step. For a harness that organizes skills, memory and security for coding agents, see ECC, which is MIT licensed and supports Claude Code, Codex, Opencode and Cursor.

Choosing the right tool

  • To run models: start with Ollama.
  • To give agents current documentation: use context7.
  • To cut token use in shell-heavy workflows: use rtk.
  • To automate the browser: use browser-use.
  • To run coding agents: consider OpenHands or ECC.

Browse the full set of developer tools and libraries and SDKs in the catalog to find more components.

Recommendation

For most readers, begin with Ollama and a small open model. Confirm that the model answers prompts and that your code can call the local API. Once that works, add one tool at a time, starting with documentation through context7 or token reduction through rtk, and measure whether each addition improves your results before adding the next.