LLM-Agents-Ecosystem-Handbook

LLM Agents Ecosystem Handbook

A practical operating manual for building, evaluating, securing, and shipping modern LLM agent systems.

[Awesome] [License: MIT] [PRs Welcome] [LLM-Friendly] [Providers]

Modern agents are not "a prompt + a tool." They are systems — with identity, memory, skills, tools, MCP integrations, guardrails, observability, evals, and a provider strategy. This handbook teaches the whole stack and ships templates, blueprints, runnable adapters, and curated examples you can adopt today.

What's in this repo

A curated, opinionated, production-oriented handbook in seven parts:

1. Concepts — Agent OS, identity, memory, skills, MCP, safety, observability — every layer of the modern agent stack
2. Provider ecosystem — adapters + docs for 24+ LLM providers (frontier APIs, fast inference, marketplaces, enterprise clouds, specialty, local runtimes), with a router for fallback chains
3. Skills ecosystem — design guide, taxonomy, maturity model, security checklist, and a curated skill catalog
4. Prompt engineering — agent prompt patterns, instruction hierarchy, context engineering, prompt-injection defense
5. Coding-agent workflows — for Claude Code, Cursor, Codex, Aider, Cline, and custom runtimes — repo instructions, prompts, review checklist, safe refactoring
6. Design docs — agent / technical design docs, ADR guide, design reviews, rollout plans, the DESIGN.md machine-readable spec
7. Curated catalog — 100+ existing agent skeletons, framework comparisons, evaluation tools, tutorials — preserved and improved

Who this is for

Modern Agent Stack

📖 Deep dive: agent_os/README.md

Provider ecosystem

The handbook ships an LLMProvider abstraction with 24+ providers across six families. Most providers go through a single OpenAI-compatible code path; specialty / local providers are first-class.

If you want a governed OpenAI-compatible control plane in front of those providers, Tuning Engines is a useful runtime option for policy enforcement, approval gates, MCP and agent tracing, and usage or cost visibility without changing the surrounding agent framework.

Quick start:

from utilities import get_provider
from utilities.provider_router import ProviderRouter

# Use any single provider
out = get_provider("groq").chat(
    [{"role": "user", "content": "Summarize MCP."}],
    model="llama-3.1-8b-instant",
)

# Or route by task class with fallback
router = ProviderRouter()
out = router.chat(messages, task_class="cheap")  # Groq → DeepSeek → Together → OpenRouter

📖 providers/README.md • providers/provider_matrix.md • providers/router_patterns.md • providers/local_models.md

Repository map

.
├── README.md • llms.txt • llms-full.txt
├── agent_os/                ← the Agent OS concept, layers, workspace examples
├── providers/               ← 24+ provider docs + adapters + router patterns
├── templates/               ← AGENTS.md / SOUL.md / MEMORY.md / SKILL.md / DESIGN_DOC / ADR / …
├── skills/                  ← design guide + taxonomy + maturity model + curated catalog + 4 examples
├── memory/                  ← memory taxonomy, distillation, security, examples
├── mcp/                     ← MCP basics, architecture, security, server catalog, examples
├── prompt_engineering/      ← agent prompt patterns, instruction hierarchy, defenses
├── coding_agents/           ← Claude Code, Cursor, Codex, workflows, prompts, review
├── design_docs/             ← agent + technical design docs, ADR guide, design.md spec
├── safety/                  ← guardrails, approvals, prompt injection, secure checklist
├── observability/           ← tracing, spans, cost/latency, dashboards
├── evals/                   ← eval design, regression / tool / memory / MCP / safety / prompt
├── blueprints/              ← production architectures by use case
├── examples/                ← end-to-end runnable agent workspaces
├── checklists/              ← agent design, prod readiness, MCP security, …
├── llm_wiki/                ← LLM-friendly index, glossary, matrices, wiki pattern
├── docs/                    ← framework comparison, best practices, beginners' guide
├── tutorials/               ← RAG, memory, fine-tuning, chat-with-X
├── utilities/               ← LLMProvider + router + provider_config
├── agents/                  ← 100+ curated agent skeletons (preserved)
├── complete_apps/, web_apps/, notebooks/, datasets/, design/, resources/, scripts/, tests/, ecosystem/
└── .github/                 ← issue / PR templates

Skills ecosystem

A curated, in-repo catalog plus a clear taxonomy and maturity model:

• skills/skill_design_guide.md — write triggers the model picks
• skills/skill_vs_tool_vs_mcp.md — when to use which
• skills/skill_taxonomy.md — domains, tags, risk
• skills/skill_maturity_model.md — experimental → production
• skills/skill_packaging.md — ship a portable skill
• skills/skill_validation.md — lint / smoke / eval
• skills/awesome_skills_catalog.md — broader ecosystem map
• skills/catalog/ — index + per-domain skills
• skills/examples/ — four full reference skills

Curated skills shipped: research-summarizer, repo-auditor, mcp-security-reviewer, agent-memory-curator, api-design-reviewer, pr-summarizer, adr-writer, incident-postmortem, sprint-planner, dataset-profiler.

Prompt engineering

A dedicated section, agent-focused:

• prompt_engineering/agent_prompt_patterns.md
• prompt_engineering/system_prompt_design.md
• prompt_engineering/instruction_hierarchy.md
• prompt_engineering/context_engineering.md
• prompt_engineering/tool_use_prompting.md
• prompt_engineering/planning_and_reflection.md
• prompt_engineering/memory_prompting.md
• prompt_engineering/prompt_injection_defense.md
• prompt_engineering/prompt_eval_methods.md
• prompt_engineering/anti_patterns.md

Templates: SYSTEM_PROMPT, AGENT_PROMPT. Checklist: agent_prompt_checklist.

Use this repo with coding agents

The handbook is itself a great surface for coding agents. Drop your favorite tool (Claude Code, Cursor, Codex, Aider, Cline) into the repo:

• llms.txt gives the agent an index in 30 seconds
• coding_agents/ has tool-specific notes + prompts
• coding_agents/prompts/ — repo audit, modernization, feature, bugfix, provider expansion, docs update, release review
• templates/CODING_AGENT_TASK.md.template — task contract template
• templates/REPO_MODERNIZATION_PROMPT.md.template — multi-phase modernization

The guidance is tool-neutral: same AGENTS.md, same workflows, regardless of harness.

Design docs

Agent + technical design docs, ADRs, reviews, rollouts, and the DESIGN.md machine-readable spec for design tokens:

• design_docs/agent_design_doc.md
• design_docs/technical_design_doc.md
• design_docs/adr_guide.md
• design_docs/design_review.md
• design_docs/rollout_plan.md
• design_docs/design_md_spec.md
• design_docs/examples/ — research / MCP / memory / provider-router worked examples

Templates: DESIGN_DOC, ADR.

Frameworks at a glance

📖 Full comparison + decision tree: docs/framework_comparison.md. Capability tags hedged: verify against current upstream docs.

Skills, MCP, and Memory in one minute

• Skills are reusable, model-loaded workflows (SKILL.md + scripts + references). Use when a task is repeatable, multi-step, and benefits from progressive disclosure. → skills/
• MCP (Model Context Protocol) is a standard for exposing tools/context to any agent. Use when integrations should be reusable (GitHub, filesystem, browser, internal APIs). → mcp/
• Memory is durable state across runs (MEMORY.md, vector stores, decision logs). → memory/

A useful rule of thumb:

📖 Decision matrix: skills/skill_vs_tool_vs_mcp.md

Guardrails & safety

Production agents need risk-tiered tool controls and human approval gates for high-impact actions.

📖 safety/README.md • safety/prompt_injection.md • safety/secure_agent_checklist.md

Observability & evals

You cannot ship what you cannot measure. The handbook ships:

• A tracing primer (observability/tracing.md) and span model (observability/spans.md)
• Cost / latency / failure analysis playbooks
• Eval design + datasets (evals/): regression, tool-call, memory, MCP, safety, prompt evals
• A curated guide to evaluation_frameworks — Promptfoo, DeepEval, RAGAs, Langfuse, Phoenix, TruLens, LangSmith, MLflow

Templates (copy-paste ready)

Merged knowledge areas (1.0.1)

This release merged seven external projects into the handbook. Each was adapted (not bulk-copied) into the structure above:

📖 Full migration plan: MIGRATION_AND_PROVIDER_EXPANSION_PLAN.md

Supported LLM providers

The utilities/llm_provider.py module exposes a single LLMProvider interface (and a backwards-compatible complete() function). Switch via LLM_PROVIDER without touching agent code; route automatically with ProviderRouter.

24+ providers across frontier / fast / marketplace / enterprise / specialty / local. See:

• providers/provider_matrix.md — capability comparison
• providers/env_vars.md — every variable
• .env.example — copy-and-fill

Contributing

Contributions are very welcome — new examples, framework updates, fixes, and translations all help. Start with:

• CONTRIBUTING.md — workflow, scope, quality bar
• .github/PULL_REQUEST_TEMPLATE.md
• .github/ISSUE_TEMPLATE/
• checklists/open_source_quality_checklist.md

Roadmap & changelog

• ROADMAP.md — what's next
• CHANGELOG.md — what shipped

License

MIT — see LICENSE.

Maintainer

Curated & maintained by Sayed Allam (oxbshw). If this handbook helped you ship, please ⭐ the repo and open a PR with what you learned along the way.