Skill_Seekers

Skill Seekers

English | 简体中文 | 日本語 | 한국어 | Español | Français | Deutsch | Português | Türkçe | العربية | हिन्दी | Русский

[Version] [License: MIT] [Python 3.10+] [MCP Integration] [Tested] [PyPI version] [PyPI - Downloads] [Website] [GitHub Repo stars] [PyPI Downloads]

🧠 The data layer for AI systems. Skill Seekers turns documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more — 18 source types — into structured knowledge assets, ready to power AI Skills (Claude, Gemini, OpenAI), RAG pipelines (LangChain, LlamaIndex, Pinecone), and AI coding assistants (Cursor, Windsurf, Cline). Prepare once, export to 22 targets.

💛 Sponsors

Bronze Sponsors

Become a sponsor · GitHub Sponsors

---

🚀 Quick Start

# 1. Install
pip install skill-seekers

# 2. Create a skill from any source
skill-seekers create https://docs.djangoproject.com/

# Optional: preview how a source will be detected without creating anything
skill-seekers detect https://docs.djangoproject.com/ --json

# 3. Package it for your AI platform
skill-seekers package output/django --target claude

You now have output/django-claude.zip, ready to use.

# Pick a different AI agent for enhancement (default: claude)
skill-seekers create https://docs.djangoproject.com/ --agent kimi
skill-seekers create https://docs.djangoproject.com/ --agent-cmd "my-custom-agent run"

🛰️ AI-driven project scan

Point scan at a project and an AI agent reads its manifests, README, Dockerfile/CI and sampled source imports — then emits one config per detected framework, plus a <project>-codebase.json for your own code:

skill-seekers scan ./my-react-app --out ./configs/scanned/
# → react.json, vite.json, tailwind.json, jest.json, my-react-app-codebase.json

skill-seekers create ./configs/scanned/react.json

If a detection has no existing preset, the AI generates a fresh config; on exit you can optionally publish it back to the community registry.

All 18 source types

skill-seekers create facebook/react            # GitHub repository
skill-seekers create ./my-project              # Local codebase
skill-seekers create manual.pdf                # PDF
skill-seekers create report.docx               # Word
skill-seekers create book.epub                 # EPUB
skill-seekers create notebook.ipynb            # Jupyter
skill-seekers create openapi.yaml              # OpenAPI/Swagger
skill-seekers create presentation.pptx         # PowerPoint
skill-seekers create guide.adoc                # AsciiDoc
skill-seekers create page.html                 # Local HTML (or a whole dir)
skill-seekers create feed.rss                  # RSS/Atom
skill-seekers create curl.1                    # Man page

# Video (YouTube, Vimeo, or local — needs skill-seekers[video])
skill-seekers create --video-url https://www.youtube.com/watch?v=... --name mytutorial
skill-seekers create --setup                   # auto-install GPU-aware visual deps

skill-seekers create --space-key TEAM --name wiki               # Confluence
skill-seekers create --database-id ... --name docs              # Notion
skill-seekers create --chat-export-path ./slack-export --name team-chat  # Slack/Discord

See the Scraping Guide for every source type and its options.

---

📦 Installation

pip install skill-seekers              # Core: scraping, GitHub, PDF, packaging
pip install skill-seekers[all-llms]    # + every LLM platform
pip install skill-seekers[mcp]         # + MCP server
pip install skill-seekers[all]         # Everything

Not sure what you need? Run the wizard: skill-seekers-setup

Install	Adds
skill-seekers[gemini]	Google Gemini support
skill-seekers[openai]	OpenAI ChatGPT support
skill-seekers[all-llms]	All LLM platforms
skill-seekers[mcp]	MCP server for Claude Code, Cursor, etc.
skill-seekers[video]	YouTube/Vimeo transcript & metadata extraction
skill-seekers[video-full]	+ Whisper transcription & visual frame extraction
skill-seekers[jupyter]	Jupyter Notebook support
skill-seekers[pptx]	PowerPoint support
skill-seekers[confluence]	Confluence wiki support
skill-seekers[notion]	Notion pages support
skill-seekers[rss]	RSS/Atom feed support
skill-seekers[chat]	Slack/Discord chat export support
skill-seekers[asciidoc]	AsciiDoc support
skill-seekers[all]	Everything

Video visual deps (GPU-aware): after installing skill-seekers[video-full], run skill-seekers create --setup to auto-detect your GPU and install the matching PyTorch variant + easyocr.

Prerequisites: Python 3.10+, Git. New here? → Bulletproof Quick Start 🎯

---

📚 Documentation

I want to...	Read this
Get started quickly	Quick Start — 3 commands to your first skill
Understand the concepts	Core Concepts
Scrape sources	Scraping Guide — all 18 source types
Enhance skills with AI	Enhancement Guide · Enhancement Modes
Export skills	Packaging Guide
Build workflows	Workflows
Look up a command	CLI Reference — all 19 commands
Configure	Config Format · Environment Variables
Set up MCP	MCP Setup · MCP Reference
Integrate with RAG / IDEs	LangChain · RAG Pipelines · Cursor · Windsurf · Cline
Handle huge doc sets	Large Documentation — 10K–40K+ pages
Understand the architecture	UML Architecture — 14 diagrams
Fix a problem	Troubleshooting

Complete documentation index: docs/README.md

---

🎯 What you get

Use case	Output	Powers
AI Skills	Comprehensive SKILL.md + reference files	Claude Code, Gemini, GPT
RAG pipelines	Chunked documents with rich metadata	LangChain, LlamaIndex, Haystack
Vector databases	Pre-formatted data ready for upsert	Pinecone, Chroma, Weaviate, FAISS, Qdrant
AI coding assistants	Context files your IDE AI reads automatically	Cursor, Windsurf, Cline, Continue.dev

Export targets (22)

skill-seekers package output/react --target claude      # → Claude Skill (ZIP + YAML)
skill-seekers package output/react --target langchain   # → LangChain Documents
skill-seekers package output/react --target llama-index # → LlamaIndex TextNodes
skill-seekers package output/react --target ibm-bob     # → IBM Bob skill directory

LLM platforms (12): claude · gemini · openai · minimax · opencode · kimi · deepseek · qwen · openrouter · together · fireworks · markdown RAG & vector (8): langchain · llama-index · haystack · chroma · faiss · weaviate · qdrant · pinecone Other (2): atlas · ibm-bob

See the Feature Matrix for per-platform support details.

Why it matters

• ⚡ 99% faster — days of manual data prep → 15–45 minutes
• 🎯 Real skill quality — 500+ line SKILL.md files with examples, patterns, and guides
• 📊 RAG-ready chunks — smart chunking preserves code blocks and context
• 🔄 Multi-source — combine docs + GitHub + PDFs + videos into one knowledge asset
• 🌐 One prep, every target — export to 22 targets without re-scraping
• ✅ Battle-tested — 3,900+ tests, 68 workflow presets, production-ready

---

✨ Key capabilities

Three-layer discovery for JavaScript SPA sites (sitemap.xml → llms.txt → headless browser rendering), automatic llms.txt detection (10× faster when present), smart topic categorization, and a lenient HTML parser fallback so broken markup still scrapes.

→ Scraping Guide · llms.txt Support

Three-stream architecture: code analysis (AST, design patterns, tests), documentation (README, docs/, wiki), and community (issues, PRs, metadata). The C3.x pipeline adds 10 GoF pattern detectors across 9 languages, usage examples extracted from tests, AI-written how-to guides, config extraction, and architecture overviews.

skill-seekers create ./my-project --preset quick          # 1–2 min, surface level
skill-seekers create ./my-project --preset standard       # balanced (default)
skill-seekers create ./my-project --preset comprehensive  # deep, exhaustive

→ Pattern Detection · How-To Guides · Test Example Extraction

Every AI call runs through one transport, in API mode (Anthropic, Google Gemini, OpenAI, Moonshot/Kimi, MiniMax) or LOCAL mode (Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — no API costs). Control depth with --enhance-level 0-3 and pick an agent with --agent.

→ Enhancement Guide · Enhancement Modes · Multi-Agent Setup

One config can pull documentation, GitHub, PDFs, videos, and more into a single knowledge asset, with conflict detection and pairwise synthesis across sources.

→ Unified Scraping

skill-seekers create <source> --index adds a stdlib-only scripts/search.py and a SQLite FTS5 index over the generated references, so an agent can find the right file#anchor before reading a large Markdown file wholesale. Off by default; the Markdown is untouched.

→ Skill Search Index

YouTube, Vimeo, and local files. Three-tier transcript fallback (subtitles → YouTube transcript API → local Whisper), plus optional visual extraction that OCRs on-screen code from sampled frames.

→ Video Guide

Quality scoring with a gate (skill-seekers quality output/react/ --threshold 7) or JSON stdout (skill-seekers quality output/react/ --json), provisional English readability metrics (informational — they never affect the score), doc-change detection with scheduled re-scrapes and notifications, streaming ingestion for very large doc sets, and incremental updates.

→ Large Documentation · Code Quality

---

🔌 MCP Integration (40 tools)

Skill Seekers ships an MCP server for Claude Code, Cursor, Windsurf, VS Code + Cline, and IntelliJ IDEA.

# stdio mode (Claude Code, VS Code + Cline)
python -m skill_seekers.mcp.server_fastmcp

# HTTP mode (Cursor, Windsurf, IntelliJ)
python -m skill_seekers.mcp.server_fastmcp --transport http --port 8765

Then just ask your assistant: "Package and upload the React skill."

→ MCP Setup · MCP Reference · HTTP Transport

---

🤖 Installing to AI agents

Skills install automatically into 19 AI coding agents:

skill-seekers install-agent output/react/ --agent cursor
skill-seekers install-agent output/react/ --agent all      # every detected agent
skill-seekers install-agent output/react/ --agent cursor --dry-run

Agent	Path	Scope
Claude Code	~/.claude/skills/	Global
Cursor	.cursor/skills/	Project
VS Code / Copilot	.github/skills/	Project
Amp	~/.amp/skills/	Global
Goose	~/.config/goose/skills/	Global
OpenCode	~/.opencode/skills/	Global
Letta	~/.letta/skills/	Global
Aide	~/.aide/skills/	Global
Windsurf	~/.windsurf/skills/	Global
Neovate	~/.neovate/skills/	Global
Roo Code	.roo/skills/	Project
Cline	.cline/skills/	Project
Aider	~/.aider/skills/	Global
Bolt	.bolt/skills/	Project
Kilo Code	.kilo/skills/	Project
Continue	~/.continue/skills/	Global
Kimi Code	~/.kimi/skills/	Global
IBM Bob	.bob/skills/	Project

Uploading to Claude

export ANTHROPIC_API_KEY=sk-ant-...
skill-seekers package output/react/ --upload   # package + upload
skill-seekers upload output/react.zip          # upload an existing zip

No API key? Package it and upload output/react.zip manually at claude.ai/skills.

→ Upload Guide

---

⚙️ How it works

graph LR
    A[Documentation Website] --> B[Skill Seekers]
    B --> C[Scraper]
    B --> D[AI Enhancement]
    B --> E[Packager]
    C --> F[Organized References]
    D --> F
    F --> E
    E --> G[AI Skill .zip]
    G --> H[Upload to AI Platform]

1. Scrape — extract every page (checking llms.txt first)
2. Categorize — organize content into topics (API, guides, tutorials, …)
3. Enhance — AI writes a comprehensive SKILL.md with examples
4. Package — bundle into a platform-ready artifact
5. Upload — ship it to your AI platform (optional)

Architecture

8 core modules + 5 utility modules (~200 classes):

Module	Purpose
CLICore	Git-style command dispatcher, source auto-detection
Scrapers	18 source-type extractors on a shared build layer
Adaptors	22 output platform formats behind one SkillAdaptor ABC
Analysis	C3.x codebase pipeline, 10 GoF pattern detectors
Enhancement	AI improvement via a single AgentClient transport
Packaging	Package, upload, and install skills
MCP	FastMCP server (40 tools, 10 tool modules)
Sync	Doc change detection and notification

→ UML Architecture · API Reference · Skill Architecture

---

🆕 New in v3.9.0

• HTML parser fallback for broken markup (#96) — severely malformed pages no longer scrape as empty; well-formed pages are byte-identical.
• Transient-failure retries — the doc scraper (#97) and MCP fetch_config (#92) now retry connection blips and 5xx with backoff; 4xx still fails fast.
• Whisper transcription fallback (#420) — local videos without subtitles finally get a real transcript.
• MiniMax image OCR + registry-driven multimodal providers (#423) — providers declare their wire protocol and image capability; China-issued keys work against the right endpoint.
• Token-lean GitHub issue defaults (#169) — GitHub skills no longer bundle full closed-issue history by default.
• Env-driven CORS across all three servers (#422, #424) — no more wildcard origins with credentials.

Full history: CHANGELOG.md

---

📈 Performance

Documentation size	Time	Output
Small (< 100 pages)	5–10 min	~2 MB
Medium (100–500 pages)	15–30 min	~10 MB
Large (500–2,000 pages)	30–60 min	~40 MB
Huge (10K–40K+ pages)	Use stream	See Large Documentation

---

🐛 Troubleshooting

skill-seekers doctor          # diagnose installation & environment
skill-seekers doctor --json   # machine-readable diagnostics for CI and agents
skill-seekers sync-config     # detect config drift

Common issues and fixes: Troubleshooting Guide · TROUBLESHOOTING.md

---

🤝 Contributing

Contributions are welcome — see CONTRIBUTING.md.

• 📋 Development Roadmap & Tasks — pick any task
• 💬 Discussions — questions and ideas
• 🐛 Issues — bugs and feature requests

---

📝 License

MIT — see LICENSE.

🔒 Security

[MseeP.ai Security Assessment Badge]

---

🌐 Ecosystem

Skill Seekers is a multi-repo project:

Repository	Description	Links
Skill_Seekers	Core CLI & MCP server (this repo)	PyPI
skillseekersweb	Website & documentation	Live
skill-seekers-configs	Community config repository
skill-seekers-action	GitHub Action for CI/CD
skill-seekers-plugin	Claude Code plugin
homebrew-skill-seekers	Homebrew tap for macOS

Want to contribute? The website and configs repos are great starting points for new contributors!