Neutron Universe

Wikipedia Deep Research Prompt

Produce a reusable template repo that adds a local Wikipedia search source to the LangGraph “Open Deep Research” stack and ships runnable Kubernetes resources. Generate files with concrete content, parameterized via {{variables}}. Follow the spec exactly.

Goal

Create a minimal, reproducible template a downstream user can clone, set a few vars, and deploy:

  • Runs open_deep_research API with a Wikipedia search backend.

  • Hosts a local Kiwix mirror of Wikipedia ZIM files.

  • Performs semantic search with txtai and returns Markdown excerpts with citations to local Kiwix URLs.

  • Selects the Wikipedia tool via search_api: WIKIPEDIA without changing graph logic.

Output format

Do not add commentary outside files. Include all files listed below.

Repository layout

open-deep-research-wiki-template/
  README.md
  Makefile
  k8s/
    resources.yaml
    kiwix.yaml
  docker/
    Dockerfile
    entrypoint.sh
    run.sh
    run_fastapi.sh
  pyproject.toml
  uv.lock        # generate a minimal lock; pin txtai + torch cpu
  langgraph.json # pass-through if needed
  src/open_deep_research/
    __init__.py
    deep_researcher.py    # integrates unchanged graph, no business-logic changes
    utils.py              # adds wiki_search + get_search_tool wiring
    wikipedia.py          # local Kiwix + txtai retrieval
    configuration.py      # SearchAPI enum includes WIKIPEDIA
    prompts.py            # unchanged
    state.py              # unchanged
    configuration_example.env

Parameterization

Use Jinja-style placeholders. Do not hardcode values.

  • {{NAMESPACE}} default: open-deep-research

  • {{DOMAIN}} default: <domain>

  • {{WIKI_HOST}} default: wiki.{{DOMAIN}}

  • {{KIWIX_BASE}} default: https://{{WIKI_HOST}}

  • {{ZIM_NAME}} default: wikipedia_en_all_maxi_2025-08

  • {{API_IMAGE}} default: localhost:32000/mm_open_deep_research:latest

  • {{TLS_SECRET}} default: <domain>-wildcard-tls

  • {{REPLICAS}} default: 1

  • {{API_PORT}} default: 8123

  • {{CPU_LIMIT}} default: 4

  • {{MEM_LIMIT}} default: 56Gi

  • {{ZIM_HOSTPATH}} default: /mnt/data/zim

File requirements

1) k8s/resources.yaml

  • Deployment + Service for API.

  • Env for threading control: FAISS_NUM_THREADS=1, OMP_NUM_THREADS=1, OPENBLAS_NUM_THREADS=1, MKL_NUM_THREADS=1, PYTHONFAULTHANDLER=1.

  • SERVER_MODE=fastapi, PORT={{API_PORT}}.

  • Resource requests: cpu 100m, mem 2Gi. Limits from params.

  • Readiness/Liveness GET /.

  • Service on port 80 -> {{API_PORT}}.

2) k8s/kiwix.yaml

  • Deployment, Service, Ingress for kiwix-serve.

  • Container ghcr.io/kiwix/kiwix-serve:latest with args kiwix-serve -p 8080 /data/zim/*.zim.

  • hostPath volume at {{ZIM_HOSTPATH}} read-only.

  • Ingress {{WIKI_HOST}} with TLS {{TLS_SECRET}}.

3) docker/Dockerfile

  • Base: python:3.11-slim.

  • Install build deps: ca-certificates curl git build-essential libgomp1.

  • Install uv via official script.

  • Set caches to /opt/cache (HF, transformers, sentence-transformers, torch, pip).

  • Copy pyproject.toml, uv.lock, src, server (if present), scripts.

  • uv sync then export venv to PATH.

  • Set XDG_CACHE_HOME and related envs.

  • Prewarm by invoking /app/src/open_deep_research/wikipedia.py to trigger model pulls.

  • ENTRYPOINT to entrypoint.sh.

4) docker/entrypoint.sh

  • SERVER_MODE switch: fastapi -> run_fastapi.sh, cli -> run.sh.

5) pyproject.toml and uv.lock

  • Dependencies include:

    • langgraph, langchain-*, openai, exa-py, tavily-python, pandas, fastapi, uvicorn, langserve, rich, httpx, beautifulsoup4==4.13.3, txtai==9.0.0, torch>=2.5,<3.

  • Configure CPU-only torch source:

    [tool.uv.sources]
torch = [{ index = "pytorch-cpu" }]

[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

  • Ensure txtai pulls neuml/txtai-wikipedia at runtime.

6) src/open_deep_research/wikipedia.py

Implement:

  • KIWIX_BASE=os.getenv("KIWIX_BASE","{{KIWIX_BASE}}")

  • ZIM_NAME=os.getenv("ZIM_NAME","{{ZIM_NAME}}")

  • Force CPU: os.environ["FAISS_DISABLE_GPU"]="1".

  • Embeddings().load(provider="huggingface-hub", container="neuml/txtai-wikipedia") behind an @lru_cache and threading.RLock.

  • wiki_query(query, percentile_threshold=0.99, max_articles=5) that:

    • Executes txtai SQL similar('{query}') with percentile filter.

    • For each hit: resolve title, fetch HTML from Kiwix (/raw/... then XML search fallback), strip to Markdown, drop references, collapse whitespace.

    • Return dicts: {title,id,score,percentile,kiwix_url,markdown}.

7) src/open_deep_research/utils.py

Add:

  • wiki_search tool with signature queries: List[str] that calls wiki_search_async, flattens results, and formats Search results:\n\n--- SOURCE i: <title> ---\nURL: <kiwix_url>\n\nSUMMARY:\n<markdown>\n\n---------.

  • get_search_tool to map SearchAPI.WIKIPEDIA to wiki_search and expose it as name="web_search" so the rest of the graph is unchanged.

8) src/open_deep_research/deep_researcher.py

  • Keep the supervisor/researcher/compression logic as-is.

  • Ensure get_all_tools(config) path picks up SearchAPI.WIKIPEDIA via utils.get_search_tool.

  • No other semantic changes.

9) src/open_deep_research/configuration.py

  • Ensure SearchAPI enum includes WIKIPEDIA.

  • Default remains selectable via config.configurable.search_api.

10) Makefile

Targets:

  • build: build API image tag {{API_IMAGE}}.

  • push: push image.

  • deploy-kiwix: apply k8s/kiwix.yaml.

  • deploy-api: apply k8s/resources.yaml.

  • undeploy: delete both.

  • test-wiki: curl https://{{WIKI_HOST}}/ readiness.

  • invoke: sample JSON call with search_api: wikipedia.

11) src/open_deep_research/configuration_example.env

Document:

KIWIX_BASE={{KIWIX_BASE}}
ZIM_NAME={{ZIM_NAME}}
FAISS_DISABLE_GPU=1
SERVER_MODE=fastapi
PORT={{API_PORT}}

Quality gates

Implement and satisfy these checks:

  1. Static: repository pip check inside the built image must pass.

  2. Runtime: container starts and GET / responds 200.

  3. Kiwix: readiness on / returns 200 after ZIM mount.

  4. Search: calling the graph with search_api=wikipedia returns at least one SOURCE block with URL: https://{{WIKI_HOST}}/content/{{ZIM_NAME}}/A/....

  5. Isolation: no external HTTP calls are required during research if Kiwix and ZIM exist.

  6. Idempotency: kubectl apply twice yields no changes.

Constraints

  • Use only the libraries listed in pyproject.toml.

  • Keep txtai CPU-only; do not enable GPU.

  • Keep tool name exposed to the graph as web_search to avoid changing planner logic.

  • Keep code Python 3.11 compatible.

Done when

  • All files are emitted with full contents using the delimiter.

  • Placeholders are present where specified.

  • A user can fill variables, apply manifests, build image, and run a local Wikipedia-backed deep-research workflow end to end.