Skip to content

Operator ETL wiki

Agentic data intake for FOIA and public comments — deterministic medallion warehouse, LangGraph orchestration, MCP allowlist, PII policy plane.

Python and SQL decide what data exists. Agents orchestrate within typed boundaries. Tests prove the invariants. No LLM API key is required for the MVP demo.

When to read: First page after you find the repo or this site. Then run verify.

Prove it (3 lines)

git clone https://github.com/khaosans/operator-etl.git
cd operator-etl
./scripts/verify.sh

Success ends with OPERATOR_ETL_VERIFY=PASS, 41 pytest, and a FOIA demo of status=complete, silver=10, quarantined=2.

Full card: QUICKSTART.md · Screenshots: TOUR.md

Gov / FOIA dashboard after a local FOIA run

Who this is for

You are… Start here
New engineer QUICKSTART → WALKTHROUGH · TOUR
FOIA / program officer FOIA guide → DASHBOARD · TOUR
Architect / reviewer WHY → FOUNDATIONS → ./scripts/verify.sh
Data engineer extending sources ADD-A-SOURCE → CLI
AI agent AGENTS.md → operator-verify

Honest scope

Proven locally (make e2e) Not this MVP
FOIA CSV → PII → silver/quarantine → gold → critic-verified insight Production Presidio PII
41 pytest + CI on every push Live GCP / BigQuery end-to-end
MCP allowlist (3 tools, no vault) Live LLM API (optional path mocked; template default)

Full audit: FINAL-REVIEW.md

Wiki map

  1. Start — QUICKSTART · GETTING-STARTED · PERSONAS · TOUR
  2. Understand — WHY · HOW-IT-WORKS · FOUNDATIONS
  3. Prove — WALKTHROUGH · TESTING · FINAL-REVIEW
  4. Use — CLI · DASHBOARD · MCP · ADD-A-SOURCE · LLM
  5. Reference — GLOSSARY · FAQ · TROUBLESHOOTING · STANDARDS
  6. Scale — SCALING · PRODUCT-UX · infra/README
  7. Contribute — CONTRIBUTING · RELEASING · SECURITY

Deep spec: Operator-ETL-White-Paper.md (not duplicated here).

Agent knowledge: okf/index.md — playbooks for Cursor/Claude; humans use this wiki.

See also