Skip to content

Operator ETL wiki

Agentic data intake for FOIA and public comments — a medallion warehouse (raw → valid rows → KPIs; PATTERNS), LangGraph orchestration, MCP allowlist, PII policy plane.

Python and SQL decide what data exists. Agents orchestrate within typed boundaries. Tests prove the invariants. No LLM API key is required for the MVP demo.

When to read: First page after you find the repo or this site. How this wiki is organized is below. To learn what we built: CONCEPTS.

The problem

Chat-with-the-warehouse demos fail three ways that matter for public comments and any audited intake: PII in the prompt, invented counts in the memo, and runs you cannot replay. Operator ETL separates deterministic ETL from bounded agents so those failures are testable instead of hoped away.

Full argument: WHY.md.

What we built

A reference architecture, not a production FOIA product: medallion layers (bronze → silver/quarantine → gold), PII vault, LangGraph, a critic that rejects uncited numbers, and three MCP tools with no vault access. Default insight is a template. Optional LLM only rewrites wording from gold KPI JSON.

Tour: CONCEPTS.md · Words: PATTERNS.md · Runtime: HOW-IT-WORKS.md · Audit: FINAL-REVIEW.md.

Prove it (3 lines)

git clone https://github.com/khaosans/operator-etl.git
cd operator-etl
./scripts/verify.sh

Success ends with OPERATOR_ETL_VERIFY=PASS, 76 pytest, and a FOIA demo of status=complete, silver=10, quarantined=2.

Full card: QUICKSTART.md · Screenshots: TOUR.md

Gov / FOIA dashboard after a local FOIA run

Why it is useful · other data · risks

Topic Page
Why the pattern exists WHY · CONCEPTS · PATTERNS
Same architecture on another CSV (311, grants, orders, …) APPLY
Residual risks (regex PII, critic digits, cloud JSON, HITL) RISKS
NIST alignment, not certification NIST

Who this is for

Full jobs-to-be-done (today vs later): PERSONAS.md.

You are… Start here
Sam — new engineer QUICKSTART → CONCEPTS → WALKTHROUGH · TOUR
Priya — FOIA / program officer FOIA guide → DASHBOARD · TOUR
Riley — data engineer / extending a feed APPLY → ADD-A-SOURCE · CLI
Jordan — architect / reviewer WHY → CONCEPTS → RISKS → NIST → FOUNDATIONS
Casey — AI agent / MCP AGENTS.md → operator-verify
Alex — decision-maker / buyer WHY → white paper → ROADMAP

How to read this wiki

Docs follow a Diátaxis split so tutorials, explanation, how-to, and reference stay distinct.

Kind What you want Where
Tutorials Do the happy path Start + Prove: Quickstart, Getting started, Tour, Walkthrough
Explanation Understand problem, design, words, transfer, risk Understand: Why, Concepts, Patterns, How it works, Apply, Risks, NIST, Models, Foundations
How-to Achieve a task Use + Scale: CLI, sources, LLM, dashboard, GCP
Reference Look up a fact Glossary, FAQ, Troubleshooting, Standards, Testing, Final review, Versioning

Personas: PERSONAS.md.

Honest scope

Proven locally (make e2e) Not this MVP
FOIA CSV → PII → silver/quarantine → gold → critic-verified insight Production Presidio PII
76 pytest + SAST/SCA CI on every push Live GCP / BigQuery end-to-end
MCP allowlist (3 tools, no vault) Live LLM API (optional path mocked; template default)

Full audit: FINAL-REVIEW.md · Residual risks: RISKS.md

Wiki map

  1. Start — QUICKSTART · GETTING-STARTED · PERSONAS · TOUR
  2. Understand — WHY · CONCEPTS · PATTERNS · HOW-IT-WORKS · APPLY · RISKS · NIST · MODELS · FOUNDATIONS
  3. Prove — WALKTHROUGH · TESTING · FINAL-REVIEW · SECURITY-HARDENING
  4. Use — RUNNING · CLOUD-AGENT · CLI · DASHBOARD · MCP · A2A · DISCORD · OBSERVABILITY · ADD-A-SOURCE · LLM
  5. Reference — GLOSSARY · FAQ · TROUBLESHOOTING · STANDARDS
  6. Scale — SCALING · MULTI-CLOUD · PRODUCT-UX · infra/README
  7. Contribute — CONTRIBUTING · REPO-HYGIENE · BUILD-HYGIENE · VERSIONING · RELEASING · SECURITY

Deep spec: Operator-ETL-White-Paper.md (not duplicated here).

Agent knowledge: okf/index.md — playbooks for Cursor/Claude; humans use this wiki.

See also