Skip to content

Foundations — why this design is simple and provable

Operator ETL keeps the hard parts deterministic: Python and SQL decide what data exists. Agents orchestrate within typed boundaries. Every important invariant has a test you can run locally with make e2e.

When to read: You need citations, the proof matrix, or bibliography backing design choices. Plain English for each word: PATTERNS.md.

Usage: HOW-IT-WORKS.md · See it work: WALKTHROUGH.md · Start here: README.md

Thesis

Government FOIA workflows need an audit trail, PII protection before release, and numbers leadership can defend. The simplest architecture that satisfies all three: medallion warehouse (deterministic ETL) + bounded agents (LangGraph + MCP allowlist) + rule-based critic (every insight number must exist in gold).

flowchart LR
  Source[Authoritative source] --> Invariant[Design invariant]
  Invariant --> Code[Code path]
  Code --> Test[pytest / make e2e]

Proof matrix

Each row links an invariant to an authoritative source, where it lives in code, and how to verify it.

Invariant Source Why it matters here Where in code How to verify
Immutable intake, validated silver, trusted gold [1] Medallion architecture FOIA audit trail; quarantine without data loss operator_etl/ bronze/silver/gold/quarantine tests/test_pipeline.py, make e2e
Safe replay on duplicate drops [2] Idempotent consumers (DDIA) GCS/Pub/Sub delivery is at-least-once ingest_files content hash tests/test_pipeline.py
Explicit orchestration, resumable runs [3] LangGraph Replace ad-hoc agent loops; HITL interrupts operator_etl_graph/ tests/test_gov_graph.py
Least-privilege agent data access [4] MCP spec Agents get typed tools, not the warehouse operator_etl_mcp/, sql/allowlist.yaml tests/test_mcp_tools.py
No PII in agent context [5] NIST SP 800-122 FOIA redaction before release — NIST.md operator_etl_policy/pii.py tests/test_pii.py
Withhold bad KPIs, don't warn-and-show [6] Goodhart's Law Metrics that become targets get gamed quality_gate in insights tests/test_quality.py
Insight numbers must match warehouse [7] Grounded generation + [11] NIST AI 600-1 confabulation Defensible memos; GAI wording still critic-gated — NIST.md operator_etl_graph/critic.py tests/test_critic.py
Human sign-off on release [8] NIST AI RMF Agents orchestrate, humans publish — NIST.md okf/decisions/agents-never-publish-prod.md, graph needs_human tests/test_gov_graph.py (quality fail), test_critic_exhausted_routes_needs_human; PII gray-zone HITL needs Presidio
Public release workflow [9] FOIA statute Domain framing for gov tab and marts FOIA guide, sql/marts/gov/ scripts/demo_mvp.sh
Limit LLM agency [10] OWASP LLM Top 10 Excessive agency → allowlist + no vault MCP MCP deny paths tests/test_mcp_tools.py

Full standards index: STANDARDS.md

What we deliberately kept simple

  • DuckDB local — zero-infra proof on a laptop; same graph and SQL marts lift to BigQuery later
  • Template insight + rule-based critic — no LLM API key required for MVP; critic rejects hallucinated numbers deterministically
  • Regex PII scanner — covers email, phone, SSN patterns; Presidio optional upgrade (SPECIFIED)
  • Three MCP tools only — get_gold_metrics, run_quality_sql, get_run_status
  • One command proof — make e2e runs OKF validate, 76 pytest tests, and FOIA demo assertions

References

  1. Databricks. (2020). Medallion Architecture. https://www.databricks.com/glossary/medallion-architecture
  2. Kleppmann, M. (2017). Designing Data-Intensive Applications. O'Reilly. Ch. 11 — stream processing and idempotent consumers.
  3. LangChain. LangGraph documentation. https://langchain-ai.github.io/langgraph/
  4. Anthropic. Model Context Protocol specification. https://modelcontextprotocol.io/
  5. NIST. (2010). SP 800-122 — Guide to Protecting the Confidentiality of Personally Identifiable Information. https://csrc.nist.gov/publications/detail/sp/800-122/final — scan before insight; vault; no PII in model payload. NIST.md.
  6. Goodhart, C. A. E. (1975). Problems of Monetary Management: The U.K. Experience. In Papers in Monetary Economics. Reserve Bank of Australia. See also Strathern: "When a measure becomes a target, it ceases to be a good measure."
  7. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. https://arxiv.org/abs/2005.11401 — Operator ETL applies the grounding principle via deterministic critic, not retrieval.
  8. NIST. (2023). AI Risk Management Framework (AI RMF 1.0). https://www.nist.gov/itl/ai-risk-management-framework — Govern, Map, Measure, Manage; human oversight before production release. Plain-language map: NIST.md.
  9. U.S. Congress. 5 U.S.C. § 552 — Freedom of Information Act. https://www.foia.gov/ — public disclosure workflow framing.
  10. OWASP. Top 10 for Large Language Model Applications. https://owasp.org/www-project-top-10-for-large-language-model-applications/ — LLM06 excessive agency mitigated by MCP allowlist.
  11. NIST. (2024). AI 600-1 — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. https://doi.org/10.6028/NIST.AI.600-1 — confabulation and GAI risks we actually mitigate; we do not claim the full profile.
  12. NIST. (2020). Privacy Framework 1.0. https://www.nist.gov/privacy-framework — identify/protect PII in the comment pipeline (not a full privacy program).
  13. Saltzer, J. H., & Schroeder, M. D. (1975). The Protection of Information in Computer Systems. Proceedings of the IEEE. https://doi.org/10.1109/PROC.1975.9939 — least privilege and fail-safe defaults (planes, MCP, fail-closed).
  14. Hohpe, G., & Woolf, B. (2003). Enterprise Integration Patterns. Addison-Wesley. Dead Letter Channel — quarantine keeps failed rows.
  15. Raasveldt, M., & Mühleisen, H. (2019). DuckDB: an Embeddable Analytical Database. SIGMOD. https://doi.org/10.1145/3299869.3320212 — local warehouse file.
  16. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. CIDR. https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf — lakehouse context for medallion layers.
  17. Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics. https://doi.org/10.1109/3468.844354 — HITL: computer suggests, human publishes.

See also