Standards and best practices¶

Operator ETL follows established patterns for agentic data systems, government FOIA workflows, and maintainable Python projects. This page is the single index — why each pattern matters and which test proves it: FOUNDATIONS.md.

Knowledge and documentation¶

Standard	Reference	How we apply it
OKF v0.1	Google Knowledge Catalog OKF	`okf/` bundle with typed frontmatter; validate with `python3 scripts/okf_validate.py okf --strict`
Progressive disclosure	OKF `index.md` pattern	Root `okf/index.md` → playbooks → code
README conventions	Make a README	Prerequisites, quick start, commands, doc map in README.md

Data architecture¶

Standard	Reference	How we apply it
Medallion architecture	FOUNDATIONS.md §1 — Databricks	Bronze (immutable) → silver (validated) → gold (aggregates) + quarantine
Layer definitions	okf/models/medallion-layers.md	DuckDB local; BigQuery on GCP
Idempotent ingest	FOUNDATIONS.md §2 — Kleppmann DDIA	SHA-256 in `ingest_files`; safe at-least-once delivery
Fail-closed quality	FOUNDATIONS.md §6 — Goodhart; pii-fail-closed.md	Withhold KPIs when gate fails

Agentic AI¶

Standard	Reference	How we apply it
Three planes	okf/models/three-planes.md	Data / Policy / Control separation
LangGraph	FOUNDATIONS.md §3	Control plane in `operator_etl_graph/`; checkpoints for resume
MCP	FOUNDATIONS.md §4, §10	Allowlisted tools only — okf/decisions/mcp-allowlist-only.md
Critic / faithfulness	FOUNDATIONS.md §7 — grounded generation	Every insight number must exist in `gold_metrics`
Agents never auto-publish	FOUNDATIONS.md §8 — NIST AI RMF	Human sign-off before FOIA release

Security and compliance¶

Standard	Reference	How we apply it
PII fail-closed	FOUNDATIONS.md §5 — NIST SP 800-122	Scan before insight; encrypted vault; no MCP vault access
FOIA workflow	FOUNDATIONS.md §9	Public comments intake and redaction queue
Secrets hygiene	SECURITY.md	No `.env`, vault, or tfvars in git
Least privilege (GCP)	White paper §12.3	Separate service accounts per workload in Terraform

Engineering decisions (ADRs)¶

Documented in docs/Operator-ETL-White-Paper.md §3:

ADR	Decision
ADR-001	Medallion bronze / silver / gold
ADR-002	LangGraph over implicit agent loops
ADR-003	MCP as agent boundary
ADR-004	Fail-closed quality gate
ADR-005	DuckDB local → BigQuery GCP — okf/decisions/duckdb-local-bigquery-gcp.md

Python and repository¶

Standard	Reference	How we apply it
src layout	PyPA src layout	Packages under `src/`
uv	Astral uv	`pyproject.toml` + `uv.lock`
Proof gate	FOUNDATIONS.md proof matrix	`make e2e` before share, deploy, or scale claims — see WALKTHROUGH.md
CI	GitHub Actions	`.github/workflows/ci.yml` — same gate as local e2e + Docker build

Implementation status¶

What is coded vs specified: okf/models/implementation-status.md

Update that matrix when shipping features; sync white paper badges on major releases.