Data Governance System Architecture: Enterprise-Scale

🎯This Week's Journey

From prompts to enterprise data governance platform.

Monday: 3 core prompts for metadata extraction, quality checks, and policy enforcement. Tuesday: automated agents for data cataloging. Wednesday: team workflows across data engineers, compliance officers, and business users. Thursday: complete technical architecture with 9 layers, multi-agent orchestration, ML-driven quality checks, and scaling from startup to 10M+ records/day.

📋

Key Assumptions

Monitor 100-10,000 data sources (databases, APIs, files, streams)

Process 10K-10M records/day with sub-second policy enforcement

Multi-region deployment with data residency requirements (US, EU, APAC)

Compliance requirements: GDPR, CCPA, SOC2, HIPAA (healthcare), PCI-DSS (finance)

Integration with existing data stack (Snowflake, Databricks, BigQuery, S3, etc.)

System Requirements

Functional

Automated metadata extraction from structured and unstructured sources
Real-time data quality scoring with ML-driven anomaly detection
Policy engine for access control, PII redaction, and retention rules
End-to-end data lineage tracking with column-level granularity
Self-service data catalog with search, tagging, and business glossary
Automated compliance reporting (GDPR Article 30, CCPA disclosures)
Multi-agent orchestration for complex governance workflows

Non-Functional (SLOs)

latency p95 ms500

freshness min5

availability percent99.9

💰 Cost Targets: {"per_record_usd":0.0001,"per_user_month_usd":50,"per_source_month_usd":20}

Agent Layer

planner

Decomposes governance workflows into tasks and routes to specialized agents

🔧 task_decomposer, resource_estimator, dependency_resolver

⚡ Recovery: If agent unavailable: queue task with backoff, If plan invalid: fallback to manual workflow, If resource exhausted: throttle new requests

executor

Executes primary governance workflows (metadata extraction, policy application)

🔧 metadata_extractor, data_profiler, policy_applier, lineage_builder

⚡ Recovery: If connection fails: retry 3x with exponential backoff, If extraction incomplete: mark for manual review, If policy conflict: escalate to admin

evaluator

Validates data quality, policy compliance, and lineage accuracy

🔧 quality_scorer, anomaly_detector, compliance_checker, lineage_validator

⚡ Recovery: If quality check fails: flag for re-profiling, If compliance violation: trigger alert and block access, If lineage broken: rebuild from source

guardrail

Enforces safety constraints (PII redaction, access control, rate limits)

🔧 pii_detector, access_control_engine, rate_limiter, audit_logger

⚡ Recovery: If PII detection fails: deny access (fail-safe), If access control unavailable: deny and alert admin, If rate limit exceeded: queue request

metadata

Extracts and enriches metadata from diverse data sources

🔧 schema_introspector, data_sampler, stats_calculator, llm_enricher (for descriptions)

⚡ Recovery: If schema locked: retry after delay, If sampling fails: use metadata only, If LLM unavailable: skip enrichment

lineage

Builds and maintains end-to-end data lineage graphs

🔧 sql_parser, log_analyzer, graph_builder, impact_analyzer

⚡ Recovery: If parsing fails: mark lineage as partial, If cycle detected: alert and break cycle, If graph too large: paginate or sample

ML Layer

Feature Store

Update: Real-time for access patterns, hourly for quality scores, daily for lineage metrics

• data_quality_score (rolling 7-day avg)
• access_frequency (requests/day per table)
• pii_density (% of PII columns per table)
• lineage_depth (max hops from source)
• user_role_embedding (vector representation)
• policy_complexity_score (rule count + conditions)

Model Registry

Strategy: Semantic versioning with A/B testing for 7 days before promotion

• pii_detector
• quality_scorer
• anomaly_detector
• metadata_enricher

Observability Stack

Real-time monitoring, tracing & alerting

0 active

SOURCES

Apps, Services, Infra

COLLECTION

10 Metrics

PROCESSING

Aggregate & Transform

DASHBOARDS

4 Views

ALERTS

Enabled

📊Metrics(10)

📝Logs(Structured)

🔗Traces(Distributed)

sources_registered

✓

metadata_extraction_time_p95_ms

✓

quality_score_avg

✓

pii_detections

✓

policy_violations

✓

lineage_edges_created

✓

Deployment Variants

🚀

Startup Architecture

Fast to deploy, cost-efficient, scales to 100 competitors

Infrastructure

✓

Serverless (Lambda/Cloud Run)

✓

Managed PostgreSQL (RDS/Cloud SQL)

✓

Managed Redis (ElastiCache/Memorystore)

✓

S3 for logs and backups

✓

Single region (us-east-1 or eu-west-1)

→Quick to deploy (<1 week)

→Low operational overhead

→Pay-per-use pricing

→Good for <100K records/day

→Single-tenant by default

Risks & Mitigations

⚠️ Source credentials leaked or compromised

Medium

✓ Mitigation: Store credentials in Secrets Manager with 90-day rotation. Use least-privilege IAM roles. Audit all credential access. Encrypt at rest with customer-managed keys.

⚠️ PII detector fails, exposing sensitive data

Low

✓ Mitigation: Fail-safe: Deny all access if PII detector unavailable. Multi-layer detection (NER + regex + manual review). Regular audits of PII detection accuracy. Immediate rollback if accuracy drops.

⚠️ Lineage graph becomes too large to query

High (at scale)

✓ Mitigation: Paginate graph queries. Limit depth to 10 hops. Add indexes on frequently queried paths. Consider graph sampling or pre-computed summaries. Offer filtering by date/source.

⚠️ Agent orchestration deadlock or infinite loop

Medium

✓ Mitigation: Timeout all agent calls (30s max). Circuit breaker pattern. Monitor agent utilization and kill hung processes. Retry with different plan if deadlock detected. Telemetry on agent execution time.

⚠️ Multi-tenant data leakage (tenant A sees tenant B's data)

Low (with proper design)

✓ Mitigation: Tenant ID in all queries (enforce at DB level). Row-level security in PostgreSQL. Separate schemas per tenant. Regular penetration testing. Audit all cross-tenant queries (should be zero).

⚠️ Cost explosion from LLM API usage

Medium (without guardrails)

✓ Mitigation: Rate limits per tenant. Cost budgets with alerts. Cache LLM responses. Use cheaper models for non-critical tasks. Batch requests where possible. Monitor cost per record.

⚠️ Compliance drift (policies not enforced consistently)

Medium (as system grows)

✓ Mitigation: Automated policy testing (unit tests for policy logic). Regular compliance audits (quarterly). Policy change log with approval workflow. Guardrail agent validates all policy changes before deployment.

🧬

Evolution Roadmap

Progressive transformation from MVP to scale

🌱

Phase 1Q1 2025

0-3 months (MVP)

Launch with 5-10 data sources

Core metadata extraction and cataloging

Basic policy enforcement (access control, PII redaction)

Single-tenant deployment

Complexity Level

▼

🌿

Phase 2Q2 2025

3-6 months (Scale)

Support 50+ data sources

Add data lineage tracking

ML-driven quality scoring

Multi-tenant architecture

Complexity Level

▼

🌳

Phase 3Q3-Q4 2025

6-12 months (Enterprise)

Support 500+ data sources

Multi-region deployment

Advanced ML (agentic RAG, anomaly detection)

SOC2 Type II certification

Complexity Level

🚀Production Ready

🏗️

Complete Systems Architecture

9-layer architecture for enterprise data governance

🌐

Presentation

4 components

Data Catalog UI

Compliance Dashboard

Admin Portal

API Explorer

⚙️

API Gateway

4 components

GraphQL API

REST API

Rate Limiter

Auth Middleware

💾

Agent Layer

6 components

Planner Agent

Executor Agent

Evaluator Agent

Guardrail Agent

Metadata Agent

Lineage Agent

🔌

ML Layer

4 components

Feature Store

Model Registry

Inference Engine

Evaluation Loop

📊

Integration

4 components

Source Connectors

Data Profiler

Policy Engine

Notification Service

🌐

Data

4 components

Metadata Store (PostgreSQL)

Graph DB (Neo4j)

Vector DB (Pinecone)

Cache (Redis)

⚙️

External

4 components

Data Sources

Identity Provider

Notification APIs

Audit Services

💾

Observability

4 components

Metrics (Prometheus)

Logs (ELK)

Traces (Jaeger)

Dashboards (Grafana)

🔌

Security

4 components

KMS

Secret Manager

WAF

Audit Logger

🔄

Sequence Diagram - New Data Source Registration

Automated data flow every hour

Step 0 of 11

Data Flow - Source Registration to Policy Enforcement

User0s

Registers new data source → Connection config (host, port, credentials)

API Gateway50ms

Validates auth and rate limits → JWT token + request payload

Planner Agent200ms

Creates task plan → Task graph (extract → profile → lineage → policy)

Guardrail Agent100ms

Validates connection config → Config + security rules

Metadata Agent3s

Connects to source and extracts schema → Tables, columns, data types, row counts

Metadata Agent2s

Samples data and calculates stats → Distinct values, nulls, min/max, histograms

Metadata Agent4s

Enriches with LLM-generated descriptions → Business-friendly column descriptions

Evaluator Agent500ms

Runs quality checks → Quality scores (0-100) per column

Evaluator Agent300ms

Detects PII → PII labels (email, SSN, credit card, etc.)

Lineage Agent2s

Parses SQL queries from logs → Source → target relationships

Lineage Agent1s

Builds lineage graph in Neo4j → Graph edges with transformation metadata

Executor Agent500ms

Applies default policies → Access rules, retention periods, redaction rules

Guardrail Agent100ms

Audits policy application → Audit log entry

API Gateway50ms

Returns success response → Source ID + metadata summary

Volume

10K-100K records/day

Pattern

Serverless Monolith

🏗️

Architecture

AWS Lambda or Cloud Run

PostgreSQL (managed)

Redis (managed)

S3 for logs

Cost & Performance

$200/mo

per month

3-5s per source

Volume

100K-1M records/day

Pattern

Queue + Workers

🏗️

Architecture

API server (ECS/GKE)

Message queue (SQS/Pub/Sub)

Worker pool (auto-scaling)

PostgreSQL + Redis

Neo4j for lineage

Cost & Performance

$800/mo

per month

1-3s per source

Volume

1M-10M records/day

Pattern

Multi-Agent Orchestration

🏗️

Architecture

Load balancer

Agent framework (LangGraph)

Event bus (Kafka)

Distributed cache (Redis Cluster)

Sharded PostgreSQL

Neo4j cluster

Cost & Performance

$3K/mo

per month

500ms-2s per source

Recommended

Volume

10M+ records/day

Pattern

Enterprise Multi-Region

🏗️

Architecture

Kubernetes (EKS/GKE)

Multi-region event streaming (Kafka)

Distributed tracing (Jaeger)

Multi-region DB replication

CDN for UI

Dedicated ML inference cluster

Cost & Performance

$10K+/mo

per month

200-500ms per source

Key Integrations

Data Sources (Snowflake, Databricks, BigQuery, S3, etc.)

Protocol: JDBC/ODBC, REST APIs, S3 SDK

Connect via secure credentials

Query information_schema for metadata

Sample data for profiling

Extract query logs for lineage

Identity Provider (Okta, Auth0, Azure AD)

Protocol: SAML 2.0, OIDC

User authenticates via SSO

IdP returns JWT with roles/groups

API Gateway validates JWT

RBAC enforced per request

Notification Services (Slack, Email, PagerDuty)

Protocol: Webhooks, SMTP, REST APIs

Policy violation detected

Notification service triggered

Alert sent to relevant stakeholders

Audit log updated

Audit & Compliance (Splunk, Datadog, CloudWatch)

Protocol: Syslog, REST APIs, agent-based

All governance actions logged

Logs forwarded to SIEM

Real-time dashboards and alerts

Long-term retention for compliance

Security & Compliance

Failure Modes & Recovery

Failure	Fallback	Impact	SLA
Source connection timeout	Retry 3x with exponential backoff (1s, 2s, 4s) → Queue for manual review	Delayed metadata extraction, no data loss	99.5%
LLM API down (metadata enrichment)	Skip enrichment, use basic metadata only → Queue for later enrichment	Degraded UX (no descriptions), core functionality intact	99.9%
PII detector fails	Deny all access (fail-safe) → Alert admin → Manual review	Service degraded, data protected	100% (safety first)
Policy engine unavailable	Deny all access → Serve cached policy decisions (if available) → Alert admin	Service degraded, compliance maintained	99.9%
Database unavailable	Read from replica (if available) → Queue writes → Alert admin	Read-only mode, no data loss	99.9%
Lineage graph too large (>1M nodes)	Paginate graph queries → Limit depth to 10 hops → Suggest filtering	Slower lineage queries, functionality intact	99.0%
Agent orchestration deadlock	Timeout after 30s → Rollback transaction → Retry with different plan	Delayed workflow completion, no data corruption	99.5%

System Architecture

┌─────────────┐
│   Planner   │ ← Coordinates all agents
│    Agent    │
└──────┬──────┘
       │
   ┌───┴────┬────────┬──────────┬──────────┐
   │        │        │          │          │
┌──▼──┐  ┌─▼──┐  ┌──▼───┐  ┌──▼────┐  ┌──▼────┐
│Exec │  │Eval│  │Guard │  │Meta  │  │Lineage│
│Agent│  │Agent│  │ Agent│  │Agent │  │ Agent │
└──┬──┘  └─┬──┘  └──┬───┘  └──┬───┘  └──┬────┘
   │        │        │         │         │
   └────────┴────────┴─────────┴─────────┘
             │
          ┌──▼─────┐
          │  Data  │
          │ Store  │
          └────────┘

🔄Agent Collaboration Flow

Planner Agent

Receives source registration request, creates task plan (extract → profile → lineage → policy)

Guardrail Agent

Validates connection config, checks security rules → Returns validation result

Metadata Agent

Connects to source, extracts schema, samples data → Returns metadata JSON

Evaluator Agent

Runs quality checks, detects PII → Returns quality scores and PII labels

Lineage Agent

Parses SQL logs, builds lineage graph → Stores in Neo4j

Executor Agent

Applies default policies (access control, retention) → Updates policy engine

Planner Agent

Aggregates results, returns success response to user

🎭Agent Types

Reactive Agent

Low

Metadata Agent - Responds to extraction request, returns metadata

Stateless

Reflexive Agent

Medium

Guardrail Agent - Uses rules + context to make security decisions

Reads context (user role, resource sensitivity)

Deliberative Agent

High

Evaluator Agent - Plans quality checks based on data type and history

Stateful (tracks quality trends)

Orchestrator Agent

Highest

Planner Agent - Makes routing decisions, handles loops and retries

Full state management (task graph, agent status)

📈Levels of Autonomy

Tool

Human calls, agent responds

→ Monday's prompts (manual metadata extraction)

Chained Tools

Sequential execution

→ Tuesday's code (extract → validate → store)

Agent

Makes decisions, can loop

→ Evaluator Agent (decides which quality checks to run)

Multi-Agent

Agents collaborate autonomously

→ This system (Planner coordinates 5 specialized agents)

RAG vs Fine-Tuning for Metadata Enrichment

Business glossaries and data dictionaries change frequently. RAG allows daily updates without retraining. Fine-tuning requires weeks of work and $5K+ per iteration.

✅ RAG (Chosen)

Cost: $100/mo (vector DB + embeddings)

Update: Daily (add new docs to vector DB)

How:

❌ Fine-Tuning

Cost: $5K/iteration + $500/mo inference

Update: Monthly (retrain on new examples)

How:

Implementation: Vector DB (Pinecone) with company glossary, industry terms, and example descriptions. Retrieved during metadata extraction. GPT-4 generates descriptions with retrieved context.

Hallucination Detection in Metadata Enrichment

LLMs hallucinate column descriptions, inventing business context that doesn't exist.

Confidence scores from LLM (logprobs). Flag if <0.7.

Cross-reference with glossary. Flag if terms not found.

Semantic similarity check. Flag if description doesn't match column name/type.

Human review queue for flagged descriptions.

0.8% hallucination rate, 95% caught by L1-L3, 100% caught by L4

Evaluation Framework for Governance Models

PII Detection Precision

97.2%target: 98%+

PII Detection Recall

96.8%target: 95%+

Quality Score MAE

3.2 pointstarget: <5 points

Lineage Accuracy

94.1%target: 95%+

Metadata Enrichment Satisfaction

4.2/5.0target: 4.0/5.0

Testing: Shadow mode: Run new models in parallel with production for 7 days. Compare outputs. Promote if metrics improve by >5% and no regressions.

Dataset Curation for PII Detection

Collect: 50K columns from real sources - De-identified and anonymized

Clean: 42K usable (removed duplicates, invalid data) - Automated + manual review

Label: 42K labeled (PII vs non-PII) - ($$21K (data labeling service))

Augment: +8K synthetic examples - Edge case generation (rare PII types, obfuscated formats)

→ 50K high-quality labeled examples. Cohen's Kappa: 0.94 (excellent inter-annotator agreement).

Agentic RAG for Policy Enforcement

Agent iteratively retrieves policy rules based on reasoning about data context.

User requests access to 'customer_email' column. Agent reasons: 'This is PII' → RAG retrieves GDPR rules → Agent reasons: 'Need consent check' → RAG retrieves consent records → Agent generates decision: 'Allow with redaction if no consent'.

💡 Not one-shot retrieval. Agent decides what rules to fetch based on data context. Handles complex, multi-step policy logic.

Continuous Learning Loop

Tech Stack Summary

LLMs

GPT-4 (OpenAI), Claude 3.5 (Anthropic), or Gemini (Google)

Orchestration

LangGraph, CrewAI, or custom agent framework

Database

PostgreSQL (metadata), Neo4j (lineage graphs), Redis (cache)

Queue

AWS SQS, Google Pub/Sub, or Apache Kafka

Compute

Kubernetes (EKS/GKE), AWS Lambda, or Google Cloud Run

ML Inference

TensorFlow Serving, TorchServe, or AWS SageMaker

Vector DB

Pinecone, Weaviate, or Qdrant

Monitoring

Prometheus + Grafana, Datadog, or New Relic

Logging

ELK Stack (Elasticsearch, Logstash, Kibana) or Splunk

Tracing

Jaeger, Zipkin, or AWS X-Ray

Security

AWS KMS, HashiCorp Vault, Cloudflare WAF

🏗️

Need Data Governance Architecture?

We'll audit your data landscape, design a governance system, and help you scale from 10K to 10M+ records/day.

No part of this content may be reproduced, distributed, or transmitted in any form without prior written permission.

Data Governance System Architecture 🏗️

From prompts to enterprise data governance platform.

Key Assumptions

System Requirements

Functional

Non-Functional (SLOs)

Agent Layer

planner

executor

evaluator

guardrail

metadata

lineage

ML Layer

Feature Store

Model Registry

Observability Stack

Deployment Variants

Startup Architecture

Infrastructure

Risks & Mitigations

⚠️ Source credentials leaked or compromised

⚠️ PII detector fails, exposing sensitive data

⚠️ Lineage graph becomes too large to query

⚠️ Agent orchestration deadlock or infinite loop

⚠️ Multi-tenant data leakage (tenant A sees tenant B's data)

⚠️ Cost explosion from LLM API usage

⚠️ Compliance drift (policies not enforced consistently)

Evolution Roadmap

0-3 months (MVP)

3-6 months (Scale)

6-12 months (Enterprise)

Complete Systems Architecture

Presentation

API Gateway

Agent Layer

ML Layer

Integration

Data

External

Observability

Security

Sequence Diagram - New Data Source Registration

Data Flow - Source Registration to Policy Enforcement

Scaling Patterns

Key Integrations

Data Sources (Snowflake, Databricks, BigQuery, S3, etc.)

Identity Provider (Okta, Auth0, Azure AD)

Notification Services (Slack, Email, PagerDuty)

Audit & Compliance (Splunk, Datadog, CloudWatch)

Security & Compliance

Failure Modes & Recovery

Multi-Agent Architecture

🔄Agent Collaboration Flow

🎭Agent Types

Reactive Agent

Reflexive Agent

Deliberative Agent

Orchestrator Agent

📈Levels of Autonomy

Advanced ML/AI Patterns

RAG vs Fine-Tuning for Metadata Enrichment

Hallucination Detection in Metadata Enrichment

Evaluation Framework for Governance Models

Dataset Curation for PII Detection

Agentic RAG for Policy Enforcement

Continuous Learning Loop

Tech Stack Summary

Need Data Governance Architecture?