Growth Retention System Architecture: Scale 10K Users

🎯This Week's Journey

From prompts to production retention engine.

Monday: 3 core prompts (churn prediction, engagement scoring, campaign generation). Tuesday: automated agent code. Wednesday: team workflows (Data → Growth → Product). Thursday: complete technical architecture. Agents, ML pipeline, scaling patterns, and compliance for 10,000+ users daily.

📋

Key Assumptions

Monitor 1K-100K active users with 5-20% monthly churn baseline

Real-time engagement scoring (<500ms p95 latency required)

Daily batch churn prediction + hourly incremental updates

GDPR/CCPA compliance for user data, HIPAA optional for healthcare SaaS

Integration with existing product analytics (Segment, Amplitude, Mixpanel)

System Requirements

Functional

Ingest user events (pageviews, feature usage, purchases) from multiple sources
Compute engagement scores in real-time (<500ms) and churn risk daily
Generate personalized retention campaigns (email, in-app, push) via LLM agents
A/B test campaign variants and measure lift in retention metrics
Provide dashboards for Growth team (churn trends, campaign performance, cohort analysis)
Support multi-tenancy for enterprise customers (data isolation, custom models)
Audit trail for all ML predictions and campaign decisions (HIPAA compliance)

Non-Functional (SLOs)

latency p95 ms500

freshness min60

availability percent99.5

churn prediction accuracy0.85

engagement score drift threshold0.05

💰 Cost Targets: {"per_user_per_month_usd":0.15,"ml_inference_per_1k_users_usd":2.5,"campaign_generation_per_user_usd":0.03}

Agent Layer

planner_agent

Decomposes retention tasks, selects tools, decides campaign strategy

🔧 segment_classifier, campaign_recommender, timing_optimizer

⚡ Recovery: If tool fails: retry 3x with exponential backoff, If all retries fail: route to manual review queue, Log failure to audit trail with context

executor_agent

Executes retention workflows (score computation, campaign generation, delivery)

🔧 feature_store.get_features(), ml_model.predict(), llm_api.generate_campaign(), email_api.send()

⚡ Recovery: If feature fetch fails: use cached features (staleness <1hr), If ML model fails: fallback to rule-based scoring, If LLM API fails: use pre-generated template, If email send fails: queue for retry (max 3x)

evaluator_agent

Validates campaign quality, checks for hallucinations, measures effectiveness

🔧 content_classifier (toxicity, relevance), brand_checker, hallucination_detector, engagement_predictor

⚡ Recovery: If quality score <0.6: reject and regenerate, If hallucination detected: flag for human review, If brand violation: auto-reject with explanation

guardrail_agent

Enforces PII redaction, policy compliance, safety filters

🔧 pii_detector (AWS Comprehend/Presidio), policy_engine (OPA/custom rules), content_filter (profanity, sensitive topics)

⚡ Recovery: If PII detected: auto-redact and log, If policy violation: block delivery and alert, If redaction fails: route to manual review

campaign_generator_agent

Generates personalized retention campaigns using LLM

🔧 llm_api (GPT-4/Claude), template_selector, personalization_engine

⚡ Recovery: If LLM timeout: use cached template with dynamic fields, If generation fails: fallback to rule-based template, If content too generic: retry with more context

scoring_agent

Real-time engagement scoring and churn prediction

🔧 feature_store.get_online_features(), ml_model.predict() (XGBoost/LightGBM), feature_importance_explainer

⚡ Recovery: If feature fetch fails: use last known score (staleness <24hr), If model inference fails: fallback to rule-based heuristic, If score anomaly detected: flag for review

ML Layer

Feature Store

Update: Real-time (streaming) + Daily batch refresh

• user_tenure_days
• events_last_7d
• events_last_30d
• feature_usage_breadth (distinct features used)
• feature_usage_depth (avg usage per feature)
• days_since_last_login
• purchase_count_lifetime
• support_tickets_last_90d
• cohort_retention_rate
• referral_count

Model Registry

Strategy: Semantic versioning (major.minor.patch), Git-backed

• churn_classifier_v3
• engagement_scorer_v2
• campaign_recommender_v1

Observability Stack

Real-time monitoring, tracing & alerting

0 active

SOURCES

Apps, Services, Infra

COLLECTION

10 Metrics

PROCESSING

Aggregate & Transform

DASHBOARDS

4 Views

ALERTS

Enabled

📊Metrics(10)

📝Logs(Structured)

🔗Traces(Distributed)

retention_campaign_sent_total

✓

engagement_score_p95

✓

churn_prediction_accuracy

✓

llm_latency_p95_ms

✓

email_delivery_rate

✓

feature_staleness_seconds

✓

Deployment Variants

🚀

Startup Architecture

Fast to deploy, cost-efficient, scales to 100 competitors

Infrastructure

✓

Serverless (Lambda/Cloud Functions)

✓

Managed services (RDS, DynamoDB, SQS)

✓

OpenAI/Anthropic APIs (no self-hosting)

✓

Single-region deployment

✓

Shared infrastructure (multi-tenant DB)

→Fast time-to-market (2-4 weeks)

→Low operational overhead

→Cost: $200-600/mo for 100-1K users/day

→99.5% SLA acceptable

→No custom ML models (use pre-trained)

Risks & Mitigations

⚠️ Model accuracy degrades over time (concept drift)

High (user behavior changes)

✓ Mitigation: Weekly offline evaluation. Auto-retrain if accuracy drops >5%. A/B test new models before full rollout.

⚠️ LLM generates inappropriate content (toxicity, bias)

Medium (LLMs are imperfect)

✓ Mitigation: Guardrail Agent with toxicity filter. Human review for flagged content. Content policy enforcement via OPA.

⚠️ PII leakage to LLM provider

Low (with proper redaction)

✓ Mitigation: PII detection before LLM call. Audit trail for all LLM requests. DPA with LLM provider. Fail-closed if redaction fails.

⚠️ Cost overruns from LLM API usage

Medium (usage spikes)

✓ Mitigation: Rate limiting per user. Cost budgets with alerts. Fallback to templates if budget exceeded. Monthly cost review.

⚠️ Feature store staleness (outdated scores)

Medium (infrastructure failures)

✓ Mitigation: Cache with TTL <1hr. Monitoring for staleness. Alert if staleness >30min. Fallback to rule-based if cache miss.

⚠️ Email deliverability issues (spam filters)

Medium (reputation management)

✓ Mitigation: SPF/DKIM/DMARC setup. Monitor bounce/complaint rates. Warm up IP addresses. Use reputable email service (SendGrid).

⚠️ Agent orchestration failures (infinite loops, deadlocks)

Low (with proper testing)

✓ Mitigation: Circuit breakers for agents. Max retry limits (3x). Timeout for each agent step (5s). Dead-letter queue for failed tasks.

🧬

Evolution Roadmap

Progressive transformation from MVP to scale

🌱

Phase 1Weeks 1-12

Phase 1: Foundation (0-3 months)

Deploy basic retention system (100-1K users/day)

Implement 3 core agents (Scoring, Campaign, Guardrail)

Integrate with product analytics (Segment/Amplitude)

Launch first A/B test (LLM campaigns vs manual)

Complexity Level

▼

🌿

Phase 2Weeks 13-24

Phase 2: Scale (3-6 months)

Scale to 1K-10K users/day

Add Planner, Executor, Evaluator agents

Deploy feature store (Feast) + model registry (MLflow)

Improve churn prediction to 85% accuracy

Complexity Level

▼

🌳

Phase 3Weeks 25-52

Phase 3: Enterprise (6-12 months)

Scale to 10K+ users/day with 99.9% SLA

Multi-region deployment (US + EU)

Self-hosted ML models (GPU instances)

Full HIPAA/SOC2 compliance

Complexity Level

🚀Production Ready

🏗️

Complete Systems Architecture

9-layer architecture from user events to retention campaigns

🌐

Presentation

4 components

Growth Dashboard (React)

Campaign Builder UI

Analytics Embed (iframe)

Mobile SDK

⚙️

API Gateway

4 components

Load Balancer (ALB/NGINX)

Rate Limiter (Redis)

Auth Service (OAuth2/OIDC)

API Router

💾

Agent Layer

6 components

Planner Agent (task decomposition)

Executor Agent (workflow engine)

Evaluator Agent (quality checks)

Guardrail Agent (PII redaction, policy)

Campaign Generator Agent (LLM-powered)

Scoring Agent (real-time inference)

🔌

ML Layer

6 components

Feature Store (Feast/Tecton)

Model Registry (MLflow)

Training Pipeline (Airflow)

Inference Service (TorchServe/TFServing)

Evaluation Framework

Drift Detector

📊

Integration

5 components

Event Ingestion (Kafka/Kinesis)

Analytics Connector (Segment API)

Email Provider (SendGrid/SES)

Push Notification (FCM/APNS)

CRM Sync (Salesforce/HubSpot)

🌐

Data

5 components

Event Stream (Kafka)

OLTP Database (PostgreSQL)

OLAP Warehouse (Snowflake/BigQuery)

Feature Cache (Redis)

Model Artifacts (S3)

⚙️

External

4 components

LLM API (OpenAI/Anthropic)

Analytics Platform (Amplitude)

Email Service (SendGrid)

Cloud Provider (AWS/GCP)

💾

Observability

5 components

Metrics (Prometheus/Datadog)

Logs (ELK/Loki)

Traces (Jaeger/Tempo)

ML Monitoring (Arize/Evidently)

Alerting (PagerDuty)

🔌

Security

5 components

IAM (Keycloak/Auth0)

Secrets Manager (Vault/KMS)

Audit Log (CloudWatch/Splunk)

PII Redactor

WAF (Cloudflare/AWS WAF)

🔄

Sequence Diagram - Retention Campaign Flow

Automated data flow every hour

Step 0 of 14

Data Flow - Event to Campaign

User event → Scoring → Campaign generation → Delivery in <2 seconds

User0 ms

Triggers event (feature_used) → Event JSON

Event Tracker SDK5 ms

Publishes to Kafka → Event stream

Kafka Consumer50 ms

Reads event, updates feature store → Updated features

Scoring Agent150 ms

Fetches features, runs ML model → Engagement score (0.32)

Planner Agent50 ms

Analyzes score, decides to send campaign → Task plan

Executor Agent10 ms

Calls Campaign Generator Agent → Campaign request

Campaign Agent800 ms

Generates personalized email via LLM → Email content

Evaluator Agent100 ms

Validates quality, checks hallucinations → Quality score (0.89)

Guardrail Agent50 ms

Redacts PII, enforces policy → Approved content

Email Service200 ms

Sends email via SendGrid → Delivery receipt

Audit Logger20 ms

Records decision + content → Audit entry

Volume

0-100 users/day

Pattern

Serverless Monolith

🏗️

Architecture

Lambda/Cloud Functions for agents

DynamoDB or Firestore

SQS/Pub-Sub for async tasks

Managed LLM APIs (OpenAI/Anthropic)

Cost & Performance

$200/mo

per month

1-2 sec p95

Volume

100-1K users/day

Pattern

Queue + Workers

🏗️

Architecture

API server (ECS/Cloud Run)

Redis for caching + queue

Worker pool (3-5 instances)

PostgreSQL RDS

Feast feature store

Cost & Performance

$600/mo

per month

500-800 ms p95

Volume

1K-10K users/day

Pattern

Multi-Agent Orchestration

🏗️

Architecture

Kubernetes (EKS/GKE) with autoscaling

Kafka for event streaming

Feast + Redis for online features

TorchServe for ML models

LangGraph for agent orchestration

PostgreSQL + Snowflake

Cost & Performance

$2K/mo

per month

300-500 ms p95

Recommended

Volume

10K+ users/day

Pattern

Enterprise Multi-Region

🏗️

Architecture

Multi-region Kubernetes

Kafka + Flink for stream processing

Tecton (managed feature store)

Multi-LLM failover (OpenAI + Anthropic + self-hosted)

Snowflake + Databricks

Global load balancer

Cost & Performance

$8K+/mo

per month

200-400 ms p95

Key Integrations

Product Analytics (Segment/Amplitude)

Protocol: REST API + Webhooks

User event → Segment API

Segment → Kafka (via webhook)

Kafka → Feature Store

Feature Store → Scoring Agent

Email Service (SendGrid/AWS SES)

Protocol: SMTP + REST API

Campaign Agent → Email Service API

Email Service → User inbox

Email Service → Webhook (open/click events)

Webhook → Analytics pipeline

CRM (Salesforce/HubSpot)

Protocol: REST API (OAuth 2.0)

Churn prediction → CRM sync job

CRM API → Update contact field (churn_risk)

Sales team views in CRM dashboard

LLM API (OpenAI/Anthropic)

Protocol: REST API

Campaign Agent → LLM API (generate)

LLM API → Campaign content

Evaluator Agent → LLM API (judge quality)

LLM API → Quality score

Feature Store (Feast/Tecton)

Protocol: gRPC (online) + Parquet (offline)

Kafka → Feature computation job

Job → Write to online store (Redis)

Scoring Agent → Fetch features (gRPC)

Daily batch → Write to offline store (Snowflake)

Security & Compliance

Failure Modes & Fallbacks

Failure	Fallback	Impact	SLA
LLM API down (OpenAI/Anthropic)	→ Use cached templates with dynamic personalization	Degraded personalization, campaigns still sent	99.5%
ML model inference timeout	→ Use rule-based scoring (engagement = events_last_7d / 10)	Lower accuracy, no churn prediction	99.0%
Feature store unavailable	→ Use cached features (staleness <1hr acceptable)	Slightly outdated scores, campaigns still sent	99.5%
Email delivery service down	→ Queue campaigns for retry (max 3x over 24hr)	Delayed delivery, no data loss	99.0%
PII detection service fails	→ Block all campaign generation until service recovers	No campaigns sent (safety first)	100% compliance
Kafka broker down	→ Buffer events in API server (max 1hr)	Delayed feature updates, eventual consistency	99.5%
Database connection pool exhausted	→ Read from replica, queue writes	Read-only mode for dashboards	99.0%

System Architecture

┌─────────────────┐
│   Orchestrator  │ ← Coordinates all agents
│  (LangGraph)    │
└────────┬────────┘
         │
    ┌────┴─────┬──────────┬───────────┬───────────┬──────────┐
    │          │          │           │           │          │
┌───▼───┐  ┌──▼───┐  ┌───▼────┐  ┌──▼─────┐  ┌──▼──────┐  ┌▼────────┐
│Planner│  │Scorer│  │Campaign│  │Evaluator│  │Guardrail│  │Executor │
│ Agent │  │Agent │  │ Agent  │  │  Agent  │  │  Agent  │  │  Agent  │
└───┬───┘  └──┬───┘  └───┬────┘  └──┬──────┘  └──┬──────┘  └┬────────┘
    │          │          │           │            │          │
    └──────────┴──────────┴───────────┴────────────┴──────────┘
                                │
                         ┌──────▼──────┐
                         │   Actions   │
                         │ (Email/Push)│
                         └─────────────┘

🔄Agent Collaboration Flow

Orchestrator

Receives user event (feature_used), routes to Scoring Agent

Scoring Agent

Fetches features from feature store, runs ML model → Returns engagement_score=0.32 (at-risk)

Planner Agent

Analyzes score, decides: 'User at-risk, send re-engagement campaign' → Creates task plan

Executor Agent

Executes task plan, calls Campaign Generator Agent with user context

Campaign Generator Agent

Retrieves similar campaigns (RAG), generates personalized email via LLM → Returns draft

Evaluator Agent

Validates quality (hallucination check, relevance score) → Returns quality_score=0.89 (approved)

Guardrail Agent

Scans for PII, enforces content policy → Redacts PII, approves content

Executor Agent

Sends email via SendGrid API, logs to audit trail

Orchestrator

Monitors delivery, schedules follow-up if no engagement in 3 days

🎭Agent Types

Reactive Agent

Low

Scoring Agent - Responds to event, returns score

Stateless

Reflexive Agent

Medium

Campaign Generator - Uses context (user profile, past campaigns)

Reads context

Deliberative Agent

High

Planner Agent - Plans multi-step campaign strategy

Stateful (maintains plan)

Orchestrator Agent

Highest

Main Orchestrator - Coordinates all agents, handles loops/retries

Full state management

📈Levels of Autonomy

Tool

Human calls, agent responds (no decisions)

→ Monday's prompts (manual execution)

Chained Tools

Sequential execution (predefined flow)

→ Tuesday's code (Score → Generate → Send)

Agent

Makes decisions, can loop/retry

→ Planner Agent (decides campaign strategy)

Multi-Agent

Agents collaborate autonomously, self-correct

→ This system (6 agents working together)

RAG vs Fine-Tuning for Campaign Generation

Campaign best practices evolve weekly. RAG allows daily updates without retraining. Fine-tuning would require monthly retraining ($5K/mo) vs RAG ($200/mo for vector DB).

✅ RAG (Chosen)

Cost: $200/mo (Pinecone)

Update: Daily (add new campaign examples)

How:

❌ Fine-Tuning

Cost: $5K/mo (retraining)

Update: Monthly

How:

Implementation: Vector DB (Pinecone) with 10K high-performing campaign examples. Retrieved top-5 similar campaigns during generation. Reranker (Cohere) to select best examples.

Hallucination Detection for Campaigns

LLMs hallucinate features ('Try our new AI assistant' when no such feature exists)

Confidence scores from LLM (<0.7 = flag)

Cross-reference product feature list (API call)

Fact-checking layer (LLM-as-judge with product docs)

Human review queue for flagged campaigns

0.8% hallucination rate, 98% caught by L1-L3

Evaluation Framework

Churn Prediction Accuracy

87.2%target: 85%+

Engagement Score MAE

0.08target: <0.1

Campaign Open Rate Lift

+18%target: +15% vs baseline

Hallucination Rate

0.8%target: <1%

PII Leakage Rate

0%target: 0%

Testing: Shadow mode: 1000 users scored in parallel (ML vs rule-based). A/B test: 10% traffic to new model for 2 weeks. Champion-challenger: gradual rollout (10% → 50% → 100%).

Dataset Curation

Collect: 50K user events + 5K campaigns - Export from production DB (anonymized)

Clean: 42K usable events - Remove duplicates, filter bots, validate schema

Label: 42K labeled (churned/retained) - ($$8K (mechanical turk + expert review))

Augment: +8K synthetic examples - SMOTE for minority class (churned users)

→ 50K high-quality training examples. 80/10/10 train/val/test split.

Agentic RAG for Campaign Personalization

Agent iteratively retrieves context based on reasoning

User segment: 'power user, low recent activity' → RAG retrieves 'power user re-engagement campaigns' → Agent reasons 'need feature usage data' → RAG retrieves 'feature adoption stats' → Campaign generated: 'You used to love [Feature X]. Check out these new updates.'

💡 Not one-shot retrieval. Agent decides what context it needs based on user profile.

Multi-Armed Bandit for Campaign Optimization

Tech Stack Summary

LLMs

GPT-4, Claude 3.5 Sonnet, or self-hosted Llama 3

Agent Orchestration

LangGraph, CrewAI, or Temporal

ML Framework

XGBoost, LightGBM, PyTorch

Feature Store

Feast (OSS) or Tecton (managed)

Model Registry

MLflow or Weights & Biases

Event Streaming

Kafka or AWS Kinesis

Database

PostgreSQL (OLTP), Snowflake (OLAP)

Cache

Redis or Memcached

Queue

SQS, RabbitMQ, or Kafka

Compute

Kubernetes (EKS/GKE) or Lambda/Cloud Functions

Monitoring

Prometheus, Datadog, Grafana, Arize (ML)

Security

Vault (secrets), Presidio (PII), OPA (policy)

Email Service

SendGrid or AWS SES

Analytics

Segment, Amplitude, or Mixpanel

🏗️

Need Architecture Review?

We'll audit your retention system, identify bottlenecks, and show you how to scale 10x while cutting costs 30%.

No part of this content may be reproduced, distributed, or transmitted in any form without prior written permission.

Retention System Architecture 🏗️

From prompts to production retention engine.

Key Assumptions

System Requirements

Functional

Non-Functional (SLOs)

Agent Layer

planner_agent

executor_agent

evaluator_agent

guardrail_agent

campaign_generator_agent

scoring_agent

ML Layer

Feature Store

Model Registry

Observability Stack

Deployment Variants

Startup Architecture

Infrastructure

Risks & Mitigations

⚠️ Model accuracy degrades over time (concept drift)

⚠️ LLM generates inappropriate content (toxicity, bias)

⚠️ PII leakage to LLM provider

⚠️ Cost overruns from LLM API usage

⚠️ Feature store staleness (outdated scores)

⚠️ Email deliverability issues (spam filters)

⚠️ Agent orchestration failures (infinite loops, deadlocks)

Evolution Roadmap

Phase 1: Foundation (0-3 months)

Phase 2: Scale (3-6 months)

Phase 3: Enterprise (6-12 months)

Complete Systems Architecture

Presentation

API Gateway

Agent Layer

ML Layer

Integration

Data

External

Observability

Security

Sequence Diagram - Retention Campaign Flow

Data Flow - Event to Campaign

Scaling Patterns

Key Integrations

Product Analytics (Segment/Amplitude)

Email Service (SendGrid/AWS SES)

CRM (Salesforce/HubSpot)

LLM API (OpenAI/Anthropic)

Feature Store (Feast/Tecton)

Security & Compliance

Failure Modes & Fallbacks

Multi-Agent Architecture

🔄Agent Collaboration Flow

🎭Agent Types

Reactive Agent

Reflexive Agent

Deliberative Agent

Orchestrator Agent

📈Levels of Autonomy

Advanced ML/AI Patterns

RAG vs Fine-Tuning for Campaign Generation

Hallucination Detection for Campaigns

Evaluation Framework

Dataset Curation

Agentic RAG for Campaign Personalization

Multi-Armed Bandit for Campaign Optimization

Tech Stack Summary

Need Architecture Review?