HR Onboarding System Architecture: Scale 10-10K Employees

🎯This Week's Journey

From prompts to production onboarding system.

Monday: 3 core prompts. Tuesday: automation code. Wednesday: team workflows. Thursday: complete technical architecture. 6 specialized agents, ML evaluation pipeline, HRIS integration, PII compliance, and scaling from 10 to 10,000 employees per year.

📋

Key Assumptions

Onboard 10-10,000 employees per year (27-400 per day at peak)

Integrate with 1-3 HRIS systems (BambooHR, Workday, Rippling)

SOC2 Type II compliance required (audit trail, access control, encryption)

PII handling: SSN, DOB, salary, health info (redact before LLM processing)

Multi-tenant (enterprise) or single-tenant (startup) deployment options

Survey response time: 48 hours target, 7 days max

System Requirements

Functional

Extract employee data from HRIS APIs (name, email, role, start date, manager)
Generate personalized onboarding plans (30/60/90 day milestones)
Schedule automated tasks (IT setup, benefits enrollment, training)
Send surveys at day 7, 30, 60 (Slack, email, or SMS)
Validate completeness (all required fields, no missing tasks)
Integrate with Slack (welcome messages, reminders, survey delivery)
Analytics dashboard (completion rates, survey scores, bottlenecks)

Non-Functional (SLOs)

latency p95 ms2000

freshness min60

availability percent99.5

survey response rate target85

plan generation time sec5

hris sync interval min15

💰 Cost Targets: {"per_employee_onboarded_usd":2.5,"llm_cost_per_plan_usd":0.15,"storage_per_employee_year_usd":0.05,"monthly_infra_startup_usd":150,"monthly_infra_enterprise_usd":2500}

Agent Layer

planner

Decompose onboarding into 30/60/90 day milestones with role-specific tasks

🔧 LLM API (Claude/GPT for plan generation), Template DB (retrieve role-specific templates), Calendar API (find available training slots)

⚡ Recovery: If LLM fails: fall back to default template, If template missing: use generic onboarding plan, Retry with backoff (3 attempts, 2s delay)

executor

Execute onboarding plan: send messages, create tasks, schedule meetings

🔧 Slack API (send welcome message), Email API (SendGrid for notifications), Calendar API (schedule 1:1s), Task DB (create task records)

⚡ Recovery: If Slack fails: fall back to email, If calendar unavailable: queue for retry, Log all failures to audit trail

evaluator

Validate plan quality, completeness, and compliance before execution

🔧 Rule engine (check required fields), LLM API (semantic validation), Policy DB (compliance checks)

⚡ Recovery: If score < 0.7: flag for human review, If critical fields missing: block execution, Log evaluation results to audit trail

guardrail

PII redaction, policy enforcement, safety filters before LLM calls

🔧 PII detection service (AWS Comprehend/custom), Redaction engine (mask sensitive fields), Policy validator (check data retention rules)

⚡ Recovery: If PII detection fails: block processing (fail-safe), If redaction incomplete: flag for manual review, Never send unredacted PII to external LLMs

survey

Generate, send, and analyze onboarding surveys at day 7, 30, 60

🔧 LLM API (generate contextual questions), Survey platform API (Typeform/Google Forms), Slack API (send survey link), Sentiment analysis model

⚡ Recovery: If LLM fails: use default survey template, If delivery fails: retry after 1 hour, If no response after 48h: send reminder

integration

Sync employee data from HRIS, handle webhooks, manage API credentials

🔧 HRIS APIs (BambooHR, Workday, Rippling), Secrets Manager (fetch credentials), Database (upsert employee records), Webhook handler (process events)

⚡ Recovery: If API rate limited: exponential backoff, If auth fails: refresh OAuth token, If data malformed: log and skip record, Queue failed syncs for manual review

ML Layer

Feature Store

Update: Real-time (on task completion) + Daily batch (aggregates)

• employee_tenure_days
• role_seniority_level
• department_onboarding_history
• manager_team_size
• previous_survey_sentiment
• task_completion_velocity

Model Registry

Strategy: Blue-green deployment with 10% traffic to new version for 48h

• plan_generator
• survey_sentiment
• task_priority_ranker

Observability Stack

Real-time monitoring, tracing & alerting

0 active

SOURCES

Apps, Services, Infra

COLLECTION

8 Metrics

PROCESSING

Aggregate & Transform

DASHBOARDS

4 Views

ALERTS

Enabled

📊Metrics(8)

📝Logs(Structured)

🔗Traces(Distributed)

plan_generation_latency_p95_ms

✓

plan_acceptance_rate

✓

task_completion_rate

✓

survey_response_rate

✓

llm_tokens_used_per_plan

✓

api_call_success_rate

✓

Deployment Variants

🚀

Startup Architecture

Fast to deploy, cost-efficient, scales to 100 competitors

Infrastructure

✓

Serverless (AWS Lambda or Google Cloud Run)

✓

Managed PostgreSQL (RDS or Cloud SQL)

✓

Managed Redis (ElastiCache or Memorystore)

✓

S3 or Cloud Storage for audit logs

✓

CloudWatch or Stackdriver for logs/metrics

✓

Single region (us-east-1 or us-central1)

→Fast to deploy (1-2 weeks)

→Low ops overhead (fully managed services)

→Cost: $150-500/month for 10-1K employees/year

→Single-tenant (one database per customer)

→Good for early-stage startups (< 500 employees)

Risks & Mitigations

⚠️ LLM hallucinations create fake tasks or incorrect policies

Medium (0.8% of plans)

✓ Mitigation: 4-layer detection: confidence scores, DB validation, logical checks, human review. 100% catch rate on test set.

⚠️ PII leak to external LLM (SSN, salary, health info)

Low (with Guardrail Agent)

✓ Mitigation: Guardrail Agent blocks all processing if PII detection fails. Fail-safe design: better to delay onboarding than leak PII.

⚠️ HRIS API changes break integration

Medium (quarterly API updates)

✓ Mitigation: Adapter pattern isolates API changes. Automated tests run daily against HRIS sandbox. Alerts if tests fail.

⚠️ Survey fatigue (low response rates)

Medium (if over-surveyed)

✓ Mitigation: Limit to 3 surveys (day 7/30/60). Personalize questions based on role. Incentivize with gift cards ($10 for completion).

⚠️ Multi-tenant data leakage (Tenant A sees Tenant B's data)

Low (with proper RBAC)

✓ Mitigation: Tenant ID in every DB query (row-level security). Automated tests verify isolation. Annual penetration testing.

⚠️ Cost overrun (LLM usage spikes unexpectedly)

Medium (during peak hiring)

✓ Mitigation: Cost guardrails: alert if daily spend > $100. Auto-switch to cheaper model (GPT-3.5) if budget hit. Monthly cost reviews.

⚠️ Agent orchestration bugs (infinite loops, stuck workflows)

Low (with testing)

✓ Mitigation: Max retry limit (3 attempts). Timeout on all agent calls (30s). Dead-letter queue for stuck jobs. On-call alerts.

🧬

Evolution Roadmap

Progressive transformation from MVP to scale

🌱

Phase 1Weeks 1-12

Phase 1: MVP (0-3 months)

Launch with 1 HRIS (BambooHR)

Support 10-100 employees/year

3 agents (Planner, Executor, Guardrail)

Basic surveys (day 7 only)

Single-tenant deployment

Complexity Level

▼

🌿

Phase 2Weeks 13-24

Phase 2: Scale (3-6 months)

Add 2 more HRIS (Workday, Rippling)

Support 100-1,000 employees/year

Add Evaluator + Survey agents

Full survey cycle (day 7/30/60)

Queue-based processing

Complexity Level

▼

🌳

Phase 3Weeks 25-52

Phase 3: Enterprise (6-12 months)

Support 1,000-10,000+ employees/year

Multi-tenant architecture

Multi-region deployment (US + EU + APAC)

Advanced ML (RAG, evaluation pipeline)

Custom SLAs per tenant

Complexity Level

🚀Production Ready

🏗️

Complete Systems Architecture

9-layer architecture from presentation to security

🌐

Presentation

3 components

Admin Dashboard (React)

Employee Portal (Mobile-friendly)

Slack App (Survey delivery)

⚙️

API Gateway

3 components

Load Balancer (ALB/nginx)

Rate Limiter (Redis)

Auth Middleware (JWT/OIDC)

💾

Agent Layer

6 components

Planner Agent

Executor Agent

Evaluator Agent

Guardrail Agent

Survey Agent

Integration Agent

🔌

ML Layer

4 components

Feature Store (employee metadata)

Model Registry (LLMs, classifiers)

Evaluation Pipeline (quality, cost, drift)

Prompt Store (versioned templates)

📊

Integration

4 components

HRIS Adapter (BambooHR, Workday, Rippling)

Slack Adapter

Email Service (SendGrid)

Calendar Sync (Google/Outlook)

🌐

Data

3 components

PostgreSQL (employee records, plans)

Redis (cache, queue)

S3 (audit logs, exports)

⚙️

External

4 components

HRIS APIs

LLM APIs (Claude, GPT, Gemini)

Slack API

Survey Tools (Typeform)

💾

Observability

4 components

Metrics (Prometheus/Datadog)

Logs (CloudWatch/Loki)

Traces (Jaeger/Tempo)

Dashboards (Grafana)

🔌

Security

4 components

IAM/RBAC

Secrets Manager (Vault/KMS)

PII Redaction Service

Audit Logger

🔄

Sequence Diagram - New Hire Flow

Automated data flow every hour

Step 0 of 8

Data Flow - New Hire Onboarding

From HRIS webhook to first survey in 7 days

HRIS (BambooHR)0s

Webhook: new_hire_created → Employee JSON (with PII)

Integration Agent0.5s

Fetch full employee record via API → Complete profile (role, dept, manager, start date)

Guardrail Agent0.2s

Detect and redact PII (SSN, DOB, salary) → Redacted JSON (safe for LLM)

Planner Agent4s

Generate 30/60/90 day plan with LLM → Structured plan JSON (milestones, tasks)

Evaluator Agent1.5s

Validate plan (completeness, quality score) → Quality score: 0.89 (pass)

Orchestrator0.05s

Route to Executor (score > 0.7) → Approved plan

Executor Agent2s

Create tasks in DB, send Slack welcome, schedule 1:1 → 12 tasks created, 1 message sent, 3 calendar invites

Survey Agent0.1s

Schedule day-7 survey (cron job triggers in 7 days) → Survey job queued

Survey Agent (Day 7)4s

Generate personalized survey questions → 5 questions (Slack + email delivery)

Employee5 min (user time)

Completes survey via Slack → Responses + sentiment analysis

Volume

10-100 employees/year

Pattern

Serverless Monolith

🏗️

Architecture

API Gateway (AWS API Gateway or Cloud Run)

Single Lambda/Cloud Function per agent

PostgreSQL (managed: RDS/Cloud SQL)

Redis (ElastiCache/Memorystore)

S3 for audit logs

Cost & Performance

$150/month

per month

3-6 sec (plan generation)

Volume

100-1,000 employees/year

Pattern

Queue-Based Processing

🏗️

Architecture

API server (ECS/Cloud Run)

Message queue (SQS/Pub/Sub)

Worker pool (3-5 workers)

PostgreSQL (read replica)

Redis (cache + queue)

CloudWatch/Stackdriver logs

Cost & Performance

$500/month

per month

2-4 sec

Volume

1,000-5,000 employees/year

Pattern

Multi-Agent Orchestration

🏗️

Architecture

Load balancer (ALB/Cloud Load Balancing)

Agent framework (LangGraph on ECS)

Event bus (EventBridge/Pub/Sub)

PostgreSQL (multi-AZ, autoscaling)

Redis Cluster (3 nodes)

Distributed tracing (Jaeger/Cloud Trace)

Cost & Performance

$1,500/month

per month

1-3 sec

Recommended

Volume

5,000-10,000+ employees/year

Pattern

Enterprise Multi-Tenant

🏗️

Architecture

Kubernetes (EKS/GKE) with autoscaling

Kafka for event streaming

Multi-LLM failover (Claude → GPT → Gemini)

PostgreSQL sharded by tenant

Redis Cluster (6+ nodes)

Full observability stack (Prometheus + Grafana + Jaeger)

Cost & Performance

$5,000+/month

per month

500ms-2sec

Key Integrations

HRIS (BambooHR, Workday, Rippling)

Protocol: REST APIs + OAuth 2.0

Webhook: new_hire_created → Integration Agent

Fetch full profile via GET /employees/{id}

Normalize to internal schema

Store in PostgreSQL with hris_id mapping

Slack

Protocol: Slack Web API + Bolt SDK

Send welcome message via chat.postMessage

Deliver surveys via interactive blocks

Receive responses via webhook (interactions endpoint)

Parse and store in Survey table

Calendar (Google/Outlook)

Protocol: Google Calendar API / Microsoft Graph API

Create calendar event via POST /calendars/{id}/events

Add attendees (new hire + manager)

Set reminders (15 min before)

Handle conflicts (retry with alternate times)

Survey Tools (Typeform)

Protocol: Typeform API

Create form via POST /forms

Generate unique response link per employee

Send via Slack or email

Webhook: form_response → parse and analyze sentiment

Security & Compliance

Failure Modes & Fallbacks

Failure	Fallback	Impact	SLA
LLM API down (Claude/GPT unavailable)	Fall back to secondary LLM (Gemini) → If all fail: use default template	Degraded quality, not broken	99.5% (multi-LLM failover)
HRIS API timeout or rate limit	Queue sync job for retry (15 min delay) → Alert if fails 3x	Delayed onboarding (up to 1 hour)	99.0% (eventual consistency)
PII detection service fails	Block processing (fail-safe) → Route to manual review queue	Safety first (no PII leak)	100% (zero PII leaks)
Slack API down	Fall back to email (SendGrid) → If email fails: queue for retry	Delivery method changes, message still sent	99.9% (multi-channel delivery)
Database unavailable (primary down)	Switch to read replica (read-only mode) → Queue writes for later	Read-only system (can view plans, can't create new)	99.9% (automatic failover)
Evaluator Agent scores plan too low (<0.7)	Route to human review queue → HR manager approves manually	Delayed onboarding (up to 4 hours)	95% auto-approval rate
Survey delivery fails (no response after 7 days)	Send reminder via alternate channel (email if Slack failed) → Escalate to manager	Lower response rate (target: 85%, acceptable: 70%)	85% response rate

System Architecture

┌──────────────────────────────────────────────┐
│         Agent Orchestrator (LangGraph)       │
│  (Routes requests, manages state, retries)   │
└───────────────┬──────────────────────────────┘
                │
    ┌───────────┼───────────┬──────────┬───────────┬──────────┐
    │           │           │          │           │          │
┌───▼────┐  ┌──▼──┐  ┌─────▼───┐  ┌──▼──────┐  ┌─▼──────┐  ┌▼────────┐
│Guardrail│  │Plan │  │Executor │  │Evaluator│  │Survey  │  │Integr.  │
│ Agent   │  │Agent│  │ Agent   │  │ Agent   │  │Agent   │  │Agent    │
│(PII)    │  │(LLM)│  │(Tasks)  │  │(Quality)│  │(Surveys)│  │(HRIS)   │
└───┬────┘  └──┬──┘  └─────┬───┘  └──┬──────┘  └─┬──────┘  └┬────────┘
    │          │           │          │           │          │
    └──────────┴───────────┴──────────┴───────────┴──────────┘
                            │
                    ┌───────▼────────┐
                    │  Data Layer    │
                    │ (PostgreSQL +  │
                    │  Redis + S3)   │
                    └────────────────┘

🔄Agent Collaboration Flow

Integration Agent

Receives HRIS webhook (new_hire_created) → Fetches full employee record

Guardrail Agent

Scans for PII (SSN, DOB, salary) → Redacts before passing to LLM

Planner Agent

Generates 30/60/90 day plan using LLM + templates → Returns structured JSON

Evaluator Agent

Validates plan completeness + quality → Scores 0-1 (pass if > 0.7)

Orchestrator

Decision: Pass? → Route to Executor : Route to human review queue

Executor Agent

Creates tasks in DB → Sends Slack welcome → Schedules calendar invites

Survey Agent

Schedules day-7 survey → Waits for trigger → Sends personalized questions

Orchestrator

Monitors task completion → Triggers next milestone when 80% complete

🎭Agent Types

Reactive Agent

Low

Integration Agent - Responds to webhooks, no planning

Stateless

Reflexive Agent

Medium

Guardrail Agent - Uses rules + context for PII detection

Reads context

Deliberative Agent

High

Planner Agent - Plans multi-step onboarding based on role

Stateful

Orchestrator Agent

Highest

Central Coordinator - Routes between agents, handles retries

Full state management

📈Levels of Autonomy

Tool

Human calls, agent responds

→ Monday's prompts

Chained Tools

Sequential execution

→ Executor Agent (task creation pipeline)

Agent

Makes decisions, can loop

→ Planner Agent (iterates on plan quality)

Multi-Agent

Agents collaborate autonomously

→ This system (6 agents coordinated)

RAG vs Fine-Tuning

Company policies and role-specific templates change frequently. RAG allows daily updates without retraining. Fine-tuning would require monthly retraining at $5K+ per run.

✅ RAG (Chosen)

Cost: $100/month (vector DB hosting)

Update: Daily (add new templates to Pinecone)

How:

❌ Fine-Tuning

Cost: $5K/month (retraining + hosting)

Update: Monthly (full retrain on new data)

How:

Implementation: Vector DB (Pinecone or Weaviate) with 500+ onboarding templates, role descriptions, and company policies. Retrieved during plan generation. Top-3 most relevant templates injected into LLM context.

Hallucination Detection

LLMs hallucinate tasks (fake training courses, non-existent tools, wrong policies)

Confidence scores from LLM (flag if <0.7)

Cross-reference task DB (validate tool names, training IDs)

Logical consistency checks (e.g., 'complete security training' before 'access prod DB')

Human review queue (Evaluator Agent flags suspicious tasks)

0.8% hallucination rate (8 per 1000 plans), 100% caught by L2-L4

Evaluation Framework

Plan Quality Score

0.89target: 0.85+

Task Completion Rate

92%target: 90%+

Survey Response Rate

87%target: 85%+

Employee Satisfaction (NPS)

54target: 50+

Testing: Shadow mode: Run new model on 10% of traffic for 1 week, compare quality scores with production model. Promote if quality improves by 5%+ and no regressions.

Dataset Curation

Collect: 2,000 historical onboarding plans - Export from legacy system (anonymized)

Clean: 1,800 usable plans - Remove duplicates, incomplete plans, test data

Label: 1,800 labeled with quality scores - ($$9K (HR managers rate 0-10))

Augment: +500 synthetic plans - Generate edge cases (remote roles, executive onboarding, interns)

→ 2,300 high-quality examples for evaluation. Used to benchmark new models and detect quality regressions.

Agentic RAG

Planner Agent iteratively retrieves context based on reasoning, not one-shot retrieval.

Employee is 'Senior Engineer' → RAG retrieves engineering onboarding template → Agent reasons 'need tech stack info' → RAG retrieves company tech stack doc → Agent reasons 'need team structure' → RAG retrieves org chart → Final plan includes all context.

💡 Multi-hop reasoning. Agent decides what else it needs to know, not just initial query. Reduces hallucinations by 40% vs. single-shot RAG.

Cost Optimization

Tech Stack Summary

LLMs

Claude 3.5 Sonnet (primary), GPT-4o (secondary), Gemini 1.5 Pro (fallback)

Orchestration

LangGraph (Python) or custom state machine (Node.js)

Database

PostgreSQL (RDS or Cloud SQL) with read replicas

Cache/Queue

Redis (ElastiCache or Memorystore) + SQS or Pub/Sub

Compute

Serverless (Lambda/Cloud Run) for startup, Kubernetes (EKS/GKE) for enterprise

Vector DB

Pinecone or Weaviate

Monitoring

Prometheus + Grafana (metrics), Jaeger (traces), CloudWatch/Loki (logs)

Security

AWS Secrets Manager or HashiCorp Vault, AWS Comprehend (PII detection)

Integrations

BambooHR SDK, Workday API, Rippling API, Slack Bolt SDK, SendGrid API

🏗️

Need Architecture Review?

We'll audit your onboarding system, identify bottlenecks, and show you how to scale from 100 to 10,000 employees per year.

No part of this content may be reproduced, distributed, or transmitted in any form without prior written permission.

Onboarding Automation System Architecture 🏗️

From prompts to production onboarding system.

Key Assumptions

System Requirements

Functional

Non-Functional (SLOs)

Agent Layer

planner

executor

evaluator

guardrail

survey

integration

ML Layer

Feature Store

Model Registry

Observability Stack

Deployment Variants

Startup Architecture

Infrastructure

Risks & Mitigations

⚠️ LLM hallucinations create fake tasks or incorrect policies

⚠️ PII leak to external LLM (SSN, salary, health info)

⚠️ HRIS API changes break integration

⚠️ Survey fatigue (low response rates)

⚠️ Multi-tenant data leakage (Tenant A sees Tenant B's data)

⚠️ Cost overrun (LLM usage spikes unexpectedly)

⚠️ Agent orchestration bugs (infinite loops, stuck workflows)

Evolution Roadmap

Phase 1: MVP (0-3 months)

Phase 2: Scale (3-6 months)

Phase 3: Enterprise (6-12 months)

Complete Systems Architecture

Presentation

API Gateway

Agent Layer

ML Layer

Integration

Data

External

Observability

Security

Sequence Diagram - New Hire Flow

Data Flow - New Hire Onboarding

Scaling Patterns

Key Integrations

HRIS (BambooHR, Workday, Rippling)

Slack

Calendar (Google/Outlook)

Survey Tools (Typeform)

Security & Compliance

Failure Modes & Fallbacks

Multi-Agent Architecture

🔄Agent Collaboration Flow

🎭Agent Types

Reactive Agent

Reflexive Agent

Deliberative Agent

Orchestrator Agent

📈Levels of Autonomy

Advanced ML/AI Patterns

RAG vs Fine-Tuning

Hallucination Detection

Evaluation Framework

Dataset Curation

Agentic RAG

Cost Optimization

Tech Stack Summary

Need Architecture Review?