From prompts to production roadmap system.
Monday: 3 core prompts for scoring features. Tuesday: automated code to run them. Wednesday: team workflows (PM, Eng, Leadership). Thursday: complete technical architecture with 6 specialized agents, ML pipeline, and production deployment from 10 to 10,000 features per month.
Key Assumptions
System Requirements
Functional
- Ingest feature requests from multiple sources (Jira, Linear, Slack, email)
- Score features across 5 dimensions using AI agents
- Aggregate scores with weighted formulas and ML models
- Generate prioritized roadmap with confidence intervals
- Provide explanations for each score and ranking
- Support custom scoring criteria per organization
- Track score changes over time with audit trail
Non-Functional (SLOs)
💰 Cost Targets: {"per_feature_scored_usd":0.05,"per_org_per_month_usd":200}
Agent Layer
planner
L3Decomposes feature scoring into subtasks and selects appropriate tools
🔧 feature_parser, criteria_loader, context_retriever
⚡ Recovery: If parsing fails: use default criteria, If context unavailable: proceed without historical data, Retry up to 3x with exponential backoff
executor
L4Orchestrates the scoring workflow across all dimensions
🔧 scorer_agent, parallel_executor, result_aggregator
⚡ Recovery: If one dimension fails: continue with others, flag incomplete, If all fail: route to manual queue, Timeout per dimension: 5 seconds
scorer
L2Scores individual dimensions using LLM reasoning
🔧 claude_api, gpt4_api, prompt_formatter, score_normalizer
⚡ Recovery: If LLM timeout: retry with different model, If invalid score: use median of historical scores, If confidence < 0.5: flag for human review
evaluator
L3Validates output quality and detects anomalies
🔧 statistical_validator, anomaly_detector, consistency_checker
⚡ Recovery: If validation fails: route to human review queue, If anomaly detected: flag and continue with warning, If quality < threshold: trigger re-scoring
guardrail
L4Enforces policy checks, PII redaction, and safety filters
🔧 pii_detector, policy_engine, content_filter
⚡ Recovery: If PII detected: redact and log, If policy violation: block and alert, If uncertain: flag for manual review
aggregator
L3Combines dimension scores into final ranking with ML model
🔧 weighted_average, ml_ranking_model, confidence_estimator
⚡ Recovery: If ML model fails: fall back to weighted average, If weights missing: use equal weights, If rank calculation fails: use score-only ordering
ML Layer
Feature Store
Update: Real-time for new features, batch daily for historical aggregates
- • feature_text_embeddings (768-dim)
- • historical_score_avg
- • historical_score_std
- • source_reliability_score
- • user_engagement_metrics
- • similar_feature_scores
- • org_scoring_patterns
Model Registry
Strategy: Semantic versioning with A/B testing for new versions
- • score_aggregator_v2
- • confidence_estimator_v1
- • anomaly_detector_v1
Observability Stack
Real-time monitoring, tracing & alerting
0 activeDeployment Variants
Startup Architecture
Fast to deploy, cost-efficient, scales to 100 competitors
Infrastructure
Risks & Mitigations
⚠️ LLM cost explosion (>$1K/day)
Medium✓ Mitigation: Set daily budget caps. Cache similar features. Use cheaper models for low-value features. Monitor cost per feature. Alert at $500/day threshold.
⚠️ Score accuracy degrades over time (concept drift)
High✓ Mitigation: Weekly evaluation against PM judgments. Retrain aggregation model monthly. A/B test new prompts. Human-in-the-loop for low-confidence scores.
⚠️ PII leakage to LLM providers
Low✓ Mitigation: PII detection before LLM calls. Redact all detected entities. Audit logs for all LLM requests. Use enterprise LLM plans with no training on customer data.
⚠️ Latency exceeds 5 seconds (user frustration)
Medium✓ Mitigation: Parallel scoring across dimensions. Cache common patterns. Use faster models for low-value features. Async processing with webhook callback.
⚠️ Agent orchestration bugs (infinite loops, deadlocks)
Medium✓ Mitigation: Timeout per agent (5 sec). Max retry count (3x). Circuit breaker pattern. Extensive testing with chaos engineering.
⚠️ Integration failures (Jira/Linear API down)
Medium✓ Mitigation: Queue-based architecture. Retry with exponential backoff. Manual upload fallback. Monitor integration health.
⚠️ Bias in scoring (favors certain types of features)
High✓ Mitigation: Diverse training data. Bias detection in evaluation. Human review of edge cases. Regular fairness audits.
Evolution Roadmap
Progressive transformation from MVP to scale
Phase 1: MVP (0-3 months)
Phase 2: Scale (3-6 months)
Phase 3: Enterprise (6-12 months)
Complete Systems Architecture
9-layer architecture from presentation to security
Presentation
4 components
API Gateway
4 components
Agent Layer
6 components
ML Layer
5 components
Integration
5 components
Data
4 components
External
5 components
Observability
5 components
Security
5 components
Sequence Diagram - Feature Scoring Flow
Automated data flow every hour
Data Flow
Feature request → Prioritized roadmap in 4 seconds
Key Integrations
Jira Integration
Linear Integration
Slack Integration
CRM Integration (Salesforce/HubSpot)
Analytics Integration (Mixpanel/Amplitude)
Security & Compliance
Failure Modes
| Failure | Fallback | Impact | SLA |
|---|---|---|---|
| LLM API down (Anthropic/OpenAI) | → Switch to backup LLM provider → If all fail, queue for retry | Degraded (slower), not broken | 99.5% |
| Scoring confidence < 0.5 | → Flag for human review → Use historical median score as placeholder | Quality maintained, some manual work | 99.9% |
| Database unavailable | → Read from replica → Write to queue for replay | Read-only mode, eventual consistency | 99.9% |
| PII detection service down | → Block all processing (safety first) → Alert on-call | Complete halt to protect privacy | 100% |
| Aggregation model fails | → Use weighted average of dimension scores | Slightly less accurate, fully functional | 99.0% |
| Queue depth > 1000 | → Scale up workers → Rate limit new submissions | Slower processing, no data loss | 99.5% |
| Cost budget exceeded ($500/day) | → Throttle LLM calls → Use cached scores for similar features | Slower, less fresh scores | N/A (cost control) |
┌──────────────┐
│ Planner │ ← Decomposes task
└──────┬───────┘
│
▼
┌──────────────┐
│ Executor │ ← Orchestrates workflow
└──────┬───────┘
│
┌───┴────┬──────────┬──────────┐
│ │ │ │
┌──▼──┐ ┌─▼──┐ ┌────▼────┐ ┌──▼────┐
│Scorer│ │Scorer│ │ Scorer │ │Scorer │ (5 parallel)
│(Value)│ │(Effort)│ │(Alignment)│ │(Impact)│
└──┬──┘ └─┬──┘ └────┬────┘ └──┬────┘
│ │ │ │
└────────┴──────────┴──────────┘
│
┌───▼────┐
│Guardrail│ ← Validates & sanitizes
└───┬────┘
│
┌───▼────┐
│Aggregator│ ← Combines scores
└───┬────┘
│
┌───▼────┐
│Evaluator│ ← Quality check
└────────┘🔄Agent Collaboration Flow
🎭Agent Types
Reactive Agent
LowScorer Agent - Responds to input, returns score
Reflexive Agent
MediumGuardrail Agent - Uses rules + context for validation
Deliberative Agent
HighPlanner Agent - Plans multi-step execution
Orchestrator Agent
HighestExecutor Agent - Coordinates all agents, handles loops
📈Levels of Autonomy
RAG vs Fine-Tuning
Hallucination Detection
Evaluation Framework
Dataset Curation
Agentic RAG
Multi-Model Ensemble
Tech Stack Summary
2026 Randeep Bhatia. All Rights Reserved.
No part of this content may be reproduced, distributed, or transmitted in any form without prior written permission.