From prompts to production forecasting system.
Monday: 3 core prompts for deal scoring, risk detection, and forecast generation. Tuesday: automated Python agents. Wednesday: team workflows across sales, ops, and finance. Thursday: complete technical architecture with ML pipeline, multi-agent orchestration, and CRM integration at scale.
Key Assumptions
System Requirements
Functional
- Ingest deal data from Salesforce/CRM APIs in real-time
- Score each deal with win probability (0-100%)
- Detect risk signals (stalled deals, ghost contacts, pricing issues)
- Generate weekly/monthly/quarterly forecasts with confidence intervals
- Provide deal-level recommendations (next best action)
- Support what-if scenarios (add/remove deals, adjust close dates)
- Audit trail for all forecast changes and model predictions
Non-Functional (SLOs)
💰 Cost Targets: {"per_deal_scored_usd":0.02,"per_forecast_generated_usd":0.5,"monthly_infrastructure_usd":800}
Agent Layer
planner
L4Decomposes forecast request into subtasks, selects execution strategy
🔧 Pipeline analyzer, Task dependency resolver, Resource allocator
⚡ Recovery: If task fails: retry with exponential backoff (3x), If agent unavailable: reassign to backup agent, If critical path blocked: notify human + fallback to manual
ingestion
L2Fetch and normalize deal data from Salesforce and other sources
🔧 Salesforce REST API, Data validator, Schema mapper, Deduplicator
⚡ Recovery: If API timeout: retry with jitter (5x), If rate limit: queue and backoff, If schema mismatch: log error + use last known good schema, If data quality < 90%: flag for manual review
scoring
L3Calculate win probability for each deal using ML model
🔧 Feature store (Feast), ML model (XGBoost/LightGBM), SHAP explainer, Confidence estimator
⚡ Recovery: If model unavailable: fallback to rule-based scoring, If features missing: impute with historical median, If confidence < 70%: flag for human review, If inference timeout: return cached score with staleness warning
risk
L3Identify deal risks (stalled, ghost contacts, pricing issues)
🔧 Activity analyzer (LLM), Pattern matcher (regex + ML), Email sentiment analyzer, Calendar engagement tracker
⚡ Recovery: If LLM fails: use rule-based fallback, If activity data incomplete: mark as 'insufficient data', If false positive rate > 20%: trigger model retraining
forecast
L3Aggregate deal scores into period forecasts with confidence intervals
🔧 Monte Carlo simulator, Confidence interval calculator, Scenario engine, Historical accuracy tracker
⚡ Recovery: If simulation fails: use deterministic sum, If confidence interval too wide (>30%): flag as unreliable, If historical accuracy < 80%: trigger model review
evaluator
L4Validate forecast quality, detect drift, trigger retraining
🔧 Accuracy calculator (MAPE, RMSE), Drift detector (KS test), Model comparator, Alert system
⚡ Recovery: If drift detected: auto-trigger retraining pipeline, If accuracy < 85%: escalate to ML team, If evaluation fails: log error + use last known metrics
guardrail
L4Enforce policies, redact PII, prevent harmful outputs
🔧 PII detector (regex + NER), Policy engine, Output sanitizer, Audit logger
⚡ Recovery: If PII detected: block output + alert security team, If policy violation: reject + log incident, If guardrail fails: fail-safe (block all outputs)
ML Layer
Feature Store
Update: Hourly for active deals, daily for pipeline
- • deal_age_days
- • stage_duration_days
- • activity_count_30d
- • email_response_rate
- • meeting_count_30d
- • champion_engagement_score
- • account_health_score
- • competitor_mentions
- • pricing_discussions
- • contract_review_started
- • decision_maker_engaged
- • budget_confirmed
Model Registry
Strategy: Semantic versioning (major.minor.patch)
- • deal_scorer_v3
- • risk_detector_v2
- • forecast_aggregator_v1
Observability Stack
Real-time monitoring, tracing & alerting
0 activeDeployment Variants
Startup Architecture
Fast to deploy, cost-efficient, scales to 100 competitors
Infrastructure
Risks & Mitigations
⚠️ Salesforce API rate limits exceeded (5,000 req/day)
High (at scale)✓ Mitigation: Use Bulk API 2.0 for large syncs. Implement exponential backoff. Cache frequently accessed data. Request higher API limits from Salesforce.
⚠️ Model drift due to market changes (e.g., recession, new competitors)
Medium✓ Mitigation: Weekly drift detection (KS test). Auto-retrain when drift >0.3. A/B test new models. Monitor forecast accuracy vs actuals.
⚠️ PII leakage in logs or model outputs
Low (with guardrails)✓ Mitigation: Guardrail agent redacts PII. No PII in prompts. Audit logs reviewed quarterly. Incident response plan for breaches.
⚠️ Agent orchestration failures (deadlocks, infinite loops)
Low✓ Mitigation: Timeout after 10 sec. Circuit breaker pattern. Fallback to partial results. Replay from checkpoint.
⚠️ Cost overruns from LLM API usage
Medium (at scale)✓ Mitigation: Set per-user rate limits. Cache LLM responses. Use smaller models for simple tasks. Monitor cost per forecast.
⚠️ Data quality issues (missing fields, stale data)
Medium✓ Mitigation: Data quality checks (completeness, freshness). Alert if quality <90%. Impute missing values with historical median. Flag low-quality forecasts.
⚠️ Vendor lock-in (Salesforce, AWS, LLM provider)
High✓ Mitigation: Abstract integrations behind adapters. Use open standards (FHIR, OAuth). Multi-cloud strategy for enterprise. Document migration paths.
Evolution Roadmap
Progressive transformation from MVP to scale
Phase 1: MVP (0-3 months)
Phase 2: Scale (3-6 months)
Phase 3: Enterprise (6-12 months)
Complete Systems Architecture
9-layer architecture from presentation to security
Presentation
4 components
API Gateway
4 components
Agent Layer
6 components
ML Layer
5 components
Integration
4 components
Data
4 components
External
4 components
Observability
5 components
Security
5 components
Sequence Diagram - Forecast Generation Flow
Automated data flow every hour
Data Flow - End-to-End
Salesforce → Forecast in 3 seconds
Key Integrations
Salesforce CRM
Email APIs (Gmail/Outlook)
Calendar APIs (Google/Outlook)
Data Warehouse (Snowflake)
Security & Compliance
Failure Modes & Recovery
| Failure | Fallback | Impact | SLA |
|---|---|---|---|
| Salesforce API down | Use cached deal data (max age: 1 hour) + alert sales ops | Degraded (stale data), not broken | 99.9% (Salesforce SLA) |
| ML model inference timeout | Use rule-based scoring (stage × amount × age) | Lower accuracy (85% vs 94%), but functional | 99.5% (model availability) |
| Feature store unavailable | Use cached features (Redis) or compute on-the-fly | Increased latency (2x), but accurate | 99.9% |
| Forecast generation fails (Monte Carlo crash) | Use deterministic sum (probability × amount) | No confidence intervals, but forecast available | 99.0% |
| Agent orchestration deadlock | Timeout after 10 sec, return partial results | Incomplete forecast, flagged as 'partial' | 99.5% |
| Database connection pool exhausted | Queue requests, return 503 (retry after) | Temporary unavailability (30-60 sec) | 99.9% |
| PII detected in output | Block output, alert security team, log incident | Forecast unavailable until manual review | 100% (safety first) |
RAG vs Fine-Tuning for Risk Detection
Hallucination Detection in Deal Scoring
Evaluation Framework
Dataset Curation
Agentic RAG for Deal Insights
Continuous Learning Loop
Tech Stack Summary
2026 Randeep Bhatia. All Rights Reserved.
No part of this content may be reproduced, distributed, or transmitted in any form without prior written permission.