The Problem
On Monday you tested the 3 prompts in ChatGPT. Cool! You saw how extraction → validation → prioritization works. But here's reality: you can't ask your PM team to copy-paste 200 feature requests every sprint. One PM spending 3 hours manually scoring features? That's $150/sprint in labor costs. Multiply that across quarterly planning and you're looking at $7,200/year just on prioritization admin. Plus the inconsistent scoring that leads to wrong roadmap decisions.
See It Work
Watch the 3 prompts chain together automatically. This is what you'll build.
Watch It Work
See the AI automation in action
The Code
Three levels: start simple, add reliability, then scale to production. Pick where you are.
When to Level Up
0-100 features/month
- Sequential processing
- Basic error handling
- Manual retry on failures
- Log to console/file
- Single API key
100-1,000 features/month
- Exponential backoff retries
- Batch processing (5-10 concurrent)
- Structured logging (Winston/Python logging)
- Rate limiting
- Multiple API keys for failover
1,000-5,000 features/month
- Conditional routing (extract → validate → score)
- State management across steps
- Automatic retry with backoff
- Parallel processing (10-50 concurrent)
- Dead letter queue for failures
- Monitoring dashboards
5,000+ features/month
- Specialized agents (extraction, validation, scoring, analytics)
- Load balancing across API providers
- Caching layer (Redis) for repeated features
- Real-time analytics integration
- A/B testing for prompt variations
- Custom fine-tuned models
- SLA monitoring and alerting
Product-Specific Gotchas
Five challenges you'll hit when automating roadmap prioritization. Here's how to handle them.
Inconsistent Feature Request Formats
Feature requests come from Slack, email, support tickets, sales calls - each with different detail levels. Your extraction prompt needs to handle all formats.
Use a preprocessing step to normalize inputs before extraction.
def normalize_feature_request(raw_input: str, source: str) -> str:
"""Normalize feature requests from different sources"""
# Add source context to help LLM understand format
context_map = {
'slack': 'This is a casual feature request from internal Slack. May lack detail.',
'sales': 'This is from a sales call. Focus on customer name and ARR impact.',
'support': 'This is from support ticket. Focus on user pain and frequency.',Reach Estimation Without Analytics
Many feature requests lack hard user numbers. You need to estimate reach from qualitative signals like 'several customers' or 'power users want this'.
Build a reach estimation heuristic that the LLM can use.
# Reach estimation rules for LLM REACH_HEURISTICS = """ Use these rules to estimate reach when exact numbers aren't provided: 1. Customer segment sizes: - Enterprise: ~50-200 users per customer - SMB: ~5-20 users per customer - Individual: 1 user per customer
Impact Scoring Subjectivity
Impact (0.25 to 3) is subjective. Different PMs score differently. You need consistent criteria the LLM can apply.
Define clear impact tiers with examples.
IMPACT_SCORING_GUIDE = """ Score impact on a 0.25 to 3 scale: 0.25 (Minimal): - Nice-to-have polish - Saves < 5 min/week per user - No competitive gap Example: Add keyboard shortcut for rarely-used action
Effort Estimation Accuracy
LLMs can't accurately estimate engineering effort without context on your codebase, tech stack, and team velocity. Overestimating effort kills good features.
Use historical effort data to calibrate estimates.
# Build effort calibration dataset
EFFORT_CALIBRATION = {
'simple_ui_change': {'typical_days': 2, 'range': (1, 3)},
'new_api_endpoint': {'typical_days': 3, 'range': (2, 5)},
'database_migration': {'typical_days': 5, 'range': (3, 10)},
'third_party_integration': {'typical_days': 7, 'range': (5, 15)},
'new_major_feature': {'typical_days': 20, 'range': (15, 40)}
}Analytics Integration for Validation
To validate RICE scores, you need to pull real usage data. But analytics APIs (Mixpanel, Amplitude) have different schemas and rate limits.
Build a unified analytics adapter layer.
from typing import Protocol, Dict
import httpx
class AnalyticsProvider(Protocol):
"""Protocol for analytics integrations"""
async def get_active_users(self, segment: str = None) -> int:
...
Adjust Your Numbers
❌ Manual Process
✅ AI-Automated
You Save
2026 Randeep Bhatia. All Rights Reserved.
No part of this content may be reproduced, distributed, or transmitted in any form without prior written permission.