openclaw-qqlobster-skills/skills/product-manager-skills/knowledge/ai-product-craft.md

AI Product Craft

Compressed decision logic for AI product managers: readiness assessment, context architecture, orchestration patterns, and validation methodology. Derived from ai-shaped-readiness-advisor, context-engineering-advisor, and pol-probe-advisor.

---

AI-Shaped Readiness

AI-First vs. AI-Shaped

Dimension	AI-First (table stakes)	AI-Shaped (defensible)
Mindset	Automate existing tasks	Redesign how work gets done
Goal	Speed up artifact creation	Compress learning cycles
AI Role	Task assistant	Strategic co-intelligence
Test	Competitor replicates by adding headcount	Competitor must redesign entire org

The 5 Competencies

1. Context Design — Build a durable "reality layer" humans and AI both trust. Treat AI attention as scarce. Persist constraints + glossary; retrieve everything else on demand. Foundational: blocks all other competencies if missing.

2. Agent Orchestration — Repeatable, traceable AI workflows (research -> synthesis -> critique -> decision -> log rationale). Version-controlled prompts. Each step shows its work. One-off prompts are tactical; orchestrated workflows are strategic.

3. Outcome Acceleration — Compress learning cycles, not just task speed. Eliminate validation lag (PoL probes in days, not weeks). Remove approval delays (AI pre-validates against constraints). Cut meeting overhead (async AI synthesis).

4. Team-AI Facilitation — AI operates as co-intelligence, not accountability shield. Review norms (AI outputs = drafts). Evidence standards (cite sources, reject "I think"). Decision authority (AI recommends, humans decide). Psychological safety to challenge AI.

5. Strategic Differentiation — New customer capabilities competitors can't replicate by throwing bodies at it. Workflow rewiring requiring full org redesign to copy. Economics competitors can't match (10x cost advantage through AI).

Maturity Levels (per competency)

• Level 1 — AI-First: One-off prompts, no structure, efficiency only
• Level 2 — Emerging: Some saved prompts/templates, scattered docs, modest gains
• Level 3 — Transitioning: Multi-step workflows, structured context, learning cycles compressing
• Level 4 — AI-Shaped: Autonomous orchestrated workflows, durable reality layer, defensible moat

Priority Dependency Chain

Context Design (foundation)
  └─> Agent Orchestration (requires context)
        └─> Outcome Acceleration (requires orchestration)
              └─> Strategic Differentiation (requires all above)
Team-AI Facilitation ──── (parallel track, required for scale)

If Context Design is Level 1-2, fix it first. Everything else is fragile without it.

---

Context Engineering

Context Stuffing vs. Context Engineering

Dimension	Stuffing	Engineering
Mindset	Volume = quality	Structure = quality
Approach	"Add everything just in case"	"What decision am I making?"
Persistence	Persist all context	Retrieve with intent
Agent chains	Share everything between agents	Bounded context per agent
Failure response	Retry until it works	Fix the structure
Economic model	Context as storage	Context as attention (scarce)

Why stuffing fails: Accuracy degrades significantly as context grows — models prioritize beginning and end, ignore the middle (Liu et al. 2023, "Lost in the Middle"). Dead ends and errors accumulate (context rot). Retries become normalized.

5 Diagnostic Questions

1. What specific decision does this support? Can't answer = don't need it.
2. Can retrieval replace persistence? Just-in-time beats always-available.
3. Who owns the context boundary? No owner = unbounded growth.
4. What fails if we exclude this? No concrete failure = delete it.
5. Are we fixing structure or avoiding it? Stuffing often masks bad info architecture.

Persist vs. Retrieve Rule

• Persist (80%+ of interactions): Core constraints, user preferences, operational glossary, non-negotiable rules
• Retrieve (<20% of interactions): Project details, historical PRDs, competitive analysis, past transcripts
• Gray zone (20-80%): Weigh retrieval latency vs. context window cost

Two-Layer Memory Architecture

Short-term (conversational): Immediate interaction history. Single session. Summarize/truncate older parts.

Long-term (persistent): Constraints registry + operational glossary + user preferences. Vector database for semantic retrieval. Two subtypes:

• Declarative: facts ("We follow HIPAA")
• Procedural: patterns ("Always validate feasibility before usability")

Research -> Plan -> Reset -> Implement Cycle

The core context rot prevention pattern:

1. Research: Agent gathers data. Context grows large and messy. Expected.
2. Plan: Synthesize into high-density SPEC.md/PLAN.md (source of truth).
3. Reset: Clear entire context window. Non-negotiable.
4. Implement: Fresh session with only the plan as context.

Why it works: Eliminates context rot, dead ends, and goal drift. Agent starts clean with compressed, high-signal context.

Efficiency Formula

Context Efficiency = (Accuracy x Coherence) / (Tokens x Latency)

Key finding: RAG with 25% of available tokens preserves 95% accuracy while cutting latency and cost.

Context Manifest Template

Always Persisted:  constraints (technical, regulatory), user prefs, glossary
Retrieved On-Demand: historical PRDs, transcripts, competitive analysis
Excluded:          meeting notes >30 days, full codebase, marketing materials
Boundary Owner:    [Name]
Next Review:       [Date + 90 days]

---

Agent Orchestration

Core Workflow Pattern

Research -> Synthesis -> Critique -> Decision -> Log Rationale

Each step must be: traceable (cites sources), bounded (own context window), version-controlled (prompts in Git), consistent (same inputs -> predictable process).

Maturity Progression

1. Ad-hoc prompts: Type into ChatGPT as needed. No reuse.
2. Saved templates: Reusable prompts, custom GPTs/Claude Projects. Manual steps.
3. Multi-step workflows: Research -> synthesis -> critique. Manual handoffs between steps.
4. Autonomous orchestration: Runs end-to-end. Traceable. Version-controlled. Auditable.

Bounded Context per Agent

Anti-pattern: Agent A passes everything to Agent B to Agent C (context window explodes to 100k+).

Fix: Each agent outputs a bounded synthesis (2-page max) to the next agent. Apply Research->Plan->Reset->Implement between agent handoffs.

Building Your First Orchestrated Workflow

1. Pick most frequent AI use case
2. Document every step you currently take manually
3. Design loop: research -> synthesis -> critique -> decision -> log
4. Implement (Claude Projects for simple; API orchestration for complex)
5. Run on 3 past examples; compare to manual process
6. Version-control prompts; train 2 teammates; iterate

---

AI Validation (PoL Probes)

The 5 Probe Types

Probe	Core Question	Timeline	AI-Specific Use
Feasibility Check	Can we build this?	1-2 days	GenAI prompt chains, API sniff tests, data integrity sweeps
Task-Focused Test	Can users complete this without friction?	2-5 days	Test AI-generated UIs, chatbot flows, recommendation quality
Narrative Prototype	Does this earn buy-in?	1-3 days	Explain AI capabilities to stakeholders via Loom/video
Synthetic Data Simulation	Can we model without production risk?	2-4 days	Test prompt logic, simulate edge cases, Monte Carlo on AI outputs
Vibe-Coded Probe	Will this survive real user contact?	2-3 days	Frankensoft stack (ChatGPT Canvas + Replit + Airtable) for workflow validation

Selection Logic

Work backwards from hypothesis:

1. What specific risk am I eliminating?
2. What's the cheapest path to harsh truth?
3. Match method to hypothesis, not tooling comfort.

Golden rule: Use the cheapest prototype that tells the harshest truth.

AI-Specific Feasibility Checks

For AI product features, feasibility checks are critical because AI capabilities are non-obvious:

• Prompt chain testing: Run 100 real examples through your proposed prompt. Measure error rate.
• API sniff tests: Verify third-party AI integrations return expected format, latency, cost.
• Data integrity sweeps: Check if your data supports the AI feature (quality, volume, format).
• Disposal protocol: Delete all spike code after documenting findings. Spike-and-delete, not spike-and-ship.

Success Criteria Template

• Pass: [Quantitative threshold, e.g., <5% error rate, 80%+ task completion]
• Fail: [Observable failure, e.g., >18% errors, users abandon mid-flow]
• Learn: [Specific insight regardless of pass/fail]

Write criteria before building. "We'll know it when we see it" is not a success criterion.

Troubleshooting Common AI Product Issues

Hallucination (output contains fabricated facts):

1. Measure: run 100+ real queries, categorize errors (factual, format, reasoning, refusal)
2. Reduce context window — strip to minimum required tokens per the 5 diagnostic questions above
3. Add retrieval with source citations — ground answers in specific documents, not parametric memory
4. Add output validation — regex/rule checks for structured fields, LLM-as-judge for open text
5. Set confidence thresholds — if model confidence is low, return "I don't know" instead of guessing

Latency (AI response too slow for UX):

1. Profile the pipeline — which step is slow? (retrieval, inference, post-processing)
2. Reduce input tokens — smaller context = faster inference. Apply persist vs. retrieve rule.
3. Use streaming — display partial results as they generate
4. Cache common queries — if 30% of queries are similar, pre-compute answers
5. Consider smaller model for simple tasks — route easy queries to fast model, hard queries to capable model

Inconsistency (same input, different outputs):

1. Lower temperature — 0.0-0.3 for factual tasks, 0.5-0.7 for creative tasks
2. Pin model version — don't use "latest" in production
3. Structured output — JSON schema or enum constraints reduce variation
4. Add few-shot examples — 2-3 input/output pairs anchor the response pattern
5. Evaluate on a fixed test set — track consistency score across versions

---

Quality Gates

AI Product Anti-Patterns

1. Prompt-and-Pray Shipping AI features with untested prompts. No evaluation framework, no error rate measurement. Fix: Run feasibility checks (100+ examples) before committing to build.

2. Context Stuffing at Scale Pasting entire knowledge bases into AI. "More tokens = better results." Fix: Apply the 5 diagnostic questions. Accuracy degrades significantly as context grows (Lost in the Middle effect).

3. No Evals Launching AI features without quantitative success criteria. "Users seem to like it." Fix: Define pass/fail thresholds before building. Measure error rates, task completion, hallucination frequency.

4. Efficiency Masquerading as Strategy "We use AI to write PRDs 2x faster — we're AI-shaped!" If a competitor matches it by hiring 2 more people, it's table stakes. Fix: Ask the replication test — does copying require org redesign?

5. Tool Fetishism "Should we use Claude or ChatGPT?" Tool debates replace workflow redesign. Fix: Tools don't matter. Workflows matter.

6. Speed Without Learning Shipping faster without validating faster. AI accelerates building the wrong thing. Fix: Compress learning cycles (PoL probes in days), not just build cycles.

7. Prototype Theater Building polished demos to impress executives instead of testing hypotheses with users. Fix: Test with users first, present findings to executives. Narrative prototypes over production polish.

8. Skipping the Reset Never clearing context between research and implementation. Context rot poisons execution. Fix: Mandatory reset after plan synthesis. Start implementation with only the high-density plan.

9. Individual AI, Not Team AI "I'm AI-shaped, but my team isn't." Can't scale; workflows die when you're on vacation. Fix: Codify review norms, evidence standards, decision authority. Team transformation > individual productivity.

10. Testing Multiple Variables One probe testing workflow + pricing + UI simultaneously. Ambiguous results. Fix: One probe, one hypothesis. Three hypotheses = three probes.

The Falsification Protocol

For every AI feature decision, complete:

"If I exclude [context/feature/test], then [specific failure] will occur in [specific scenario]."

If you can't complete the sentence, you don't need it. Vague failures ("AI might not fully understand") are not valid.

Minimum Viable AI Product Checklist

• Hypothesis written before building
• Feasibility check run (100+ examples, error rate measured)
• Context architecture defined (persist vs. retrieve vs. exclude)
• Success criteria quantified (pass/fail/learn thresholds)
• Disposal date set for probes (spike-and-delete)
• Context boundary owner assigned
• AI outputs treated as drafts (human review protocol)
• Learning cycle measured (before vs. after AI intervention)