AI Feature A/B Testing

Why This Matters

Every time you ship an AI feature, you are making a bet. A new prompt, a different model, a tweaked temperature: any change can improve your product dramatically or degrade it silently. Traditional software changes produce deterministic outputs: if you add a button and users click it more often, the button is better. AI features do not work this way. The same user, the same query, the same model can produce three different responses in three consecutive calls. Quality is often a matter of opinion. The impact of an AI response on a user's downstream behaviour may not appear for days or weeks.

Building a rigorous experimentation practice for AI features is one of the highest-leverage skills an AI product team can develop. Teams that experiment well ship better products faster and break fewer things for users.

Everyday Analogy

Imagine you are a chef experimenting with two different soups at a restaurant. For traditional food, you could use objective measurements: saltiness, temperature, cooking time. For a complex dish with dozens of interacting flavours, a slight change in one spice changes the entire character of the soup in ways that are hard to measure with instruments. You need diners to taste both versions, and their preferences might only crystallise after they have finished the full meal. AI experimentation is the same: the quality of an LLM response interacts with context, user intent, and downstream behaviour in ways that require careful, patient measurement.

Why AI Experiments Are Different

1. Non-Deterministic Outputs

Traditional software is deterministic. Given the same inputs, it produces the same outputs. This makes a control group meaningful: you know exactly what every control user experienced.

LLMs are non-deterministic. Two users in your control group asking the same question may receive different answers. Temperature, sampling settings, and internal model randomness all introduce variance. This means your "control" is already a distribution of outputs, not a fixed state. When you compare treatment to control, you are comparing two distributions, which requires more sophisticated statistical thinking.

Practical implication: Always log the actual response, not just whether the user was in the treatment group. You need to analyse the distribution of outputs, not just the distribution of outcomes.

2. Subjective Quality

When you change a button colour, you measure clicks. When you change an LLM prompt or model, you need to measure whether the response was good. Goodness is multidimensional: accuracy, helpfulness, tone, brevity, safety, relevance. Different users weight these dimensions differently. There is rarely a single objective metric.

Practical implication: Define your quality dimensions before the experiment begins. Do not try to measure everything; focus on the two or three dimensions that matter most for the specific use case.

3. Delayed Feedback

When a user reads an AI-generated summary, the impact on their behaviour may not appear immediately. Did the summary help them complete a task? Did they trust the information and act on it? Did a slightly wrong response erode their trust over weeks? These effects take time to manifest in your data.

Practical implication: Run AI experiments for longer than you think is necessary. A two-day experiment may show no effect even when a two-week experiment would reveal a significant one.

4. Entanglement

When you change a traditional UI element, it only affects users who see that element. When you change an LLM, you affect every user, every use case, and every downstream interaction simultaneously. A prompt change that improves customer support responses may degrade code generation responses because the same model serves both.

Practical implication: Scope your experiments carefully. Test one change, for one use case, with a clearly defined user segment. Avoid simultaneous model changes across multiple features.

5. No True Rollback

When you ship a new version of traditional software and revert, users go back to exactly what they had before. When you update a model or prompt, the original state may be hard to recover precisely, especially if you are using a managed API where the provider updates the underlying model.

Practical implication: Version your prompts explicitly. Store the exact prompt text, model version, and inference parameters for every experiment. Treat prompts as code artefacts with the same version control discipline.

Types of AI Experiments

Prompt Experiments

The most common type. You hold the model constant and vary the system prompt, few-shot examples, instruction phrasing, or output format. Prompt experiments are low-risk and fast to run, since no model deployment is required.

Example: Testing whether adding "Think step by step" to your customer support prompt improves task completion rates versus the baseline prompt.

Model Swap Experiments

You compare the same prompt across different models, for example GPT-4o versus Claude Sonnet versus Gemini Pro. Model swaps have higher stakes: different models have different latency, cost, and safety profiles. They also require you to ensure the prompt is appropriate for each model's specific instruction format.

Example: Evaluating whether switching from one model to a newer version improves factual accuracy on your specific query distribution without increasing latency or cost.

Feature Experiments

You test an AI feature being on versus off. This is the most straightforward comparison because the control group experiences your existing product and the treatment group gets a genuinely new capability.

Example: Testing whether adding an AI-powered writing assistant to an email tool increases drafts saved and emails sent.

UX Experiments

You hold the AI model and prompt constant and vary how the AI output is displayed. Does showing the response in a streaming typewriter style feel more trusted than displaying it all at once? Does adding confidence indicators change how users act on AI recommendations?

Example: Testing whether showing AI-generated answers with a "Sources" section increases click-through rates on those answers compared to showing answers without source attribution.

Experiment Design

Defining Your Hypothesis

A good experiment hypothesis is falsifiable and specific. Follow this structure:

"We believe that [change] will cause [metric] to [increase/decrease] by [magnitude] for [user segment] because [rationale]."

Weak hypothesis: "We think the new prompt will be better."

Strong hypothesis: "We believe that adding three product-specific few-shot examples to our search summarisation prompt will increase the percentage of searches that result in a product page click-through by at least 8% for users searching for consumer electronics, because examples will calibrate the model's response style to our catalogue's language."

Randomisation Strategy

Randomly assign users, not requests, to experiment groups. If you randomise at the request level, the same user may experience both variants, contaminating your results. Use a consistent hash of the user ID to ensure stable assignment for the duration of the experiment.

For features where user state matters (personalisation, memory, long-running conversations), consider waiting until users start a fresh session before assigning them to a new experiment group.

Minimum Detectable Effect

Decide before running the experiment what the smallest improvement is that would be worth shipping. This minimum detectable effect (MDE) drives your required sample size. A 0.5% improvement in a critical conversion metric may be worth detecting, but it requires a much larger sample and a longer experiment than detecting a 5% improvement.

Use a power calculator to determine sample size. Most AI feature experiments require at least 80% statistical power at p < 0.05, meaning you need enough users to distinguish your MDE from random noise with high confidence.

Guardrail Metrics

Always define guardrail metrics alongside your primary metric. These are metrics that must not degrade, even if your primary metric improves. For AI features, common guardrails include:

Shadow Testing

Shadow testing, sometimes called a dark launch, is the practice of running a new model or prompt in parallel with your production model, logging the outputs of both, but only serving the production response to users.

How It Works

User Request
    │
    ├──▶ Production Model ──▶ Response served to user
    │
    └──▶ Shadow Model    ──▶ Response logged (not served)

Shadow testing allows you to:

• Compare output quality between production and the candidate model before any user impact
• Discover new failure modes and edge cases in your specific query distribution
• Build up a large offline evaluation dataset for automated quality scoring
• Estimate latency and cost of the new model under real traffic patterns

Offline Evaluation

Once you have shadow logs, you can score them using LLM-as-judge evaluation. Feed both the production and shadow responses to an evaluator model alongside the original query, asking it to rate which response is better across dimensions like accuracy, completeness, and tone. This gives you a preference score that quantifies quality differences before touching production traffic.

Important caveat: LLM-as-judge is not ground truth. It reflects the evaluator model's preferences, which may not match your users' preferences. Use it as a signal, not a verdict. Validate significant findings with human raters.

Canary Releases

A canary release gradually routes a small fraction of traffic to the new model or feature, monitoring for problems before full rollout.

Canary Traffic Progression

What to Monitor at Each Stage

At 1%, you are primarily watching for catastrophic failures: server errors, extreme latency spikes, safety violations, and obviously broken outputs. Your sample size is too small for statistical significance on subtle quality metrics.

At 5–25%, you have enough volume to begin detecting smaller quality regressions. Start monitoring your primary experiment metrics, guardrail metrics, and user-facing signals like thumbs-down rates and retry rates.

At full rollout, you continue monitoring to catch delayed effects and novelty bias.

Metrics for AI Experiments

Selecting the right metrics is the most important decision in your experiment design. Use a layered approach.

Implicit Behavioural Signals

These are signals users generate through their behaviour, without being asked to provide feedback.

Explicit User Feedback

Explicit signals are more precise but harder to collect at scale.

Automated LLM-as-Judge Evaluation

Run a separate LLM evaluation pipeline that scores each response across your quality dimensions. Typical dimensions:

• Correctness: Is the factual content accurate?
• Completeness: Does the response fully address the query?
• Conciseness: Is the response appropriately brief or does it pad unnecessarily?
• Safety: Does the response avoid harmful, biased, or inappropriate content?
• Tone: Does the response match your product's voice and the user's context?

Score responses on a 1–5 scale and track the distribution across your experiment groups.

Business Metrics

Connect AI quality to business outcomes.

Statistical Considerations

The Multiple Testing Problem

Running many experiments simultaneously inflates your false positive rate. If you run 20 experiments at p < 0.05, you should expect one false positive even if no experiment had a true effect. Apply a correction such as the Bonferroni correction or use a Bayesian approach to manage this risk.

Novelty Bias

When users first encounter a new AI feature, they often interact with it more than they will long-term, simply because it is new. This inflates your short-term metrics and makes the treatment look better than it will perform over time. Run your experiments long enough to capture post-novelty behaviour (typically two to four weeks for high-engagement features).

Network Effects and Contamination

In products where users interact with each other's AI-generated content, such as social platforms, collaborative tools, or shared knowledge bases, a user in the control group may be influenced by content generated by a treatment group user. This violates the independence assumption of standard A/B tests. Consider cluster-based randomisation (assign entire teams or communities to the same group) for these products.

Building an Experimentation Culture

Hypothesis-Driven Development

Every significant AI change should start with a documented hypothesis. Before you write a line of code, write down: what you are changing, what you expect to happen, why you expect it, and how you will measure it. This habit prevents the common failure mode of shipping changes and then cherry-picking whichever metric happens to look good.

Learning, Not Just Winning

Many experiments will show no effect or negative effects. These are not failures; they are learning. A negative result that eliminates a poor hypothesis is as valuable as a positive result. Create a shared experiment log where the whole team can review results, including non-significant and negative results.

The Experiment Backlog

Maintain a prioritised backlog of experiments the same way you maintain a feature backlog. Prioritise experiments by: expected impact if positive × confidence in hypothesis × ease of running. Run the highest-priority experiments continuously; idle experimentation capacity is wasted learning opportunity.

Tooling

Practical Example: A Search Summarisation Experiment

Context: You have an AI-powered search feature that summarises product documentation. You want to test whether switching to a newer model version improves answer accuracy.

Step 1 — Shadow test for two weeks. Run both models on all production queries. Log every response. Run offline LLM-as-judge evaluation. Result: the new model scores 12% higher on accuracy and 8% higher on completeness in the offline evaluation. Latency is 15% higher at p95.

Step 2 — Define the experiment. Primary metric: task completion rate (user visits the documentation page and does not immediately return to search). Guardrails: latency p95 must stay below 3 seconds; safety violation rate must not increase.

Step 3 — Canary at 1%. Deploy new model to 1% of users. Monitor for 48 hours. No errors, latency p95 is 2.4 seconds (within guardrail), no safety violations detected.

Step 4 — Staged rollout to 10%. After five days, task completion rate is up 6.2% (p = 0.03). Retry rate is down 4.1% (p = 0.07 — trending positive but not significant yet). Continue rollout.

Step 5 — Full rollout at three weeks. Task completion rate is +7.8% (p < 0.01). Retry rate is −5.3% (p = 0.04). Business metric: support ticket creation rate decreased 3.1% (lagging metric, approaching significance). Decision: ship.

Step 6 — Document the learning. Log hypothesis, method, results, and business impact in the experiment log. Identify follow-on hypotheses.

Try It Yourself

1. Design a minimal A/B test for a prompt change: define the metric (AI-047), the guardrail metric that must not regress, the traffic split, and how long you'd run it given your daily volume. Small volume means a long test, so compute it honestly.
2. Spot the novelty effect: for any AI feature you use, ask whether week-one enthusiasm or week-four retention is the truer signal, then check which one a naive A/B readout would capture.

Key Takeaways

• AI experiments require a different mental model from traditional A/B tests because outputs are non-deterministic and quality is multidimensional.
• Shadow testing is the safest first step: compare models offline before touching production traffic.
• Canary releases let you catch catastrophic failures at 1% traffic before they affect everyone.
• Layer your metrics: implicit signals, explicit feedback, automated evaluation, and business outcomes together give you a complete picture.
• Novelty bias, the multiple testing problem, and delayed feedback all require discipline to manage correctly.
• An experimentation culture, with documented hypotheses, shared learning, and a prioritised experiment backlog, compounds the value of every individual experiment.

Looking Ahead

In AI-056, we turn to the legal and compliance landscape for AI products, covering the EU AI Act, GDPR for AI features, intellectual property risks, and how to build a compliance framework your product team can actually work with.