Online evaluations are the process of testing and measuring the performance of AI and Large Language Model (LLM) features with real users in a live production environment. Unlike offline evaluations, which use static datasets, online evaluations provide feedback based on actual user interactions, which is crucial for understanding the true quality and impact of a feature.

A/B testing and canary releasing are two common methods for conducting online evaluations.

• A/B Testing: A portion of your users (e.g., 50%) is shown a new version of a feature (the “variant”), while the rest see the existing version (the “control”). You then compare performance metrics between the two groups.
• Canary Releasing: A new feature is rolled out to a small, targeted subset of users called a “canary cohort.” If the feature performs well with this group, it’s gradually rolled out to the rest of your user base. This minimizes the potential negative impact of a buggy or poorly performing feature.

For AI features, this means you can test a new prompt, a different model, or updated logic on a small group before deploying it to everyone.

Moving from an idea to a fully deployed AI feature involves a multi-stage evaluation process that shifts from a controlled, offline environment to the dynamic, real-world setting of production.

Offline evaluation is the first step in developing an AI feature. You create a “golden dataset” of representative inputs and their ideal outputs. You then run your new prompt or model against this dataset and score its performance using metrics like:

• Semantic similarity: Does the output mean the same thing as the ideal output?
• Style and tone adherence: Does the AI follow instructions for voice and style?
• JSON validation: If you expect structured data, is the output valid?

Offline scoring is essential for catching obvious errors and ensuring basic functionality. However, it can’t predict how a feature will perform with the near-infinite variety of real-world user inputs. A prompt that scores perfectly on your test set might fail spectacularly in production.

This is where online evaluation takes over. The process involves routing a segment of live traffic to your new AI feature, capturing the interactions, and scoring them in real-time.

1. Establish Canary Cohorts: First, you define a small group of users to receive the new version. This could be 1% of all users, users from a specific region, or even just your internal team. The key is to isolate the new feature’s impact.
2. Stream Traces and Evals: As the canary cohort interacts with the feature, you need to log, or “trace,” everything: the user’s input, the prompt sent to the LLM, the model’s response, and the latency. These traces are then fed into an evaluation system.
3. Score Online Performance: The evaluation system scores the live interactions. This scoring can be done in several ways:
   • Implicit feedback: Tracking user behavior like clicks, conversions, or time on page. For example, did the user accept the AI-generated text or immediately delete it?
   • Explicit feedback: Asking users directly for feedback, often with “thumbs up/down” icons.
   • AI-based evaluation: Using another, more powerful LLM (like GPT-4) as a “judge” to score the quality, relevance, or helpfulness of the production model’s output against a predefined rubric.
4. Visualize with Dashboards: The scores, traces, and key metrics are streamed into a real-time dashboard. This allows your team to monitor performance, compare the new version against the old one, and spot any anomalies as they happen.
5. Automate Rollbacks: Based on the dashboard metrics, you can set up automated guardrails. If the error rate for the new feature spikes or its quality score drops below a certain threshold, the system can automatically roll back, routing all traffic back to the stable version.

This entire pipeline allows you to test in production safely and make data-driven decisions about whether to proceed with a full rollout.

The unpredictable nature of LLMs makes traditional software testing methods insufficient. A tiny change to a prompt can have significant and unforeseen consequences on the output, making online evaluation a necessity, not a luxury.

Here’s why it’s critical for shipping modern AI features:

• Subjective Quality: There’s often no single “correct” answer for an LLM’s output. Offline metrics can’t capture the nuance of whether a response is helpful, creative, or on-brand. Live user feedback is the ultimate source of truth.
• Discovering Edge Cases: Users will interact with your AI in ways you never anticipated. Online evaluation helps you discover and address these edge cases before they affect your entire user base.
• Building Trust: Shipping buggy or unhelpful AI features erodes user trust. A rigorous online evaluation process acts as a safety net, ensuring a higher quality bar for what you release.
• Increasing Velocity: While it may seem like an extra step, a well-implemented evaluation pipeline actually increases your development velocity. It gives you the confidence to experiment and ship changes faster, knowing you have guardrails in place to catch problems automatically.

Setting up a robust online evaluation pipeline requires careful planning. Here are some best practices to follow.

• Start with a clear hypothesis: Before you launch a test, define what you are trying to improve. Are you aiming for higher user engagement, lower costs, or better factual accuracy? This will determine the metrics you need to track.
• Combine different feedback types: Don’t rely on a single metric. A complete picture emerges when you combine implicit user behavior (e.g., copy-pasting the AI’s response), explicit feedback (e.g., user ratings), and automated AI-based evaluations.
• Ensure robust tracing: Your ability to debug and improve is only as good as your logs. Capture detailed traces of each interaction, including timestamps, user IDs, prompts, responses, and any metadata that might be relevant.
• Automate promotion and rollback: The goal is to create a system where a new feature can be promoted from a 1% canary to a 100% rollout automatically if it meets performance targets. Similarly, the system should automatically roll back if performance degrades.
• Use dedicated platforms: While you can build this pipeline yourself, several platforms (such as Langfuse, Arize AI, and Traceloop) specialize in LLM observability and evaluation. These tools provide pre-built dashboards, tracing, and scoring mechanisms to help you get started faster.

To run an A/B test or a canary release, you need a reliable way to control which users see which version of your feature. This is where a service like Kinde becomes essential.

Kinde’s feature flags allow you to manage user access to features without changing your code. You can create a flag, for example, named new-ai-prompt, and define its value for different groups of users.

Here’s how you could implement a canary release for a new AI feature using Kinde:

1. Create a feature flag in Kinde: You could create a string flag called ai-prompt-version with a default value of v1.
2. Implement logic in your application: In your code, you would use a Kinde SDK to check the value of this flag for the current user. The application would then use the corresponding prompt version.
3. Define your canary cohort: You can configure the flag in Kinde to return a value of v2 for a specific segment of users. This could be done by enabling it for a particular organization, for individual users on your testing team, or by using other user properties.
4. Monitor and expand: As you monitor your evaluation dashboards and gain confidence in the v2 prompt, you can gradually update the feature flag rules in Kinde to roll it out to a larger percentage of users, eventually making v2 the new default for everyone.

By using feature flags, you separate the act of deploying code from the act of releasing a feature, giving you fine-grained control over who sees what and when. This is a foundational element for safely and effectively testing your AI features in production.

• About feature flags
• Manage feature flags