A fan-out architecture is a pattern where a single input or trigger initiates multiple downstream processes that can run in parallel or in sequence. In modern software, especially with AI, this often means a single user request might “fan out” to several different AI models, agents, or microservices to generate a comprehensive response.

For example, a request to “draft a blog post about our new feature” could fan out to:

• An agent that outlines the post.
• A second agent that writes the content based on the outline.
• A third that generates a relevant hero image.
• A fourth that suggests SEO keywords and a meta description.

This approach creates powerful, multi-step workflows, but it also introduces complexity. Measuring the return on investment (ROI) of this complexity is critical to ensure you’re building something that is not only functional but also efficient, cost-effective, and user-friendly.

Measuring the success of a fan-out system requires a multi-layered approach that looks at task performance, user experience, and cost. This is achieved by establishing clear Key Performance Indicators (KPIs), testing against a curated “golden set” of scenarios, and running regular evaluations (evals).

KPIs for a fan-out architecture fall into three main categories.

• Task-Level KPIs: These measure the raw success and accuracy of each individual task in the workflow.
  • exact match: Did the output perfectly match the expected result? This is useful for deterministic tasks with a single correct answer.
  • pass@k: In how many instances did one of the top k generated responses meet the quality criteria? This is ideal for non-deterministic or creative tasks where multiple good answers exist.
  • compile rate: For code generation tasks, does the resulting code compile and run without errors?
• Experience KPIs: These measure how the end-user perceives the system’s performance. A technically perfect system that feels slow or clunky is still a failure.
  • first-token latency: How quickly does the user start seeing a response? Low latency creates the perception of speed, even if the full task takes longer.
  • abandonment rate: How often do users navigate away or cancel the process before it completes? A high abandonment rate is a strong signal of poor user experience.
• Cost KPIs: These track the resources consumed to achieve a result, which is crucial for managing the financial ROI of using AI models.
  • tokens/resolved: What is the total number of input and output tokens consumed by all models to successfully complete the entire fan-out workflow? This is a direct proxy for the cost of the operation.

A golden set is a small, carefully curated collection of test cases that represent the most critical and common user journeys, or “golden paths.” This set includes a variety of inputs and the corresponding expected outputs or quality benchmarks. The goal is to create a durable, representative sample of reality that is fast and cheap to test against.

Evals are the process of running this golden set through your system to calculate your KPIs. The most effective way to prevent quality degradation over time is to run these evals automatically before any new code is merged into the main branch. This practice, known as pre-merge evals, acts as a CI/CD pipeline for your AI’s quality, ensuring that a code change that improves one agent doesn’t accidentally break another or drive up costs.

Without a clear measurement framework, it’s easy to lose control of a fan-out system. You might find yourself building an impressive technical demo that is too slow, too expensive, or too unreliable for real-world use.

Measuring ROI helps you:

• Prevent quality drift: Ensure that the user experience and output quality remain high as you add complexity and make changes.
• Control costs: Keep a close eye on token consumption and other resource usage to ensure the feature remains profitable.
• Make data-driven decisions: Use objective KPIs to decide where to invest your engineering efforts—for example, optimizing a slow agent or improving the accuracy of an unreliable one.
• Align technical performance with business goals: Connect low-level metrics like pass@k to high-level business outcomes like user retention and profitability.

• Start with a small golden set: Don’t try to cover every edge case at first. Focus on the 5-10 most important “golden paths” that deliver the most value to your users.
• Automate your evals: Integrate your evaluations directly into your source control workflow. A failing eval should block a pull request just like a failing unit test.
• Balance your KPIs: Don’t over-optimize for one metric at the expense of others. A super-fast, cheap system that produces incorrect results is useless. Aim for a healthy balance across task, experience, and cost KPIs.
• Version your eval sets: As your product evolves, so will your definition of a “golden path.” Treat your golden set like code—version it, document changes, and have a process for updating it.

Implementing, testing, and monetizing a fan-out architecture requires robust infrastructure for user management, access control, and billing.

• Feature flags: Safely rolling out a complex fan-out system is critical. With Kinde’s feature flags, you can release a new workflow to a specific subset of users (e.g., internal testers or a beta group). This allows you to measure your KPIs in a production environment and compare them to the existing system, de-risking the launch and ensuring a positive ROI. You can even manage your flags programmatically using the Kinde Management API.
• Usage-based billing: The tokens/resolved cost KPI is not just an internal metric; it’s a direct input for your pricing model. With Kinde’s billing engine, you can implement sophisticated usage-based or metered billing. For example, you can offer different subscription tiers that include a certain number of “AI workflow runs” per month and use the API to report metered usage for each customer, directly tying your costs to your revenue.

• About feature flags
• Manage feature flags through the Kinde API
• About billing
• Add metered usage for a customer via API