Intercom Evals and Releases Adds Safer Testing for Fin Updates.

Intercom is giving support teams a more disciplined way to change how its Fin AI agent behaves. The company has introduced Evals and Releases, two products designed to test proposed changes before launch, validate them with live traffic and detect problems once they reach customers.

The update addresses a central problem in deploying AI for customer support: a seemingly minor instruction or content change can affect answers across thousands of conversations. Conventional software testing checks predictable inputs and outputs. AI agents are less deterministic, so teams also need to assess qualities such as accuracy, policy compliance and tone across varied customer scenarios.

Intercom outlined the products in an official blog post and subsequently announced them on LinkedIn. Together with its existing Monitors product, they form what the company calls “Eval-driven delivery.”

Evals lets teams run scenarios based on their real customer conversations before a change goes live. Fin’s responses are then scored against criteria defined by the business.

That distinction matters. Rather than relying only on a generic measure of whether an answer appears helpful, a company can evaluate behavior against its own expectations. Those expectations might cover how Fin responds in a particular support situation or whether its behavior follows instructions set by the team. Intercom’s announcement does not specify a fixed library of criteria, indicating that teams establish the standards used for scoring.

Using real conversations should also make tests more representative of production conditions. Support queries frequently contain incomplete descriptions, unusual phrasing and account-specific context that carefully written test prompts can miss. Replaying scenarios derived from actual interactions gives teams a way to expose regressions before customers encounter them.

For small businesses, this could turn AI quality assurance into a repeatable workflow rather than an occasional manual review. A team changing Fin’s guidance can run relevant scenarios, inspect the scores and decide whether the update is ready for live use.

Passing a controlled evaluation does not guarantee that a change will improve results in production. Releases covers that next stage by controlling how Fin changes reach customers and supporting A/B tests on live traffic.

Teams can compare a proposed version with the existing behavior, then use the live results to determine whether the update worked before expanding it to everyone. Intercom has not detailed the traffic controls, statistical methodology or rollout options in the supplied announcement, but the core model resembles progressive delivery in software development: limit exposure, collect evidence and broaden deployment only when the new version performs as intended.

The resulting workflow has three parts:

• Before launch: Evals tests behavior using scenarios based on real conversations.
• During rollout: Releases compares changes through A/B tests on live traffic.
• After deployment: Monitors evaluates every live conversation in real time.

That last layer is important because production behavior can reveal edge cases absent from pre-launch testing. Monitors provides continuing oversight after a release, while Evals and Releases help teams prevent weak changes from reaching a full audience in the first place.

The launch also fits Intercom’s broader effort to build operational controls around AI support, following its work on Operator, an AI agent for managing customer support operations. The focus is shifting from merely deploying an AI agent to managing its behavior as an evolving production system.

Intercom has not provided pricing, plan requirements or a detailed availability schedule in the supplied materials. Teams evaluating the products will need to confirm those details directly with the company.

What is Intercom Evals and Releases?

Evals and Releases are two products for managing changes to Intercom’s Fin AI agent. Evals tests proposed behavior using scenarios based on real conversations, while Releases uses A/B tests on live traffic to validate changes before a full rollout.

How do Evals and Releases work with Monitors?

Evals checks Fin before a change goes live, Releases controls and measures the live rollout, and Monitors evaluates every production conversation in real time. Intercom calls this combined process Eval-driven delivery.

What do Evals and Releases cost?

Intercom did not disclose pricing or plan requirements in the announcement materials provided. Customers should consult Intercom for current access and packaging details.