Test and monitor Quackback AI

Test the AI agent with real customer questions before launch, then track resolutions, escalations, AI CSAT, action outcomes and Copilot usage in Admin > Analytics.

JM
James Morton
Written By James MortonLast updated 3 days ago

Once the AI agent is live, check how it performs with real customers and how much your team uses Copilot. This article covers testing the AI agent before launch and the Quackback AI reporting in Analytics.

Test before you launch

Before you rely on the AI agent with customers, try it yourself:

  1. Make sure the agent is not paused. On Admin > Settings > Agent, the button reads Pause Agent while the agent is live and Resume while it is paused. Also turn on the Messages tab in Admin > Settings > Widget. Messenger replies need the Support inbox.
  2. Open your widget on your site or portal and ask the questions your customers actually ask.
  3. Pull a few real transcripts from your inbox and replay the customer's first message.
  4. Check the answers against your help center. If an answer is wrong or missing, fix the article first, then adjust Guidance if the agent still behaves the wrong way.

Tip: Ask at least one question the agent shouldn't be able to answer, and confirm it hands off to your team instead of guessing.

Quackback AI analytics

Open Admin > Analytics and go to the Quackback AI section, which covers the AI agent and Copilot. Pick a time period at the top, the same as the other analytics sections.

Outcomes

How the customer-facing AI agent handled the conversations it joined:

  • Conversations involved: how many conversations the AI agent took part in
  • Resolved: split into confirmed resolutions (the customer said so) and assumed ones
  • Escalated: how often the AI agent handed off to a human
  • Pending: conversations the AI agent is still working on
  • AI CSAT: satisfaction ratings on conversations the AI agent handled
  • System errors: turns that failed for a technical reason

Actions

How often each action was attempted, completed, or failed.

Copilot

How your team used Copilot over the same period. If nobody used it, the section says there was no Copilot activity.

Improve from what you see

  • High escalation rate: look at the handed-off conversations. Missing articles are the most common cause.
  • Low AI CSAT: read the low-rated conversations and add Guidance rules for the patterns you find.
  • System errors: on self-hosted installs, check that your AI endpoint and model settings are valid.

Was this helpful?

Your feedback shapes what we write next.

← Previous

Knowledge sources

Next →

Build a workflow

Related articles

AI & Automation overviewA map of Quackback AI (the AI agent and Copilot), skills, connectors and workflows, where each lives in Settings, and what each needs to run.Configure the AI agentTurn on, pause and configure the AI agent that answers customers in Messenger: name, tone, knowledge, guidance rules, skills, actions and handoff to your team.Use CopilotHow Copilot helps teammates in the inbox: answering questions about a conversation, drafting replies and polishing your own, plus the separate AI translation and attribute features.Give Quackback AI actionsBuilt-in Quackback AI actions, how to set Allow, Ask or Never per action for the AI agent and Copilot, and how to connect your own tools from a remote MCP server.