FAQs: AI Scoring

The agreement rate measures how often the answers selected by Virtual Supervisor match those selected by a human evaluator. It is calculated both per question and as an overall metric for the evaluation form. A higher agreement rate indicates stronger alignment between AI-generated evaluations and human judgment, while lower agreement highlights areas that may need refinement.

Teams use the agreement rate to identify unclear questions, inconsistent scoring logic, or gaps in evaluation guidance, and to improve how effectively the auto-complete evaluation form performs at scale.

How to calculate and use the agreement rate

1. Evaluate interactions using the form
   • Go to Menu > Analytics > Analytics Workspace > Interactions.
   • Open an interaction and navigate to the Quality Summary tab.
   • Click Create Evaluation.
   • Select the Agent Auto-Complete evaluation form.
   • Choose a human evaluator to manually review the auto-completed answers.
   • Click Create to generate the evaluation.
2. Review and update the evaluation
   • Review each question in the evaluation.
   • Use the transcript as evidence to update any incorrect automated responses.
   • Submit the completed evaluation.
3. Test the form across multiple interactions
   • Repeat this process for at least 20 different interactions to ensure reliable agreement data.
4. Review agreement metrics
   • Go to Conversation Intelligence > Quality Management > Evaluation Forms.
   • Open the latest published version of the evaluation form.
   • Review the overall agreement rate and the agreement rate for each question.
5. Refine the form
   • Identify questions with low agreement rates.
   • Update question wording, answer options, or scoring logic based on observed discrepancies.
   • Retest the form as needed until the desired agreement rate is achieved.

Q: Do current reports include data from auto-complete evaluations?
No. At this time, auto-complete evaluation data is not included in Quality Management reports.

Q: When will auto-complete evaluation data be available in reports?
Genesys is targeting support for auto-complete evaluation data in existing Quality Management reports, as well as in the upcoming question-level reports, by mid-Q2 2026.

Q: What does this mean for supervisors and analysts?
Until reporting support is available, auto-complete evaluation data will not appear in dashboards or exported reports. Once the update is released, supervisors and analysts will be able to view and analyze auto-complete and manually completed evaluations together, providing a more comprehensive view of quality performance.

Q: Will any action be required to access this data once it’s available?
No. Once reporting support is released, auto-complete evaluation data will be included automatically in applicable reports.

No, Quality Policies don’t support Agent Auto-Complete evaluations.

For more information on how to generate an agent auto complete evaluation, see How do I generate an Agent Auto Complete Evaluation?

You can generate evaluations using an Agent Auto-Complete evaluation form in two ways: manually (per interaction) or automatically using the AI Scoring Rules Management API.

Learn more

• For information on creating manual evaluations, see Create a new evaluation.
• For guidance on designing, testing, and fine-tuning auto-complete evaluations, see AI Scoring best practices.

A charge for AI Scoring is incurred whenever a quality evaluation form includes one or more AI-Scoring-enabled questions and that form is used to evaluate an interaction. Charges apply regardless of whether the AI-Scoring-enabled questions are ultimately answered.

The only exception is when the evaluation encounters an AI Scoring–related error at the evaluation level. In those cases, no charge is generated.

Q: Do current reports include data from auto-complete evaluations?

A: Not yet. Currently, reports do not include data generated from auto-complete evaluations. Support for this data is planned for both existing Quality Management reports and the new question-level reports, with availability targeted for mid-Q2 2026.

Q: What does this mean for supervisors and analysts?

A: Until reporting support is released, auto-complete evaluation data will not appear in dashboards or exported reports. Once the update becomes available, you’ll be able to review and analyze auto-complete evaluations alongside manually completed evaluations, providing a more complete picture of overall quality performance.

Q: Will any action be required to access this data once it becomes available?

A: No. Once the reporting update is released, auto-complete evaluation data will be included automatically in all applicable reports—no configuration changes or additional setup required.

No, Quality Policies do not support Agent Auto-Complete evaluations.

For more information on how to generate an agent auto complete evaluation, see How do I generate an Agent Auto Complete Evaluation?

You can generate an evaluation for a specific interaction in one of two ways:

Generating Auto-Complete Evaluations Using AI Scoring Rules Management

To automate the generation of evaluations at scale, configure an Agent Scoring Rule using the AI Scoring Rules Management API.

Step 1: Create an Agent Scoring Rule

Use the following API:

POST /api/v2/quality/programs/{programId}/agentscoringrules

Example Request:

POST /api/v2/quality/programs/bd27fab3-6e94-4a93-831e-6f92e664fc61/agentscoringrules HTTP/1.1
Host: api.inindca.com
Authorization: Bearer *******************
Content-Type: application/json

Example JSON body:

{
“programId”: “bd27fab3-6e94-4a93-831e-6f92e664fc61”,
“samplingType”: “Percentage”,
“submissionType”: “Automated”,
“evaluationFormContextId”: “14818b50-88c0-4cc5-8284-4ed0b76e3193”,
“enabled”: true,
“published”: true,
“samplingPercentage”: 97
}

Field Explanations

• programId – ID of the Speech & Text Analytics (STA) program.
• evaluationFormContextId – The contextId of the automated evaluation form to use.
• samplingPercentage – Percentage of interactions that should automatically generate evaluations.
• enabled – Must be true for the scoring rule to be active.
• published – Must be true for the rule to take effect.
• submissionType – Set to "Automated" to ensure evaluations are auto-generated.

Once the rule is active, evaluations will automatically be created for interactions that meet the rule’s criteria.

Additional Resources

• For instructions on creating manual evaluations, see Create a new evaluation.
• For guidance on designing, testing, and tuning auto-complete evaluations, see AI Scoring Best Practices.

The following table shows the AWS region mappings used by Genesys Cloud for AI scoring with Bedrock models.

Yes. To learn how to use AI Scoring effectively and get the most accurate results, see AI scoring best practices.

Include a question about summarizing outcomes or confirming satisfaction before ending the interaction.

Example: “Did the agent confirm customer satisfaction or summarize next steps before closing the conversation?”
AI marks Yes when the agent checks resolution or restates next steps clearly. This confirms that the customer’s issue was addressed before the call or chat ended.

Ask whether the agent acknowledged or explained any pause longer than a set threshold.

Example: “Did the agent avoid unnecessary dead air or long silences without explaining the reason?”
AI marks Yes when the agent explains pauses (for example, “I’ll place you on a brief hold while I check this”). Unexplained silence longer than 15 seconds is marked No.

Compliance questions should reference required statements that appear in the transcript.

Example: “Did the agent comply with mandatory disclosure or compliance statements (for example, terms, disclaimers, or legal requirements)?”
AI marks Yes when mandatory phrases—such as legal disclaimers or security verifications—are found. Keep help text specific to your industry’s compliance standards.

Write questions that specify the elements of a complete escalation explanation.

Example: “Did the agent explain the escalation process clearly, including who to contact, what information is needed, and expected response times?”
AI marks Yes when all three elements are present. Include examples in the help text to illustrate acceptable responses.

Ask whether the agent clearly described an action or next step that resolves the customer’s concern.

Example: “Did the agent provide a clear resolution step, such as a refund, replacement, or troubleshooting instruction?”
AI marks Yes when specific actions are stated. General apologies or vague responses without follow-through are not considered resolutions.

Replace emotional judgments with observable acknowledgment.

Example: “Did the agent acknowledge the customer’s issue before moving to resolution?”
AI marks Yes when the transcript includes acknowledgment phrases such as “I’m sorry,” “I understand,” or “That must be frustrating.” Avoid vague words like “empathetic” or “polite,” which require human interpretation.

Yes. Personalization improves customer engagement and can be objectively measured.

Example: “Did the agent use the customer’s name at least once during the conversation?”
AI marks Yes when the transcript shows the name provided by the customer being used. Include guidance in the help text so evaluators understand when name usage is required.

Ask direct questions focused on whether the agent allowed the customer to finish speaking.

Example: “Did the agent allow the customer to finish speaking without interruptions?”
AI marks Yes when the agent listens fully before responding. Short, polite interjections like “Sorry, please continue” after a pause are not counted as interruptions. Avoid combining this with other behaviors in the same question.

Include questions that ask whether the agent verified at least one customer credential before discussing account details.

Example: “Did the agent verify the customer’s identity before addressing account-specific concerns?”
AI marks Yes when a credential such as date of birth, account ID, or phone number is confirmed in the transcript. This helps ensure compliance and security in agent interactions.

To evaluate greetings, write transcript-based questions that ask whether the agent used a standard greeting phrase at the start of the interaction.

Example: “Did the agent greet the customer using a standard phrase such as ‘hi,’ ‘hello,’ or ‘good morning’?”
AI marks Yes when a polite greeting is detected. Avoid subjective wording like “Was the agent friendly?” as AI can only recognize the presence of the greeting, not tone or emotion.

Yes, agent and evaluation centered evaluation daily limits (50 per day), apply to both regular and AI driven evaluations. The limit can be increased on an individual customer basis to either 100 or 200 per day, by creating a customer care ticket with this request.

Yes, if you intend to use AI scoring to evaluate calls. AI Scoring requires interactions to be transcribed.