This article explains how to build analytics-powered Cognigy.AI flows that track flow steps and allow detailed success or failure analysis via a Microsoft Power BI analytics dashboard.

Requirements

• Good understanding of Cognigy.AI flows
• Good understanding of the Cognigy.AI OData Analytics endpoint
• Microsoft Power BI Knowledge (Also possible in Excel or any other Data visualization tool that can access an OData endpoint)

Introduction

   Watch Cognigy Sessions Episode "Analytics & OData" for a technical deep dive

Monitoring a virtual agent's successes and, more importantly, failures is critical for understanding the value it brings to the business and ensuring this value is consistently increased. The industry standard for bot performance is to set goals, keep track of intents and monitor NLU scoring, that is, the number of times an intent was triggered, average intent score, number of confirmed orders, etc. These typical metrics can be monitored as part of the standard dataset available via the Cognigy.AI OData Analytics endpoint and are ready to be used out-of-the-box for any bot deployment with Cognigy.AI.

The visual flow concept that Cognigy.AI delivers provides conversation designers with an industry-leading graphical editor, placing conversation design into a visual map with flow nodes becoming key reference points throughout conversations. In some situations, drawing conclusions about flow performance via the standard goal and intent data is insufficient. To achieve a more direct view of conversation flow performance, we can monitor steps that occur during conversations and throughout our flows that outline a conversation path for any session. Visualization dashboards can then be designed in parallel to Cognigy Flows that best present the successes and failures based on the actual conversation paths that have taken place. Conversation designers can then use these dashboards to identify weaknesses and further improve the bot's performance with a deeper understanding of actual conversations.

Let's take a look at how we can achieve this.

Task 1: Preparation

Location: Cognigy.AI

Before thinking about dashboards and visualization, it pays to consider the design best practices for Cognigy.AI. Conversation designers should consider a few simple steps to make the flows more understandable, which will, in turn, lead to clearer linking between data and conversational flow.

Please take the time to review our Best Practices section before proceeding with this concept. 

In particular, the requirement for clear and logical naming conventions, from flow names through to node labels, is a requirement, as this same convention will be used below.

Task 2: Create Flow Steps

Location: Cognigy.AI

Steps are the name we give to a point in the flow that we want to highlight as an important phase in the conversation. This is typically linked to a single execution of a flow, but a step can be created for any node or intent that occurs in a flow. This could be the first welcome message, a question that is asked, or an action that is taken. Any nodes or intents that have steps configured in the respective settings menu will be added to the executed step list when activated by the flow in the order they are activated.

Configuring Nodes as Steps

Any node in the flow chart can be configured as a step by adding a value to the "Analytics Step" field in the editing menu. This could also be a dynamic value using CognigyScript or Tokens. Question nodes will automatically be assigned a default label in the "Analytics Step" field. However, all other nodes must have the step name manually added.

Configuring Intents as Steps

When using the Cognigy NLU, any intent in the intent list can be configured as a step by adding a value to the "Analytics Step" field in the advanced settings menu. 

Task 3: Connect and Prepare the OData Records

Location: Microsoft Power BI

Once you have added your step and analytics nodes to your flow, it's time to conduct test conversations. It pays to have a good set of sample data that provides examples for many of the different outcomes so that you can see what you want to visualize in your dashboards. Once you have done this, import your OData analytics data to Microsoft Power BI (Watch this Cognigy Sessions Episode to learn how). 

Accessing the OData Records

Refer to the documentation page for details on how to connect to the OData records.

For this example, import the following records to Power BI:

• Steps:
  • Contains a list of all steps that exist in any agent, one record per step.
  • Includes the step entity reference ID
  • Includes the step label from the Cognigy UI
  • Includes flowName, projectName, and snapshotId.

• ExecutedSteps:
  • Contains a record of each step that has been executed
  • Includes the step entity reference ID
  • Includes the step label from the Cognigy UI
  • Includes the parent step entity reference ID that we executed prior
  • Includes ProjectID, ProjectName, flowName, inputId, userId, and sessionId.

• Sessions:
  • Contains a record of each session that has taken place
  • Contains a field called stepPath which is a CSV list of all steps executed in that session
  • Contains a field called stepsCount which is a numerical count of all steps executed in that session
  • Includes goals, sessionId, localeName, ProjectId, and projectName.

Manipulating the Data

The Conversations record must be manipulated to extract a column for each executed step. To do this, open the table in the Power query editor by right-clicking on the record and selecting "Edit Query" as shown below:

Once in the query editor, select the column "stepPath" then "Split Column" then "By Delimiter". Set the delimiter to "Comma" and click OK.

This will result in a new column being created for every step that was executed throughout a given conversation containing the entityReferenceId of that step.

To replicate this table with the given step label as created in the Cognigy.AI UI, we need to create new columns for each step that we wish to monitor and look up the label name by the entityReferenceID from the "Steps" record. The entityReferenceId is used by default because the step label may change at any time by a user in the Cognigy.AI UI. Therefore, we reference it to a list of the current step labels where the entityReferenceId never changes.

To achieve this, select the record of the conversations in Power BI and click "New Column". Use a formula similar to the following to retrieve the step label from the "Steps" record:

Step1 = LOOKUPVALUE('Steps'[label],Steps[entityReferenceId],'Conversations'[stepPath.1])

Adding an END Step

In addition to this, to create a new step at the end of each session to indicate when a session ended, use the following formula:

Step1 = IF('Conversations'[stepsCount]=0,"END",LOOKUPVALUE('Steps'[label],Steps[entityReferenceId],'Conversations'[stepPath.1]))

This formula checks the steps count to see if the conversation ended in the previous step. If so, the word END will be placed as the current step. Ensure you update the step count comparative value to reflect each step. For example, Step 1 checks for a stepCount of 0, Step 2 checks for a step count of 1, etc.. An example of Step 2 is below:

Step2 = IF('Conversations'[stepsCount]=1,"END",LOOKUPVALUE('Steps'[label],Steps[entityReferenceId],'Conversations'[stepPath.2]))

Create a new column for each step you wish to monitor in your analytics dashboard with the same formula pattern as explained above. The result should look something like this:

Once this is complete, you are ready to build some dashboard visualizations.

Task 4: Create Data Visualizations

Location: Microsoft Power BI

We can now use this data to build a graphical visualization in Microsoft Power BI.

There are many different ways to view this data using built-in Power BI visualizations and the extended visuals available via the Power BI visuals marketplace. For monitoring multiple steps in a flow, the "Pie Chats Tree" is an excellent visual to add to your dashboards that can be used in conjunction with filters to show monitoring for selected steps.

Visualization 1: Decomposition Tree / Conversation Flow

This visual is a basic flow chart using a standard flow chart graph from the PBI set called the Decomposition Tree. This visual allows a single flow path to be displayed simultaneously from start to end. This makes it great for understanding the volumes of certain use cases. Remember that multiple charts can be created as a cluster for your dashboard, so multiple decomposition trees could be used to highlight certain routes with your conversations.

Data Points Used:

• Analyze: Count of _id (Conversations)
• Explain by: Step1, Step2, Step3, Step4, ... etc

Visualization 2: Pie Charts Tree / Conversation Flow

This visual is a more complex flow chart tree called the Pie Charts Tree and is available on the Power BI marketplace to download and install. This visual allows any number of tree branches of a conversation to be shown at once and includes a small pie chat at the end of each branch explaining the volume each branch holds. It also allows branches to be collapsed and manipulated to view only the required paths. Caution with large datasets as this visual could quickly become too large to handle.

Data Points Used:

• Categories: Step1, Step2, Step3, Step4, ... etc
• Value: Count of _id (Conversations)

Visualization 3: Pie Charts Tree / Reverse Conversation Flow

This visual is a reverse take on the same flow chart tree described previously. It's built by adding the same steps but in reverse order. It is used to assess the step paths causing the largest volumes of dropped conversations in conjunction with the END step. The END step is added as a supplementary step, as described in the data manipulation task above.

Data Points Used:

• Categories: .... Step3, Step2, Step1
• Value: Count of _id (Conversations)

Visualization 4: Pie Charts Tree / Destination Step Analysis

This visual uses the same Pie Charts and Tree graphs as described earlier but with data from the Executed Steps record. It's built by adding the destination and origin steps as categories and counting the number of occurrences. The destination and origin steps are manually generated fields created by referencing the entityReferenceId and parentStep values to their respective labels in the Steps collection. This visual allows monitoring of the reasons for a single step being reached and the volume that is being generated from each parent step.

Data Points Used:

• Categories: Destination, Origin
• Value: Count of _id (ExecutedSteps)