This article explains how to build analytics powered Cognigy.AI flows that track flow steps and allow detailed success or failure analysis via a Microsoft Power BI analytics dashboard.

Requirements

• Good understanding of Cognigy.AI flows
• Good understanding of the Cognigy.AI OData Analytics endpoint
• Microsoft Power BI Knowledge (Also possible in Excel or any other Data visualization tool that can access an OData endpoint)

Introduction

   Watch Cognigy Sessions Episode "Analytics & OData" for a technical deep dive

Monitoring the successes, and more importantly the failures, of a virtual agent is critical for understanding the value it is bringing to the business and ensuring this value is consistently increased. The industry standard for bot performance monitoring is to set goals, keep track of intents and monitor NLU scoring i.e. number of times an intent was triggered, average intent score, number of confirmed orders etc. These typical metrics are can be monitored as part of the standard dataset available via the Cognigy.AI OData Analytics endpoint and are ready to be used out-of-the-box for any bot deployment with Cognigy.AI.

The visual flow concept that Cognigy.AI delivers, provides conversation designers with an industry leading graphical editor, placing conversation design into a visual map with flow nodes becoming key reference points throughout conversations. In some situations, drawing conclusions about flow performance via the standard goal and intent data is simply not enough. To achieve a more direct view of conversation flow performance, we can can monitor steps that occur during conversations and throughout our flows that outline a conversation path for any session. Visualization dashboards can then be designed in parallel to Cognigy Flows that best present the successes and failures based on the actual conversation paths that have taken place. These dashboard can then be used by conversation designers to identify weaknesses and further improve the bot performance with a deeper understanding of actual conversations.

Let's take a look at how we can achieve this..

Task 1: Preparation

Location: Cognigy.AI

Before getting started with thinking about dashboards and visualization, it pays to first consider the design best practices for Cognigy.AI. Conversation designers should consider a few simple steps to make the flows more understandable which will in turn lead to clearer linking between data and conversational flow.

Please take the time to review our Best Practises section before proceeding with this concept. 

In particular, the requirement for clear and logical naming conventions, from flow names through to node labels is a requirement as this same convention will be used below.

Task 2: Create Flow Steps

Location: Cognigy.AI

Steps are the name we give to a point in the flow that we want to highlight as a important phase in the conversation. This is typically linked to a single execution of a flow, but step can be created for any node or intent that occurs in a flow. This could be the first welcome message, a question that is asked or an action that is taken. Any nodes or intents that have steps configured in the respective settings menu will be added to the executed step list when activated by the flow in the order they are activated.

Configuring Nodes as Steps

Any node in the flow chart can be configured as a step by adding a value to the "Analytics Step" field in the editing menu. This could also be a dynamic value using CognigyScript or Tokens. Question nodes will automatically be assigned a their default label in the "Analytics Step" field, however all other nodes must have the step name manually added.

Configuring Intents as Steps

When using the Cognigy NLU, any intent in the intent list an be configured as a step by adding a value to the "Analytics Step" field in the advanced settings menu. 

Task 3: Connect and Prepare the OData Records

Location: Microsoft Power BI

Once you have added your step and analytics nodes within your flow, it's time to carry out some test conversations. It pays to have a good set of sample data that provides examples for many of the different outcomes so that you have good sight of what you want to visualize in your dashboards. Once you have done this, import your OData analytics data to Microsoft Power BI (Watch this Cognigy Sessions Episode to find out how). 

Accessing the OData Records

Cognigy version 4.2.0 adds new OData records to the existing line up. The original "Records" and "Conversations" tables are renamed to "Inputs" and "ChatHistory" respectively. We also have 3 new tables to assist with building visualizations with the step concept. Please refer to our documentation page for details on how to connect to these records.

For this example, import the following records to Power BI:

• Steps:
  • Contains a list of all steps that exist in any agents, one record per step
  • Includes the step entity reference ID
  • Includes the step label from the Cognigy UI
  • Includes flowName, projectName, snapshotId and more..

• ExecutedSteps:
  • Contains a record for each step that has been executed
  • Includes the step entity reference ID
  • Includes the step label from the Cognigy UI
  • Includes the parent step entity reference ID that we executed prior
  • Includes ProjectID, ProjectName, flowName, inputId, userId, sessionId and more..

• Conversations:
  • Contains a record for each session that has taken place
  • Contains a field called stepPath which is a CSV list of all steps executed in that session
  • Contains a field called stepsCount which is a numerical count of all steps executed in that session
  • Includes goals, sessionId, localeName, ProjectId, projectName and more..

Manipulating the Data

The Conversations record must be manipulated to extract a column for each step that was executed. To do this, open the table in the Power query editor by right clicking on the record and selecting "Edit Query" as shown below:

Once in the query editor, select the column "stepPath" then "Split Column" then "By Delimiter". Set the delimiter to "Comma" and click OK.

This will result in a new column being created for every step that was executed throughout a given conversation containing the entityReferenceId of that step.

In order to replicate this table with the given step label as created in the Cognigy.AI UI, we need to create new columns for each step that we wish to monitor and lookup the label name by the entityReferenceID from the "Steps" record. The reason the entityReferenceId is used by default is that the step label may change at any time by a user in the Cognigy.AI UI, therefore we reference it to a list of the current step labels where the entityReferenceId never changes.

To achieve this, select the conversations record in Power BI and click "New Column". Use a formula similar to the following to retrieve the step label from the "Steps" record:

Step1 = LOOKUPVALUE('Steps'[label],Steps[entityReferenceId],'Conversations'[stepPath.1])

Adding an END Step

As an addition to this, in order to create a new step at the end of each session to indicate when a session ended, use the following formula:

Step1 = IF('Conversations'[stepsCount]=0,"END",LOOKUPVALUE('Steps'[label],Steps[entityReferenceId],'Conversations'[stepPath.1]))

This formula checks the steps count to see if the conversation ended on the previous step. If so, the word END will be placed as the current step. Ensure you update the step count comparative value to reflect each step. E.g. Step 1 check for a stepCount of 0, Step 2 check for a step count of 1, etc.. An example of Step 2 is below:

Step2 = IF('Conversations'[stepsCount]=1,"END",LOOKUPVALUE('Steps'[label],Steps[entityReferenceId],'Conversations'[stepPath.2]))

Create a new column for each step that you wish to monitor in you analytics dashboard with the same formula pattern as explained above. The result should look something like this:

Once this is complete, you are ready to build some dashboard visualizations.

Task 4: Create Data Visualizations

Location: Microsoft Power BI

We can now use this data to build a graphical visualization in Microsoft Power BI.

There are many different ways to view this data using both built in Power BI visualizations and the extended visuals available via the Power BI visuals market place. For monitoring multiple steps in a flow, the "Pie Chats Tree" is an excellent visual to add to your dashboards that can be used in conjunction with filters to show monitoring for selected steps.

Visualization 1: Decomposition Tree / Conversation Flow

This is visual is a basic flow chart using an standard flow chart graph from the PBI set called the Decomposition Tree. This visual allows a single flow path to be displayed at one time from start to end. This makes it great for understanding the volumes of certain use-cases. Remember that multiple charts can be created as a cluster for your dashboard so multiple decomposition trees could be used to highlight certain routes with your conversations.

Data Points Used:

• Analyze: Count of _id (Conversations)
• Explain by: Step1, Step2, Step3, Step4, ... etc

Visualization 2: Pie Charts Tree / Conversation Flow

This visual is a more complex flow chart tree called the Pie Charts Tree and is available on the Power BI market place to download an install. This visual allows any number of tree branches of a conversation to be shown at once and includes a small pie chat at the end of each branch explaining the volume that each branch holds. It also allows branches to be collapsed and manipulated to view only the paths that are required. Caution with large datasets as this visual could quickly become to large to handle.

Data Points Used:

• Categories: Step1, Step2, Step3, Step4, ... etc
• Value: Count of _id (Conversations)

Visualization 3: Pie Charts Tree / Reverse Conversation Flow

This visual is a reverse take on the same flow chart tree described previously. It's built by adding the same steps but in reverse order. It is used to assess the step paths causing the largest volumes of dropped conversations in conjunction with the END step. The END step is added as a supplementary step as described in the data manipulation task above.

Data Points Used:

• Categories: .... Step3, Step2, Step1
• Value: Count of _id (Conversations)

Visualization 4: Pie Charts Tree / Destination Step Analysis

This visual uses the same Pie Charts Tree graphs as described earlier but with data from the Executed Steps record. It's built by adding the destination step and the origin step as categories and counting the number of occurrences. The destination and origin steps are manually generated fields created by referencing the entityReferenceId and parentStep values to their respective labels in the Steps collection. This visual allows monitoring of the reasons for a single step being reached and the volume that is being generated from each parent step.

Data Points Used:

• Categories: Destination, Origin
• Value: Count of _id (ExecutedSteps)