AI

Research methods for developing AI powered services

AI is already widely deployed in back-end systems to analyse complex data (web stats, retinal scans). When used with speech, voice, text and facial recognition (Siri, Alexa, chatbots, Gmail smart compose etc.), and gadgets that automatically collect and analyse user data (Google’s Pixel 3 smartphones, smart fridges etc), it could be the genie is already out the bottle. 

Artificial intelligence and ‘machine learning’ will be used in areas such as radiology, dermatology and pathology to ‘improve clinical care’ – and to get through more time-consuming work than health professionals can,

NHS England chief executive Simon Stevens

A previous post looked at how chess playing computers are at the forefront of development. Though designed to play strategy games, the champion AphaZero is more versatile and considered a  “broad” AI capable of both supervised and unsupervised, reinforcement learning. Besides games, it has learnt to model complex proteins as AlphaFold. 

AI and big data are quickly becoming widespread and capable of doing things that couldn’t be done before.

Chatbots like IBM’s Watson and Microsoft’s Azure are designed to be customised for different customer service roles and AI development platforms like Google’s Tensorflow already create live services. The number of services described as smart increases, and even traditional services increasingly have an automated component.

Requirements, process and constraints are already known for existing services, but AI will present new challenges and questions. Risks are reduced and benefits increased when research begins before development, and the codebase reflects the world beyond the view of the organisation or IT department. 

This post explores current user research methodologies useful for informing the development, training and testing of intelligent decision support systems (IDSS) and AI/human interfaces. New ones will be needed to support user centred design for this emerging technology.

In the Turing test, a person communicates via a text terminal with two hidden conversational partners: another human and a computer. If the person cannot distinguish between the human and the computer, then the computer would be considered to be behaving intelligently.

Usability First

Formative and summative user research
(aka generative and evaluative)

    • Formative explores requirements and context
    • Summative, the efficacy of a design or build in meeting them
    • Formative research typically gives way to summative as development progresses and there’s something to test

Formative research

Methods and outputs

  • What are the attitudes and beliefs about smart, new technologies?
e.g. automated systems might be less accessible, trusted or tolerated 
Method- 
Qualitative research: ethnographic and attitudinal studies, user and stakeholder interviews
Contextual enquiry: observing interactions
Quantitative research: surveys
 
Output  –
Insight report, anecdotal evidence
  • What might be the challenges and effects of introducing smart technologies for both the provider and users
Method – 
Literature review, impact assessment
User and stakeholder interviews
 
Output  – 
Experience map showing highs and lows across service touch-points,
  • Will the system be accessible to people with disabilities or communication requirements

     

Method –
Specialist accessibility audit and gap analysis
Ensuring research demographic spans the Gov.uk digital inclusion scale 
 
Output  –
Accessibility report, ideas for alternative user journeys 
  • Is a controlled natural language (CNL) required and if so, what is its vocabulary?
Method –
Analysis of call logs: Bag-of-words (BoW)

Interviewing and observing interactions

Output  –
Lexicon, or a more formal controlled vocabulary e.g. Regular expressions (Regex
  • What are the conditions and choices for the user, and the available paths in the system 
e.g. most simply the menu and submenu options for a website’s navigation – decision trees
Method –  
Analysis of call logs etc., card sorting
 
Output –
Scripts, questions and phrases, tree diagrams, navigational hierarchy  
  • What will be helpful feedback for both the system and the user (positive and negative)?
e.g. “From how you have described your enquiry the HMRC tax helpline might be more relevant.” 
Method – 
Interviewing and observing frontline operations
 
Output –
Scripts for system response, model questions and answers
  • More generally, what are the use cases, devices, connectivity, user stories, flows and navigation
Method – 
Numerous
 
Output – 
Use cases and user stories suitable for a product backlog, process (user) flow diagram, 
Tree diagrams can map systems and show user options and paths

Summative research – testing

Methods and outputs

  • Evaluating the “smoothness” of stepping through the process across service touch points and in realistic timeframes
Method – 
Usability testing, diary studies, service logs and anecdotal evidence
 
Output – 
Issue report , completion times
  • Evaluating the transitions (handovers) between different contexts/modalities 
e.g. a shopping list compiled on a phone, order placed with a pc, the “shop” calculated and receipt issued at point of sale, stock systems updated, and delivery signed for on a pda.
Method –
Usability testing, Amazon’s Mturk (also used to evaluate use cases), analysis of drop off rates at boundaries
 
Output –
Drop off and bounce rates
  • Evaluating the efficacy and efficiency of automated processses
Method –  
Usability testing, interviews, questionnaires, observing frontline operations
analysis of server logs, bounce rates, completion rates, anecdotal reports
 
Output –
Service metrics
“Life, the Universe, and Everything. There is an answer. But, I’ll have to think about it.
The Answer to the Great Question… Of Life, the Universe and Everything… Is… 
Forty-two,’ said Deep Thought, with infinite majesty and calm.” 

 

The Hitchhiker’s guide to the galaxy

Leave a Reply

Your email address will not be published. Required fields are marked *