Chatbot apps for communication and social interaction therapy

The challenge

Enabling at-home speech, language and social interaction therapy

Speech, language and communication difficulties are a common and unfortunate symptom of neurological conditions like Parkinson's disease or dementia, or in those recovering from a stroke.

For the one in 100 children with autism disorders, social and communication difficulties often present themselves at a very early age and can be carried throughout their entire lives.

These difficulties can severely affect a child's development, and can lead a person of any age to withdraw and socially isolate themselves, with further implications on their mental and physical health.

Unfortunately, support for communication recovery, management or improvement can be expensive and time consuming, and isn't often able to support a person's day-to-day activities outside of a clinical setting.

Our response

We’ve developed a unique smartphone chatbot app for at-home social and communication therapy.

Our researchers at the Australian eHealth Research Centre, together with the University of Queensland, developed Harlie, a smartphone app that uses AI technology and natural language processing algorithms to chat with humans – much like Apple's Siri or the Google Assistant does.

The difference with Harlie, however, is that the app's complex programming can provide therapy, education and virtual companionship for those who have special needs, are afflicted with a health condition or are even just lonely.

Rather than simply respond to questions, Harlie can ask insightful questions of the user, and can interpret and appropriately respond to declarative and exclamatory statements, such as: "It’s my first day at school today," or, "the kids teased me today."

Sample of dialog between chatbot app Harlie and a person with autism.

Key:

• green rectangles represent dialog generated from Harlie
• blue rectangles represent responses from the user
• grey rectangles indicate pauses in the timeline.

Sample 1 dialog

Harlie: Hi, I am Harlie. Thank you for testing me
Pause
Human: Hi Harlie on my name is *
Pause
Harlie: Hi! Its delightful to speak with you. How are you?
Pause
Human: I'm good thanks
Pause
Harlie: Okay, thanks for letting me know. Would you like to give a voice sample now?
Pause
Human: No I'm good thanks.
Pause
Harlie: That's okay. I hear this is a mens shed. Are you part of a community group?
Pause
Human: Umm yes I am.
Pause
Harlie: I hear that community groups often offer the opportunity to craft and build things. Is that something you enjoy doing?
Pause
Human: Oh yes I do this is really exciting.
Pause
Harlie: I see. What kind of things do you wish you could make one day?
Pause
Human: Wooden train.
Pause
Harlie: What project would you recommend to a beginner like me?
Pause
Human: Uh huh a bird feeder.
Pause
Harlie: What supplies would I need to buy for my toolkit?
Pause
Human: Hammer, nails, good supply of wood.
Pause
Harlie: Thanks for the advice. Thanks for the great chat, but I ahve to be off now. Bye.
Pause
Human: Nice speaking with you.

Sample 2 shows an excerpt of the main dialog

Human: I'm good thanks
Harlie: Okay, thanks for letting me know. Would you like to give a voice sample now?
Human: No I'm good thanks.
Harlie: That's okay. I hear this is a mens shed. Are you part of a community group?
Pause
Human: Umm yes I am.
Pause
Harlie: I hear that community groups often offer the opportunity to craft and build things. Is that something you enjoy doing?
Pause
Human: Oh yes I do this is really exciting.
Pause
Harlie: I see. What kind of things do you wish you could make one day?
Pause
Human: Wooden train.
Pause
Harlie: What project would you recommend to a beginner like me?
Pause
Human: Uh huh a bird feeder.
Pause
Harlie: What supplies would I need to buy for my toolkit?

Show image description

Harlie can converse on a variety of topics, from travel and music to family and pets, in a judgement-free environment. During chats, Harlie can capture information on the user's speech, including articulation, vocabulary range and the length of mid-sentence pauses. Harlie can process and pass this information, with consent, to health and research teams to aid further research and provide more targeted therapy.

A screenshot of Harlie Chatbot smartphone app showing a conversation about bullying.

The conversation includes the following dialogue:

[Date and time stamp]
Chatbot: How are you today? (chatbot icon)
[Date and time stamp]
Person: People are being mean to me (person icon)
[Date and time stamp]
Chatbot: Are you being bullied? (chatbot icon)
(image of older person grabbing a younger person with hands showing an intimidating closed-fist gesture).

Show image description

The AI technology backing Harlie is the result of more than five years' collaborative research conducted by computer scientists, clinical and social researchers, as well as individuals who suffer from health conditions, their carers and their wider community.

Thanks to this AI, Harlie's capabilities can continue to evolve through input from clinicians, teachers and app users themselves.

The results

We’ve received funding to trial the technology behind Harlie in a variety of settings

We’re working with the Autism CRC and Curtin University on a pilot study, investigating the use of chatbots like Harlie to support adult tertiary students on the autism spectrum.

Our team has also received funding for more activities involving AI:

• a chatbot helping to build bullying-resistance skills for children on the autism spectrum
• a chatbot to complement genetic counselling sessions, working with Melbourne Genomics Health Alliance
• an automated call system to interact with and support individuals recently discharged from hospital.

In 2019, Harlie was a merit recipient for Research and Development Project of the Year at the Australian Information Industry Association’s Queensland iAwards.