UBank chatbot drives home loan growth

UBank has seen a 15 percent increase in the number of people completing home loan applications after the creation of an experimental chatbot to guide people through the process.

The NAB-owned digital bank last year added IBM Watson to its LiveChat website function to step customers through the process of applying for a home loan online.

After just 10 months - and with the chatbot, known as Robochat, still officially in beta - the project has already achieved “outstanding results”, according to UBank’s head of digital and technology Jeremy Hubbard.

“We’ve had some really good uptake of Robochat,” Hubbard told IBM Think 2018 in Las Vegas.

“We see four out of five people happy to engage with Robochat, it’s answered over 22,000 questions, and it’s now answering the questions at a rate of 80 percent accuracy on the first attempt.

“We see the average length of a conversation being around four-to-five interactions with Robochat, and we can see from the logs customers are getting the data that they need to proceed with the application form.”

Crucially for UBank, the project is converting a higher number of prospective home loan applicants, getting them in the front door and through the process of applying for a loan.

“As a business, over the last couple of years we’ve been really focused on growing our home loan book, and one of the ways we do that is through improving our conversion rate - that is, people that start the process [and who] make it all the way through to the end,” Hubbard said.

“As a digital bank, one of the key ways we do that is by improving the digital experience.

“We’re seeing on average of about a 15 percent uplift [in the conversion rate] for those customers that engage with Robochat, which for us is an outstanding result."

UBank has previously discussed some of the mechanics of Robochat. It was essentially built over a six week period, or eight weeks if you include some pre-work around security and risk.

The mechanics of user experience

Hubbard said Robochat was handling “somewhere less than 100 interactions” per day.

“We’re only a mid-sized bank and our home loan volumes are relatively low,” he said.

What this means is UBank has had plenty of time to pore through Robochat’s logs to really understand what happens when home loan prospects try to engage.

The bank already gets some indication from a post-chat survey on the chatbot, which is used for an immediate measure of customer satisfaction.

“On a scale of one to five, we’re seeing an average of about 3.5 in terms of their satisfaction on Robochat, which is good but not outstanding I would say,” Hubbard said.

However, the scores given by users don’t always match up with how Robochat executes.

“I can say it can be sometimes painful when you see Robochat do the wrong thing, and sometimes really frustrating when it looks like Robochat’s done the perfect thing and still the [customer satisfaction] score comes through as a one-star,” Hubbard said.

While Robochat managed to answer questions correctly 8 out of 10 times - this has doubled in the 10 months since launch - Hubbard noted that user experience still varied wildly.

“When you look at that, it’s an average 8/10 it’s getting right,” he said.

“But for some customers, perhaps depending on their grammar or whether English is their second language, or the slang they happen to use, they might be getting much worse than 8/10. They might get 2/10 or 1/10 correct answers, while plenty of other customers are getting 10/10 questions answered.

“My biggest challenge is that inconsistency - 80 percent is really good but for some people it’s really terrible and for others it’s great, so [there’s] ongoing work there.”

UBank also found its augmentation of the Watson system led to reductions in the system’s accuracy.

“One of the things you’ve got to be careful of is adding more questions as part of training Watson [for Watson to be able to respond to],” Hubbard said.

“We got to 80 percent accuracy and then we kept adding more questions and it would drop back into the 70s.

“We were trying to work out why have we gone backwards and not forwards.”

The phenomenon has a name: “intent overlap”.

In Watson-speak, an intent is what the user wants to achieve from the question they’ve asked. UBank has mapped all the questions that users might ask Robochat against 40 intents. That then triggers the answers to those questions.

“There’s so many questions that Watson starts to get confused on the difference between one intent and the next intent because the questions start to overlap each other,” Hubbard said.

“I think there’s a new tool coming out, but certainly there’s a script IBM helped us with where we could analyse all the questions and look for the potential of that intent overlap.”

Hubbard said the team monitored the questions that customers asked Robochat, and manually added in any significant variations.

“My experience is Watson is not great at automatically adding those new questions in, so we have been manually monitoring and adding those questions in - you can do that through an analysis tool at the backend,” he said.

“We review the additional questions that have been asked, make sure they’re relevant to add to that set, and then add them.”

Analysing the chatbot logs have also turned up some insights into how well UBank explains its home loan products on its website.

“It’s interesting - we see a lot of people use Robochat on page one of the application form where we find out really they’re still just researching the product,” Hubbard said.

“I think it’s very clear that there’s an ‘apply now’ button on our product page and when you click that you’re starting a [home loan] application.

“So it’s strange to us - why are they asking questions about the product when they’ve gone into the next phase of the journey?

“We need to look at that and work out what aren’t we answering on our product pages, in our brochure-ware, before they get into the application process.”

The chatbot interface also exposes some of the residual challenges in making it easy for customers to apply online for a product that would typically be done face-to-face with a banker.

“One of the more complex bits in Australia is we need to ask customers to provide in the application form their general living expenses. This is a real sticking point in a digital channel," Hubbard said.

“We’ve tried through UX as well as calculators to make that simpler - we see them engage quite a bit with the chatbot in that space.

“That’s where we’d normally see a banker help out face-to-face a lot.”

Future applications

While Robochat remains available for us only for home loans, Hubbard said there was pressure from the business to expand it across all financial products.

“I think that’s on the roadmap,” he said.

“Right now it’s also only unauthenticated response or interactions we’re doing. Clearly it would be nice to do authenticated ones and we’re getting closer to being able to do that.”

Robochat is also just answering questions about home loans. Hubbard indicated UBank wanted to look at being able to take data from the chats to pre-fill a loan application on the customer’s behalf.

Last year, Hubbard suggested to an Australian-focused Watson event that the future could also hold applications of Robochat on voice-enabled platforms like Alexa.

Work on that has advanced, but has been put on hold owing to user experience challenges.

“We prototyped voice activation pretty early on,” Hubbard said.

“It’s technically really straightforward to do, so I think shortly after we launched literally in two hours we plugged it into Alexa, Google Home etc.

“What we found was that the user experience was not great for lots of reasons, but not least of which really simple things like we’ve got lots of acronyms and things that just don’t read out well in voice, and also it’s very hard for a customer to work out what they can ask [through a voice mechanism].

“So we’ve kind of put that on hold at the moment.

“What we’d like to do is actually help a customer complete a home loan application by actually taking the application via voice. It’s just down to how do we design that experience.”

Ry Crozier is attending IBM Think 2018 in Las Vegas as a guest of IBM.