Deploying Directory Assistance Automation Solutions

grassquantityAI and Robotics

Nov 15, 2013 (3 years and 10 months ago)

182 views

1

Deploying Directory Assistance Automation Solutions

February 2007

Krishnan Srinivasan, Senior Manager,

Professional Services, Menlo Park, CA

2

Motivation for Automation


Service draw large volumes
(hundreds of millions of calls / year)


Automation helps to


Save costs associated with operator work time and positions


Improve operator productivity


Provide a consistent user interface and encourages callers to be
more specific about their requests


Challenges posed by Automation


Technical


(high accuracy with low false accept rate)


User acceptance


(getting used to automated service)


ROI to the provider

(get reasonably quick return on investment)


Operator acceptance

(provide seamless service along with automation)

3

Full
Automation

Partial
Automation

Voice
Store
and
Forward

Types of Automation


Role of Speech Recognition

4

Res

Biz/Gov

DA Automation


High Level Call Flow


Locality


Listing Type


Listing Name


Last Name


First Name


Multi
-
Tier Confirm

Caption/Location/Mixed

Disambig

Name/Address

Disambig

Number/Information


Release

5

All Calls

Factors that Influence Automation Rates

System Losses

Service Scope

City, Listing Losses

Automated Calls

VUI Gain

Caption Navigation

Losses

6

Automation Requests


FRL Strategy

Automate top N listings (typically 2
-
10 K)


Advantages


Smaller grammar resulting in good in
-
grammar recognition
accuracy


Simple dialog that blends with partial automation


Disadvantages


FRL churns (weekly, monthly, seasonal)


High out of domain rate from non
-
FRLs resulting in higher false
accepts


High maintenance cost to keep FRLs updated


Limited coverage


Limits on how much automation can be achieved

7

Automating Requests


Full Coverage Strategy

All listings are potentially automatable


Advantages


Good coverage


Lower churn than FRL


Automated builds


Increased potential for automation


Challenges


Bigger footprint


Need good recognition/search technology


Sophisticated dialog to navigate through complex listing captions


Need for sophisticated confirmation and error recovery

8

What makes for a good DA Application?

Optimized Caller
Experience


Caller
-
Driven Call
-
Flows


Intelligent, configurable dialog engine enables solution to
dynamically determine the most effective dialog strategy

Higher levels of
Accuracy and
Understanding


Higher coverage of natural requests with advanced
technologies


More effective, natural interaction using intelligent dialog
engine


Higher performance for noisy environments, wireless

Dramatically Lowers
Maintenance


Automates the creation and maintenance of system
grammars


Reduces maintenance by over 90% compared to manual
approaches and substantially reduces error rates

Simplified
Administration &
Performance Tracking


Provides application management and reporting
capabilities for improved business intelligence

9

Design Driven by Caller Expectations


Maintain User Confidence


Confident that they can complete the task and get accurate information


User Satisfaction is directly correlated with the user confidence



Provide the most accurate number/information


Don’t just release *any* number, but rather the *right* number


Minimize the number of repeat calls due to releasing the incorrect number



Maximize efficiency


Only ask questions that are absolutely necessary to resolve to final listing


Decide to “partially automate”, when likelihood of a successful interaction
is low


Truly efficient DA interactions are the ones that are completely automated

10

Key Elements of Directory Automation


Listing Normalization


Directory databases are designed for printed form and for lookup
by human operators. The listings are usually hierarchically
indented for presentation on print.


Need to adapt/normalize listings for automated search


Listing Recognition and Search


Recognize and understand callers “naturally spoken” query


Search for the requested listing in the database


Listing Disambiguation


Progressively narrow down the search till a single “releasable”
listing is found


Update Process


Database changes everyday

11

Listing Normalization: Examples

Standard Abbreviations

LTD


Limited

Non
-
Standard Abbrev.

SVC


Service

Abbreviation by Context

DISTR



School District


Film Distributors

Acronyms

,
-

_ : ; ’ ”

Acronym by Context

Revolucion

Acronym
-
ize Biz Names

University of California San Francisco
-
> UCSF

Morphology

ACTIVATE,ACTIVATING,ACTIVATION,

ACTIVATIONS

Numbers

KDFC 102.1

7
-
11

Call Signs

KDFC 102.1

MIX 106.1

Captions

Utilities
-
Inquiries & Complaints

12

Listing Normalization: Processing Steps

DB

Clean & Normalization

Distortion Analysis

Process for TTS

MetaDB

Grammars

TTS Prompts

13

Listing Recognition and Understanding
(from Nuance DA)




Burger King


Restaurant”


Uman Enterprises
.


U M A N Enterprises.”

“Um, is there a
Burger


King

in
Atherton
?”

“I want
Burger King


What was said?

What it means

(650)564
-
2348

(321)432
-
6232


Safeway

in

Crescent Park
.”

(602)932
-
8427


Safeway

near

the
corner of Alice


Identify
Salient
Segments

that contribute to meaning


Identify
noise
words

SLM,

SSM

14

Listing Search & Routing
(from Nuance DA)


SSM combined with Unsupervised Learning

Grammar Rules


(I'm looking for)


(I'd just like)

(I'd like)

(I'd really like)

(I'll have)

(I'm after)

(I'm searching in)


(sears)

(sears roebuck)

(sears retail stores)


(sports department at sears)


(movies at the regal theater)



. . .

Finite State:

Thousands of grammar rules
manually created and maintained

Statistical Semantic Models:

unsupervised learning


Statistical unsupervised learning, based
on actual customer calls

PLUS

15

Data Driven

Intelligent

Disambiguation

Listing

Recognition

Listing Disambiguation
(from Nuance DA)

Location?

(City)

Biz/Gov


Listing ?

Res Listing ?

Caption
-
1 ?

Location ?

(Street)

State

Disambig

Confirm

Error

Retry

“John Smith”

“San Jose”

“San Jose”

“Safeway”

“Pharmacy”

“McArthur drive”

“Safeway”

Caption
-
2 ?

“Bakery”

“Prescription refill”

“UnitedAirlines in

San Francisco”

What Name ?

“San Jose”

“Residential listing

in San Jose”

“Safeway in San Jose”

“Safeway”

16

Update Processor
(from Nuance DA)


Components


Formatter


Raw data reformatted to follow
standard structures


Normalization


Spelling correction,
acronym/abbreviation expansion,
context
-
sensitive modification
Statistically learned rules (Nuance
Blue origination)


Manual rules (from DA deployments)


Applicable to any domain


DA, EDA


Trainer


SLM training from normalized data
-

for
recognition of wide
-
ranging input


SSM training from normalized listing data
and field data


for semantic routing


Generic capability usable by Call Steering,
EDA



Compiler


Simply generation of the grammars in
a fast search format

Searchable

(Listing) Data

Update Processor

Formatter

Normalization

Trainer

Compiler

SLM

SSM

Rule

Pack

Raw

(Listing) Data

17

Architecture
(from Nuance DA)

Nuance DA Vxml

(Reference
Implementation)

Vxml Platform

ASR, TTS

DB Interface

Searchable

Listing Data

Update

Processor

DB Feed

OAM
-
Mgt

(Mgt Stn)

Component

Core Technology

DA Technology

Third
-
Party

NVP reference

EDA app

OSD

Disambig Engine

SSM

V
-
Builder

Admin


Data

Service Control

Alarms

Logging

DA, EDA
reports

Log

Data

Operator

Action

Server

Operator

Workstation

SWMS

OSI

OSW

18

Performance Tuning


Dialog Tuning


Grammar coverage (SLM)


Search Optimization (SSM)


Automated Pronunciation Tuning


Model and Language Adaptation


Parameter Tuning


Grammar weights


Recognition parameters


Text to Speech Tuning


Pronunciation


Stress and intonation

Performance tuning during pilot deployment

19

Usability


Iterative Usability


Based on simulated dialog (Wizard of Oz simulation)


Application not required


Refines design


Evaluative Usability


Based on pilot system


Select users from representative population


Call Monitoring


Interviews


Post Deployment Health Checks


Based on deployed system


Call Monitoring


Interviews


20

Implement & Rollout
(from Nuance DA)


Requirements


Business


User


Application


System


Platform Activation


Release 1:Dev/Int/Tst


Locality Auto


Straight Line Listings


VSF + Screen Pop


Data Collection App


Basic Call Flow


Biz/Gov/Res


Locality


Biz/Gov Listing


Last/First Name


Release 1


Pilot


(Live Caller Launch)



Tune


Locality


Listing




Release 2:Dev/Int/Tst


Captions Navigation


Location Disambig


TTS Confirmation


Number Release


Release 2
-

Pilot

(Live Caller Launch)



Tune


Locality


Listing


Caption


Location


Data Collection


Live Traffic


Transcription



Implement Tagging with Operator Actions


Data Normalization


21

Thank You!


Q & A