©
2008 The MITRE Corporation. All rights reserved.
A National Resource Working in the Public Interest
Approved for Public Release
An Expertise Finder
Application Built on
Enterprise Search
Robert Joachim, rjoachim@mitre.org
MITRE Corporation, McLean VA
Gilbane San Francisco 2008
June 19, 2008
MITRE Corporation
Fortune Magazine “100 best companies
to work for” (2002
-
2008)
Computerworld “100 best places to
work in IT” (2005
-
2008)
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
2
Overview
About MITRE, our Intranet, our enterprise search
architecture
MITRE Expertise Finder implementation
–
Expertise finding models (APQC)
–
MITRE expertise finding history
–
This product
Interface details
How its built / How it works
–
System validation / Usage metrics
–
Nearer and longer term enhancements
–
Conclusion / Recommendations
Background
–
Sources / Resources
Other recent ‘real world’ expertise finder implementations
–
Commercial (COTS) software for expertise finding
–
Community finding prototype example
–
Use of social bookmarks at MITRE
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
3
About MITRE
About MITRE
–
Not
-
for
-
profit; operates 3 Federally Funded Research and
Development Centers (FFRDCs), for DoD, FAA, and IRS
–
Application of expertise in systems engineering, information
technology, operational concepts, and enterprise
modernization
–
6000 employees located at Bedford, MA, McLean VA, plus other
domestic and international sites; 65% of staff have Masters or
Ph.D. degrees
Our role
–
Problem solving / rapid response for our sponsors
–
‘Reachback’ into the corporation for knowledge is key
Long standing history of information sharing practices
–
Embedded and reinforced in our corporate culture
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
4
MITRE Intranet & Enterprise Search
MITRE Intranet is called the ‘MII’
–
MITRE Information
Intranet
–
Early adoption of web technology & web search
–
Our intranet consists of multiple content repositories on
various platforms, including
Oracle Portal & Oracle application servers
Intranet content server & multiple distributed content servers
Microsoft SharePoint for team site management and collaboration
Listserv lists for collaborative communication
Google Enterprise is MITRE’s intranet search engine
–
The expertise finder application described here is based on
Google enterprise search
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
5
MITRE Google architecture
Expertise Finder
Application interfaces
MITRE intranet search &
‘focused search’ interfaces’
GSA 5005
Database crawls
Technical Exchange
Meeting Search
Social Bookmarks
XML feeds
MITRE Intranet
server &
SharePoint
document
libraries
-
URLs: 400K
Content repositories
(2.2 M URLs total)
Intranet
Web
-
enabled file
system +
distributed MITRE
Webservers (40)
-
URLs 1.4 M
MITRE List
messages
-
URLs: 450K
Email List Search
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
6
MITRE Expertise Finding History
Current system is based, in part, on earlier MITRE research
and prototype work
–
MITRE staff: Maybury, House, D’Amore
First developments of this system, based on Google
enterprise search results, were also prototypes
–
Then, released as a pilot project to collect user feedback
–
Subsequently productized
–
Then enhanced over multiple releases
Additional functional search and display features
Additional content resources for expertise identification
Architectural focus
–
created using
–
Service
-
oriented componentized architecture
Loosely
-
coupled building block pieces, that can be swapped in/out,
if necessary (“What if we were to replace
--
our enterprise search
system , our staff directory system”, etc.)
–
Extensible
Can be extended to other content repositories or could be used for
alternate ‘finding’ applications
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
7
Expertise finding systems
--
characterization
APQC (formerly American Productivity Quality Center)
characterization of Expertise Finding systems:
APQC ‘Model 1’
--
Linking knowledge seekers with
knowledge providers
–
No
a priori
designation of ‘experts’
–
This is our approach
APQC ‘Model 2’
–
assigned discipline managers are
responsible for knowing levels of expertise in their area
APQC ‘Model 3’
–
designated ‘validated’ experts
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
8
Main view
MITRE Expertise Finder
What it looks like
Helps answer the question:
“Who at MITRE knows about
topic
X
“
Results are based on Google
relevancy ranking, in
conjunction with author/owner
attribution & document counts
Organizational view
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
9
Main view
Expertise Finder
–
Results details
Source options
Display options
Email contact options
Person, with job title and link to phonebook
Content ‘evidence’ (with
title, links to object &
repository, ‘keywords in
context’, object date)
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
10
Expertise Finder
–
Results details
Organizational view and content display by organization
Bubble size and position
indicates contributors
and contributions by
corporate center or
division
Clicking on a single
bubble displays people
and content from that
organization
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
11
How it works
Expertise Finder results
(staff and organizations)
4
LDAP
(staff / organization
lookup)
3
Key MITRE Web
-
based repositories
contributing to Expertise Finder
Staff attribution
Resource
Author / owner
attribution
Web
-
enabled file system
(Employeeshare transfer
folders,
‘about
-
me’ resumes)
Standard User
Identifier (SUI)
in folder path
CommunityShare
(MS SharePoint)
MS Office Property Author
Field
MITRE blogs
SUI (email name)
MITRE Sourceforge
HTML Author
meta tag
MITRE List Messages
SUI (email name)
MITRE Technical Exchange
Meetings
Meeting Point of contact
MITRE Institute Courses
Course instructor (future)
Onomi social bookmarks
Bookmark contributor (future)
2
MII Google
(Google Search
Appliance)
User
Query
MII Google query / results
1
http query
XML
Expertise Finder
(Java
-
based application on Oracle
Application Server)
1. Based on keyword query, retrieves
ranked results set from MII Google
search (XML output)
2. Identifies author / owner attributes
3. Performs LDAP lookup for full staff
name and organization
4. Returns results set, ranked by
contributions, with hits ‘evidence’ and
keyword context
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Resource and staff attribution details
Page
12
2
Resource
Author / owner
attribution
Person
Metadata
Quality
CommunityShare (MS SharePoint)
MS Office Property Author
Field
Varies
MITRE blogs
Standard user identifier
(email name)
Excellent
MITRE Sourceforge (Software
projects)
HTML Author meta tag
Excellent
MITRE List Messages
Standard user identifier
(email name)
Excellent
MITRE Technical Exchange Meetings
Meeting Point of contact
Excellent
From
step 2,
previous
slide
MITRE Institute Courses
(future resource)
Course instructor
Excellent
Social bookmarks (Onomi)
(future resource)
Bookmark contributor
Excellent
HTML pages from distributed
webservers
HTML Author meta tag
Varies
MS Office documents from distributed
webservers
MS Office Property Author
Field
Varies
Web
-
enabled file system
(Employeeshare transfer folders,
‘about
-
me’ resumes)
Standard User Identifier
(SUI = email name) in folder
path
Excellent
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
13
This system architecture: Advantages / Disadvantages
Advantages
–
Uses full
-
text indexed content (concepts are ‘fluid’
–
especially
for new technologies, products, projects)
–
Uses the same online content contributors share in the course of
their day
-
to
-
day work
–
No requirement for users to maintain a registry of expertise
–
Incentivizes staff to share content online in open repositories
–
Incentivizes staff to use correct metadata, especially authorship
metadata
Disadvantages
–
Results are only as good as the quality of authorship metadata
used to associate information objects with staff
–
Results are dependent on the underlying search system
relevancy ranking
Although, we also force specific repository results by sending
multiple parallel queries to multiple content repositories
–
Users could ‘game’ the system (by arbitrarily putting large
numbers of documents online)
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Query characteristics
--
observations
Best performing queries
–
Specific (‘term specificity’): Products, programs, projects,
standards
–
query terms that are ‘good discriminators’
–
Query examples:
Standards/Compliance: IEEE
-
1061, fisma
Products: Cognos, AppWorx
Projects: Next
-
generation airspace
Topics: ontologies, second life, biometrics
Worst performing queries
–
Extremely general terms, whether single or multiple words
–
Query examples:
‘ Engineering’
–
in a corporation where a majority of staff function
in some engineering capacity and are performing engineering
-
related tasks
‘Software’
–
in a corporation of where a significant portion of our
work focuses on some aspects of software engineering
–
But
–
consider
–
these very general queries may not perform
well in general full
-
text retrieval anyway
Page
14
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
15
Results evaluation/validation methods
Validation methods
–
Informal: Send in a query based on a topic where the user knows
a set of experts/specialists, see how many come back in results
Many staff take this on themselves, as a check to see if they are
included in results
–
Informal: when a user submits an email to contacts identified,
informal email probe to that user
“Was this system helpful in identifying knowledgeable staff”
“Did you get an answer”
–
Metrics: Continued usage by staff
Query metrics have held steady over time, paralleling general search
query metrics
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Usage metrics and query analysis
Usage metrics tracking by
–
Basic usage
From Google query logs
--
Expertise Finder interface queries are
coded with a specific parameter for identification in query logs
From internal WebTrends web analytic reports (user visits, page
views)
–
Query analysis
We can identify specific queries sent to the system by analyzing
query log data
Monthly usage metrics
–
General MITRE Google queries: 60K
-
75K queries per month
–
Expertise Finder queries: 2K
-
4K queries per month
–
Usage ratio average of general search to the expertise finder
application
--
~25 : 1
Page
16
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Expertise Finder enhancements
New enhancement in development: ‘Presence awareness’
identification using a Microsoft Office ActiveX Web service
System enhancements under consideration
–
Limit by date/date range (to find most staff based on most
recent contributions)
–
Limit by more detailed level of contributing repositories
Beyond just ‘Documents and Webpages’ and ‘Lists’
E.g., SharePoint, Technical Exchange Meetings, Social Bookmarks
–
Permit user to limit and/or sort by staff classification/role, e.g.,
‘AC’ Technical; PRO ‘professional level support’; PSS
Administrative support
Page
17
Shows staff availability
online, free/busy status,
access to Office
Communicator chat,
other tools
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Longer term: where we may be going
Exploration of social networking for expertise finding
–
Use of staff profiles based on social networking models (similar
to MySpace, Facebook, LinkedIn)
–
Use social network connections as an additional dimension in
expertise finding
Hybrid approach
–
base expertise finding on
–
Text from document content
As we are doing
–
there will continue to be value in identifying
expertise from content objects
and
–
Staff profiles, which may be
User
-
generated, auto
-
generated, or a combination
–
Let the user decide, per query, how to focus the results based
on these resources
Page
18
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Conclusion/recommendations: expertise
finding implementation
If you are considering expertise finding implementation
Consider which APQC model fits your organization’s
environment and requirements
If implementing a software application
–
Evaluate metadata quality for staff attribution
–
Decisions
Staff identification based on registry vs. content, or hybrid
Build vs. buy
Build in conjunction with enterprise search
–
Use of service
-
oriented architecture
For swap
-
in/swap out of code base, directory resources, and
content resources
For future feature enhancements
Page
19
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Background info
Page
20
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
21
Sources/Resources
Google Scholar results
–
MITRE expertise finding research
–
MITRE authors: M. Maybury, D. House, R. D’Amore
http://scholar.google.com/scholar?q=Maybury,+House,+D%E2
%80%99Amore
Ackerman, Mark and McDonald, David “Just Talk to Me: A
Field Study of Expertise Location” Proceedings of the 1998
ACM conference on computer supported cooperative work,
Seattle, November 14
-
18, 1998
–
http://portal.acm.org/citation.cfm?id=289506
Hughes, Gareth and Crowder, Richard “Experiences in
designing highly adaptable expertise finder systems”
Proceedings of the DETC 03, Chicago, September 2
-
6, 2003
–
http://eprints.ecs.soton.ac.uk/8206/
Expertise Locator Systems: Finding the Answers. APQC
Publications: 2003
–
http://www.apqc.org/portal/apqc/ksn?paf_gear_id=contentgear
home&paf_dm=full&pageselect=detail&docid=123338
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
22
Sources/Resources
Maybury, Mark Expert Finding Systems, MITRE
Corporation: MITRE Technical Report, MTR 06B00040,
September 2006
–
http://www.mitre.org/work/tech_papers/tech_papers_06/06_111
5/06_1115.pdf
Maybury, Mark “Discovering Distributed Expertise” AAAI
Fall Symposium Series,
Regarding the “Intelligence” in
Distributed Intelligent Systems,
November 9, 2007
–
http://www.mitre.org/work/tech_papers/tech_papers_07/07_073
0/07_0730.pdf
Damianos, Laurie, et al. Onomi: Social Bookmarking on a
Corporate Intranet, MITRE Corporation, May 2006
–
http://www.mitre.org/work/tech_papers/tech_papers_06/06_035
2/06_0352.pdf
Author contact: rjoachim@mitre.org
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
23
Sources/Resources: Expertise finding recent
real
-
world implementations
Presented or cited at Enterprise Search Summit, New York,
May 20
-
21, 2008
–
“Mining additional value from enterprise search”, Trent
Parkhill, Haley & Aldrich
Based on search product: Coveo
–
“Search connections in context”, Oz Benamram, Morrison &
Foerster
Based on search product: Recommind
–
Google, Inc. internal expertise finder
Based on search product: Google enterprise
Montague Institute, January 17, 2008
–
“Enterprise mashups for expertise location”, Qin Zhu, HP Labs
Based on search product: Inktomi
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Commercial software products for
enterprise expertise finding
--
examples
Enterprise search with expertise finding components
–
Autonomy IDOL
–
FAST (Partnering with AskMe)
–
Endeca
–
Recommind
–
Microsoft SharePoint Enterprise search (with MOSS07
“Knowledge Network”)
Dedicated/specialized systems
–
TACIT ActiveNet
–
AskMe
–
Triviumsoft SEE
-
K
Page
24
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Community Finding (Prototype)
Page
25
Based on MITRE
Expertise Finder code
Identifies MITRE online
communities (Email lists
& SharePoint
communities)
Uses search results of
–
List messages
–
Documents associated
with a community
–
Community
descriptions
©
2008 The MITRE Corporation. All rights reserved.
5
-
Nov
-
13
Approved for Public Release
Page
26
“Onomi” Social Bookmarks
–
Tagging/Sharing
of Web Resources by MITRE staff
Onomi
(rhymes with
‘Taxonomy’)
–
based on
open
-
source tool Scuttle
–
Lets MITRE staff
bookmark content
resources and tag content
of interest with topical
terms
–
Helps me “find this again”
–
Builds communities of
interest
–
And
–
contributes to
expertise finding:
Users tagging/contributing
are ‘experts’ in the
content/topics bookmarked
and tagged
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Σχόλια 0
Συνδεθείτε για να κοινοποιήσετε σχόλιο