Application Built on

observancecookieΑσφάλεια

5 Νοε 2013 (πριν από 4 χρόνια και 8 μήνες)

213 εμφανίσεις

©
2008 The MITRE Corporation. All rights reserved.

A National Resource Working in the Public Interest

Approved for Public Release

An Expertise Finder
Application Built on
Enterprise Search

Robert Joachim, rjoachim@mitre.org

MITRE Corporation, McLean VA

Gilbane San Francisco 2008

June 19, 2008

MITRE Corporation

Fortune Magazine “100 best companies
to work for” (2002
-
2008)

Computerworld “100 best places to
work in IT” (2005
-
2008)

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
2

Overview


About MITRE, our Intranet, our enterprise search
architecture



MITRE Expertise Finder implementation

–
Expertise finding models (APQC)

–
MITRE expertise finding history

–
This product


Interface details


How its built / How it works

–
System validation / Usage metrics

–
Nearer and longer term enhancements

–
Conclusion / Recommendations


Background

–
Sources / Resources


Other recent ‘real world’ expertise finder implementations

–
Commercial (COTS) software for expertise finding

–
Community finding prototype example

–
Use of social bookmarks at MITRE


©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
3

About MITRE


About MITRE

–
Not
-
for
-
profit; operates 3 Federally Funded Research and
Development Centers (FFRDCs), for DoD, FAA, and IRS

–
Application of expertise in systems engineering, information
technology, operational concepts, and enterprise
modernization

–
6000 employees located at Bedford, MA, McLean VA, plus other
domestic and international sites; 65% of staff have Masters or
Ph.D. degrees


Our role

–
Problem solving / rapid response for our sponsors

–
‘Reachback’ into the corporation for knowledge is key


Long standing history of information sharing practices

–
Embedded and reinforced in our corporate culture

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
4

MITRE Intranet & Enterprise Search


MITRE Intranet is called the ‘MII’
–

MITRE Information
Intranet

–
Early adoption of web technology & web search

–
Our intranet consists of multiple content repositories on
various platforms, including


Oracle Portal & Oracle application servers


Intranet content server & multiple distributed content servers


Microsoft SharePoint for team site management and collaboration


Listserv lists for collaborative communication


Google Enterprise is MITRE’s intranet search engine

–
The expertise finder application described here is based on
Google enterprise search

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
5

MITRE Google architecture

Expertise Finder

Application interfaces

MITRE intranet search &
‘focused search’ interfaces’

GSA 5005

Database crawls

Technical Exchange
Meeting Search

Social Bookmarks

XML feeds

MITRE Intranet
server &
SharePoint
document
libraries

-

URLs: 400K

Content repositories


(2.2 M URLs total)

Intranet

Web
-
enabled file
system +
distributed MITRE
Webservers (40)

-

URLs 1.4 M

MITRE List
messages

-

URLs: 450K

Email List Search


©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
6

MITRE Expertise Finding History


Current system is based, in part, on earlier MITRE research
and prototype work

–
MITRE staff: Maybury, House, D’Amore


First developments of this system, based on Google
enterprise search results, were also prototypes

–
Then, released as a pilot project to collect user feedback

–
Subsequently productized

–
Then enhanced over multiple releases


Additional functional search and display features


Additional content resources for expertise identification


Architectural focus
–

created using

–
Service
-
oriented componentized architecture


Loosely
-
coupled building block pieces, that can be swapped in/out,
if necessary (“What if we were to replace
--

our enterprise search
system , our staff directory system”, etc.)

–
Extensible


Can be extended to other content repositories or could be used for
alternate ‘finding’ applications

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
7

Expertise finding systems
--

characterization

APQC (formerly American Productivity Quality Center)

characterization of Expertise Finding systems:



APQC ‘Model 1’
--

Linking knowledge seekers with
knowledge providers

–
No
a priori

designation of ‘experts’

–
This is our approach



APQC ‘Model 2’
–

assigned discipline managers are
responsible for knowing levels of expertise in their area



APQC ‘Model 3’
–

designated ‘validated’ experts


©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
8

Main view

MITRE Expertise Finder

What it looks like


Helps answer the question:
“Who at MITRE knows about
topic
X
“



Results are based on Google
relevancy ranking, in
conjunction with author/owner
attribution & document counts


Organizational view

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
9

Main view

Expertise Finder
–

Results details


Source options




Display options


Email contact options


Person, with job title and link to phonebook



Content ‘evidence’ (with
title, links to object &
repository, ‘keywords in
context’, object date)

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
10

Expertise Finder
–

Results details

Organizational view and content display by organization


Bubble size and position
indicates contributors
and contributions by
corporate center or
division

Clicking on a single
bubble displays people
and content from that
organization


©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
11

How it works

Expertise Finder results

(staff and organizations)

4

LDAP

(staff / organization

lookup)

3

Key MITRE Web
-
based repositories
contributing to Expertise Finder

Staff attribution

Resource

Author / owner
attribution

Web
-
enabled file system
(Employeeshare transfer
folders,

‘about
-
me’ resumes)

Standard User

Identifier (SUI)

in folder path

CommunityShare

(MS SharePoint)


MS Office Property Author
Field

MITRE blogs

SUI (email name)

MITRE Sourceforge

HTML Author

meta tag

MITRE List Messages


SUI (email name)

MITRE Technical Exchange
Meetings

Meeting Point of contact

MITRE Institute Courses


Course instructor (future)


Onomi social bookmarks

Bookmark contributor (future)

2

MII Google

(Google Search

Appliance)

User

Query

MII Google query / results

1

http query

XML

Expertise Finder

(Java
-
based application on Oracle
Application Server)

1. Based on keyword query, retrieves
ranked results set from MII Google
search (XML output)

2. Identifies author / owner attributes

3. Performs LDAP lookup for full staff
name and organization

4. Returns results set, ranked by
contributions, with hits ‘evidence’ and
keyword context

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Resource and staff attribution details

Page
12

2

Resource

Author / owner
attribution

Person
Metadata
Quality

CommunityShare (MS SharePoint)


MS Office Property Author
Field

Varies

MITRE blogs

Standard user identifier
(email name)

Excellent

MITRE Sourceforge (Software
projects)

HTML Author meta tag

Excellent

MITRE List Messages


Standard user identifier
(email name)

Excellent

MITRE Technical Exchange Meetings

Meeting Point of contact

Excellent

From
step 2,
previous
slide

MITRE Institute Courses

(future resource)

Course instructor

Excellent

Social bookmarks (Onomi)

(future resource)

Bookmark contributor

Excellent

HTML pages from distributed
webservers

HTML Author meta tag

Varies

MS Office documents from distributed
webservers

MS Office Property Author
Field

Varies

Web
-
enabled file system
(Employeeshare transfer folders,
‘about
-
me’ resumes)

Standard User Identifier
(SUI = email name) in folder
path

Excellent

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
13

This system architecture: Advantages / Disadvantages


Advantages

–
Uses full
-
text indexed content (concepts are ‘fluid’
–

especially
for new technologies, products, projects)

–
Uses the same online content contributors share in the course of
their day
-
to
-
day work

–
No requirement for users to maintain a registry of expertise

–
Incentivizes staff to share content online in open repositories

–
Incentivizes staff to use correct metadata, especially authorship
metadata


Disadvantages

–
Results are only as good as the quality of authorship metadata
used to associate information objects with staff

–
Results are dependent on the underlying search system
relevancy ranking


Although, we also force specific repository results by sending
multiple parallel queries to multiple content repositories

–
Users could ‘game’ the system (by arbitrarily putting large
numbers of documents online)


©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Query characteristics
--

observations


Best performing queries

–
Specific (‘term specificity’): Products, programs, projects,
standards
–

query terms that are ‘good discriminators’

–
Query examples:


Standards/Compliance: IEEE
-
1061, fisma


Products: Cognos, AppWorx


Projects: Next
-
generation airspace


Topics: ontologies, second life, biometrics


Worst performing queries

–
Extremely general terms, whether single or multiple words

–
Query examples:


‘ Engineering’
–

in a corporation where a majority of staff function
in some engineering capacity and are performing engineering
-
related tasks


‘Software’
–

in a corporation of where a significant portion of our
work focuses on some aspects of software engineering

–
But
–

consider
–

these very general queries may not perform
well in general full
-
text retrieval anyway



Page
14

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
15

Results evaluation/validation methods



Validation methods

–
Informal: Send in a query based on a topic where the user knows
a set of experts/specialists, see how many come back in results


Many staff take this on themselves, as a check to see if they are
included in results

–
Informal: when a user submits an email to contacts identified,
informal email probe to that user


“Was this system helpful in identifying knowledgeable staff”


“Did you get an answer”

–
Metrics: Continued usage by staff


Query metrics have held steady over time, paralleling general search
query metrics



©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Usage metrics and query analysis



Usage metrics tracking by

–
Basic usage


From Google query logs
--

Expertise Finder interface queries are
coded with a specific parameter for identification in query logs


From internal WebTrends web analytic reports (user visits, page
views)

–
Query analysis


We can identify specific queries sent to the system by analyzing
query log data



Monthly usage metrics

–
General MITRE Google queries: 60K
-

75K queries per month

–
Expertise Finder queries: 2K
-

4K queries per month

–
Usage ratio average of general search to the expertise finder
application
--

~25 : 1

Page
16

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Expertise Finder enhancements


New enhancement in development: ‘Presence awareness’
identification using a Microsoft Office ActiveX Web service









System enhancements under consideration

–
Limit by date/date range (to find most staff based on most
recent contributions)

–
Limit by more detailed level of contributing repositories


Beyond just ‘Documents and Webpages’ and ‘Lists’


E.g., SharePoint, Technical Exchange Meetings, Social Bookmarks

–
Permit user to limit and/or sort by staff classification/role, e.g.,


‘AC’ Technical; PRO ‘professional level support’; PSS
Administrative support

Page
17

Shows staff availability
online, free/busy status,
access to Office
Communicator chat,
other tools

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Longer term: where we may be going


Exploration of social networking for expertise finding

–
Use of staff profiles based on social networking models (similar
to MySpace, Facebook, LinkedIn)

–
Use social network connections as an additional dimension in
expertise finding


Hybrid approach
–

base expertise finding on

–
Text from document content


As we are doing
–

there will continue to be value in identifying
expertise from content objects


and

–
Staff profiles, which may be


User
-
generated, auto
-
generated, or a combination

–
Let the user decide, per query, how to focus the results based
on these resources

Page
18

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Conclusion/recommendations: expertise
finding implementation


If you are considering expertise finding implementation



Consider which APQC model fits your organization’s
environment and requirements



If implementing a software application

–
Evaluate metadata quality for staff attribution

–
Decisions


Staff identification based on registry vs. content, or hybrid


Build vs. buy


Build in conjunction with enterprise search

–
Use of service
-
oriented architecture



For swap
-
in/swap out of code base, directory resources, and
content resources


For future feature enhancements

Page
19

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Background info


Page
20

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
21

Sources/Resources


Google Scholar results
–

MITRE expertise finding research

–

MITRE authors: M. Maybury, D. House, R. D’Amore
http://scholar.google.com/scholar?q=Maybury,+House,+D%E2
%80%99Amore


Ackerman, Mark and McDonald, David “Just Talk to Me: A
Field Study of Expertise Location” Proceedings of the 1998
ACM conference on computer supported cooperative work,
Seattle, November 14
-
18, 1998

–
http://portal.acm.org/citation.cfm?id=289506


Hughes, Gareth and Crowder, Richard “Experiences in
designing highly adaptable expertise finder systems”
Proceedings of the DETC 03, Chicago, September 2
-
6, 2003

–
http://eprints.ecs.soton.ac.uk/8206/


Expertise Locator Systems: Finding the Answers. APQC
Publications: 2003

–
http://www.apqc.org/portal/apqc/ksn?paf_gear_id=contentgear
home&paf_dm=full&pageselect=detail&docid=123338


©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
22

Sources/Resources


Maybury, Mark Expert Finding Systems, MITRE
Corporation: MITRE Technical Report, MTR 06B00040,
September 2006

–
http://www.mitre.org/work/tech_papers/tech_papers_06/06_111
5/06_1115.pdf


Maybury, Mark “Discovering Distributed Expertise” AAAI
Fall Symposium Series,
Regarding the “Intelligence” in
Distributed Intelligent Systems,
November 9, 2007

–
http://www.mitre.org/work/tech_papers/tech_papers_07/07_073
0/07_0730.pdf


Damianos, Laurie, et al. Onomi: Social Bookmarking on a
Corporate Intranet, MITRE Corporation, May 2006

–
http://www.mitre.org/work/tech_papers/tech_papers_06/06_035
2/06_0352.pdf



Author contact: rjoachim@mitre.org

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
23

Sources/Resources: Expertise finding recent
real
-
world implementations



Presented or cited at Enterprise Search Summit, New York,
May 20
-
21, 2008

–
“Mining additional value from enterprise search”, Trent
Parkhill, Haley & Aldrich


Based on search product: Coveo

–
“Search connections in context”, Oz Benamram, Morrison &
Foerster


Based on search product: Recommind

–
Google, Inc. internal expertise finder


Based on search product: Google enterprise


Montague Institute, January 17, 2008

–
“Enterprise mashups for expertise location”, Qin Zhu, HP Labs


Based on search product: Inktomi




©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Commercial software products for
enterprise expertise finding
--

examples


Enterprise search with expertise finding components

–
Autonomy IDOL

–
FAST (Partnering with AskMe)

–
Endeca

–
Recommind

–
Microsoft SharePoint Enterprise search (with MOSS07
“Knowledge Network”)



Dedicated/specialized systems

–
TACIT ActiveNet

–
AskMe


–
Triviumsoft SEE
-
K

Page
24

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Community Finding (Prototype)

Page
25



Based on MITRE
Expertise Finder code



Identifies MITRE online
communities (Email lists
& SharePoint
communities)


Uses search results of

–
List messages

–
Documents associated
with a community

–
Community
descriptions

©
2008 The MITRE Corporation. All rights reserved.

5
-
Nov
-
13

Approved for Public Release

Page
26

“Onomi” Social Bookmarks
–
Tagging/Sharing
of Web Resources by MITRE staff

Onomi
(rhymes with
‘Taxonomy’)

–

based on
open
-
source tool Scuttle

–

Lets MITRE staff
bookmark content
resources and tag content
of interest with topical
terms

–
Helps me “find this again”

–
Builds communities of
interest

–

And
–

contributes to
expertise finding:

Users tagging/contributing
are ‘experts’ in the
content/topics bookmarked
and tagged