The Use of Data in Performance Management

fivesenegaleseManagement

Nov 10, 2013 (3 years and 5 months ago)

60 views

The Role of Data in

Performance Management


Paul M. Lefkovitz, Ph.D

President, Behavioral Pathway Systems

Netsmart


October 15, 2012




Agenda


The Role of Data in Performance
Management


Evaluating Your Personal Use of Data in
Performance Management


Evaluating Your Organization's Use of
Data in Performance Management

THE ROLE OF DATA

IN PERFORMANCE
MANAGEMENT

Data, Data, Everywhere and

not a Drop of Information


Leaders aspire to
be data
-
driven


We
are deluged
with
data


However, most
decisions in
behavioral health
are
NOT

data
-
driven

How Come????

Barriers to the Use of Data


Personal Factors


Orientation


Skill/Experience


Organizational Factors


Performance Management Philosophies


Processes/Procedures


Lack of a Context


How helpful would a thermometer be as a
measure of your health, if you didn’t know
that 98.6 were normal?

Performance Data Utilization Scale


Measures the attitudes regarding the use
of formalized data in your organization


Brief 10
-
item scale


5 point Likert
-
type scale, resulting in a
maximum score of 50


Intended to provide a global measure of
the extent to which an organization may
be characterized as “data
-
driven”.

Is your organization data
-
driven?

Sources of Data


High
-
Tech


Information
system


Decision support
technology


Marketing tools for
measurement of
patient satisfaction


Human resources
tools for the
measurement of
organizational climate


Pharmacy department
systems


ORYX Data



Low/No
-
Tech


Home grown
measurement tools


Internal and external
focus groups


Community needs
studies


Staff or patient
surveys


FOCUS
-
PDCA and
other performance
improvement
methodologies



Types of Data


“Balanced” approach
-
4 Domains


Financial Measures


Well
-
developed and resourced


Clinical Measures


Poorly developed and resourced


Operational Measures


Inconsistently developed and resourced


Organizational Climate


Inconsistently developed and resourced

Financial Measures


Financial metrics measure quality of
financial management


Financial reports are well utilized


Wide array of standardized measures


Current Ratio


Days Cash on Hand


Days in Accounts Receivable


Bad Debt


Management and General Expenses as a
Percent of Total Expenses


Unit Costs

Clinical Measures


Standardized tools (e.g., BASIS
-
32, CAFAS) offer
the most highly validated data


But such tools may not deliver the “message”


Many “scores” offer no inherent information


Naturalistic, descriptive measures may be more
effective, e.g. homelessness decreased from 7% to
2% of those served rather than a 16.28 point
reduction on the “Residential Stability Scale”
(made
-
up)


Other naturalistic measures: involvement with
criminal justice system, employment, discharge
status


No
-
Shows


Client satisfaction



Operational Measures


Access


Productivity


Service Utilization


Client/Staff Ratios


Staffing Patterns


Injuries


Medication Errors


Service Delivery Patterns



Organizational

Climate Measures


Research shows that management of
human resources is more related to
performance than strategy or other factors


Staff Turnover


MHCA Customer Survey System


Press Ganey


Gallup Q
-
12


Minnesota Satisfaction Questionnaire


Survey of Employee Engagement


BPS Organizational Climate Survey


Data Requirements


for Different Audiences


“Burden of Proof” concept in the
American system of jurisprudence
offers a useful paradigm for thinking
about data requirements


Beyond a Reasonable Doubt


Clear and Convincing


Preponderance

Burden of Proof & Functions of Data

Burden of Proof

Characteristics

Functions of
Data

Preponderance


i潯oe

d慴愠捯汬散瑩潮


卭慬氠獡浰汥

獩se


卵扪散e楶攠浥慳畲e


Basic data analysis


Confounding variables

Performance
Improvement

Clear and Convincing


Systematic data collection


Moderate sample size


l扪散瑩t攠浥慳畲e


剥汩慢汥⁤慴愠慮慬祳楳


i業楴敤⁣潮景畮o楮i

Advocacy

Beyond a Reasonable
Doubt


Rigorous data collection


Robust

sample size


Concrete measure


周Tr潵杨od慴愠慮慬祳楳


乯N捯湦潵湤楮朠癡物慢汥v (潲o
物獫
-
慤橵獴敤⁡湡e祳楳i

剥杵污瑯gy

Funding/

Scientific

Discovery

EVALUATING YOUR
PERSONAL USE OF DATA
IN PERFORMANCE
MANAGEMENT

Personal Orientation to Data


Comfort with “numbers”


Orientation to how information is used in
making decisions


Awareness of data orientation?


Data Orientation Scale


Brief 11 point scale that measures
orientation to data in planning and decision
-
making


What is your personal orientation to
data?


Data Orientation


Data Disciple


Data Doer


Data Dabbler


Data Doubter


Data Dismisser


Data Disciple


Data plays a central role in planning and
decision
-
making


Is methodical and makes few mistakes


Relies on data
-
may have difficulties when
quick, spontaneous decisions are required


Data disciples should celebrate their
comfort with data but remind themselves to
be open to other sources of input, including
hunches

at times, data may lead to the
wrong conclusions


Data Doer


Comfortable with data and is used in
planning and decision
-
making


Carries out expected activities to gather
necessary data but may not go to great
lengths to always have relevant data


Is open to data presented by others and
reinforces the use of data


Data Doers should exploit their balance
between respect for data and openness to
other influences in planning and decision
-
making





Data Dabbler


Data is viewed as one of many sources of
inspiration in planning and decision
-
making


Can be open to data when presented but
may not always actively seek out data


Data Dabblers should try to ensure that
they have acquired sufficient information
prior to making important decisions.


Identifying others that they can rely on to
provide relevant data would be helpful.

Data Doubter


Decisions from the “gut” tend to be favored


May ignore helpful data that is readily
available


Others might feel impeded in
communicating information relating to
decisions


Data Doubters should recognize that their
“gut” is subliminally drawing upon data and
try to get in touch with what it is


Identifying others that they can rely on to
provide relevant data would be important.

Data Dismisser


Data tends to be regarded with suspicion and
disdain


May not want to be controlled by facts and
information


May feel data needs to be “perfect” in order to be
useful


Others may feel frustrated by lack of openness to
new information


Data Dismissers should try to understand
sources of their discomfort

Control?
Perfectionism?


Helpful to partner closely with trusted associate
that does value data and can help communicate
information in a more palatable way

EVALUATING YOUR
ORGANIZATION’S USE OF
DATA IN PERFORMANCE
MANAGEMENT

Evaluating Your

Organization’s Use of Data

in Performance Management


Performance Management Inventory


Is
your
organization
asking the right questions?


Is your
organization
using the right tools?


Is your
organization
supported by the right
analyses?


Is your
organization
generating the right reports?


Is your
organization
involving the right people?


Is your
organization making
the right changes
?


Is data being used effectively in the management of
your organization?

Is Your
Organization
Asking the
Right Questions?


All organizations establish priorities
--
not everything
can receive equal attention


Is the organization measuring performance in the
most vital areas?


Alignment between stated priorities and
measurement efforts is important
-
it ensures that
the organization knows what it needs to know.


1. Is

you
r organization

asking the right questions?


Is your system gathering information that is driven by the strategy and vision of the
organization? Please enter the ten most important organizational priorities in Column A. In
Column B, check those areas
in which relevant performance data is available.


A. Key Organizational Priorities

B. Availability

of Data


1.




2.




3.




4.




5.




6.




7.




8.




9.




10.






Item 1 Score:











Interpretive Notes:

This score is
based on the number of
key organizational priorities for which data is available
.
A
high score (above 7) suggests a significant degree of
harmony between the organization’s priorities and the data
generated by the performance measurement system.


Is Your
Organization

Using
the Right Tools?


Effort on the part of clients


Effort on the part of staff


Helpful information provided


Materials easy to handle/manage/store


Modest costs


Acceptance by staff


Acceptance by clients


Validated tools


Prompt availability of data


Philosophical alignment


2. Is your organization

using the right tools?


Please evaluate your performance management system tools (surveys, forms, inventories,
tests, etc.) by checking
all

of the following applicable items:




Tools involve acceptable effort on the par
t of clients



Tools involve acceptable effort on the part of staff



Information provided is helpful



Materials are easy to handle/store/manage



Inventory, survey, instrument costs
are modest



Tools are accepted by staff



Tools are accepted by clients



Tools have been validated



Data is available quickly

no great delays for scoring



Tools r
eflect the clinical philosophies of the staff




Item 2

Score:









Interpretive Notes:

This item score
is

the number of items
checked. A high score reflects good selection of
measur
ement instruments. A score of 5

or lower would
suggest that the benefits of the tools may be outweighed by
the effort involved or ot
her factors.





Is Your
Organization
Supported

by
the Right Analyses?


Data must be aggregated


Manual aggregation may be sufficient for simple
applications


Computerized analysis


Faster and easier


Can provide many more possibilities for “slicing and dicing” the
data


Ad hoc analysis and decision support



3.

Is your organization

supported by the right analyses?


Which statement most accurately describes your performance measurement system? Enter
the most appropriate rating from 0 to 10 in the space provided below.


Description

Scale

Computerized data analysis is
conducted that routinely provides detailed
reports with external comparative benchmarks as well as customized
supplemental analyses upon request

10

Computerized data analysis is conducted that routinely provides detailed
reports with external comparative
benchmarks

8

Computerized data analysis is conducted that provides basic reports with
external benchmarks or more detailed reports without external benchmarks

5

Manual data analysis is conducted that provides basic summary information


2

No data analysi
s is conducted


0



Item 3

Score:









Interpretive Notes:


Performance measurement is a highly
technical process that requires the proper expertise and
support. A score below 5 is likely to leave many questions
unanswered.




Is Your
Organization

Generating the
Right Reports?


Reports communicate aggregated results


May consist of tables, graphs, narrative text


Professional appearance


Well
-
organized


Clearly written


Limited use of jargon


Good balance of narrative text and tables


Relevant information


Comfortable
-
Not overwhelming


Not too lengthy


Not too sparse


Easy to understand

4. Is your organization

generating the right r
eports
?


Describe the reports generated by your performance measurement system. From the list
that appears below, please check all that apply.



Professional appearance


Well
-
organ
ized


Clearly presented


Limited use of jargon/technical language


Good balance of n
arrative
text and tables


Relevant information


Comfortable

not overwhelming


Not too lengthy


Not too sparse


Drives decisions



Item 4

Score:









Interpretive Notes:

This item score
is

the number of items
checked.

Performance measurement reports
must

be “user
-
friendly” to have value. If less than 6 items from the list are
checked, the value of your performance measurement
system may be compromised. Reports may be going righ
t
into the file (or the trash
) rather than being utilized.





Is Your Organization

Involving the Right People?


Performance measurement is an
organization
-
wide activity


Proper involvement of staff at all levels
in selecting indices and reviewing data
is critical


Senior leadership


Middle management


Line staff




5
. Is your organization

involving the right people?


Are you getting the full use of the data that is gathered? Please use the following scale to
describe how performance measures are selected and how the findings are disseminated
and employed within the orga
nization. Enter your rating in the space provided below.


Description

Scale

Representative staff members at all levels within the organization are
regularly involved in the selection, review, and response to performance data

10

Senior leadership, middle
management, and occasionally line staff are
involved in the selection, review, and response to performance data

8

Senior leadership and middle management are predominantly involved in the
selection, review, and response to performance data; Verbal reports

regularly
go to line staff

5

Senior leadership and middle management are exclusively involved in the
selection, review, and response to performance data

2

Senior leadership is exclusively involved in the selection, review, and
response to performance da
ta

0



Item 5 Score:









Interpretive Notes:

Performance management data can
provide the rationale and direction for helpful organizational
change. The more outcomes data are shared among leaders
and non
-
leaders, the more effective it

can be in guiding the
change process.






Is Your Organization

Making the Right Changes?


Expectations of the performance
measurement system


Resistance to organizational change


The ultimate litmus test of a performance
measurement system is whether it leads
to organizational improvement


System should stimulate the ongoing
implementation of improvement
initiatives

6.

Is your organization

making

the right changes?


In the past year, how many organizational improvement initiatives originated
directly

from
the data derived from your performance measurement system? The score is the number of
concrete initiatives that were i
mplemented, up to a maximum of 10.




Item 6

Score:









Interpretive Notes:

If the answer
to this question is “none”,
your

performance measurement system may not be adding
value to th
e organization. A
system
’s contributions are
largely measured by

the
concrete ideas for improvement
that emanate from th
e resultant data.





Questions?

Paul M. Lefkovitz, Ph.D.

President, Behavioral Pathway Systems

www.bpsys.org

plefk@bpsys.org

877
-
330
-
9870