Survey of the State of Analytics in UK Higher and Further Institutions 2013

previousdankishSoftware and s/w Development

Nov 25, 2013 (3 years and 11 months ago)

107 views

Survey of the State of Analytics in UK Higher and Further Institutions 2013





1















Survey of the State of
Analytics in UK Higher
and Further Institutions
2013

A white paper

By
Adam Cooper,
Stephen Powell,
Li Yuan and Sheila MacNeill


Survey of the State of Analytics in UK Higher and Further Institutions 2013





2

Survey

of the State of Analytics in UK
Higher and Further Institutions 2013

By
Adam Cooper, Stephen Powell, Li Yuan and Sheila MacNeill


Introduction

An informal survey was undertaken by Cetis in May and June 2013. Subscribers to a number of email
circulatio
n lists
-

with members coming largely from institutional IT, administration and educational
technology responsibilities
-

were invited to respond.

The purpose of the survey was to:



Assess the current state of analytics in UK FE/HE.



Identify the challenges
and barriers to using analytics.

For the purpose of the survey, we defined our use of “analytics” to be the process of developing
actionable insights through problem definition and the application of statistical models and analysis
against existing and/or
simulated future data. In practical terms, it involves trying to find out things
about an organisation, its products services and operations, to help inform decisions about what to do
next.

Various domains of decision
-
making are encompassed in this definit
ion: the kinds of decision that is
readily understood by a business
-
person, whatever their line of business; questions of an essentially
educational character; and decisions relating to the management of research. The line of questioning
was inclusive of t
hese three perspectives. The questions asked were:

1.
Which education sector do you work in (or with)?

2. What is your role in your institution?

3. In your institution which department(s) are leading institutional analytics activities and
services?

4. In your institution, how aware are staff about recent developments in analytics?

5. Do the following roles use the results of statistical analysis such as correlation or significance
testing rather than simple reporting of data in charts or tables?

6. W
hich of the following sources are used to supply data for analytics activities?

7. Which of the following data collection and analytics technologies are in place in your
institution?

8. Please name the supplier/product of the principle software in use (e.g
. IBM Cognos, SPSS,
Tableau, Excel)

9. Which of the following staff capabilities are in place in your institution?

10a. What are the drivers for taking analytics based approaches in your institution?

10b. What are the current barriers for using of analytic
s in your institution?

The survey is informal in that:



no attempt was made to have a balanced sample, either in terms of institutional type or
respondent role;



it relied on voluntary participation, so will suffer from selection bias;



multiple responses
from a single institution may have occurred, but these cannot be identified;



we surveyed the knowledge of individuals rather than the actual state of an organisation.

These facts, coupled with the small number of responses, means that the resulting data ca
nnot be
assumed to represent the true state of affairs but to be indicative. The report is written as a stimulus
Survey of the State of Analytics in UK Higher and Further Institutions 2013





3

both for discussion and for more thorough investigation into some of the areas where the survey
responses hint at an issue.

Terminology: a “res
ponse” is a single submission of the survey and an “item response” pertains to a single
question.


Basic Facts

There were 26 responses.




Concerning the State of Knowledge

The line of questioning sought the knowledge of the respondent and not the actual state of affairs in a
given institution. Although this fact limits the extent to which responses can be interpreted in relation to
the question as asked, it does allow us to
form a picture of the way knowledge of analytics activities is
distributed. We can form a response to the question: what does “don't know” look like?

Technical note: most questions contain multiple parts and for these a “don't know rating” is calculated as

the
proportion of parts within that question that were answered “I don't know”. Question 3 contained no options
so has a don’t know rating of 0 or 1 only.


The mean don't know rating for each response was calculated and found to have a median value of
0.3
13 indicating that half of the sample had more than around 31% “don't know” item responses. 25%
of the responses had around 17% or less “don't know” item responses. The conclusion is clearly that
there is a lot of uncertainty. This is particularly striking

because the respondents were self
-
selecting; it
would be expected the people with the least knowledge would be less likely to respond.

Survey of the State of Analytics in UK Higher and Further Institutions 2013





4


The state of unknowing varies between
questions. This plot shows the mean over all
responses for each question.

It see
ms to indicate that there is a better
awareness that there is something going on, both
within the institution and generally (questions 3
and 4) than there is about the way an
institutional response is being implemented
(questions 5, 6, 7, 9). A more detail
ed analysis
appears to indicate that questions 5 and 6 are a
particular hot
-
spot of uncertainty, with a probable
cluster of responses where the item responses for
both questions were “don't know” whereas other
i瑥浳 牥獰r湳敳n睨敲攠歮k睮⸠䥮f浡湹n捡獥猬

捥牴ci湴礠i猠獰s敡搠o癥爠浡湹n煵敳瑩o湳n
睩瑨w畴ua湹n灡瑴t牮 扥b湧ne癩摥湴n


Leadership of Analytics Activities

In your institution which department(s) are leading institutional analytics activities and services?
(multiple selection was possible)


The dom
inant leaders reflect the centres for key
institutional data and the IT services to handle it.
There were only 4 “don't know” responses and the
“other” item responses show some partial overlap
睩瑨w瑨t 獰散楦i敤e捡t敧e物r猺s?C敮e牡l
摥灡牴r敮e?,

?A捡摥浩d

灬a湮楮朠o爠煵rli瑹t
(A偑F, 却p
摥湴d䥮景牭a瑩o渠(o敧e獴特⤢,

?卣pool猠
l敡摩湧dl敡牮i湧na湡l祴y捳c fq l敡摩湧di湳瑩瑵瑩o湡l
a湡l祴y捳c

?q敡c
桥爠q牡i湩n朠A牥r?,
?Co浰lia湣攠
啮楴, iea牮i湧n卥牶楣p猢
,

?偯捫整猠睩瑨w渠捥牴ci渠
灲oj散e猢
.


Awareness

In your institution, how aware are staff about recent developments in analytics? (6 sub
-
questions, according to
staff role)

An impression of the general level of awareness is given by summation over all staff roles given in the
question: (Executive Managem
ent, Academic Managers, IT Service Staff, Teaching Staff, Library Staff,
Research management).

Survey of the State of Analytics in UK Higher and Further Institutions 2013





5


A stacked bar plot, ordered by increasing number
of “Very aware” responses, shows the different
l敶el猠o映灥牣敩癥搠a睡牥湥獳s睩瑨w渠敡捨c獴s晦
杲o異u



These results are likely to be biased because of the distribution of backgrounds of respondents and this
may mean that we should not assume the item responses that were not “don't know” are
representative.

The tentative conclusion seems to be that, unlike
many initiatives with a strong data and IT element,
executive management awareness is strong. In general, the management and support roles appear to
be more aware than the academic staff but the similarity between IT and library staff may not have
been exp
ected. This data suggests anyone planning to develop analytics at an institutional or sectorial
level should not make too many assumptions about awareness, rather they should investigate the state.

Use of Statistical Analysis

Do the following roles use the

results of statistical analysis such as correlation or significance testing rather than
simple reporting of data in charts or tables? (7 sub
-
questions. according to staff role)

Summation over all staff groups (Executive management, Academic management, IT

services staff,
Teaching staff, Library staff, Facilities and estates staff, Finance and purchasing) allows us to capture
the overall impression.

Survey of the State of Analytics in UK Higher and Further Institutions 2013





6


The general levels of “don't know” are really
牡瑨t爠桩杨g t桡琠摯敳e瑨t猠瑥ll 畳㼠偲m扡扬礠瑨tt
牥灯牴猠
a湤n癥牢vl 捯浭畮u捡瑩o湳n摯 湯琠牥晥爠
瑯 瑨t猠歩湤ko映a湡l祳楳y t桥瑨t爠r桩猠浥m湳n瑨tt
i琠i猠湯琠o捣c牲i湧no爠r桡琠i琠i猠扥b湧n摯湥na湤n湯琠
捯m浵mi捡瑥搠睯畬搠牥煵楲攠晵牴桥爠
i湶敳ni条瑩o渮 A牧畡扬礬 睨w捨敶c爠r猠瑨t 捡獥 睥w
獨s畬搠扥b捯湣敲湥搻n瑨t猠歩k
搠o映a湡l祳楳yi猠o晴f渠
湥捥獳s特 瑯 摥捩摥d睨整桥爠r桥h瑡扬b猠o映摡ta
a湤n捨c牴猠牥rll礠獩杮s晹 a co浰敬li湧n捡獥s景爠
a捴co渮 䙵牴桥牭o牥r 睩摥爠畳攠o映瑨t獥s睥wl
-
敳ea扬b獨敤sa湤n牯畴u湥n獴s瑩獴s捡l 浥瑨m摳d睩瑨w渠
a渠i湳瑩瑵瑩o渠獨s畬搠灲o扡bl礠灲散e摥d畳u

o映
浯牥ra摶d湣敤n灲敤e捴c癥v浥瑨m摳d

q桥h景llo睩湧w灬o琠摲ill猠摯w渠i湴n 瑨t 牯l敳ea湤n
獨s睳⁴桡琠瑨t 灡瑴t牮 i猠獩浩ma爠r捲o獳s獥癥牡l
牯l敳e扵琠桡猠晡捩ci瑩敳e敳eat敳ea湤n
晩湡湣支灵牣桡獩湧sa琠a渠數e牥浥r A牧畡扬礬
瑨t獥sa牥r猠獨s畬搠扥b浡歩湧n浯牥r畳u

o映
獴s瑩獴s捡l 浥瑨m摳d瑨t渠瑨t 摡瑡 i湤楣n瑥猠獩湣s
瑨t獥sa牥r猠a牥r瑨t o湥猠睨敲攠瑨t猠i猠浯獴s敡獹s
瑯 摯 睩瑨w畴u瑨t 捯浰l數e瑹⽳畢瑬整礠o映瑥a捨楮朠
a湤nl敡牮i湧no爠r桥h畮捥牴ui湴n敳ei渠數e敲湡l
獴牡瑥杩挠晡捴c牳㬠瑨t猠浡礠b攠a渠a牴敦a捴co映瑨t
牥獰r湤
敮e po灵污瑩o渠獩浰s礠湯琠歮k睩湧w睨w琠
杯敳eo渠i渠瑨t獥s晵湣瑩潮al a牥r献



Survey of the State of Analytics in UK Higher and Further Institutions 2013





7

Sources of Data

Which of the following sources are used to supply data for analytics activities? (11 sub
-
questions, according to
data source)

Again, we must be cautious about
interpreting the results; there is a large “don't know” fraction and the
distribution of respondent roles/departments will have introduced bias that cannot reasonably be
estimated.

The data is ordered according to the number of “yes” responses for each sub
-
question and shown in the
following chart. Sector data includes data from the national student survey, HESA, UCAS etc.


Although we should be cautious in assuming
anything about the actual situation that the “don't
know” responses mask
-

景爠r硡浰l攠瑨tt

瑨t 摯渧琠
know fraction can be assumed to be “no” or that it
捡渠扥ba獳畭敤e瑯 桡癥v瑨t 獡浥m灲o灯牴ro渠o映
“yes” and “no” as these definitive responses
-

瑨t猠
灬p琠獵杧敳e猠a 晥眠灲o癩獩o湡l 捯浭敮e猺



VLE data is widely used yet it only gives a
partial an
d ambiguous view on learner
activity. Given the apparent lack of use of
even basic statistical methods, are we
being over
-
confident about the value of
this data?



Collectively, VLE, attendance, and library
data would give a more balanced picture
of engageme
nt yet the second two are
much less widely used.



Given the availability of sector data, what is
the explanation that only half of the
responses indicated it is used?

The apparently low use of labour market data may
be a missed opportunity.




Survey of the State of Analytics in UK Higher and Further Institutions 2013





8

Technologies

in Place

Which of the following data collection and analytics technologies are in place in your institution? (5 sub
-
questions,
according to technology)


There is some consistency across the more
common technologies given in the question: data
visualisati
on, dashboards and data warehouses.
The less commonly used technologies are those
with the largest number of “don't know” responses
although it is plausible that some “don't know”
responses for “predictive analytics” occurred
扥捡畳攠瑨t 瑥牭 i猠i浰牥捩獥.

A猠扥景牥r 睥
獨s畬搠湯琠a獳畭攠瑨t琠瑨t 牥ra瑩癥v灲o灯牴楯渠o映
祥猯湯 牥獰r湳敳n杩癥猠a渠a灰po硩浡瑥 敳ei浡瑥
o映瑨t 灲opo牴楯渠o映i湳瑩瑵瑩o湳n瑨t琠a捴畡ll礠
敭灬e礠瑨t獥s瑥捨湯lo杩g献 䙯爠摡瑡 癩獵vli獡tio渠
a湤n摡獨扯a牤猬 i琠浩杨琠扥b牥r獯湡扬b 瑯 a獳s
浥m
that many “don't know” responses should be
substituted with “no”, given that these are user
-
晡捩湧c浡湩n敳ea瑩o湳no映a湡l祴y捳c


Survey respondents were asked to provide
information about products/suppliers as textual
item responses. Each response coul
d contain any
number of product names. The following plot
shows the number of item responses for products
that were named two or more times. More than 1
supplier may have been specified in a response.

SPSS is now part of the IBM Cognos offering,
although i
t may be used independently of the
suite.





Survey of the State of Analytics in UK Higher and Further Institutions 2013





9

Drivers for Analytics

What are the drivers for taking analytics based approaches in your institution? (multiple selection was possible)


The survey responses match the topics typically
discussed in the
literature on analytics in
universities and colleges. The following suggested
drivers attracted lower levels of positive response
than they may deserve:



The low value for research excellence may
reflect the survey group, which contains
few participants wit
h research
management responsibility and an
unknown number of responses from
research intensive institutions.



Student recruitment seems to be a missed
opportunity. The 50% response rate for
this driver should be verified.

Resource utilisation (excluding “h
uman resource”)
i猠a湯瑨t爠捡湤楤n瑥 景爠癥物晩捡瑩o渮 䥦f瑨t <㘰%
牡瑥 i猠捯牲散e, i琠獵杧敳e猠o灰p牴r湩ni敳e景爠
敦晩捩c湣礠獡癩湧猠a牥r扥b湧n浩獳敤mi渠瑯o 浡湹n
i湳瑩瑵瑩o湳n







Survey of
the State of Analytics in UK Higher and Further Institutions 2013

1

Barriers to Adoption

What are the current barriers for using of analytics in your institution? (multiple selection was possible)


Bearing in mind that survey respondents were self
-
selecting, and

so may have been
disproportionately positive in their attitude
towards analytics, the results seem to indicate
that several common barriers to IT and data based
initiatives arising from the attitude of people are
not prevalent. The issue of lack of senior

management report, which is often seen as a
significant challenge, is not a major factor. This
matches the earlier
-
noted level of awareness that
is perceived among executive management. Staff
and student acceptance is also not seen as an
issue. This may b
e due to there being little
practical effect on day
-
to
-
day teaching, learning,
research, and support because few analytics
initiatives have been rolled
-
out at institutional
level. Even if we assume the survey is accurate, it
is plausible that attitudes cou
ld change
dramatically.

Whereas the attitude of people is not generally
perceived as a barrier, both IT and analytical
capabilities are. Among the respondents, there is a
slightly greater sense that the analytical
capabilities
-

specialised analysts and tr
aining
-

are
common barriers than IT
-
related aspects
-

standardised data and IT resources.


A more detailed inspection shows some evidence that the IT and analytical barriers occur together
whereas the attitude
-
based barriers are apparently randomly
distributed. Three respondents only
identified specialised analysis and training as barriers. No response identified zero barriers.

Respondents who stated “other” supplied: “time”, “enough staff resources and priority to drive forward”,
“No idea, as we are
n't there yet”, and three counts where the field was left empty.

Acknowledgements

This report was produced by Adam Cooper and edited by Stephen Powell, and the survey was developed
by the Cetis Analytics Team: Li Yuan, Stephen Powell and Sheila MacNeill. A
ll are members of Cetis and
the work was supported by Jisc.

Survey Form, Data and Source Code

These are all available from
GitHub

(
https://github.com/arc12/Cetis
-
Analytics
-
Survey
-
2013
):

Survey form (PDF).

Raw data (CSV).







Survey of
the State of Analytics in UK Higher and Further Institutions 2013

2

Source code for R[1] with the clus
ter package[2]. This is intended for processing the raw data using knitr,
which is conveniently done using
RStudio

(http://www.rstudio.com).

A slightly extended version of this report containing some additional exploratory plots.

[1]: R Core Team (2012). R
: A language and environment for statistical computing. R Foundation for
Statistical Computing, Vienna, Austria. ISBN 3
-
900051
-
07
-
0

[2]: Maechler, M., Rousseeuw, P., Struyf, A., Hubert, M., Hornik, K.(2013). cluster: Cluster Analysis Basics
and Extensions.

R package version 1.14.4







Survey of
the State of Analytics in UK Higher and Further Institutions 2013

3

A
b
out this White Paper

Title:

Survey of the State of Analytics in UK Higher and Further Institutions 2013

Authors:

Adam Cooper, Stephen Powell, Li Yuan and Sheila MacNeill

Date:

October 2013

URI:


http://publications.cetis.ac.uk/2013/884



Text Copyright © University of Bolton 2013

This work is licensed under the Creative Commons Attribution 3.0.
http://creativecommons.org/licenses/by/3.0/






About Cetis

Cetis is the
Centre for Educational Technology, Interoperability and Standards
.
Our staff are
globally recognised as leading experts on education technology innovation, interoperability and
technology standards. For over a decade Cetis has pr
ovided impartial strategic, technical and
pedagogical advice on educational technology and standards to funding bodies, standards agencies,
government, institutions and commercial partners.

Cetis are active in the development and implementation of open sta
ndards and have been instrumental
in developing and promoting the adoption of technology and standards for course advertising, open
education resources, assessment, and student data management, opening new markets and creating
opportunities for innovation.

Our work includes a wide range of activities from representation at national
standardisation bodies, facilitation of online and face
-
to
-
face events to production of a range of formal
and informal publications.


http://www.cetis.ac.uk