Disciplinary and sub-disciplinary norms and benchmarks in Computer Science research

weedyhospitalElectronics - Devices

Nov 25, 2013 (4 years and 4 months ago)


Disciplinary and sub
disciplinary norms

and benchmarks in Computer Science research

This document has been put together from collective or individual contributions from all
professorial staff

December 10, 2012

Computer Science is a very diverse discipline, which makes it virtually impossible to
establish norms or benchmarks. If one considers the way Mathematics, Physics and
Engineering are today inter
related (one could even add Philosophy), and realise that th
separation into autonomous disciplines has happened over five centuries, it is easier to
understand why that separation has not taken place within Computer Science, which has
barely five decades of existence as a discipline.

The result is that one ca
n find within Computer Science sub
disciplines that are still very
close to Mathematics, others which are theoretical in that they study computational structures
and languages, and others which focus on the engineering of software systems.
arity is also becoming more rather than less widespread. For example,
bioinformatics intersects with biological sciences, human
computer interaction intersects
with psychology, embedded systems intersects with engineering (in some universities,
embedded s
ystems can be found in engineering departments), and there is intersection with
the arts via multimedia studies, etc.

This diversity is reflected in the research cultures of the different sub
disciplines. Therefore,
our answer to the specified questions
uses a very broad brush.

Our recommendation is that any form of assessment in relation to performance be done by a
panel of experts in the relevant sub

The optimum forms of research output (e.g. single
author monograph/journal

In Computer Science, the optimum forms of research output are articles
in refereed journals or in high
quality, selective refereed conferences. However,
impact factors of journals or conference rankings are not a proxy for quality and
should not be used i
n a formulaic way. Books published by reputable publishing
houses and chapters in volumes edited by prestigious scientists can also be valued and
normally give evidence of scholarship.

The best outlets (e.g. Oxford University Press for a monograph/ Natur
e or The
Economic Journal for a paper).

Many journals and
conferences are specific to sub
disciplines and, therefore, it is impossible to establish a hierarchy.

The most esteemed category of authorship or participation (e.g. single
author, PI
named first

in the group).

There are no rules that can be applied across sub
disciplines, especially in interdisciplinary areas. Normally, the order and number of
authors is not relevant.

The top grant awarding Councils, Trusts.

No ranking exists amongst awardin

The ballpark a
mounts that might separate distinguished/professor
level from
routine funding.

Essentially, levels of funding do not necessarily reflect research
quality or scholarship. Typically, except for a few sub
disciplines, research in
Computer Science does not require expensive equipment. Often, funding is sought
for networking and for s
upporting research collaborations, for which the amounts are
relatively small. Higher
levels of funding can also be associated with the priorities of
awarding bodies and, therefore, they do not necessarily reflect good

outstanding awards, honours, medals etc.

distinctive to the discipline.

Turing Award and the Gödel Prize.

The importance attached to keynote addresses, invitations to speak at specific
conferences or gatherings, etc.

Invitations to speak at confere
nces or other scientific
gatherings provide evidence of esteem but their absence does not indicate lack of
esteem. Membership of programme committees of selected refereed conferences can
also provide evidence of esteem but their absence does not indicate
lack of esteem.

International or national Fellowships of particularly high standing, visiting
Professorships, etc.

Several professional organisations award fellowships.
Fellowships gives evidence of peer esteem but their absence does not indicate lack
esteem. Fellowship of the Royal Society is a clear indicator of esteem at the highest

The importance attached to membership of editorial boards.

Membership of
editorial boards is normally considered as a service and there is no particular stan
associated with it. It gives evidence of peer esteem but its absence does not indicate
lack of esteem.

Any other salient indicators you think important.

There is a proliferation of
indexes available based on citation counts. However, the correlat
ion between citation
counts and quality or impact of publications is often poor.

Any pitfalls to bear in mind when assessing research performance, for example,
types of research activity which by their nature take significantly longer to reach
fruition t
han superficially similar types of activity.

One particular pitfall to bear in
mind is that Computer Science is sometimes prone to fashions: one can witness
surges of activity in areas with only superficial research content and hardly any

It wou
ld be helpful to have a ballpark indicator of disciplinary norms (e.g. two
published papers per annum; one significant monograph every five years) but
also a sense of what might constitute a significant step
change in performance;
commensurate with upward
advancement within the professorial grade.
Publication rates are very sub
discipline specific and are no proxy for quality. The
considered opinion of three distinguished academics will always be more reliable than
any metric.