Disciplinary and sub-disciplinary norms and benchmarks in Computer Science research

weedyhospitalElectronics - Devices

Nov 25, 2013 (3 years and 8 months ago)

79 views

Disciplinary and sub
-
disciplinary norms

and benchmarks in Computer Science research


This document has been put together from collective or individual contributions from all
professorial staff


December 10, 2012



Computer Science is a very diverse discipline, which makes it virtually impossible to
establish norms or benchmarks. If one considers the way Mathematics, Physics and
Engineering are today inter
-
related (one could even add Philosophy), and realise that th
eir
separation into autonomous disciplines has happened over five centuries, it is easier to
understand why that separation has not taken place within Computer Science, which has
barely five decades of existence as a discipline.


The result is that one ca
n find within Computer Science sub
-
disciplines that are still very
close to Mathematics, others which are theoretical in that they study computational structures
and languages, and others which focus on the engineering of software systems.
Interdisciplin
arity is also becoming more rather than less widespread. For example,
bioinformatics intersects with biological sciences, human
-
computer interaction intersects
with psychology, embedded systems intersects with engineering (in some universities,
embedded s
ystems can be found in engineering departments), and there is intersection with
the arts via multimedia studies, etc.


This diversity is reflected in the research cultures of the different sub
-
disciplines. Therefore,
our answer to the specified questions
uses a very broad brush.


Our recommendation is that any form of assessment in relation to performance be done by a
panel of experts in the relevant sub
-
discipline.




The optimum forms of research output (e.g. single
-
author monograph/journal
article/etc.)

In Computer Science, the optimum forms of research output are articles
in refereed journals or in high
-
quality, selective refereed conferences. However,
impact factors of journals or conference rankings are not a proxy for quality and
should not be used i
n a formulaic way. Books published by reputable publishing
houses and chapters in volumes edited by prestigious scientists can also be valued and
normally give evidence of scholarship.




The best outlets (e.g. Oxford University Press for a monograph/ Natur
e or The
Economic Journal for a paper).

Many journals and
conferences are specific to sub
-
disciplines and, therefore, it is impossible to establish a hierarchy.




The most esteemed category of authorship or participation (e.g. single
-
author, PI
named first

in the group).

There are no rules that can be applied across sub
-
disciplines, especially in interdisciplinary areas. Normally, the order and number of
authors is not relevant.




The top grant awarding Councils, Trusts.

No ranking exists amongst awardin
g
bodies.




The ballpark a
mounts that might separate distinguished/professor
-
level from
routine funding.

Essentially, levels of funding do not necessarily reflect research
quality or scholarship. Typically, except for a few sub
-
disciplines, research in
Computer Science does not require expensive equipment. Often, funding is sought
for networking and for s
upporting research collaborations, for which the amounts are
relatively small. Higher
-
levels of funding can also be associated with the priorities of
awarding bodies and, therefore, they do not necessarily reflect good
-
quality
scholarship.




Any
outstanding awards, honours, medals etc.

distinctive to the discipline.

The
Turing Award and the Gödel Prize.




The importance attached to keynote addresses, invitations to speak at specific
conferences or gatherings, etc.

Invitations to speak at confere
nces or other scientific
gatherings provide evidence of esteem but their absence does not indicate lack of
esteem. Membership of programme committees of selected refereed conferences can
also provide evidence of esteem but their absence does not indicate
lack of esteem.




International or national Fellowships of particularly high standing, visiting
Professorships, etc.

Several professional organisations award fellowships.
Fellowships gives evidence of peer esteem but their absence does not indicate lack
of
esteem. Fellowship of the Royal Society is a clear indicator of esteem at the highest
level.




The importance attached to membership of editorial boards.

Membership of
editorial boards is normally considered as a service and there is no particular stan
ding
associated with it. It gives evidence of peer esteem but its absence does not indicate
lack of esteem.




Any other salient indicators you think important.

There is a proliferation of
indexes available based on citation counts. However, the correlat
ion between citation
counts and quality or impact of publications is often poor.




Any pitfalls to bear in mind when assessing research performance, for example,
types of research activity which by their nature take significantly longer to reach
fruition t
han superficially similar types of activity.

One particular pitfall to bear in
mind is that Computer Science is sometimes prone to fashions: one can witness
surges of activity in areas with only superficial research content and hardly any
impact.




It wou
ld be helpful to have a ballpark indicator of disciplinary norms (e.g. two
published papers per annum; one significant monograph every five years) but
also a sense of what might constitute a significant step
-
change in performance;
commensurate with upward
advancement within the professorial grade.
Publication rates are very sub
-
discipline specific and are no proxy for quality. The
considered opinion of three distinguished academics will always be more reliable than
any metric.