RAE games

These are grim times for Welsh astronomy. The cancellation of Clover follows on from a surprisingly bad RAE result for Cardiff. Peter Coles has analysed the RAE results several times over. In this first post, he listed straight weighted mean scores (in which Cardiff came 35th). In a second post, he introduced “research power”, meaning volume times score, which brought Cardiff up to 22nd. Then on January 29th, when HEFCE announced its funding algorithm (7,3,1,0 for buckets 4,3,2,1 respectively) he gave another league table showing expected relative funding, with Cardiff now 27th. (Note however that the Welsh and Scottish funding councils have not yet announced their funding algorithms…)

Last week the RAE published the sub-profiles on which the final profiles were based – i.e. we now have separate profiles for research outputs, for environment, and for esteem. I downloaded the UOA19 (Physics) table, scraped the numbers, and played plotting games with Topcat. To help you play your own games I am attaching a .doc file which is really a CSV file in disguise … Unfortunately WordPress won’t let me upload a VOTable (its XML) or even a plain .txt file, but it does allow .doc files. You can convert the .doc file into plain text, and then Topcat or Excel will read it in.

So here is one interesting thing that jumped out at me – environment scores seem to have been quite crucial. The figure displayed here shows the research outputs score (bue dots) and the environment score (red dots) plotted in turn against the overall score. Compared to research outputs, environment shows a larger range, a larger dispersion, and gradient which is distinctly larger than unity. The red dot way off the correlation is Loughborough – environment score 1.1 even though it scored 2.66 on research outputs. On overall score, Loughborough came 32nd. If its score had been as good as its outputs score it would have been 14th. Cardiff was actually slightly rescued by its environment score; it scored outputs=2.22 and environment=2.74. (Edinburgh had a fairly consistent 2.8 and 3.0).

Results from RAE

It wouldn’t be wise to overinterpret individual scores. But it does look like the panel had more marked opinions about the quality of research environment, or perhaps allowed themselves bolder judgements. Any other patterns emerging ?

9 Responses to RAE games

  1. telescoper says:


    I very much appreciate your supportive comments, not least because they contrast with the offensive remarks made by one of my former colleagues on my own blog concerning this matter.

    I saw the Cardiff profile some time ago but wasn’t aware that the others were now public. In fact HEFCW and SFC both have published their funding allocations; HEFCW did so a few weeks ago (using the same formula as HEFCE) and those from SFC were published within the last week or so.

    Cardiff’s poor score is compounded by the fact that the Welsh Assembly Government is not as generous to Higher Education as we would wish; it is they that determing the funds HEFCW can dish out. Anyway, Cardiff overall got a 5% cut in its QR funds. We’re expecting in Physics & Astronomy to lose about 15% of our current QR income. The University as a whole will suffer a cash cut this year, which almost certainly means “restructuring” of some departments.

    I’ve commented before on my own blog about the very low outputs score for Cardiff. It is clear from the profile that not one single paper from the Astronomy Instrumentation Group was graded 4*, not even the papers highly cited papers from Boomerang.

    The panel must have felt that the Cardiff contribution to such things wasn’t world-leading, but how they came to that conclusion is a mystery. Perhaps they felt that Boomerang would have done just as well without any detectors on it.

    Suffice to say that I think we were very hard done by, and I think quite a few others think so too. However, there is no right of appeal so we just have to get on with it.

    The cancellation of Clover has added to the misery, although there remains a chance that alternative partners may be found. The Clover team are working very hard on that (and on doing as much as they can with what funding is left).

    Although these are setbacks, Cardiff remains in my view a strong department and a great place to work. It also, fortunately, remains very popular with students.


  2. telescoper says:

    Looking again at the scores it reinforces my opinion that the Physics panel was exceptionally harsh in its judgement on the outputs not just for Cardiff, but across the board. A large group of strong departments has their red dot above the blue one with an offset of about 0.4 between their esteem scores and their quality scores.

    Why the panel chose to stick the boot into physics in this way is something I really don’t understand.

  3. […] here, there, elsewhere, et cetera and passim. Andy Lawrence (e-astronomer) has now written a blog post about the latest publications from HEFCE  (commenting on the Cardiff situation with a generosity […]

  4. Michael Merrifield says:

    Oh for heaven’s sake, Peter. As I have been at pains to point out, the “offensive remarks,” for which I have already apologized, were intended as an (admittedly very poor taste) joke. The butt of this joke was intended to be those institutions who boosted their standings by submitting large numbers of lower-quality staff in order to boost their “research power,” since it is largely simply a measure of how many staff one submits (as plotting one quantity against the other will quickly reveal). Since Cardiff did not play this game, and you are obviously a researcher of high standing, it is pretty clear that neither you nor your institution were the intended subject.

    Incidentally, one of my colleagues here came up with a much better witticism in the same vein by calculating “research impotence,” which is calculated like research power, but weights inversely with the scores assigned to the different categories in the RAE process, thereby measuring which departments have the largest numbers of weak staff in their return. Entertainingly, but perhaps unsurprisingly, this table has many of the same institutions near its top who proudly proclaimed their performance on the research power league table.

  5. Michael Merrifield says:

    By the way, isn’t part of the reason that the output scores show a much smaller scatter when plotted against overall score than the environment that output counted 60% of the overall score while environment only counted 20%, so you are much closer to plotting x against x in the former case?

    The larger range is certainly telling, though. I guess it may be partly a psychological effect that it is easier to make harsh judgements about impersonal infrastructure than individuals. It would be interesting to see whether the same effect occurs in other panels.

  6. andyxl says:

    Michael – you are right that we shouldn’t pay too much attention to the scatter. But I would emphasise the difference in slope as well as range – the environment is a kind of amplifier of the output score. This could be a real effect – there really are marked differences in our research environments. It could be due to panel psychology – for example for outputs the panel had 4-3-2-1 distribution in mind as a kind of definition of international quality etc. Or it could tell us about who is best at playing the game. I suspect many Departments just didn’t take describing their environments seriously enough.

  7. Michael Merrifield says:

    Perhaps also because the idea of a profile makes less sense for the environment, since it is a more global property of the institution, so there is less regression to the mean than one gets when averaging together lots of individuals’ outputs.

  8. andyxl says:

    Ah yes. The good old Central Limit Theorem.

  9. insinkerator says:


    […]RAE games « The e-Astronomer[…]…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: