How to do a programmatic review

November 21, 2008

Today I had a query from a journalist asking what I thought of Astronet, the grand roadmap review for European astronomy. (Not to be confused with the real Astronet of course…). Its hard not to make comparisons with the US Decadal Review of Astronomy and Astrophysics, and of course the STFC Programmatic Review. I am not going to dig up and debate the results of these reviews, but comment on the process. (Maybe this will cheer up “blogs forever”).

European astronomers have long been jealous of the US decadal review. It has been very important in setting a genuine strategic and practical vision for US astronomy, and has been hard nosed about setting priorities. However its success has gradually made it a scarier process; behind the scenes it can get pretty tough. Since Roger Blandford agreed to chair the latest review, you can see both the excitement and the terror lurking behind his eyes.

Astronet tried to defuse some of that tension by tackling the problem in two stages – first produce the science roadmap, banning any mention of concrete items; only then follow on to make the shopping list. However this has possibly been a factor in Astronet being a tad gentlemanly. Its hard to divert the inertia of the steamrollers if everybody is being carefully polite. So possibly Astronet has been a little anodyne. However, thats second order; mostly I think Astronet has been a great success – in providing a forum for long range scientific debate, an opportunity for issues to be aired, a place where funding agency officials and scientists mingle freely, and a method for a large number of scientists to keep abreast of whats going on and whether it makes sense. The danger that various national funding agencies and treaty organisations like ESO and ESA could duplicate lots of work or do inconsistent things or waste potential opportunities is always very real. So I think Astronet has been a valuable planning exercise.

STFC attempted something similar with a committee of a handful of people supplied with existing paperwork plus questionnaires from a subset of the items under consideration; before the post-xmas fuss they did not take evidence or opinions from projects under consideration or the community at large, whereas Astronet has involved widespread consultation, many sub panels, public meetings, published reports and so on (before the conclusions.. not after).

The advantage of the “handful of people” method is that it is cheap and rapid, and it is possible to take radical decisions, whereas the “involve everybody” method is slow, expensive, exposes you to lobbying, and tends to inertia. This is exactly why STFC did it the focused way.  But … if you go for the quick-radical method, then (a) you can make serious mistakes, and (b) when the community find out whats going on there is an explosion of discontent. This is exactly what we saw between November and June this year, with STFC in lots of hot water. There were other problems of course, but I think this is fundamentally why there was an explosion. It is likewise not a coincidence I think that things have been a lot calmer since the post-hoc “consultation panels”.

Frustratingly, one of STFC’s two predecessor organisations, PPARC, did undertake a decadal-review like process, around 2000-2001, and it was very successful. It involved setting up a number of advisory sub-panels in various areas, and allowing free form opinion submission, to help inform PPARC’s science committee. STFC disbanded these committees, wanting to simplify the review process. Biiiiiig mistake.