How to do a programmatic review

Today I had a query from a journalist asking what I thought of Astronet, the grand roadmap review for European astronomy. (Not to be confused with the real Astronet of course…). Its hard not to make comparisons with the US Decadal Review of Astronomy and Astrophysics, and of course the STFC Programmatic Review. I am not going to dig up and debate the results of these reviews, but comment on the process. (Maybe this will cheer up “blogs forever”).

European astronomers have long been jealous of the US decadal review. It has been very important in setting a genuine strategic and practical vision for US astronomy, and has been hard nosed about setting priorities. However its success has gradually made it a scarier process; behind the scenes it can get pretty tough. Since Roger Blandford agreed to chair the latest review, you can see both the excitement and the terror lurking behind his eyes.

Astronet tried to defuse some of that tension by tackling the problem in two stages – first produce the science roadmap, banning any mention of concrete items; only then follow on to make the shopping list. However this has possibly been a factor in Astronet being a tad gentlemanly. Its hard to divert the inertia of the steamrollers if everybody is being carefully polite. So possibly Astronet has been a little anodyne. However, thats second order; mostly I think Astronet has been a great success – in providing a forum for long range scientific debate, an opportunity for issues to be aired, a place where funding agency officials and scientists mingle freely, and a method for a large number of scientists to keep abreast of whats going on and whether it makes sense. The danger that various national funding agencies and treaty organisations like ESO and ESA could duplicate lots of work or do inconsistent things or waste potential opportunities is always very real. So I think Astronet has been a valuable planning exercise.

STFC attempted something similar with a committee of a handful of people supplied with existing paperwork plus questionnaires from a subset of the items under consideration; before the post-xmas fuss they did not take evidence or opinions from projects under consideration or the community at large, whereas Astronet has involved widespread consultation, many sub panels, public meetings, published reports and so on (before the conclusions.. not after).

The advantage of the “handful of people” method is that it is cheap and rapid, and it is possible to take radical decisions, whereas the “involve everybody” method is slow, expensive, exposes you to lobbying, and tends to inertia. This is exactly why STFC did it the focused way.  But … if you go for the quick-radical method, then (a) you can make serious mistakes, and (b) when the community find out whats going on there is an explosion of discontent. This is exactly what we saw between November and June this year, with STFC in lots of hot water. There were other problems of course, but I think this is fundamentally why there was an explosion. It is likewise not a coincidence I think that things have been a lot calmer since the post-hoc “consultation panels”.

Frustratingly, one of STFC’s two predecessor organisations, PPARC, did undertake a decadal-review like process, around 2000-2001, and it was very successful. It involved setting up a number of advisory sub-panels in various areas, and allowing free form opinion submission, to help inform PPARC’s science committee. STFC disbanded these committees, wanting to simplify the review process. Biiiiiig mistake.

27 Responses to How to do a programmatic review

  1. Tom Kerr says:

    Dear Andy,

    Six months ago, andyxl was a most eagerly consulted and sought topical blog of the day for the pro astronomer – full of incisive, “and occasionally informed comment”.

    In recent months it has gone from the must read for the whole community to the discussions of the vaguely interesting to the few.

    Now, it’s your blog and you have every right to post whatever you want in it, and I don’t in any way want to criticise your right to write as you like, but I’m going to anyway.

    I’ve really enjoyed your recent posts about living in the Stanford area and what the hell are you doing by talking about scientific politics again? I mean, come on. I’m sure I will be censored but let’s not return to the basics after everyone is sick and tired of the STFC mess and would actually like to read about something else for a change.

    Please go back to “normal service” again. 😉

    I’ve enjoyed your recent posts. It’s nice to get away from the politics once in a while.


  2. Michael Merrifield says:

    I think I would take a more moderate line (well, there’s a first time for everything!), and say that both approaches are right.

    The problem with going down the full-community route is that, as you pointed out, one ends up in proportional representation hell, trying to please all the people all the time, with no-one having the authority to take difficult decisions. If you don’t take difficult decisions, you fundamentally aren’t setting strategy.

    It is surely appropriate to go the Astronet route to map out the research landscape, make sure that all voices are heard in putting forward the cases for different research priorities, and set out in broad terms what areas make up the ensemble of exciting research options. But ultimately the decisions need to be taken by a much smaller group who can synthesize the information on the landscape, factor in the non-scientific issues such as the corresponding political landscape, and decide which of the many possible strategies to pursue, and where losses need to be cut.

    Personally, I don’t think that restoring the advisory panels is the way to go: even in their previous incarnation, there was a perception (and perhaps a reality) that their advice was only used as reverse-engineered justification when it happened to coincide with the decisions that the Executive had taken. If STFC is serious about restoring community engagement, it needs to set up panels of community members with delegated powers to actually make the strategic decisions within their own disciplines, after full briefing on the political as well as the scientific realities. Obviously, these decisions need to be moderated at a higher level to ensure the strategic direction of the overall organization, but the default should be that the power lies with the community-populated panels, with a “veto” from higher up requiring explanation and justification to be disseminated back to the community. This decision-making process requires members of the panel to be trusted to take a broad view and not simply fight their own partisan corners, but we already manage to do this perfectly effectively and professionally in, for example, allocating grant resources, so why not on broader strategic decisions?

    This seems to me to strike the best balance between the two extreme models that you describe, by engaging the scientific stakeholders properly in the decision-making process, while avoiding the deadlock that would surely result if strategic decisions were truly devolved to the whole community.

  3. andyxl says:

    Mike – one the problems with setting up sub-panels with teeth across all the relevant disciplines is that for STFC thats potentially a large list – astronomy, space science, particle physics, nuclear physics, condensed matter physics, biology, chemistry, commercial users of Diamond (they hope..) and all the sub-disciplines thereof. As well as the scope there is a difference in style. For us, and PP, decisions on the facilities is intimately tied up with a strategic view of our whole discipline; but for most of the others they are just facilities, and often they positively don’t want any smell of interfering in their subject thank you very much. So its difficult to imagine how STFC can do it in a uniform manner.

    Tom : more pumpkins and mountain lions in due course. By the way, I’m tired of the politics too; this post wasn’t meant to be “lets get cross about STFC all over again” – it was meant to be about how you do strategic planning – a very difficult problem.

  4. Tom Kerr says:

    Andy – my comment was purely tongue in cheek given a comment on your last blog entry! Actually, your blog was a very useful resource during the STFC crisis as it was a place that seemed to attract commentary from several UK astronomers. Sometimes it was very difficult to follow the goings on from the other side of the planet but your blog and its readers certainly helped clarify the issues. It’s good to read about the other stuff as well though!

  5. Michael Merrifield says:

    Andy — I have heard this assertion quite a few times before, not least in the argument as to why astro and PP grants should be treated differently from mainstream physics, but have seen precious little evidence to back it up. It certainly is far from universally true, as many astronomers I know view telescopes as facilities that enable them to get on with science, and many physicists I know take a keen interest in the development of the instrumentation that will enable them to do the science they want to do, such as the work done within my own department on beamline proposals for Diamond.

    In any case, I can’t really see a conflict. If there aren’t difficult strategic decisions to be taken within a discipline, then the associated committee just won’t have much work to do; where there are, the stakeholders will be fully engaged because they have a real say in the outcome. Personally, I would hope that there are difficult decisions to be taken all across the spectrum, as it would surely be rather dull science if a discipline is so unimaginative that there is only one idea as to what the next facility should be.

  6. Richard Wade says:

    Coincidentally, Keith and I met with Roger Blandford on Friday to discuss the upcoming Decadal Survey. The first thing we did was to give him a hard copy of the Astronet book.

    I think Mike is correct to say that both approaches are needed and in this respect the Astronet exercise has been a great example of wide community input into the definition of the scientific challenges and landscape. As for the other approach, what Mike describes was pretty much what STFC intended to do when it set-up Science Board, PPAN and PALS. The mistake it made (OK not the only one) was to stand down the old PPARC Advisory Panels before it and PPAN had figured out how they wanted to interact with the community.

    Note that the Programmatic Reviews (started under PPARC of course) where intended to review existing facilities rather that to survey future opportunities and as such were/are very different animals from either the Decadal Surveys or the Astronet process. You will be pleased to know that Keith and I spent far more time talking to Roger about Astronet than we did about the Programmatic Review.

  7. Martin E. says:

    Andy: A bunch of US-oriented observations… (too long again, sorry in advance)…

    The Decadal Review doesn’t have ‘teeth’ – it’s all advisory – but it works. So that bite isn’t so important.

    Sometimes it works ‘better’ than intended: JWST was top of the list, so JWST is the #1 NASA priority until it’s done, whatever it costs. That ~$4B cost just happens so rule out ever having 3 Great Observatories spanning IR/optuv/X-ray ever again: unintended consequences.
    [The only way out is to cap Big missions at ~$2B, rather than ~$4B, assuming current budgets, which may be optimistic in the current economic meltdown.]

    As a result of the JWST situation, there has been lots of talk of more realistic costing for the Blandford panel, of having options for what to do in case Mission X doesn’t look feasible, and of dividing the budget into ~3 lines: Small/Medium/Large (Tall/Grande/Venti might actually be more appropriate, for once), with firewalls between their funding pots.
    I’m a fan of this approach, and of more open competitions down the line out of the always too many good ideas. The US Explorer program is a great example, but needs re-invigorating to get a good launch rate. If doubling the Explorer budget slows down the larger missions, so be it.
    Even ground-based could benefit from open competitions for smaller projects. E.g. Last round we all went for big telescopes, closing out options for long baseline interferometry. IF the Magdalena Ridge folks (including Chris Haniff in Cambridge UK) are right, then sensitive milliarcsecond near-IR imaging is feasible at much lower cost than one conventional 8-meter. They may not be right, but the consensus process tends to foreclose smaller scale bets on potentially extraordinary advances such as this.

    The last decadal was structured with panels that predetermined the outcome, by being wavelength based, AND divided into ground and space, so opt and ir were separate, and doubled up between ground and space. That didn’t leave radio or high energy folks much territory, and makes it unsurprising that JWST was #1 in space and GSMT (correct acronym?) was #1 on the ground. Being #2 behind JWST (the Con-X slot) is like coming 2nd in the Oxford and Cambridge boat race – dead last.
    So, as every good politician knows, it’s how you set up the committees and who you put on them that decides 90% of the outcome.

    Apparently Roger has his main panel together and is now setting up Science based panels, more like Astronet. Maybe there will also be wavelength panels, but I take it as a positive development. Of course, science panels could be: Inflation, Dark Ages, Large Scale Structure, Star formation and Exoplanets. How you divide the pie makes a huge difference. The above slicing of cosmology is not too exaggerated: I actually saw happen in one mini-Decadal I was on – each method of measuring H0 was considered a separate topic, roughly equal to all quasar/AGN astrophysics. Now that I’m older and wilier I’d try to prevent that re-occuring, but science balance is tough too.

  8. Michael Merrifield says:

    Richard — which of us should be more worried when we find ourselves in agreement? Next thing you know, you’ll be buying more of my papeweights! ( for anyone who may be short of Christmas inspiration.)

  9. Michael Merrifield says:

    Sorry. That should have been a gratuitous advertising link.

  10. Richard Wade says:

    Mike- Are you sure that listing STFC as one of your customers is a good marketing ploy?

  11. Michael Merrifield says:

    We thrusting business tycoons can’t afford to be fussy about our clientele, Richard…

  12. Paul Crowther says:

    So, as every good politician knows, it’s how you set up the committees and who you put on them that decides 90% of the outcome.

    Richard, on this question.. it seems the DTI was responsible for the make-up of STFC Council (3 academics, 3 Execs, 3 non-Exec/non-academics), but who originally decided how the STFC’s committee structure would be organised and who decided on appointments to Science Board, PPAN and PALS? Just curious..

    RAS Fellows have just had a mailshot including a letter from El Presidente, including a paragraph on the subject of the Prog Rev, copied verbatim below so that Mike doesn’t feel left out.

    .. I don’t wish to rehearse all the angry debate that lasted for many months until the final Programmatic Review seemed to bring the situation in terms of facilities almost back to where we started. Some significant programmes have been lost and there are still angry and bewildered scientists around. Our international partners must now view us with suspicion or at least question our reliability…

  13. andyxl says:

    Richard – I gave a seminar here in Stanford on Thursday, but apparently Roger thought visiting you guys was more important than going to my seminar. Would you Adam and Eve it.

  14. Richard Wade says:

    Paul-The advisory structure for STFC (below Council) was proposed by a Merger Project Board comprised of representatives from both PPARC and CCLRC. The structure was discussed and agreed with both Councils and with DIUS’s predecessor. Members were selected using the PPARC process (now the STFC process available on our web site) and agreed by the new STFC Council.

  15. Paul Crowther says:

    Richard- Many thanks. I looked it up and under`The STFC Process’ we get..

    In deciding the membership of its advisory committees and specialist panels, STFC consults widely with its community on an on-going basis through the STFC web site.

    Postings on websites isn’t really how I would go about `wide consultation’, but there you go. The rest looks eminently sensible of course, involving the committee chair, STFC director and STFC CEO for membership.

    On the subject of STFC Council, the summary of Keith Mason’s witness testimony to Wakeham includes

    The Council was not designed to be representative of the scientific community, as Council obtained its scientific advice from the Science Board.

    So far so clear, but if STFC Council wasn’t meant to be representative of the community, and with only 2 non-exec, UK-based academics aboard – Mike Edmunds and my boss Keith Burnett – how could it claim to be able to judge the proposed membership and balance of Science Board etc? or for that matter take care of governance issues in a way that is sufficiently independent of the Executive?

    You may know that the IoP has formally responded to RCUK about Wakeham. This (unpublished) document welcomes the appointment of two additional non-exec members of Council – why is the CEO on the selection panel by they way? – but still argues that the overall make-up of Council remains unbalanced, proposing a reduction in Executive membership on Council so that the non-exec membership form a clear majority, plus ensuring a better coverage of STFC science – so far academics are only from astro and condensed matter physics. In my humble opinion, the IoP have it spot on.

  16. Richard Wade says:

    Paul- I think that some of the statements that people make about the makeup of Council are based on misconceptions of how such bodies work. Firstly they seldom if ever vote as they don’t have elected officers and other decisions are arrived at by consensus where possible. Secondly the members of the Executive would be present whether they were members or not (this was true in PPARC and probably is in every other Research Council). The Council can have closed sessions without the Executive if it wishes and this has happened in STFC as it did in PPARC. So the only effect of having fewer Executive Members would be to have more people in the room and I’m not sure that was ever a recipe for better decision making. That said I am happy that we are adding two further science members as the current coverage, as you point out, is rather limited.

  17. Rob Ivison says:

    Martin E. – interesting to see your take on how the “#1 JWST; #2 Con-X” situation developed on your side of the pond.

    In Europe, ESA opted for a thematic approach to Cosmic Vision 2015-25. With committees populated with X-ray astronomers and the high-energy Universe as one of a small handful of themes, the success of the Xeus proposal was never in much doubt, even putting aside the millions spent on Xeus-related development work before the mission faced review or competition.

    This is not intended as a criticism of Xeus, by the way – it is a potentially great observatory and its science complements that of the missions I’ve been pushing (FIRI, SPICA). However, I would like to see the UK take a firm line on procedures and peer review in the organisations to which it pays vast subscriptions (having re-established the integrity of our own peer review process, of course).

  18. Tom Kerr says:

    Mike – I would never, ever follow a gratuitous advertising link in someone’s blog, but would you consider a discount for an ex-Nottingham guy? 😉 You may have solved a christmas present dilemma I’m currently experiencing.

  19. Michael Merrifield says:

    Hi Tom — usually I charge chemists extra, so let’s call the basic price even…

  20. Michael Merrifield says:

    As for the other approach, what Mike describes was pretty much what STFC intended to do when it set-up Science Board, PPAN and PALS. The mistake it made (OK not the only one) was to stand down the old PPARC Advisory Panels before it and PPAN had figured out how they wanted to interact with the community.

    This comment got me to wondering why so many astronomers do not feel enfranchised in the process, since, as Richard points out, they are certainly represented in these peer-based committees. I think the disquiet arises because the decison-making process isn’t devolved quite far enough. Fundamentally, particle and nuclear physicists won’t have much to say on what are essentially internal astronomy decisions, so the number of people on PPAN actively involved in deciding the future for astronomy is very small indeed. Reinstituting advisory panels would be one way to address this issue, but unless there is some transparent way in which PPAN is required to consider explicit recommendations and justify those occasions on which it decides not to follow these panels’ advice, I don’t think that setting up such talking shops will convince most stakeholders that they have a real say in the decisions that can define their careers.

  21. Martin E. says:

    You UK folks are so used to committees with teeth. But the US Decadal works despite merely expressing a wish. There is somehow a social expectation that the government agencies should follow what the community wants. It certainly doesn’t work 100%, but it guides effectively.
    Why is there this different culture?

    ps Andy: Maybe your blog should undergo mitosis into one for astro-politics, and one for Deep Thought & Interesting Stuff?

  22. Keith A. says:

    Martin – isn’t the crucial fact in the US that any agency which doesn’t follow the Decadal priorities ends up having to explain to a congressional committee why not ?

    I have a couple of concerns with these big programmatic reviews. The first, channeling Freeman Dyson, and agreeing with Martin is that the reviews tend to recommend the big, expensive projects at the expense of smaller, nimble proposals. The second is that they are likely to drive all countries towards working on the same, trendy issues rather than diversifying. This seems to be a particular problem for ESA which has in several cases ended up pursuing missions which are similar to those of NASA rather than putting resources into scientific proposals where ESA will be making unique contributions. You can fill in your own examples here.

  23. Steve W says:

    To respond to a comment by Richard Wade further up: he says it makes no difference whether the members of the Executive are officially members of Council or not, as they would be present at meetings anyway. I find this view a little worrying. It makes a great deal of difference if you are actually a member of a committee, or simply ‘in attendance’. In the latter case, in effect, you only speak when spoken to – which conjures up a delicious image in my mind.

  24. Paul Crowther says:

    I too was struck by Richard’s reponse to my comment about the membership of Council.

    I’m sure that a consensus is always sought by the panel chair. However, instances must surely have arisen during the last 18 months when there were differences of opinion among Council members about the (difficult) strategic decisions that were necessary, both in preparing for CSR07, and implementing STFC’s less-than-ideal allocation.

    On such occasions I suspect a different composition of Council may not have followed the same approach as that what was taken, whether for the better or not. BTW anyone care to explain why the present CEO of STFC should sit on the selection panel for new Council members..?

    I also share Mike M’s fears about the effectiveness of the forthcoming advisory panels to PPAN. These are far better than no panels at all, but presently lack any genuine ‘bottom-up’ influence. PPAN brushed-off scientific advice from the ad-hoc panels when it suited them, and will likely do the same without better connectivity between panels.

  25. Richard Wade says:

    Steve- I didn’t say that it made no difference whether the Eecutive were members of Council or not. I said that they would be in the room whether they were members or not and so that the effect would be that there would be more people in the room. STFC Council is getting two extra members and as I said, the Council can meet without the Executive if it wants. So in effect it makes no real difference. I’m not sure what your experience of Council and Board meetings is but at most of the ones I attend people do only speak when invited, members or not.

    Paul- The CEO’s are always present for Council member interviews. This was true in PPARC is true in STFC and I would expect to be the case in other Councils. The Chair of the Council is also present. Why don’t you apply and then you can try it for yourself.

  26. Paul Crowther says:

    Richard, thanks for your support but Council has too many astronomers already 😉

  27. Tom Kerr says:

    A chemist? Me? I’ve never felt so insulted…. 😉

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: