Cosmic Convergence

I have a cute diagram for you. But first, as Frankie Howerd would say, The Prologue. Some background is necessary in case you get the wrong take-home message once I unveil the interestin’ picture…

Periodically somebody frightens us with tales of how The Data Deluge Threatens Science. (Yes, guilty.) Cynics will suggest that Moore’s law runs even faster, so computers will always be good enough – whats the problem? Actually, I think the interesting things are those that are not improving exponentially – last mile bandwidth, disk I/O speed, and human resource. The first two are a bit of a bummer for individuals, but fixable if you are a pro data centre – tune your TCP buffers, open multiple channels, hang thirty discs off your motherboard, etc. So this drives us to a service architecture. But so does the third thing : people effort. Developing interfaces to the data, and curating it as it comes in, takes work. A lot of this scales per archive rather than per bit. The real problem, and why we need the VO, is the number of archives not the number of bits. Its a Tower of Babel thing.

Enough of the VO lecture. The size of each major astronomical archive certainly is growing fast. But so is the size of a typical hard drive. As part of some work with Bob Mann and and Mark Holliman, I was collecting some data on these things, when suddenly it occurred to me not just to talk in general terms but to actually plot one on top of the other. Exhibit A therefore shows (i) the evolution of the size of PC disks, from the wikimedia commons page here , and (ii) the total size of the ESO Science Archive, divided by 200 – data kindly provided by Paolo Padovani. The step function in 1999 is real – its when the VLT switched on. Otherwise, they track each beautifully. Over two decades, while volumes have increased by factors of tens of thousands, the number of PC drives needed to hold the ESO data has stayed the same within a factor of two.

I guess both of these things, like a number of others, are loosely driven by a combination of integrated circuit technology and economics. But the fact that they are so close seems surprising. Maybe there is some kind of weird technology central limit theorem thing going on.

I believe

The blog awakens. One interesting thing is that even when I haven’t written anything for weeks, there are still people reading it. Seems to be a random assortment of old posts.

I guess I should write something about the Parliamentary Select Committee report  but Peter already blogged it  and didn’t get any debate, so maybe it needs a bit more thought, aka come up with provocative line.

Instead I shall continue my occasional travelogue series. I know you love it. Here I am in Naples for the twice-yearly IVOA interop. Keywords for Napoli are : chaos; pizza; chaos; antiquity; noise; sfogliatella; chaos; religion; volcano; chaos; and chaos. I do like the chaos. But its exhausting.

There are churches and shrines everywhere, packed into streets so narrow that Google Maps can’t name them. The only one with more than about twelve feet of space in front of it is the Cattedrale San Gennaro. Twice each year a miracle occurs  at this location. A vial containing the dried blood of San Gennaro himself spontaneously liquefies. A few years back I was lucky enough to witness this miracle. Well. I say witness. Actually I was craning my neck to see over the heads of about eight million people, and could just about see some blokes in funny costumes waggling something.

So is this what I believe ? Do us a lemon, John. No. I believe in Hollywood. Bear with me.

Having a few hours spare I went to see the National Archaelogical Museum. This contains some marvelous frescoes and mosaics, some preserved in Pompeii. Here are two favourites : the famous portrait of Sappho, which actually probably isn’t; and a lovely mosaic of fishes.

There are also rooms and rooms of rather dull and pompous Roman statues. However I did rather enjoy the gallery of emperors, and spotted a lovely contrast. On the left is Marcus Aurelius, the Philosopher-Emperor. As we know from watching Gladiator, he was a wise and just ruler as well as a deep thinker. I have his “Meditations” and even read about half of it one day before I got hungry. And golly gosh his nobility just leaps out of the bust. One the right is his son Commodus, who as we also know from Gladiator, was a snivelling shit, a nasty piece of work, and a dastardly cheat. Gibbon dates the beginning of the Decline from his reign. And blow me, his bust looks just like Joaquin Phoenix in the movie. I definitely wouldn’t trust this man.

All temptation to point out resemblances to current personalities will be avoided.

Vizier sees all

Here I am in Heidelberg for another Virtual Observatory meeting. Nice place. Pleasant journey, apart from German Bloke With Kilt at Edinburgh airport. Normally, one sees kilts in two circumstances. First, young Scottish chaps at weddings, ceilidhs etc. Two, American tourists on the Royal Mile. So this was a third category. Hairy German biker type with long hair and earrings who I guess decided it looked tough. Thing is, he hadn’t worked out how one sits politely when wearing a skirt. Either that or he was a very confident chap. Except that he had no particular reason to be so. Oh no, stop, enough ! Don’t look ! Out vile jelly !

Talking of the all-seeing eye, my CDS chums tell me they know about your lunch breaks. The Vizier servers can tell how long a given IP address is connected, and when it comes back. So our Alsatian friends know how long you have for lunch. Apparently it is very clear that the Spanish have the longest lunch breaks and the Brits have the shortest…

Time Loop Chaos

New experience this morning. I talked at a conference by remote video link. Its the first time I’ve given a talk sitting on my bed. Appropriately the conference in question was .Astronomy-2009, held in Leiden. (Thats “dot-astronomy”.) I was sorry not to be there, as I have never been to Leiden, it looks like a fun meeting, and it claims to be an un-conference, which is a bit Lewis Carroll. You can follow the meeting on Ustream and on Twitter.

This turned out to be much harder than I expected. I had sent them a PDF of my talk, and called in by skype. So far so good. The organisers suggested I follow my own talk on Ustream so I could see when my slides changed. However… this turned out to have a three second delay. As a result my own voice was overlapping itself in the most gharssly manner and it was hellish hard to concentrate. I suspect I sounded like a complete buffoon. My kids tell me I was loud as well as stumbling, and apparently sounded like Ron Weasley’s Dad on the telephone.  Hopefully the written words made some sense.  If  you want to see the PDF, I have uploaded it to my ROE page.

Meanwhile, I have a link for all those people who over the years I have heard complaining that Swindon is the most mind-numbingly boring town in the world, with the possible exception of Newport Pagnell. In fact, I will have you realise, Swindon is at the very bleeding edge of the internet revolution.

Astro widgetery

Apologies to all you PC and Linux folk – this one is for Mac fans. It was no surprise to find that the Fermi first light sky map was today on Astronomical Picture of the Day (APOD). However I didn’t need to go to the web site to check it out. I have a Widget that sucks down the latest APOD every day. For the non-Macees, widgets are tiny little applications (few hundred K typically) that run inside the Dashboard. They are apparently dead easy to write (not that I have tried yet …) as they are basically web pages, written with HTML and Javascript, but instead of being run by a browser, they run inside the Dashboard runtime environment.

As a result, there are now hundreds of these things, including a whole bunch of astro related ones. As well as APOD, you can get several different planetarium style viewers, including a free version of Starry Night, widget versions of the Vizier catalogues and ADS literature search, two different solar image viewers, and the latest Astronomical Telegrams. Those links are to the web-pages. To find the widgets, go to the Apple download page, browse, and enjoy. Its rather addictive, as they are so easy to download and try out. As well as all that astro stuff, there are lots of amusing things, like a Universal Translator, a Roman Numeral Clock, and regular stuff like weather forecasts etc.

A lot of this stuff you can of course also get with VO tools like VODesktop, Aladin, or PaperScope. The advantage of the widgets is they take about 3 minutes to understand.  The advantage of the VO tools is that you can carry on and do some real work with what you find.

If you want to know how to develop widgets, go here.

C-day Plus One

Here I am in the very thick of the madding crowd : Heathrow Terminal Three, en route to San Francisco. Yesterday STFC Council met and finally sealed the fates of various projects (*). The outcome is described in a pdf file you can get here. STFC folk will brave up to the crowd and explain it all at the Town Meeting on July 8th … but there ain’t really any surprises, so not sure how that will go. For me, its a mixed story; the approach to UKIRT/UKIDSS is much saner than before; the WFAU/CASU stuff is peripheral, outside the core work, and shouldn’t really have been in the review at all; and AstroGrid as expected is sacrificed. The previous two days I have been running the twice yearly AstroGrid Consortium Meeting, and we have planning how to “gift wrap” our product as professionally as we can, on the assumption that our early closure would indeed be announced. The referee’s decision is final; no barricades will be manned or TV crews invited; but if I hear that there are no cuts, only “unfulfilled aspirations”, I will spit. E-science fans may note that the long term situation is much more complex; the Astronet facility roadmap strongly recommends continued investment in the VO, and the international drive in this direction is undiminished; all we have done is relinquish the UK lead in this area. Hey, you can’t have everything. No really, you can’t. Probably we have just displaced the cost of dealing with the global data management infrastructure elsewhere. Watch this space.

So whither STFC ? They seem to be doing a Geoffrey Boycott, sticking doggedly at the wicket and adding a run or so every few minues. The Select Committee report was vicious, with many truths but also unecessary personal attacks on KOM. The Government response seemed to be a stubborn and patronising denial of all the points … but … you will note that there is now an organisational review of STFC underway. Is this a deflection, or a lining up for the firing squad ? Time will tell.

You have only until July 9th to make your input to the review

Meanwhile, lets look on the bright side ….

• PPAN did make SOME changes
• DIUS is looking hard at whether STFC is the right structure
• MoonLite is being opened up to very public scrutiny
• Advisory Panels are being re-invented
• The competent ex-PPARC bureaucracy is re-asserting itself

---

* posted next day, stateside

Peace breaks out in Vienna-on-Sea

This week your roving reporter is in Trieste. Its in Italy, but it looks like Austria. Until 1918 it was in Austria, and its full of beautiful neo-classical buildings, including a huge impressive square facing straight out to the ocean. They call it Vienna-on-Sea. Eating here is weird. Like anywhere in Italy you can get spaghetti vongole or grappa, but you also get Leberkase, Schweinhaxe, and Weissbier. (The latter two known locally as Stinco and Weizen.)

The excuse for being in Trieste is the twice yearly “interoperability workshop” of the International Virtual Observatory Alliance (IVOA). Arriving at this meeting an air of gloom hung like a fog, as there seemed to be two irreconcilably different approaches to specifying how you access database tables (TAP/QL and TAP/PARAM if you want to know) and it seemed that for the first time in its history the IVOA process might not converge, and two VOs might emerge, one either side of the Atlantic. However …. while email debates turn so easily into flame wars, once you are drinking beer together, you can’t keep up the hostilities. It wasn’t easy… but its look like we have converged on an awkward but workable compromise.

But it gets better. If QL and PARAM look like warring camps, just try Google and Microsoft. In one session we had talks about Google Sky (Ryan Scranton) and World Wide Telescope (Jonathan Fay). They are both fantastic pieces of software. Primarily they are for education and outreach rather than pro work, but it won’t stay that way – WWT is already built on VO protocols (you can do a USNO-B cone search at the position you have panned to…) and they are hooking up with the Harvard IIC folk to build a “WWT-pro” version. Meanwhile as I mentioned some months back, VO hackers have already been building VO plugins for Google Sky using the Google Keyhole Markup Language (KML). Now here is the stunner .. the guy from Microsoft said that WWT would support KML. Wow ! Peace and harmony.

But wait. It gets better. One of the longest standing wars is of course PC vs Mac. Well… when the Microsoft guy arrived at the podium, he had his talk on a Macbook. Sure, it was a Mac running VISTA using Bootcamp… but still. Microsoft guy. On a Mac. I hear the angels singing.

Repent Harlequin

Time passes, heading towards the closing hour for the STFC consultation. Who will be turned off, and who is the ticktock man anyway ?

I am in Strasbourg, running the bi-ennial Technology Forum for the European Virtual Observatory, as part of the VOTECH project. But actually I am spending half the time on email, preparing or helping to prepare multiple submissions to STFC to protect various projects – AstroGrid, UKIRT (read UKIDSS), WFAU/CASU, XMM.

It would be a depressing week if it weren’t for the fact that Strasbourg is a groovy place and the food is nice. I have of course been eating Tartes Flambees nearly all week. Just don’t call it pizza or the locals get cross.

vo-joke.xml

<?xml version=”1.0″ encoding=”UTF-8″?>
<vor:Resource created=”2008-02-25″ status=”active” updated=”2008-02-25″ xmlns=”http://www.ivoa.net/xml/VOResource/v0.10&#8243; xmlns:jok=”http://www.ivoa.net/xml/JOK/v0.8&#8243; xsi:schemaLocation=”http://www.ivoa.net/xml/VOJokeService/v0.8″&gt;
<title xmlns=”http://www.ivoa.net/xml/VOResource/v0.10″>XML and Violence Joke</title>
<shortName xmlns=”http://www.ivoa.net/xml/VOResource/v0.10″>XML-Violence</shortName&gt;
<identifier xmlns=”http://www.ivoa.net/xml/VOResource/v0.10″>ivo://wfau.roe/XML-JOK1</identifier&gt;
<curation xmlns=”http://www.ivoa.net/xml/VOResource/v0.10″&gt;
<publisher ivo-id=”ivo://wfau.roe/JOKES”>Andy Lawrence</publisher>
<creator>
<name>Duncan Law-Green, Leicester</name>
</creator>
<contributor>keith Noddle, Leicester</contributor>
<version>1</version>
<contact>
<name>Andy Lawrence</name>
<email>al@roe.ac.uk</email>
</contact>
</curation>
<content xmlns=”http://www.ivoa.net/xml/VOResource/v0.10″&gt;
<subject>XML</subject>
<subject>Humour</subject>
<description>XML is like violence. If its not working for you, you are not using enough of it.</description>
<referenceURL>https://andyxl.wordpress.com</referenceURL&gt;
<type>Geek Joke</type>
<contentLevel>Research</contentLevel>
</content>
<vr:interface qtype=”GET” xmlns:vr=”http://www.ivoa.net/xml/VOResource/v0.10&#8243; xsi:type=”vs:ParamHTTP”>
<vr:accessURL use=”base”>https://andyxl.wordpress.com/this post/</vr:accessURL>
<vs:resultType xmlns:vs=”http://www.ivoa.net/xml/VODataService/v0.5″>laughter</vs:resultType&gt;
</vr:interface>
<vs:facility xmlns:vs=”http://www.ivoa.net/xml/VODataService/v0.5″>Leicester XML Factory</vs:facility>
<vs:instrument xmlns:vs=”http://www.ivoa.net/xml/VODataService/v0.5″>multi-channel joke generator</vs:instrument>
<coverage xmlns=”http://www.ivoa.net/xml/VODataService/v0.5″&gt;
<spatial>
<region xsi:type=”vs:AllSky”/>
<regionOfRegard>0.14</regionOfRegard>
</spatial>
<spectral>
<waveband>Blue</waveband>
</spectral>
<temporal>
<startTime>Big Bang</startTime>
<endTime>Big Crunch</endTime>
</temporal>
</coverage>
<cs:capability xmlns:cs=”http://www.ivoa.net/xml/JokeSearch/v0.3″&gt;
<cs:maxSR>1</cs:maxSR>
<cs:maxRecords>1000</cs:maxRecords>
<cs:verbosity>false</cs:verbosity>
</cs:capability>
</vor:Resource>

Jim Gray tribute meeting

Tony Linde just emailed me to bring to my attention a Tribute to Honour Jim Gray, on May 31st in Berkeley.

Jim was a giant of the database world, and a good friend to the Virtual Observatory world in general, and Johns Hopkins and the SDSS archive in particular. He also helped us here in Edinburgh in our development of the WFCAM science archive, which is based on Microsoft SQL Server.

In January 2007, he disappeared without trace while on a sailing trip off the Californian coast. This was one of my own posts about Jim.