CERN Accelerating science

Abstract

I have also my modest anniversary to celebrate. In July this year it will be 20 years that I got hooked on computing. This is my personal perspective of how have I lived this period.

I am physicist by education. Actually until the summer of 1981 I had not had access to a computer, ever. Not even a Commodore or Spectrum.

All started with FORTRAN

In spring 1981 I was in my last year of studies. Juan Antonio Rubio, head at the time of High Energy Physics department at "Junta de Energía Nuclear", was head-hunting students to work in his group as part of a yearly recruitment exercise. He came to our university to tell us about particle physics and CERN, and invited in exchange those interested to submit CVs to his consideration.

I wonder what was it. Maybe the magnetic attention I was paying to his talks (they were very good), maybe the prospect to pursue a "research career" rather than recycling myself into another field... Whatever it was, I applied and I became one of the lucky ones.

By the summer the course was over, and I joined in Rubio's group. My first assignment was simple enough:

"Here are the notes to a FORTRAN tutorial attended by another colleague; here is a punched card machine that all of us use regularly. However being a newcomer we will save you of that hassle; we have installed one terminal (to a UNIVAC mainframe). Learn the language, learn how to send batch jobs via the terminal, come back when ready"

I shared this curious behaviour with my supervisor, who then replied to me with an ample smile:

Ah Miguel, I am afraid I forgot to tell you. This brand new terminal is one of the two in the whole building, and staff needing it are expected to book through this notebook their slots. You must have been bypassing the reservations of everybody over the past 10 days!"

Although I had not realised by then, I had become a computoholic.

Discovering the pleasure to help

So I had a frantic startup at the UNIVAC of my institute, but I had not really been charmed into the intricacies of Assembler, 36 bits or the guts of the Operating System.

I slowly found out that my proficiency at the FORTRAN language and the literal devoring of the computer documentation (just delivered) turned me into one of the possible references to "difficult questions". I don't say I was not flattered by that feeling. Being able to offer advice to my own colleagues!

Maybe due to these regular demands for help, but also because I had to learn about the "CERN Programming environment" in order to participate in the software developments of the group, I started to acquire the basic notions of the CERNlib of the time: HYDRA, HBOOK, HPLOT, etc. And the corresponding manuals.

I had been assigned to share an office with another colleague (and good friend of mine) Carmen Albajar. Maybe because we both liked systematic approach to problems and had a higher level of "self-organization", we started compiling relevant documentation. To such extent that we built a de-facto "documentation reference library" for the group. Colleagues offered us some of their (unique) copies such that everybody would know where to find them when needed.

I guess that one of my leading focus in supporting users seeded there. Good Quality Documentation is the Valet of Good Quality Computing.

Meca of Computing

My second computing "encounter" happened in the summer of 1982. My supervisor, Pedro Ladrón de Guevara, took me to CERN. He was member of several NA collaborations and there was some field work which had to be prepared at the experimental area.

Pedro was good friend of a certain Julius Zoll. So I was introduced to a very charming, sympathetic fellow; sometimes perhaps dogmatic in his computing opinions but who was I to do anything else than to learn from a lifetime experience? Only a while later I learnt I had met one of the ancillary references to CERN computing.

A visit to the Computer Centre at building 513 was an obliged visit after the large halls holding the fixed-target experiments ("Here is the heart to our research"). I found out that the impressive computing facilities portrayed in James Bond' films were not invented (actually I wonder if some film director of the time would have considered the CERN Computer Centre as an scenario for a futuristic spy movie; the means were there). I discovered a hall filled with two-meter high IBM "cupboards" holding several hundred megabytes each, a quite respectful collection of powerful CDC boxes, and some impressive printing/plotting equipment.

When we came out of the gallery, he brought me to a modest location (for the anecdote, my own office now) to get hold of some manuals and pose a couple of questions. I read there "Program Enquiry Office". When we came out I asked my supervisor, and he explained:

While you develop your programs at these mainframes, sometimes they will blow up at execution time. You will think you did not change anything which could explain the error and that the machine just failed.

So you would phone or contact this place and they will invite you to bring along a dump of the aborted program. After some time of tracing the logs by hand, they will find out which is the instruction which caused the failure..."

I found that kind of work fascinating. Sherlock Holmes and the Lost Bit.

Designing GUIs for "GUYs" is a serious business

Short after I was proposed to join in the Spanish team from my institute, participating in the Mark-J experiment at DESY. During the next three years I learnt the basics of High Energy Physics life: running programs and making shifts. For a doctoral student that is.

Computing-wise I got acquainted with MVS/NEWLIB and the GEP graphical package (both very fine pieces of work by the DESY/Rechnenzentrum staff); and also the JES3 and JCL horrible technobabble of divine meanings to put away mortals at once. In the collaboration it was customary not only to do software analysis in the corresponding area of research or thesis topic, but also contribute to some task of common usefulness for the Collaboration.

I opted to team up with a Dutch colleague in order to develop a tape staging pooling system based on our IBM maxidisk allocations; nothing to do with today's complex environment, but still with enough challenge to complement night thesis fun with some programming bursts.

This little project taught me than user satisfaction builds more on robust and intuitive user interfaces and meaningful error/fault messages, than on an infallible back-end. Faults are (better) tolerated as long as (scientist) users are given valid (scientific) explanations.

My second lesson is that it is physically impossible to meet conflicting user requirements of more than two users at a time.

A User Community on its own rights

Until May 1986 I was at DESY in an experiment of some 40 members working on location, who were actually the colleagues interacting on a day-to-day basis. We knew each other by first name.

Then I was reassigned to the spanish team working at CERN, this time joining the UA1 Collaboration. I kept busy for a while automating our (institute) software chain under MVS/Wylbur, scripting JCL stuff and so on. Nothing significant really in the context of a collaboration five times as large. Although reluctantly I learnt VM (I always thought it was a loss compared to MVS/Wylbur and specially NEWLIB) and REXX. I explored the possibilities of the UA1/Merlin graphics system, and came to terms with HTV (oh! How badly I was missing GEP...).

I lived like some others my first user migration: CERN was leaving behind MVS and was asking users to embrace CERNVM. Critical me (but I was not the only one; it was a habbit in UA1), I found that DD was not really getting the act right by proposing users three different interfaces to migrate information from MVS to VM: one to move flat ASCII files, one for binary, one for MVS libraries; of course the three written by different authors. I undertook to glue them into one for the good of UA1 mankind, and "MVSGET" was born at our user "U" disk. With a friendly panel and understandable choices, so I thought.

It occurred to me that such interface might be good enough to save other CERN users some migration hassles. I approached Harry Renshall (as I had seen so much from him in the news and CNLs) to offer my goodie for the sacred VM "P" disk. "MVSGET" did not make it, but I somehow impressed the gentleman because few months later I was invited to apply at DD for a user-support position (created by the departure of a colleague).

I did not think twice about it. The board turned down the application of a very young and gentle belgian lady and opted for me. Harry confessed to me later that he had helped the choice by stating that either he would carry over his support duties with an older-but-experienced Marquina or he would resign at once and let the support work be done on its own.

I remember Harry in my night swears ever since.

King Salomon's job

Moving to DD in July 1988 was not as simple as it could have been. Some months before I had had the initiative to script the UA1 simulation chain at the newborn Cray/UNICOS system. A good move for UA1 because there was a major source of CPU power unexploited at our fingertips. The pleads of my colleagues turned my move into a liaison between UA1 and DD, representing UA1 in their never-enough requests for CPU.

I learnt what was the cost of representing user interests "against" one's own colleagues; that it is not straight-forward to match demands to available resources and that it was tough to be sandwitched between both and still come up with certain satisfaction of an accomplished job.

But this was not my only newcomer problem. I suddenly found myself with the power to modify things that only months before I had felt so critical about. I had a password to modify the "P" disk. The read/write password! What a misfortune, because I started "improving" (that was my justification) feverishly everything I could think of. And worse, I was informing users about it.

My closer friends worried for my health. The more humorous ones just nicknamed me the "DD Lighthouse", because my office window was at the side of the entrance to building 513 and was welcoming visitors at almost any time at night.

A Library is not only a collection of books

Federico Carminati definitely understood how to take advantage of the situation. He was the CERN Program Library manager in the User Support group in 1989, and needed some silly youngster to undertake the UNIX builds. He was responsible for the Vax/VMS, Cray and Apollo versions (besides the maintenance of MVS and CDC for external sites); but needed really a hand for the CERNVM and forthcoming AIX, DEC, HP and SunOS4. So he wicked me at some bus trip returning from I don't remember where.

Given my CERNVM fluency he naturally delegated the VM port to me and entrusted me with the development of automated and portable procedures to produce all the UNIX builds. And there I was, turned into a "CN Lighthouse" and facing the impossible job to manage multiple ports from a single set of shell scripts.

In 1990 Rene Brun took over a reshaped User Support group under the CN seal; it was called rather "Application Software" (which changed in fact the entity and scope of the unit for the following seven years). Federico found a golden opportunity to escape into better duties, took over GEANT and left me stuck with CERNlib and my own masochistic pride.

Of course, the more ports we incorporated, the more companies expressed interest in getting into the bandwagon: Alliant, Convex (32 and 64 bits), NEC Supercomputers, NeXT, you name it. Everybody wanted to sell to High Energy Physicists. And more HEP institutes expressed desire to have their local variants of computer brands and compilers to be supported by the capital of the empire.

I became the perfect example of "give him more since he still digests", and my mailbox started becoming a black hole (a lot coming in, nothing coming out). I also particularly enjoyed the library builds the weekend before a release. Delightful hundreds of thousands of lines of Fortran code not passing the validation tests on a Sunday evening...

Four years later luckily somebody thought it was a good time to offer me a change...

From VM to UNIX by the hand

In the fall of 1993 CN division started considerations of how to terminate an otherwise healthy mainframe era, since the user demands for CPU capacity were about to exceed by several orders of magnitude what any possible extension of the IBM system (3081, 3090, 9000) would allow. The Programme of Work (POW) of that year addressed an internal reorganisation, and identified among others the need to start a migration campaign to get users into the unavoidable distributed UNIX environment. Chris Jones was mandated to lead the "Desktop Computing Infrastructure" group and I was invited to join in. My job would be to sell VM users UNIX well enough that they would happily and unconditionally abandon it three years later.

Since the word "users" was in my job description, Harry pleaded the "User Consultancy Office" unit to be transferred under my supervision rather than have it shut down as an undesirable result of the reorganisation.

Well not quite that. The truth is that I had entered in DD with a User Support hat and that I had remained in such a role ever since. Harry my DD mentor trusted me enough to be able to carry over his "vision" on how to assist users, and I did appreciate and enjoy that challenge.

Chris kept me busy for the following three years. The beginning of the migration campaign was not easy. I had been mandated to run something promisingly called the "User Migration Task Force". One of my initial reports at some internal meeting started with the following joyful remark:

"Dear colleagues, I am afraid I have not made much progress in the run of the User Migration Task Force. Unfortunately I have been given the Task, but not the Force..."

Chris jokes with me about that lapidary statement ever since. He actually made something better than that, because he filed solid resourcing requests for UMTF at the following POW of 1994. It came up well and UMTF could fly. The whole of DCI group responded at each area of responsibility and, together with the rest of the division and PDP group in particular we managed to set the user countdown to the end of June 1996.

Although the replacement services showed initial glitches, CN had managed to have several thousand users switching from CERNVM to the AIX-based cluster RSPLUS in quite an orderly fashion. I think the day we officially shut down the system I recorded the first sign of computing nostalgy. In spite of the champaign at the Computer Room.

User Support: part II

Jürgen May became our next Division Leader in 1997, and we became IT. Coming from DESY, he had solid convictions for the existance of a unit looking specifically after user issues as a "raison d'etre", and not as part of its service provision mandates. There were internal considerations of all sorts speaking in favour and against such initiative, and under what kind of umbrella the "User Consultancy Office" should stand.

In the end Jürgen pleaded a chance for the idea among the divisional management and offered me the undisguised honour of becoming its advocate. Since then I have tried to work hard to stand up to expectations, by both users and colleagues.

Manuel Delfino followed Jürgen two years later, revalidating with the next turn of divisional management the place created for such a unit. I can only be indebted on behalf of myself and my team for giving us a regular chance to prove or show our possible added values, and to let us work on our strong convictions of helping users to our best possible.

Wrap-up to 20 years of supporting users

I have tried to give a dynamic style to the story, which is partly mine, partly my living of CERN computing. To those people I have mentioned I ask for their acceptance of my rather colloquial descriptions. To those people I have not I ask for their forgiving, since I have all in my memories. To all I ask for your tolerance in the freedom (and imprecisions) I have given to myself in describing events.