What went wrong with Intute? And – my role before it’s demise

Warning – this post gets personal towards the end.  If you don’t like that sort of thing, you’d better look away now.

A brief bit of background.

In the early days of the Internet it was not very easy to find websites.  There was no Google, and the few search engines which existed were crude and incomplete.  Some libraries responded to this situation by creating lists of good websites on various subjects.  This resulted in a lot of duplicated effort, and one response to this was the funding, in 1995, of several Subject Based Information Gateways under the eLib Programme.  These included SOSIG (Social Science Information Gateway), OMNI (Organising Medical Networked Information) and EEVL (in 1996 this stood for: Edinburgh Engineering Virtual Library.  Later on, when it also took on Maths and Computing, it stood for the Enhanced and Evaluated Virtual Library).

These gateways were quite useful and relatively popular within the academic community, and some other subject gateways started up in other areas.  JISC then funded the Resource Discovery Network (RDN) with a general remit to bring more co-ordinated effort to the situation and to extend the existing gateways to cover other subject areas.  Intute was formally launched in July 2006 after the merger of the eight semi-autonomous “hubs” that had formed the Resource Discovery Network.

A restructuring and rebranding was then undertaken to create a service with a more uniform identity and appearance, better cross-searching facilities, and more focused technical and management teams. This process also incorporated the Virtual Training Suite, a series of continually updated, free online Internet training tutorials for over 65 subject areas.

Over the years, a fairly large number of other projects and sometimes services also developed out of the subject gateways, the RDN and Intute.

My role in all of this was as manager of EEVL, which became one of the Hubs.  At one stage, EEVL encompassed about 10 full-time-equivalent posts plus a number of volunteers, based in various UK universities, including Heriot-Watt, Birmingham, Ulster, Cranfield, Nottingham, Imperial College and elsewhere.  We worked on a number of projects, and not just an Internet resource catalogue.  Some, but by no means all, of these projects are detailed here.  At it’s height, there must have been over 30 fte staff working on RDN/Intute.  I stopped contributing to Intute in September 2005 but remained involved in one of the spin-off projects (TechXtra) as manager, and I then became involved in slightly related projects such as ticTOCs (which developed into JournalTOCs) and Gold Dust.

Intute was closed in July 2011, although the Intute website will remain until 2014.  The Intute Repository Search demonstrator has been rebranded as the Institutional Repository Search.  There’s more about the history of Intute here.

The service called TechXtra, which arose out of something originally called EEVL Xtra, which in turn developed from PerX , SAD1, SPP and a couple of smaller JISC projects, still exists.  TechXtra is a fairly basic, but useful, free service which can help you find articles, books, the best websites, the latest industry news, job announcements, technical reports, technical data, full text eprints, the latest research, thesis & dissertations, teaching and learning resources and more, in engineering, mathematics and computing.

End of the background.  If you want more detail, scroll down my own Background page and you’ll find various papers, magazine articles, links and presentations.  Or read this or this or this or this or this.

I originally intended to write this What went wrong with Intute post ages ago, but put it aside, and I hope that I can still remember enough of the detail for it to be accurate.  If anyone who was involved points out any factual errors in this post, I’ll be happy to edit them.  There was a feature article the other day in Searcher entitled The Decline and Fall of the Library Empire, which mentioned the sort of things that the subject gateways, the RDN and Intute worked on (see the quote below), and this got me thinking once more about writing this post.

“Remember those heady early days when we thought we were going to catalog the web? … Librarians around the world were supposed to select and catalog “good, librarian-certified” web resources.  …  And there were lots of collaborative projects to develop “librarian-built” directories of web resources. … Many of these projects were grant-funded and died off when the money ran out. Some still linger — used mostly by librarians, as they have always been — as the rest of the world rushes right by our (sometimes) carefully tended websites and directories on the way to Google, Bing, and other search engines.” [Steve Coffman]

To get back to the original question of what went wrong with Intute, an easy answer, which would also contain some truth, would be to say that nothing in particular went wrong.  Intute, and also its predecessors, was relevant to those times, helped numerous Internet users find and access quality resources of various kinds, and was pretty popular.  I don’t have the stats to hand, but in terms of hits, usage was relatively impressive when compared to other JISC services and similar initiatives elsewhere in the world.  So it could be said that Intute served a purpose, and then the need for it reduced when general search engines improved, Google and Bing took over, and the Wikipedia developed, and so Intute passed it’s sell-by date and was then closed.

However, that easy answer is not the whole story.  I was amazed that Intute continued for as long as it did.  I reckoned that, the way it failed to develop beyond a limited search tool with an associated set of Internet tutorials, it had outlived it’s purpose by 2008, if not before.  I believe that there were faults in the overall concept, some of which dated back to around the time when the RDN was formed, in 1998.

I was also amazed by the amount of funding that went into the various gateways, the RDN, Intute and associated projects.  For a while, the amount in £s was seven figures per annum.  I certainly didn’t complain about this at the time, because I believed that there were ways that this funding could be put to very good uses.

The other notable thing about the gateways, the RDN and Intute was the number of very capable people who were involved and the great amount of effort they put into the various projects.  It’s almost certainly the case that Intute lasted as long as it did because it was so well managed and had such positive people working for it.  Despite this, I would suggest, Intute essentially missed the boat, so to speak, and this was it’s main failure.  I’ll get back to this later in this post.

Other faults started to appear even in the time of the subject gateways.  A relatively very minor issue, but one which we discussed on many occasions, was – what sort of websites should be included in the gateway directories?  It was generally agreed that only quality and substantive websites should be included, because that was the purpose of the gateways – to direct people to quality information.  However, there were various very useful websites which were not in themselves substantive – so, should these be included?  Usually, they were not, and I felt that this was not helpful for users.  One example of many was the Internet Resources Newsletter which I edited.  Though it had over 30,000 subscribers, was aimed at academics and students and was well-regarded, it was not included in Intute’s catalogue. I never got miffed about this, but it did rankle a bit.  There were many other useful, but in themselves unsubstantial, resources which were not included, and because of this, for me, the usefulness of the resulting Intute service was reduced.  On various occasions I’d be talking to students about Intute or the gateways, and I’d know about a website which was relevant to the students, but I’d also know that it wasn’t included in Intute.

A bit more important were some issues that resulted from the expansion of the gateways into the RDN.  At EEVL, we’d previously been focussed entirely on the subject of Engineering.  When we joined the RDN we took on Maths, and eventually Computing as well.  The information structure of those three subjects is very different, and in turn, the information structure of engineering, for example, is especially different from the information structure of subjects in the social sciences.  What works well for one, doesn’t always work well for another.

For example, a typical engineering student, lecturer or practitioner tends on the whole to look for information on pretty specific subjects.  One example of thousands might be: photovoltaic window design.  That’s the sort of thing that a third or fourth year engineering student might be interested in.  As can be seen here, it is too specific a subject for finding anything in Intute, however.

In 2008 I remember giving a presentation to 30 Construction Engineering MSc students about finding information for their dissertations.  Well, in theory Intute was a relevant service for them, and was certainly aimed at that type of student, but when I checked Intute for websites to help them, I found that in each case their chosen dissertation topics were too specific for Intute and in every case Intute returned “Your search for [xxxx yyyy] produced no records”.  So, with some sadness, I didn’t mention Intute in the presentation.

For subjects in the social sciences, though, Intute and the gateways worked much better.  On the whole, students and others when searching for resources in the social sciences tend to look for more general subjects.  A student might be looking for material on, for example, economics in the Middle East, or political history of Australia, and Intute was quite good at directing them to good quality websites and starting points in such subjects.

A bit earlier, I mentioned Maths, and how EEVL expanded to include that subject when we joined the RDN.  When it comes to information resources, maths is a peculiar subject.  What we found was that the EEVL Internet resource catalogue was fairly good at directing non-mathematicians towards quality maths resources, but almost completely useless at directing mathematicians towards relevant resources.

To explain that further, a student in economics might have been stuck when it came to understanding, for example, algorithms.  Search for ‘algorithms’ in Intute and you find some good basic websites that should help the student.  For mathematicians, however, most of those results would be far too basic.

I remember talking to the person in charge of adding maths resources to EEVL whose husband happened to be a maths lecturer.  I asked her if her husband found the maths part of EEVL of use.  The answer was “Not in the slightest.  He is researching topics which are far too specific for finding resources by searching EEVL, and in any case he knows most of the other researchers who are involved in his topics, and simply follows their journal and conference papers.”

So, we were aware that there were problems with the concept of the gateways.  There were various things that could have been done to solve these, and other, issues, which might have helped develop Intute into a longer-lasting useful service, and I believe that in not doing these things, and in concentrating on only an Internet resource catalogue and a set of Internet tutorials, this is essentially where Intute went wrong.

It’s also where things start to get a bit personal.

I mentioned, above, some of the projects that various of us within the gateways, the RDN and Intute were involved in.  Some of these had great potential but one or two seemed to be a waste of time.  One of the most pointless exercises, IMHO, was something called Secondary Homepages in Mathematics.  You can read about that project here.  There were some other projects which I felt distracted us from developing in a relevant way, and these were usually related to enhancing the Internet resource catalogues which were unfortunately regarded as being at the core of the gateways, the RDN and Intute.

At the time, I felt that a human produced Internet resource catalogue, which involved paying people to catalogue Internet resources into a slowly-expanding searchable and browsable directory, was not only not what we should have been mainly concentrating on, but was also something doomed to become redundant in the near future.  Not only was cataloguing the best parts of the Internet an almost impossible task, but the resulting product failed to answer typical searches in some subjects, and also, it was an extremely expensive exercise to employ people to catalogue resources.

There are two ways of looking at What went wrong with Intute, then.  The first way is to say that nothing went wrong, and that it was bound to fold sooner or later; and the second one is to say that, had it developed in a different way, it might have become a vital service for students and academics, and therefore what went wrong was that it didn’t develop in the correct way.

Instead of concentrating on cataloguing the best parts of the Internet, an exercise I reckoned was becoming more-or-less pointless by 2005, Intute could have looked more closely at some of the spin-off projects, taken a giant leap of faith in a new direction, turned it’s services completely on their head, and developed into a vital service that might well have still been around today.

I wanted to suggest this course to the Intute management and had many new ideas to propose, some of which were admittedly partially-baked, but was prevented from doing so, under threat of disciplinary action if I did so, by my boss at that time.  Not only that, but I remember him saying that our institution’s contribution to Intute should be only to catalogue Internet resources.  The best I can say about his attitude on the matter is that either he had little vision, or that he thought Intute was doomed to close in the near future anyway, so why not just keep a couple of people employed on it until that time when it inevitably folded.

Of the various spin-off projects that had potential, SAD1 and the Subject Portals Project showed a lot of promise, and several excellent people were involved.  These projects attempted to produce subject portals.  There was a lot of discussion about what was, or wasn’t, or should have been, or should not have been, a subject portal.

In my mind, we were working towards developing a service which would initially have been similar to what was produced not long afterwards in Ireland – the Irish Research eLibrary (IReL) – i.e. common and deep access to electronic resources from multiple sources including numerous bibliographic databases and other catalogues.  This could then have developed further into, essentially, a ‘Summon for all’ type service for UK higher and further education institutions.  i.e. instead of individual institutions spending £20k or £30k each per year (or whatever the figure is) on individual commercial web-scale discovery services, this could have been achieved nationally, with subject-based flavours.

I was hoping that the searchable part of Intute could have been developed to be a single unified search of books, e-resources such as articles, theses and dissertations, the latest research, items in repositories (taken from what eventually became the Institutional Repository Search) etc, personalised to produce results/resources which were actually available to users depending on their institution, with additional subject slant options also depending on the user.

An ambitious task, yes.  But remember that Intute and the various projects got through a lot of funding, and we had some fantastic techies within the organisation.  The important thing was to get away from wasting money on cataloguing the Internet.  This could have been achieved by drastically reducing the amount of effort going into Intute’s Internet resource catalogue, and replacing it with a much, much smaller, and more static guide to essential top websites, which would in turn have been a slightly expanded directory of those resources included in the Virtual Training Suite.  The VTS could also have been developed further.

The big idea, however, was not so much the above, but rather how the whole thing, existing services plus various other new elements, could be presented to end-users.  The big idea was to automatically produce personalised websites for users of Intute (which could have been known as the POG (Personalised Online Gateways)).  These would have personalised URLs, preloaded subject content and emphasis depending on the user’s subject background, a facility for each user to write a blog within his/her MOG (My Online Gateway), a facility to enable discussions with other users in similar subject areas, a facility to bookmark items/websites/papers within the MOG, and more.  Some people are still suggesting similar ideas.

Another element of the big idea was how it could be marketed to students and staff, which involved using core students at institutions to spread the word to their friends via ‘personalised’ flyers.  The details don’t matter now, but there were some good marketing ideas.

Many elements of the POG/MOG idea were half-baked.  Some elements have since appeared in a much more coherent form as parts of Delicious, Facebook, Mendeley and other services.  These either didn’t exist, or were largely unknown to us at the time (remember that I’m talking about spring 2005).

A further part of the big idea was to develop the broad search engine part of EEVL along similar lines to what has become Scirus, and have this included as one part of the discovery search.

Another piece of the jig-saw would have seen the Internet Resources Newsletter merged into the POG service as a current awareness element.

The end product would have allowed users to easily find, access and be alerted to, and discuss or share, and bookmark, relevant and available-to-them articles, books, websites, and other scholarly resources, within a personalised and unique website.

In May 2005, I think it was, I put a lot of the above POG/MOG and marketing ideas into a document called ‘A big idea for The Big Idea’ (Intute had employed a PR agency called TBI at that time), and I was looking forward to presenting it at a meeting of Intute staff to which I’d been invited as manager of the EEVL Hub.  I was hoping that some of the excellent minds within Intute would take my ideas, futher bake them, and come up with a practical plan to implement at least some of them.

In the event, my boss prevented me from attending that meeting.  In fact he sent me an email saying that not only would he not sanction my travel expenses to attend the meeting, but that he specifically would not allow me to attend.  When I asked him why, he would only say that he reckoned it was more appropriate for himself and another person (a colleague of mine) to attend.

Well, when that happened, I started an official grievance against him via the university’s Human Resources section, because I reckoned he was stopping me from doing what I was being paid to do (I was at the time part-funded via Intute).

When he found out about the official grievance, he wrote me an email forbidding me from contacting directly anyone in Intute by any means, on pain of disciplinary action.  Somewhere, I think I may still have a copy of that email!

So, things turned nasty and personal, and I was unable to present my big idea to the others at Intute.

Now then – what I’m saying is not that what went wrong with Intute was that I was not allowed to present my ideas to the rest of Intute!  What I’m trying to say is that where Intute went wrong was it’s choice to concentrate on cataloguing the best parts of the Web, rather than on other possibilities including, perhaps, some of those things I’ve talked about above, and I’ve tried to explain this within a personal context.

So – what happened to the official grievance against my boss?  Well, Human Resources were really good.  They listened.  They sympathised.  They couldn’t believe some of the nonsense that had been going on.  They didn’t take sides – that’s not what they are there for.  They explained at length about the grievance process.

I soon realised that not only would the process likely consume most of my energy for months and months, but that at the end of the day and however justified my grievance may or may not have been judged, the end result at best would likely have been a fudge.

Instead of proceeding further with the grievance, therefore, it was eventually agreed that my boss would have no further input into four things I was working on (PerX which developed into TechXtra, and ideas which became the ticTOCs and Gold Dust projects), and that after ten years of involvement I would drop out of Intute which he would still be involved with.  In other words – a compromise.

Here’s a suitably final word from Intute written when the service closed.