Interlibrary Loan and Z39.50 Item Order
Digest of Discussion on ZIG list, 15-27 July 1999
Leif Andresen, Danish National Library Authority E-mail: firstname.lastname@example.org
Rob Bull, Crossnet Systems Limited E-mail: email@example.com
Rocco Carbone, occasionally @ Finsiel S.p.A. E-mail: firstname.lastname@example.org
Ray Denenberg, Library of Congress E-mail: email@example.com
Kevin Gladwell, British Library E-mail: firstname.lastname@example.org
Jacob Hallén, NetGuide, TerraTel AB E-mail: email@example.com
Sebastian Hammer, Index Data ApS E-mail: firstname.lastname@example.org
Mary E. Jackson, Association of Research Libraries E-mail: email@example.com
Randy Menakes, Ameritech Library Services E-mail: firstname.lastname@example.org
Ruth Moulton, Consultant E-mail: email@example.com
Barbara Shuh, ILL ASMA, National Library of Canada E-mail: firstname.lastname@example.org
Lennie Stovel, RLG E-mail: email@example.com
Mike Taylor, TECC Ltd. E-mail: firstname.lastname@example.org
Mike Wheatley, MLW Ltd E-mail: Mike.Wheatley @virgin.net
Mark Wilson, The Library Corporation E-mail: email@example.com
Johan Cornels Zeeman, CGI E-mail: Joe.Zeeman@cgi.ca
On 15 July 1999, Randy Menakes wrote:
I was really surprised to learn how misinformed Sebastian Hammer is of the ISO ILL standard (10160/10161) and the work of the ILL Protocol Implementors Group (IPIG) when he wrote:
"....It's also why we tend to turn away from stand-alone ILL which is what the IPIG is still pushing (carried by e-mail, even). The binding to location and availability information has to be rock-solid, and the feedback has to be immediate and positive. Again, interactivity is the key."
I'm not sure who else he's speaking for when he says "we tend to turn away from stand-alone ILL...", but it made me wonder if others suffer from similar misunderstanding.
Sebastian Hammer asked to insert a clarification: Sorry, I should have been more explicit about the intentions - I meant to be mildly provocative, not inflammatory. The intent is not to shun ISO ILL as a whole (hence the reference to "stand-alone" ILL), but to use Z39.50 ES:Item Order as the recommended carrying mechanism, at least for the initial exchanges. The approach is based on the specification in
Given this clarification, I still don't speak "for" anyone, but I believe that my statement reflects the public profiling effort taking place in this country at the moment.
Sebastian's use of the term 'e-mail' implies human beings sending e-mail messages back and forth. What the IPIG Profile mandates is BER-encoded ILL APDU's (in ASN.1) using SMTP as the supporting communication service. Correctly implemented, this can be very close to the level of interaction available in direct-connect, session-oriented behavior. The IPIG Profile also strongly encourages, but does not require, use of direct transport over TCP. Most implementations of the ILL protocol will support both.
I would also point out that the work of the IPIG and of the ZIG are hardly at odds or incompatible. I believe all of us doing development using the ISO ILL Protocol are also utilizing Z39.50 for the kind of precision search and retrieval that Sebastian discusses. I don't see any reason for supporters of either standard to "turn away" from the other. Sebastian, what alternative would you recommend for an ILL standard? Item Order? If so, don't forget that ISO 10161 is the standard cited in Z39.50 Extended Services for identifying the requested item? (And trust me, Sebastian, Item Order doesn't begin to handle the practice of Interlibrary Loan in the real world.)
Earlier today I talked to Mary Jackson, Senior Program Officer for Access Services at the Association of Research Libraries and the chair of the ILL Protocol Implementors Group. Mary asked if the back-to-back IPIG and ZIG meetings in Stockholm next month might not be a perfect opportunity for the two groups to share their work and ideas. Both meetings are preceded by a tutorial that might be the best forums for interchange.
Rob Bull joins the discussion: No - what Sebastian is referring to is the use of Z39.50 Item Order as a carrier for the ILL PDU, and there is a profile on the Z39.50 Web pages which specifies just this:
- Z39.50/ILL Profile 1 Profile for the Use of Z39.50 Item Order Extended Service to Transport ILL Protocol APDUs
Secondly, in the real world, there is for example the German National Library system which is successfully using ILL (ie, BER encoded ISO 10160/10161) transported on Z39.50 Item Order and has been doing so for quite some time in an operational environment.
The use of Z39.50 Item Order has been quite a topic of discussion in the current ONE-2 (OPAC network in Europe) project, in which there are partners in both camps - those who want to use ILL PDUs carried on Z39.50 item order, and those who want to use ILL PDUs on SMTP. There are various logistic and technical arguments being discussed for both.
It is not the use of ISO 10160/10161 parameters as prescribed in the IPIG profile that are being discussed/objected to, but the carrier mechanism.
It is quite likely however that both could be implemented and tested out in ONE-2 - this will then provide positive feedback to the respective communities, which I am sure will be of significant interest.
Ruth Moulton responds to Rob's remarks: So are these implementations using Z39.50 to transport all the APDUs in ISO ILL (answer, shipped etc), as well as the ILL-Request message ?
Ray Denenberg: Yes
Ruth: - that seems heavy - iso-ill/z39.50/tcp/ip why not use TCP/IP to transport the APDUs directly ?
Ray Denenberg: There were compelling arguments for this profile, at the time it was developed, several years ago. One argument was that each ILL APDU would result in a task package in the ES database, which would greatly facilitate statistic reporting.
Ruth: Or perhaps they are not implementations of the full ISO ILL protocol ?
Rob Bull answers Ruth: The original intention in ONE-2 was to consider all ILL APDUs - but in order to keep within the ONE-2 budget, the project is probably going to concentrate on handling non-returnables and ILL APDUs relevant to this. (There are other focal issues in ONE-2 being researched/developed such as the OPAC-holdings schema, document delivery and Z39.50 update).
Back to Ruth: Also, they can only use ISO ILL to interwork with systems that have implemented Z39.50 as a transport mechanism - where as if you use TCP/IP then you can access pretty much all of the implementations of ISO ILL (although IPIG mandates use of e-mail and only recommends use of TCP/IP, most implementations do both)
Ray Deneberg: Just to be clear, there are two relevant profiles,
1. Z39.50/ILL Profile 1: Profile for the Use of Z39.50 Item Order Extended service to Transport ILL Protocol APDUs
2. Z39.50/ILL Profile 2: Profile for the Use of Parameters from the ILL-Request APDU in Z39.50 Item Order
The assumption at the time was that any system that would implement ILL would also have implemented Z39.50 ... not a priori as a transport mechanism, but that adapting it as such wouldn't be too burdensome.
That's my recollection. Fay (and Joe) can speak more eloquently on this, though I'm not sure where Fay is.
Sebastian Hammer: Actually, a better answer would be "possibly". The profile you mention explicitly mentions, I think, that you may wish to run part of the "session" over a different medium. The ILL tracking phase might be a good place to switch to a different medium (even "email").
Remember that the ISO ILL state machine is a different sort of beast from the one in Z39.50. It is not connection oriented, and an instance of the machine (I forget what name the ILL standard uses) may have a lifespan of months or even years.
Mike Wheatley comments: The ILL Protocol is defined as OSI Application Layer protocol so it is not really relevant to say 'it is not connection oriented' any more than to say it is store-and-forward oriented. The 'instance' you refer to is the ILL Transaction and indeed it could last for years waiting for publication of the requested item!
Ray Denenberg inserts: You're quite right; these are two terms that are so overloaded now they are virtually useless. I think what Sebastian is pointing out is that the ILL protocol attempts (implicitly) to recover from lost, repeated, and out-of-order messages, and the implication is that it therefore expects lost, repeated, and out-of-order messages.
Mike Wheatley continues: And that's very wise since it sounds like folks planning to use Z39.50 as transport may also use something else for say the Tracking phase of the same ILL Transaction. In fact the Z39.50/ILL Profile 1 page 3 paragraph 1 says "there is no requirement in either the ILL protocol or this profile that all APDUsrelating to a single transport service." So there is a distinct possibility that out-of-order messages can occur - and the ILL protocol is designed to handle that.
I see that as a strength - not as an unnecessary complexity.
Ray Denenberg replies: A strength, and most definitely a significant complexity. I think you're a bit over sensitive here, Mike -- Noboby has claimed it's an *unnecessary* complexity.
Back to Sebastian: There is no assumption that the Z39.50 and ILL state machines are in any way related - the Z39.50 server simply acts as a gateway - with the added benefit that you can refer to a result set item to specifiy the item, rather than relying on transfering the bibliographic information.
Mike Wheatley responds to Sebastian: That benefit is something I tried to get added into the ILL Protocol when the standard was being developed but I was defeated by the arguments of others. It was argued that as the origin system would have received the bibliographic details of the result set item anyway - so it could simply pass the bibliographic details to the ILL Application system. There was a strong desire to keep the protocols very clearly separated. Maybe this requirement could be met by specifying an external for 'supplemental-item-description'?
Rob Bull : On the international scale this may well be the case, but in the scope of (say) the Danish scene, the DanZig profile is considering ILL/Z39.50/TCP at the national level.
If my memory serves me correctly, in the German system, ILL is carried as ILL/Z39.50 on item order and the ILL data is stored in a database.
This database can also be searched (using EXT.1 attribute set) and ILL PDUs can be retrieved on search response (piggyback) or present response as externals on task package records.
Ruth continues: It seems cleaner to me to have a Z39.50 engine for z39.50 work and an ILL engine for the ILL protocol in a single system, rather than layer the one over the other, using perhaps Z39.50 item order to link between the two.
Rob Bull: One of the points raised in ONE-2, is that having established a search and retrieval association from client to server, there is an existing association/connection in place to handle ILL messages.
Ruth responds to Rob: Thanks for your informative reply - looks like there are two ISO(ish) ILL worlds out here, which won't talk to each other - sigh.
Rob continues: I trawled through the IPIG meeting minutes, and noted that there never seemed to be any discussion (as far as I cound see) concerning using ILL on Z39.50 Item order, despite the profile on the LOC web page being there for some time now. The exception is when Poul Henrik attended an IPIG meeting last year and gave information about possible intentions of ONE-2.
Mike Wheatley reponds: The path taken by IPIG followed in the wake of the work done by IFOBS and EWOS/EG-LIB which produced ISO ISPs for use of ILL Protocol over ACSE (connection oriented) and MHS (store-and-forward). The work on these ISPs preceded that by ZIG on use of Item-Order.
With the demise(?) of OSI it seemed logical for the IPIG profile to utilise basic internet transport mechanisms that were in widespread use rather than layering the ILL Protocol on yet another complex protocol such as Z39.50. This seems a wise decision in the light of some views on the future of Z39.50.
Rob Bull reacts to Mike's response: This seems to contradict the effort/discussion on the ZIG a couple of years ago regarding the use of ILL/Z39.50 and to the production of the two profiles available since then. - as to the future of Z39.50 it seems to be growing in ever wider domains and applications.
Maybe at the time this was being discussed, folk's Z39.50 implementations were at Z39.50 version 2 - and they were not supporting extended services; and hence looked for an alternative rather than looking ahead to get Z39.50-1995 implemented.
Ray offers a comment: (you mean "on Z39.50 Extended Services"?) I think this is truly unfortunate. The ZIG did a significant amount of work, and worked hand-in-hand with the ILL heavyweights to come up with the model as reflected in these two profiles, both for Item Order and for using Z39.50 as a transport for ILL APDUs. It would be a real shame to simply discard this work, for what seems to me to be a lack of understanding.
Mike Wheatley replies: My feeling at the time of work on Item-Order was that come what may the ZIG wanted ES:Item-Order whatever the ILL requirement was. And to avoid the possibility that someone in ZIG would invent another load of new ASN.1 for use with Item-Order a rear-guard defence was implemented to define a profile for Item-Order to use the ILL APDUs as they were already defined. That was a successful tactic. (others may disagree on this view!)
Ray Denenberg disagrees with Mike: Yes, I certainly disagree. The "ZIG wanted ES:Item-Order whatever the ILL requirement was"??? "Ordering an Item" and "interlibrary loan" are significantly different functions. Nobody ever contemplated modeling interlibrary loan in terms of Z39.50.
The ZIG fully understood and appreciated the difference between ordering an item and interlibrary loan and thought Extended Services was appropriate for ordering an item and the Interlibrary Loan protocol was appropriate for interlibrary loan. (It's true that the ILL protocol does model that de-generate and paradoxical case of ILL, known as the "non-returnable loan"; the ZIG never understood arguments to the effect that the ILL protocol was better for that.) Furthermore the ZIG fully appreciated the effort that had gone into the development of the ILL request, in terms of specifying the item, and utilized this in the development of the Item Order request.
"Ordering an item" is not the issue. How to layer the ILL protocol is. The convolution is that the suggestion to layer ILL over Z39.50 just happens to employ Item Order (but not for ordering an item), so it leads some to conclude that the suggestion is to use Item Order instead of the ILL protocol.
Mike continues: At the time things were left very open. But I don't think anyone at the time seriously thought of using Z39.50 ES as transport for the full ILL Protocol.
Ray Denenberg again disagrees: That is exactly what people were seriously thinking. I'm not taking a position here, just trying to provide the benefit of my recollection.
Mary Jackson comments on Rob trawling the IPIG minutes: Back in 1993 or 94 I attended the ZIG meeting in Gainesville to talk about the early efforts of the IPIG and how it might relate to Item Order. I left that meeting with the impression that Item Order was designed for non-returnables (or read: orders to commercial document suppliers). You are correct that this topic has not specifically been on the IPIG agenda, in part because so many IPIG members were struggling with their basic implementations. I have added this topic to the August IPIG meeting to stimulate a discussion of how the two standards should/could be working more closely together.
Rob's message continues: The German system has been running for over 18 months as far as I know, and again I am not aware of any correspondence with IPIG. I do remember correspondence with Fay Turner when the software was being developed because when we ran the ILL ASN.1 spec at that time through an ASN.1 compiler we posted several syntax errors to Fay.
Mike Wheatley: There have been numerous attempts to get more involvement from European implementors of the ILL Protocol with the work of the IPIG.
Rob Bull: I can honestly say that no one has ever contacted Crossnet regarding our ILL software apart from the time we liaised with Fay Turner regarding the ASN.1 of ILL, despite the fact that our UNIX toolkit which still supports Z39.50/ILL request/status being freeware available for over 2 years now.
Ruth Moulton: Consider yourself contacted - you would be really welcome as members of IPIG and EFILA, see their web pages for how to join, (ipig is www.arl.org/access/naildd/ipig/ipig.shtml) or email them on firstname.lastname@example.org and email@example.com
Many implementors are doing interoperability testing within IPIG and really want to include as wide a group of people as possible.
EFILA would like to get a european perspective on ILL and promote interworking where possible, and is intending to document implementation experience and profile information, if you can't get to the forum on 3rd August in stokholm then look out for their next meeting, I think it's going to be in Germany.
Mike Wheatley: You must have missed all the announcements and invitations to join IPIG (and attend EFILA) meetings that went out on many lists a while ago!
Please do contact Barb Shuh so that she can add links to information on the Crossnet products from the ILL ASMA site.
And the same goes for anyone else with ILL Protocol products and implementations.
I've just looked on the Crossnet website - and then re-read your email again - and now realise that I may have misunderstood what you said above.
Am I right that your toolkit provides support simply for transporting just 2 of the 20 ILL Protocol APDUs via Z39.50 ES:Item-Order? And it does not provide an ILL Protocol Machine? Nor an ILL system application?
Rob Bull answers Mike: The toolkit contains provisional support for all ILL PDU ASN.1 - it has all been compiled through an ASN.1 compiler and all the encoding/decoding code is provided - the 2 PDUs you refer to have been used in the German Library project because this was the main reason for developing the software, and these PDUs have been brought in line with same style as the rest of the software API specification - provision of trace functions trace functions, checking of PDU optional/compulsory parameters etc. But I stress that this is a primarily a Z39.50 toolkit - not an ILL toolkit.
Notwithstanding this, it doesn't prevent anyone from extracting the ILL portion and the TCP/IP connectivity part if they want to - we will be doing this ourselves in ONE-2. In addition, we will be deploying other ILL PDUs in ONE-2 and giving these a thorough test.
Mike's comments continues: I would also ask what is the extent of the support of the ILL Protocol by the German system that you referred to? I suspect it implements only part of the ILL Protocol.
IPIG members have been concerned with the implementation of the full ILL Protocol for use in operational services with both new and legacy ILL application systems. This implies a broader requirement than that of the specific implementations you mention.
Rob Bull responds to Mike's comment: There should be budget scope to create a suitable ILL toolkit in ONE-2, providing there is a requirement to do so; other aspects such as state machine can also be extracted/reused from our Z39.50 toolkit as these components are quite versatile.
The other factor is that we know all this code works under the Windows platform as well, which we licence in ZedKit for Windows.
Back to Mike's comments: But despite this and holding meetings in Europe (London last September and Stockholm next month) there has been little input from some of those in Europe who have done some implementation and whose experiences would have been very welcome.
Hopefully the meetings at Stockholm will provide the opportunity for those previously absent to meet to discuss these ILL Profile issues across the table (or bar)!
Rob Bull will not be at the IPIG meeting in Stockholm, but will be at the ZIG tutorial. If anyone wants to ask me then about our experiences in implementing ILL I am quite open about this.
Mike Wheatley: Unfortunately I will not be at either meeting - but hope that others can take up your offer.
Rob continues on Scope of ONE-2: So far as the ONE-2 project is concerned, it is scoping its ILL work in handling non-returnables, and we plan to use the relevant sections of the IPIG profile for this. The issue is the transport mechanism, and as said before, both with/without Z39.50 will probably be implemented.
Mike Wheatley: As long as smtp is supported it will ensure interoperability at the transport level with IPIG Profile implementations.
Rob: The technical arguments for using Z39.50 as a transport for ILL have been seen to work well elsewhere and this prevails in the minds of a number of ONE-2 partners, whereas other ONE-2 partners have decided to use ILL via SMTP.
Mike Wheatley: I was not aware that Z39.50 had been used as transport for the full ILL Protocol in any significant volumes other than in closed groups which make use of just one suppliers implemetation of the ILL Protocol. A rather limited form of interoperability. But feedback of any experience is always welcome.
Rob: And are we so alone in this ? the UNIverse project specification document for interLibrary loan also lists Z39.50/Item order as a requirement for a request being placed - something is going to have to act as a gateway here as well I guess.
Ruth Moulton joins the discussion: The universe server acts as the gateway, it takes the Item Order and, if the responder is an ISO ILL one, manages an ISO ILL transaction over e-mail or tcp/ip
Mike Wheatley adds: Another aspect is that the Z39.50/ILL Profiles only define certain very limited aspects of the implementation. There are many more aspects which require detailed and specific profiling to ensure interoperability. Was this defined for the implementations using Z39.50 as transport? Do they successfully interoperate?
Mary Jackson provides some historical background on IPIG: Several years ago I contacted the Germans to see if they would be interested in joining the IPIG and participating in the discussions. Their response was that they would watch the IPIG process but not participate in it. As part of the ARL Global Resources Program, I am involved in the German Resources Project and have has a series of conversations with Goettingen library staff about their use of Item Order, and possible use of the Protocol. Again, interest has been expressed.
As an ILL librarian, I want to be sure that systems can handle book loans (returnables) and copies of journal articles (non-returnables). I am also aware that many patrons find citations from many different sources - a search of a Z39.50 catalog is not the only way of identifying bibliographic information. ILL systems need to handle Z39.50-generated cites as well as cites written on the back of the proverbial paper napkin.
The IPIG encourages this continued dialogue between ZIG and IPIG members.
Back on thread of TCP vs e-mail, Ruth: If you mean there is a tcp connection open, it's not terribly expensive to open a tcp connection on another port...
Rob: The feeling here in ONE-2 is that using Z39.50 you get an ES:response which gives some confirmation/acknowledgement of the ES:request going out; whilst not imposing any state machine on the ILL aspect.
Ray: Yes. (I mentioned the task packages supporting statistic gathering in an earlier message, but this feature you mention was certainly as important, or maybe more important.) I've been involved with the ILL protocol since 1983 (possibly longer than anyone else still around) and this has been, historically, the most controversial aspect, the inability of a system to determine if a message has been successfully recieved by the intended recipient, and this model provided an elegant solution.
Mike Wheatley: Firstly I thought that telecommunications services these days were supposed to be so much more reliable than they were a decade or two ago. One can normally send messages and they get there with impressive reliability. I don't often have to check that my emails get there - they almost invariably do.
Rob Bull: In real life, emails do have a nasty habit of getting lost, bounced, denied because mail boxes are full etc:- even if they are a low percentage.
Ruth Moulton: The Notary internet e-mail services specify how to implement delivery reports and message disposition notifications. These allow one to see that the message was delivered, or not, at both smtp and user (rfc822) level. (Equivalent to DRs and IPNs in X.400). So this makes the missing message argument specious since we would not be dealing with everyday mailboxes but mail services implemented as part of iso ill applications (either using existing ones or doing their own thing), where notary services can be part of the requirements...
Ray Denenberg: The issue isn't so much *whether* a message is succesfully received, but *when* and in *what order*.
Ruth Moulton: Of course MDNs and DRs will tell you this, if smtp/rfc822 e-mail is the transport protocol.
Ray: Say you send an ILL request, then want to cancel it. This is a complex problem for the ILL protocol, and to make a long story short, you can't effectively send the Cancel request until you're sure that the responder has received the ILL request (and you might not know that until you get a book in the mail -- too late to cancel). The ILL state tables are complex, and necessarily so, and the use of Z39.50 doesn't attempt to alter them. It does add some reliability though. You can send an ILL request and follow it up with a search on the task package, to see if it arrived.
Ruth Moulton: With Cancel, it's not just a question of if/when the request arrived, but what stage is the responder at supplying the item. If it has to come off a shelf there is probably a couple of days window in which cancel can happen. This is why Cancel service has a confirmation message.
My view of cancel has always been that if the requester wishes to cancel a cancel message should be sent off immediately, before waiting for knowledge about the fate of the request message. It is always acknowledged that Cancel may or may not be actioned, hence the cancel-reply message...
Mike: If it's important that the message got through then one has to add a check of some kind when there appears to be a reason to suspect it didn't. (After an interval with no response one sends another message and asks Did you get my email of 10th inst?).
Rob Bull: This to me is a flawed situation; I would always place a high degree of importance on these types of service, and when many Internet commercial services nowadays give you a formal, near- immediate response to a transaction; the use of Z39.50 extended services is analogous to this - the extended services response gives the user a positive feedback of transaction acknowledgement, not based on some arbitrary time factor where if there is no response the user has to start pressing a retry button.
Mike Wheatley: If your business requirement is to have such positive timely feedback that an APDU has been receved by an ILL system partner then you should implement tcp/ip as the transport mechanism for the ILL Protocol. That way you will know that the ILL system partner has received the APDU.
Use of Z39.50 ES Item Order as transport for ILL Protocol does not give that assurance.
Indeed the APDU from the ES Package may have been passed into a batch of APDUs for processing at some later time. Z39.50/ILL Profile 1 operational model resembles the store-and-forward environment (says section 7.6). How quickly the 'forward' is done is not defined in the Profile at all - so no guarantees there on the receipt by the ILL suystem partner.
Back to Mike's earlier thread: Secondly the ILL operational process itself currently proceeds successfully without confirmation of receipt of request. For instance the BL DSC processes millions of requests a year without sending an acknowledgement for each request received. The exception is when something goes wrong - then the requester chases the original request after a suitable time-delay.
Rob Bull: The Z3950 extended services model enhances this: an extended services database can enable users to readily track their transaction(s) held in the ES DB - again the benefit here is to users because the Z39.50 model allows the ES DB to be potentially searched by the user to see how things are progressing.
Mike Wheatley: Searching the ES DB to see how things are progressing is unnecessary as this information is provided via the ILL Protocol by sending a Status-Query and getting a Status-or-Error-Report. This ILL protocol status report is likely to be more meaningful (and more up-to-date) than the information on the ES DB which will merely reflect the last APDUs exchanged (using Z39.50 as transport) which does not necessarily reflect the latest state change of the partner system.
I believe that a read of the Z39.50/ILL Profile 1 document's Section 3 'Model' (particularly the paragraphs on page 3) show that searching the ES DB to establish the ILL Transaction Status is not a wise thing to do!
Ray Denenberg: "paragraphs on page 3" doesn't much narrow it down (since that's most of the model section), and I can't readily see that this is discouraged. Could you elaborate, please?
Mike Wheatley : Sorry for the vagueness... page 3 para 1 starting "It is expected that..." shows that mixed transport mechanisms are allowed in this profile.
Page 3 para 2 starting "It is not the intention of this..." shows that there are no mandatory requirements to maintain the info in the ES DB. It does say "well-behaved system will ensure..." ie provides guidelines - but these may or may not be followed by individual implementors.
Back to Mike's earlier thread: Better to simply send an ILL Status-Query and let the ILL Protocol system tell you the latest news
Ray Denenberg: The Status-Query doesn't necessarily give you the "latest news!" I wouldn't claim that an ES search is better, but neither can I see how the ILL Status-Query is better. They both suffer the same limitation, that when the client gets the information it may already be obsolete.
Michael Wheatley: The Status Query provides the most-recent information as held on the ILL system at the moment it is sent.
The ES DB can only provide the information last received and stored - which may be some time ago.
Furthermore if other transport mechanisms are also used then the ES DB may well not have the most recently exchanged APDUs on it. So for instance (from page 3 para 1) the ES DB may well not have any info on the tracking phase of an ILL transaction.
The certain way is to get the ILL Transaction info with a Status Query.
PS I am not arguing that Item Order should never be used. It certainly has its place in providing facilities to meet specific business requirements! But the full ILL needs.
Lennie Stovel: The ILL transactions do not have to be in an ES database for them to be searched by the user to see how things are progressing. This is just a straight Z39.50 search. It just needs an attribute set and a record syntax. (I'll concede that these already exist for ES databases, although it's not clear to me how to build the interface to let the user use them.)
Mike's third point: Thirdly a couple of points about the ILL Protocol:
3.1. the ILL protocol is robust in that it is resilient to lost and out-of-sequence APDUs. It has facilities for repeating APDUs when one party suspects that previously sent has gone astray.
Ray Denenberg: Yes, and the proposed use of ES can avoid a lot of this complexity.
Mike' point 3.2: One ILL partner system can query the status of the other system.
So having sent a request a system could send a status query after a suitable delay(?) to ensure that the ILL Application itself had processed it.
Hence I see the ILL Protocol as not having the necessity to determine that a message has been successfully received rather than not having the ability to determine this.
Ray Denenberg: And the ILL protocol over Z39.50/ES might be seen "as not having the necessity to determine that a message has been successfully received" as well as "having the ability to determine this".
Mike: Using Z39.50 ES alone would not provide the level of certainty of an ILL Staus Query since the ES:response could have been sent by any application that picked up the package not necessarily the ILL application. Or indeed the package may be lost before being processed to the ILL Application.
Ray Denenberg: Right. And nobody is suggesting using Z39.50 ES alone.
Mike: So I would argue that the use of Z39.50 ES is not a satisfactory way of achieving the requirement you stated.
As for statistic gathering I cannot see the relevance of use of Item-Order to this. The statistics needed concerns counts of successful (and unsuccessful) requests, time to fill requests, etc. These are derived by the ILL applications not by looking at a database of ILL APDUs. Or am I missing your point here?
Rob: You can also do an Extended services request right after Init if you want to, without having to do search/retrieval beforehand, and the Init can be used for authentication purposes. As I understand it, IPIG refers to using Z39.50 access control prompt 1 for authentication.
Ruth: Yes it does, IPIG uses the Z39.50 access control prompt 1 external to carry access control information. It can be sent in an ILL-Request message or any other message - this latter feature is to get round the fact that there is no authentication in TCP, and since an ILL transaction might last for months it will use more than a single TCP connection over time. The same mechanism can be used when e-mail is used to transmit the APDUs of course.
Apart from my dislike of having so much piggy backing of protocols on each other, I think the main sadness is that there are now two sets of systems that can't interwork, despite using the same protocols!
Mike Wheatley: I strongly agree with Ruth on this one - it's a real mistake that should be rectified. Please rejoin the party!
Sebastian: It's true that they can't interoperate on the wire, but our hope is that it will be a fairly simple task for anyone with a Z39.50 implementation to also support the processing of at least the ILL Request PDU carried in the ES: Item Order. If there is already an existing, stand-alone ILL implementation, it might be a simple matter of gatewaying, or invoking the same underlying routines. Further, the provision of gateway services should be fairly straightforward.
Mike Wheatley: Surely we want to avoid 'gatewaying'. This seems to be saying that there is going to be yet another 'interface' that system developers will have to implement. And before doing that we will have to standardise it - so that only one will have to be implemented by each developer.
I have always thought it best to try and keep the application protocols cleanly separated. That would then allow whichever protocols are appropriate to be selected and used together to meet the requirements in the best way. Carrying ILL APDUs over Z39.50 ties that implementation firmly into Z39.50 - and is that wise seeing that this discussion started from questioning Z39.50s future?
Perhaps there will soon be applications using XML and ILL. Will we need another profile defined for this?
I think the point is that there will not always be a Z39.50 implementation associated with every ILL application. So using Z39.50 as a transport mechanism is not going to be universal.
Hence ILL implementations using Z39.50 ES as transport will limit the sytems they can interoprate with. So it does not form the basis for a profile that leads to wide interoperability.
Using a more basic (and much more widely implemented) protocol for transport such as tcp/ip or smtp seems to be a much wiser choice.
Sebastian: I think for anyone trying to set up a national ILL initiative at the moment, it's a very tough call which way to go. In Denmark, we tried to survey the landscape over a long period of time, but we saw a number of different organisational models in place, as well as a number of different practical approaches to using ISO ILL. Everyone we asked had their own take on how to do things, and nothing seemed immediately interoperable.
Mike Wheatley: The call is easier now that the IPIG Profile has stabilised and a considerable number of significant implementors are adopting it (OCLC, RLG, BL, etc). Which specifies that ILL using smtp as transport is mandatory and using tcp/ip is also recommended.
Back to Sebastian: My sense is that broad interoperability of different ISO ILL implementations is much less commonplace than it is for Z39.50... it's entirely understandable, since the application - and the protocol - are arguably more complex, and has more bindings to local policies and workflow arrangements.
Mike Wheatley: Nevertheless interoperability between several ILL systems is already being achieved via the IPIG Profile. And that will soon lead to the next level of interoperability testing at the operational level (which revealed so many problems between Z39.50 applications when the local behaviours, etc became apparent).
Ruth: The main reason is that implementation has been lagging behind. Most of the members of IPIG are in the process of beginning interop testing. There has been good interoperability at the level of exchanging APDUs and to some extent protocol machines.
The next bit is the harder one, i.e. will the systems inter-operate at an operational/administrative level. Given the willingness of most IPIG members to make this work, I think your statement above will soon be disproved. The British Library appears to be most advanced (here the IPIG will shoot me down in flames no doubt), they are being actively involved in interoperability testing with other systems, at an operational level.
Ruth continues on earlier thread: I know of some systems that using Z39.50 for searching and then Item Order to launch an ISO ILL transaction using the ISO standard. (Of course other ILL standards could be plugged in at this point) - I thought this was the purpose of item order ?
Rob: I'm not sure Item order has any preferred use, we've used it in several private applications with purpose-specific profiles (not using ILL).
Mike Wheatley: PS All the discussion has been good - but the real issue is still one of interoperability - best first step towards success is to agree on implementing at least one common transport mechanism. I don't think Z39.50 is the right choice for this.
Thread on Toolkits:
Sebastian Hammer:Speaking of toolkits, I guess I should mention to any YAZ-users interested in doing ILL that they can do that within the same environment if they wish. We use an ASN.1 compiler to generate encoders/decoders and interface structures, and ISO ILL runs through that just fine. The results have been used on pretty much any platform, although we only actively support Unix and Windows.
Of course, if you want to do things the IPIG way (and despite what else I have said, I think you should) I believe you will need to Base-64 encode the BER-encoded packages to produce a mail-friendly encoding, then add the MIME-wrappers and whatever other mail-headers are required. Finally, you will need to either interact with some form of mail handling agent, or else implement SMTP from scratch (who complained about piggy-backed protocols?). This stuff could be rolled into the toolkit, and we may do so if there is any interest - or better yet if someone volunteers good code for the job.
Ruth Moulton admits to having a (sheepish grin)
Barbara Shuh: Ruth Moulton introduced us in IPIG to YAZ a couple of years ago, when we were looking for a ASN.1 compiler (something more reliable than SNACC).
Thus, I've long had a link to YAZ in the Interlibrary Loan Application Standards Maintenance Agency (ILL ASMA) web site < http://www.nlc-bnc.ca/iso/ill/readweb.htm> and recommend to anyone that asks. I've even included the reference in the handout I'm preparing for the EFILA Tutorial on the ILL Protocol being held 2-3 August in Stockholm.
But I am happy to receive some information direct from source on its applicability to use for ILL applications, and will incorporate some of your text in the existing sources.
[A Note to ]ROB BULL: On the other hand, I don't have any information on the toolkit that you mentioned. I remember It being mentioned in passing a couple of years ago, but I never managed to find a link to it. So please send me more information and I'll add links on the ILL ASMA site.
BTW, Sebastian, the e-mail scenario you suggest is covered in the IPIG Profile, Clause 6.9. There's a link on <http://www.nlc-bnc.ca/iso/ill/ipigprfl.htm > And I, for one, would welcome any toolkit development that would make implementation of protocol-based applications easier. Let's talk about this over coffee sometime in Stockholm.
Mark Wilson: TLC (The Library Corporation) also offers a full ISO ILL Toolkit that implements much more than the ASN1 - it understands the requirements of the protocol and its State Machine.
The paragraph below about the difficulties of implementing ILL on SMTP is simply not true - it took us (and others implementing this protocol) less than a day to add store and forward to our direct connect application.
Ruth responds as well to Sebastian: There is free software out there for doing base64 encoding (not hard any way) the mail layer should be left to do the encoding/decoding and format the necessary headers etc. Sendmail application can be used to do the smtp on Unix, I expect there are POP sources too...
Sebastian Hammer: I was always wondering about this... things like sendmail tend to use file-based queuing systems which are not particularly fast on the scale of proper, online network applications.... but POP (IMAP, too?) is essentially a polling mechanism, and a relatively resource-expensive one at that. Do people really foresee POP playing a role in this and still think of this as a potentially interactive solution?
Even when POP is not involved, it's often less then obvious when a mail agent chooses to use direct SMTP to another host or when the mail gets routed someplace else - possibly via different protocols (eg. UUCP). SMTP compares to TCP/IP a little bit like your national postal service compares to radio or telephony... it really works remarkably well, considering, and most mail gets through in roughly the order it is sent... but only because of a billion-dollar, more or less state-subsidized machinery of semi-automatic, semi-manual processes is *forced* to make it happen. That stamped envelope sure doesn't have any natural inclination to move from point A to point B.
TCP is actually a reliable, sequenced stream protocol... email, magically, is exactly the opposite.
I guess SMTP has one thing going for it... like HTTP, it usually gets through even the tightest firewall, and this makes it appealing to some. But how much do you want to bet that firewalls three years from now will be culling incoming email messages for signs of unwanted application protocols? That is, after all, exactly what firewalls are intended to do.
Jacob Hallén wrote: I think you are badly misrepresenting email as a transport mechanism. It is a set of services that have been marshalled into performing delivery of messages in all sorts of thinkable and unthinkable environments for a long time. This means that the environment you employ the service in plays a very important role in the reliability.
When used in the form of SMTP between applications it has the same reliability as a Z-connection or a direct socket link using TCP.
When used in a store-and-forward environment, like with POP, IMAP or UUCP, reliability drops. How much depends on the exact setup. However, these setups allow communication with units that are not constantly online on the Internet.
When used in environments where you change protocols, transport mechanism, operating systems, character sets and whatnot, your reliability will suffer, but such problems are getting less and less common.
While it probably is impossible to find ILL units in Sweden and Denmark that are only linked to the Internet by a dial-up line, the situation is very different in most parts of the world, and will be for many years to come. Given such constraints, Internet mail, with its different transport mechanisms appears to be the only deployed technology that gives the dependability for direct connections and the flexibility for store-and-forward transport that are needed.
Sebastian Hammer responded in reference to the first paragraph in Jacob's last message:
This is not really related to the discussion about Item Order (except for the last paragraph). I'm not trying to make light of the accomplishment of internet email as a message delivery system - only as an interactive, system-to-system protocol. SMTP is not inherently slow, but the applications and daemons which implement it serve their own master, not you.
It is true as you say that when two applications talk SMTP, this is about as efficient as any other TCP/IP protocol. However, that was *not* what Ruth was describing. She talked about handing a protocol package off to Unix "sendmail" or other email handling agents, and that is not necessarily a reliable form of delivery. That is "email", not "SMTP", and it is often slow and unpredictable when compared to something like HTTP or Z39.50 over TCP/IP. Indeed, if an error occurs, the way you are most likely to find out is by a human-readable email message from someone named "MAILER-DAEMON" in your mailbox (or that of your ILL application) within a few seconds, often hours, sometimes days.
In order to re-establish the reliability that you abandon by layering your application protocol on top of email, you will need to make the latter robust to lost or out-of-sequence messages. ISO ILL has added this robustness to the application layer: It has to anyway, to make the very long life-span of the state machine practical. But that doesn't make email a clever solution.
I appreciate the fact that email-based services can be provided where there are no permanent Internet connections (although you cannot run an "SMTP" server if you don't have a permanent connection). Here, a library with serious ILL activities and no real Internet connection is not worried about ISO ILL - it is worried about getting set up with a proper Internet connection.
Jacob Hallén: Cost constraints and availability of infrastructure make access to proper Internet connections a dream in over 90% of the countries of the world. When developing international standards and implementing them, we need to consider those who cannot afford to go to the meetings and their needs.
Sebastian Hammer: Yes, but are those countries really on the verge of implementing fully-automated, networked ILL systems between their libraries? I'm a great believer in providing access to information and services to all regions, but that doesn't always mean you have to design for the lowest common denominator. I helped design and develop a pretty nice email-to-Z39.50 gateway (EU-funded EUROPAGATE PROJECT) which allows you to query remote databases using Z39.50 by sending off emails with CCL commands from any old email-application. That's a different approach, and probably more realistic for the 90% of the world you talk about than to expect them to implement either Z39.50 OR the IPIG profile.
But again - this is not even the real reason some people wish to use Item Order to carry certain ILL APDUs. That has more to do with a desire to provide a fully integrated, interactive service within an operational practice where it is the ILL requester's job to figure out who can provide the item in a timely fashion - BEFORE sending the ILL request.
Jacob Hallén: I certainly see the use of Item Order to carry an ILL request. It is a nice feature to be able to search for items, order them, get a confirmation that the order has been placed, and for some materials even get the item (or a pointer to the item) returned.
However, Z39.50 does not easily lend itself to the subsequent exchange of messages, since this (as defined in the ISO ILL standard) is based on a peer-to-peer relationship between the communicating parties. To be able to communicate effectively, both parties must be able to initiate communication.
Aside from Mike Taylor: Couldn't you fake this up in Z39.50 using access control messages with otherInfo blocks attached? I appreciate that this is not the most elegant of all possible approaches, but it is from the same stable as the current effort to wedge Z39.50 into HTTP as a transport layer.
Jacob Hallén: To do full ILL with Z39.50 requires the use of something totally different from the ISO ILL standards. Such a beast does not exist, as far as I know, and would take years to develop.
Sebastian Hammer: No-one is arguing that you should exchange all of the messages through Z39.50 Item Order. Only the ones where it makes sense to do so.
Joe Zeeman: Actually the model of ILL over Z39.50 treats Z39.50 as just another carrier of ILL APDUS, like email. Using Z39.50 to carry all APDUs relating to a transaction is quite feasible, as long as the requesting ILL application treats Z39.50 like a POP server that it has to connect to in order to fetch its mail by searching the ES database.
Not that I would recommend such an approach. Proper email is cheaper and easier.
Jacob Hallén: I also seriously doubt that any Internet Service Provider would be willing to support a Z39.50 target for this purpose, no matter how much you paid them.
Joe continues: This is one of the two profiles for ILL over Z39.50. The other one allows, in essence, a Z39.50 OPAC to gather ILL request information from users and send them to a library's ILL/DD Requesting system, whence they are treated like requests entered at the local user interface. I've never seen this as an ILL system per se, but as a kind of remote user interface to an ILL system.
Sebastian Hammer: I think I've said my piece on this... maybe wildly out of place, and probably grossly unfair to the hard-working IPIG folks. I am sure that the rationale for picking email/SMTP in the context where it happened was sound. But I think it is very relevant for the IPIG or the ZIG to know the range of requirements out there, and that this or that proposed solution may be considered problematic by some people.
Ruth Moulton: The mmencode software springs to mind for B64. I'll have to hunt round and look out some references, i'll try to do it next week - honest! and I think MS mail api does both smtp and the MIME (b64 and headers) for you too, but I'm not an MS windows/nt expert.
Rocco Carbone. We have an ILL Toolkit. It will be released in source code under GPL before the end of this year. I will announce the official release to both the ILL list and the ZIG list. Please be patient.
During last days I silently followed the long debate "ILL over SMTP" against "ILL over Z39.50". Let me explain what it's happening here, and some personal opinions. Several months ago we decided to implement "ILL over Z39.50" due to several reasons (marketing opportunities, projects, business plans, user requests, etc). But I think we voted for Z39.50 as Transport mechanism because we already have a Z39.50 implementation. I think this was the main factor for our decisions. So, as result of our developing efforts, we now have an operational ZETA server able to unwrap ILL-Request APDUs conveyed in the Z39.50 ES/Item-Order APDUs.
I hate the heavy process of wrap/unwrap protocols-over-protocols. I think there is a third way: "ILL over TCP/IP", and I expect an RFCXXXX "Using the Interlibrary Load ISO 10160 Protocol in the Internet Environment", similar to RFC1729. This, for example, will allows to have in a near future lite ILL implementations directly over TCP/IP. If anyone is interested we started an operational Z39.50 Test Server supporting the ILL-request APDUs conveyed by mean of Z39.50 ES/Item-Order. This ZETA Server is still in pre-alpha stage, but we are enough confident it can be useful for interoperability tests to the whole ZIG community. Let me know if you have problems with this service.
Ruth Moulton responded to Rocco: The use of a direct network connection, i.e. ISO ILL over TCP/IP has been discussed by the IPIG group, and is mentioned in the ISO 10161 standard (in general terms). In fact it is favored by many members of IPIG, and many (if not all!) of their implementations use it.
I don't believe there are plans to publish an RFC on the subject, but if you look at the IPIG Profile you'll see rules for using TCP/IP for transmitting ILL APDUS. (www.nlc-bnc.ca/iso/ill/ipigprfl.htm)
A Practical Note from BL:
Kevin Gladwell:I thought it might be useful to let you know the British Library's experience with Email versus direct connect. Since 1995 we have provided ILL both via email and telnet. In June this year we received 120,000 requests via email and 100,000 requests via telnet. Therefore they are slightly in favour of Email - does this help the debate?
Leif Andresen asked Kevin: How many requests via BlaiseWeb in June this year?
When people are using email: do you have any idea of how many who send an email without verification i BLAISE-Web og BLAISE-Line and how many who send an email after verification?