--------------------------------------------------------------
Willpower Information logo Willpower Information
Information Management Consultants
--------------------------------------------------------------

Criteria for the evaluation of thesaurus software
by Jochen Ganzmann

Originally published in International Classification 17(1990) No. 3/4, p. 148-157, 23 refs., 1 appendix. and reprinted here by permission of Dr Ganzmann, the International Society for Knowledge Organization (ISKO) and the current publishers, Ergon Verlag.

Abstract

The growing number of thesaurus programs, especially for microcomputers, calls for criteria which facilitate the decision on the appropriate tools for thesaurus construction, application and maintenance. Following a brief outline of the determinants of the criteria, i.e. the changing environment and the routines of thesaurus management and application, criteria are discussed in detail. Only the most important requirements relating to central functions of thesaurus work are touched upon, further criteria being included in a check-list attached. Applying the criteria to existing thesaurus software in a final short assessment, the author concludes that some programs reveal substantial shortcomings. -- (Author)

A. The Need for Criteria

Over the last ten years we have seen a renewed interest in the thesaurus which is reflected both in the literature on the topic (Note 1) as well as in a steadily growing number of thesauri (4).

The reasons are manifold and complex. Most important have been the impact of artificial intelligence in relation to the concept of knowledge representation, dissatisfaction with the results of free-text retrieval and finally advances in computer technology which entailed decreasing costs, easier handling and better performance both in respect to software and hardware. These advances, especially the advent of the microcomputer, have greatly facilitated the previously tedious task of development, management and application of thesauri (5).

Over the last decade the number of thesaurus programs (i.e. software for both development/updating and application) in general and especially for microcomputers has been rapidly increasing. The programs available today can be classified

either
by the type of computer for which they have been developed:

or
by the functions which they fulfil:

Stand alone software systems, very often for microcomputers, can facilitate the construction and maintenance of thesauri to be used in mainframe computers where the handling can still be laboursome and the updating procedures are often time-consuming. Of course this implies some transfer of the thesaurus data into a retrieval system in order to use the thesaurus in indexing and retrieval.

For the end-user it has nowadays become easier and also less expensive to develop, maintain and use thesauri even in small-scale institutions and, equally important, with the advent of the microcomputer his independence from computer experts has considerably increased. At the same time, however, the rise in number of programs and the above specified diversification of thesaurus software types have more and more imposed the burden of decision-making on him.

Knowledge of the requirements caused by the changing conditions in which thesauri are used nowadays and of the criteria that are to act as a measure-stick in the decision which software is best suited to one's specific needs has become more important than ever before.

Checklists of criteria can be helpful since they can identify possible pitfalls and help reduce decision-making based on uncertainty as to the essential requirements of a software type. That there is an increasing need for orientation has been my experience at several seminars on thesaurus software held at the Lehrinstitut für Dokumentation (LID) for which the checklist to which this paper refers has been originally developed. The checklist had these end-users in mind and consequently it is above all concerned with defining criteria for microcomputer applications but can to a great extent also be applied to mini and mainframe computers.

I will outline those criteria from the checklist which seem to be most important taking into account the functions of thesaurus software and the changing environment of thesauri already mentioned. These determinant factors for the development of the criteria will be presented (chapter B) before a discussion of the criteria (chapter C). A short look at the present state of the art of thesaurus software in which I will try to make an assessment in how far programs measure up to the defined requirements will conclude the paper (chapter D).

B. Determinants for the Definition of Criteria

B.1 The Thesaurus and its Environment

Over the last thirty years the thesaurus concept has undergone a change. Traditionally and even today the thesaurus has been foremost an indexing language developed by individual institutions for their specific needs. Due to this the main functions of the thesaurus have remained relatively the same over the decades: its purpose was to optimize indexing and retrieval in a given environment with a circle of users with specific needs.

There have, however, been factors affecting this traditional concept, especially over the last ten years:

These factors which cannot be seen isolated but rather as interrelated determinants have resulted in various trends (6-11):

  1. Growing importance of multilingual thesauri which facilitate cooperation among institutions from different countries in the exchange of data and in indexing and retrieval of documents in a common database. Multilingual thesauri can also help users of online data-bases to search in their own language for documents indexed in a foreign language. Finally, multilingual thesauri have gained increasing importance since they can be integrated into expert systems supporting machine translation and retrieval (3,6).
  2. Renewed and increased interest in the compatibility or integration of various indexing languages. The creation of so-called switching languages/intermediate lexicons and mapped or integrated thesauri (12) admits, above all, an easy integration of data downloaded from other systems into one's own database, supports retrieval in the search in various document collections (either in-house as with office automation or in online databases (8,9)) indexed with various indexing languages (either of one or different natural languages), and can suggest amendments concerning the specificity and scope of the vocabulary (12).
  3. Increased importance of additional differentiation of relations between concepts in thesauri which can be integrated into expert systems as knowledge bases, for instance for improvement of automatic indexing and so-called intelligent information retrieval (8,9,13).

These changes in the environment of the thesaurus must be taken into account when the functions of software are defined.

B.2 The Tasks of Thesaurus Software

Considering what has been said so far, the specific functions of thesaurus programs can be related to tasks associated with the following complexes:
a) The Construction and Maintenance of the Thesaurus
Construction and updating of a thesaurus includes a variety of tasks and activities:

Word-material is selected and recorded, sometimes parts of existing thesauri must be integrated, information as to the source of terms, language, systematic grouping has to be recorded. Terms are controlled in respect to clarification of meaning (homonyms, definitions) and relations must be defined between terms.

Finally, thesauri must be updated regularly with regard to relations and terminology due to changes in the terminology of a given subject-field, to changing indexing and retrieval practice, to failing adequacy of the vocabulary's scope or specificity.

b) The Output of the Thesaurus
The output of the thesaurus (or parts of it) is necessary in all phases of thesaurus work, either on the screen, by the printer or also into a file in case the data are to be transferred to a word processor or for integration purposes to another system (thesaurus maintenance or retrieval system).

The display of vocabulary usually follows standards and conventions which have developed in more than thirty years. Most thesauri have at least one alphabetical and one systematic section, often KWIC/KWOC or hierarchical displays, sometimes even graphical displays are added. The representation of three types of relations (equivalence, hierarchical, associative relationship) in accordance with the respective standards on thesauri and their construction (14,15) has also become a common trait of most, though certainly not all thesauri (4).

c) Indexing and Retrieval with the Thesaurus
All thesaurus programs support the construction, maintenance and printing of thesauri.

It is possible to index documents and search for relevant information in a retrieval program when a printed thesaurus is used.

The integrated thesaurus can support specific tasks associated with indexing and retrieval, by acting as an interactive instrument for orientation about potential index and search terms, a tool for consistency controls regarding the data used in indexing and searching and a tool supporting update routines.

d) Exchange, Integration and Compatibility of Vocabularies
This aspect refers to all of the above-mentioned complexes of thesaurus work; it is, however, given special attention here because it is gaining more and more importance in the near future and entails a variety of specific tasks and functions, above all:

C. The Definition of Criteria

With regard to the criteria for the evaluation of thesaurus software the author could refer to various sources:
  1. the literature on thesaurus construction and maintenance which mentions some of the possible applications and advantages of the computer (2,3,16,17).
  2. the check-lists of three authors concerned with the development of criteria for the evaluation of thesaurus software (11,18,19).

Whereas the general literature on thesauri lacks systematization of criteria and mentions only some aspects relating to the evaluation of thesaurus software, the checklists have been more specific in their definitions of criteria. Still, these checklists have either been developed for and applied to the evaluation of stand-alone programs (11,18), been restricted to a limited set of criteria (19), or more specific criteria are only made explicit in the discussion and evaluation of a thesaurus program (11).

The approach chosen here differs from that of the three authors (11,18,19) in that it integrates criteria for stand-alone and integrated thesaurus software and adds criteria not mentioned in these check-lists.

The critena in my check-list are categorized as follows:

  1. general criteria (i.e. criteria applicable to any type of software):
  2. criteria relating to the specific tasks of thesaurus software
Since the purpose of this paper is the definition of criteria for thesaurus software with respect to the specific requirements of a tool for the construction, maintenance and updating it will concentrate on aspects relating to these functions of the software. General aspects to be considered with any type of software will be left aside in the following discussion though they have been included in the checklist since they may well affect a user's decision, too. (Note 2). Ergonomic aspects have been considered in the following discussion only where they can be referred directly to specific functions of thesaurus software.

In the following I will concentrate on the most important criteria to measure the capability of tools for thesaurus construction and application under several headings relating to the process of construction and application of the thesaurus and discuss the criteria with respect to their relevance concerning the different tasks and functions. Taking into account that the various functions of thesaurus software mentioned are not equally important I have differentiated between basic requirements, i.e. requirements fundamental to any thesaurus work, and desirable requirements which might be important in more specific applications.

C.1 Structural Definitions

Structural Definitions refer to those features that determine above all the degree of complexity and differentiation which the software will admit in the construction of a thesaurus and therefore to criteria highly important in the evaluation of thesaurus software, namely term and term related attributes and the number and types of relations that can be defined between terms.

C.1.1 Term and Term Related Attributes

Any thesaurus software program must help the user to create the terminology suited to his specific purpose. Two decisive factors determining the degree of flexibility are length and number of fields.

Basic requirements for any thesaurus program are, besides the term field, a field for scope notes to define the meaning of terms and a field for notations to facilitate the structuring of vocabulary and a systematic display.

Some thesaurus programs restrict the length of the term field to 50 characters or less which can be problematic for two reasons:

Since the selection of vocabulary should not be dictated by restrictions in the length of the term field it should admit at least 50 characters and -- ideally -- be user-definable.

Scope notes should preferably be of variable length to provide for long definitions, and the field for notations should ideally be user-definable in order to admit even more complex notations (like, for instance, the long MESH notations).

Apart from these basic requirements there are further desirable features, depending on the size, planned display and the languages to be used:

C.1.2 Relations

With regard to the types of relations the demands of individual institutions can conflict with the standards which recommend the representation of three types of relations (synonymy, hierarchical relation, relatedness). Many institutions have chosen to ignore these standards and/or have defined additional relations (4). To meet these demands any thesaurus program should ideally let the user define freely the type of relations he wants to use.

If the relationships cannot be defined freely the following requirements apply:

With regard to the number of relations individual terms can have, there must be no restrictions (as long as they do not affect consistency (cf. C.2.)). This applies especially to the relationship between terms and their broader terms (polyhierarchy as a characteristic trait of thesauri) and to the more specific equivalence relations described above -- semantic factoring into various (three or more) factors or representation of homonymous terms or broad concepts by alternative terms in the thesaurus should be made possible if the vocabulary requires it.

C.2 Input of Terms and Relations

Capture of data, modification and deletion of terms and their relations are the typical work routines, when thesauri are constructed and maintained with a computer program. Most important are the mode of input and consistency controls relating to vocabulary and relations. Display of data on the screen does also refer to these phases of work but will be treated in connection with the general aspects relating to the output of data (cf. C.3.1).

The mode of input should ideally be possible by keyboard as well as by data capture from outside in batch mode. Batch input of thesaurus data is becoming increasingly important since thesaurus construction can involve the adaptation of parts of already existing thesauri and additionally there is, as already pointed out, a growing interest in the integration and merging of machine-readable thesauri. At present there is no generally accepted exchange format for thesauri (Note 3) so that the batch input of structured thesaurus data generated by another program depends to a great extent on the ability of the source system to create the format needed by the target system.

Consistency controls concerning the relation structure as well as the terminology are of prime importance in thesauri. Any program for the construction and maintenance of thesauri must include consistency checks, to avoid illogical features in the thesaurus terminology and structure. The most important checks are those that prevent:

These consistency checks must of course be applied when terms are entered and the relations are defined but any time a term or relation is modified or deleted these changes must be reflected correctly and consistently in the thesaurus, too.

C.3 Output of Terms and Relations

As has been already said, output of data is necessary in all phases of thesaurus construction and maintenance and must be possible on the screen, by the printer or into a file. In addition it can be referred to the output of the thesaurus on the screen when the thesaurus is applied in a retrieval system. The specific requirements of an integrated thesaurus concerning orientation and support of indexing/retrieval are defined in the section on indexing and retrieval (cf. C.4).

The output of data must optimally support the routines of construction and updating by presenting the vocabulary from various angles, in various forms and freely definable layout (for the print).

C.3.1 Display on the Screen

In any of the phases of thesaurus construction and maintenance, display on the screen is necessary, above all:

Selection of words for specific modifications or checks according to certain criteria will largely depend on how differentiated the description of a word can be made in terms of its attributes, that is on the structural definitions. Output of terms according to the various relevant criteria (source, notation, facet, language, type of relation) should be possible as well as a combination of criteria (including strings). It should also be possible to select certain words that have been previously marked for specific purposes (deletion, modification, change of relationship etc.).

The display forms on the screen must -- for orientation purposes -- include an alphabetical display (with or without relations and term attributes) and a systematic display of terms. Desirable requirements are a KWIC display for large vocabularies with many compound terms and a hierarchical display to check the consistency of the hierarchical relations.

As concerns the possible forms of interaction navigation through the semantic net of the thesaurus should be made as easy as possible, and editing functions should ideally also apply to the term lists in order to facilitate the routines of construction, modification and deletion.

C.3.2 Output by the Printer

The output of word lists by the printer is necessary for several reasons:

As has been pointed out the ability of a thesaurus program to present the vocabulary from various angles depends above all on how differentiated the term description (term related attributes) can be made. The selection should be possible according to the various criteria of selection as defined for the screen thesaurus. Again all selection criteria should be freely combinable for the output.

As regards the display form of the vocabulary (or selected parts of it) a program for the construction of thesauri must, in accordance with the standards, at least be able to generate a display in an alphabetical array (including the scope notes and indications of relations between terms) and in a systematic display (with the notations necessary for systematic ordering and indication of the hierarchical level of a term, ideally also with inclusion of node labels for a faceted display in a classaurus). Additionally the multitude of known variations and various additional forms of display (e.g. broad subject groups, ordering by facets) (17) should also be generated by a computer program, above all the two types of display useful for structuring and controlling consistency of large vocabularies with many hierarchical levels and many compound terms, namely hierarchical displays as well as KWIC or KWOC lists. These additional forms of display cannot usually be generated by microcomputers, except for one defined term and its upper and lower terms (hierarchical display) or a defined string (KWIC, KWOC).

Optionally the program should be able to generate a graphical presentation of the thesaurus. The standard thesaurus programs do, however, not offer that option.

More specific questions of display relate to the details of presentation of the vocabulary. The user should be given as much freedom as possible so that he can print a thesaurus according to his own needs. Especially the layout for the print of the thesaurus should be freely definable with respect to pagination, typographic differentiation of descriptors and non-descriptors, line pitch etc.

Free selection of the types of relations to be included or omitted in the print (e.g. only synonymous terms, no implicit relations or relations not defined in the standards etc.) and free definition of their external representation by abbreviations (either by user-defined or standard national or international reference codes) can be important, since it facilitates the output of the thesaurus for a variety of applications (e.g. comparability of thesaurus structure for integration or compatibility) while admitting the maintenance of a very complex thesaurus with a multitude of specific relations (e.g. for use as a knowledge base).

C.3.3 Output into a File

Transfer of data to other systems will ask for a special format of interchange which the program must be able to generate. The transfer of thesaurus data to a word processor or of word lists into thesaurus software is usually no problem as long as the program can generate standard exchange formats (e.g. ASCII) whereas the transfer of structured thesaurus data into a retrieval system or a thesaurus maintenance program is much more difficult since there is no generally accepted exchange, format for thesaurus data and the input formats required by the individual thesaurus maintenance programs and retrieval programs vary greatly. As has been said, thesaurus software should ideally be able to generate some or various formats necessary for this transfer, but usually these devices will have to be programmed on demand.

C.4 Indexing and Retrieval Functions of Integrated Thesauri

In addition to the requirements mentioned so far integrated thesaurus modules in retrieval programs demand specific functions in order to gain an advantage over the printed thesaurus.

For easy orientation on the vocabulary and the semantic relations among terms it must be possible to display the thesaurus on the screen both alphabetically (both as a single word-list and with indications of relationships) and systematically while the user is indexing and searching. Orientation on the term best suited for the representation of a given concept is greatly facilitated when navigation through the thesaurus, i.e. the quick switching from one term to semantically related terms and browsing through both the alphabetical and systematic display is easy.

The function of control of input must be supported by the software as well, that is the program must check whether a term used for indexing is contained in the electronic thesaurus and if necessary reject it (unless otherwise defined by the user). Furthermore the software should enable the user to select descriptors directly from the online thesaurus without having to enter them. Of course a program should automatically replace those terms contained in a thesaurus but not admitted for the representation of concepts. This will be the case whenever equivalencies between a given set of terms can be represented by one term only whereas the others serve as lead-in vocabulary, e.g. non-descriptors (normal synonymy, compound terms -- provided the program admits semantic factoring) and also terms from a secondary language in the case of multilingual thesauri.

The updating of vocabulary can be greatly facilitated when the frequency of use of thesaurus terms in indexing and retrieval can be verified. Statistics on whether and how often a term has been used for indexing and searching can help identify failures of a vocabulary with regard to specificity and scope of vocabulary. The system must admit global changes, i.e. whenever a descriptor that has been used in indexing (and retrieval) is replaced by another term better suited to represent a concept (either a former non-descriptor, a completely new term or a term of broader scope), the former entries in the index must be replaced automatically by the new descriptor in the inverted file.

One specific advantage of the online thesaurus in retrieval is its ability to facilitate the formulation of a search strategy. Automatic generic search (i.e. searches for all documents indexed with terms belonging to a specific hierarchy) is a function common to many retrieval systems and can be regarded as a basic requirement of thesaurus software. The user should, however, also have the opportunity of automatic inclusion of all terms defined as related terms of a specific search term to optimize recall.

D. The State of the Art

As the discussion of criteria relevant to the evaluation of thesaurus software has shown there are a variety of features relevant to almost any application of thesaurus software whereas others will largely depend on the user's specific environment and requirements (KWIC, definition of layout, multilingual thesauri etc.).

Certain qualifications can be made as to vital and basic elements any thesaurus software should have. Especially important are those features which determine the structure and complexity of the vocabulary and features related to consistency control and display of the vocabulary since the value and applicability of the thesaurus will largely depend on these features.

This means above all:

A short glance at software packages available suggests that most programs fulfil the basic requirements for thesaurus software.

But in several of the programs some of the basic requirements are not fulfilled (Note 4):

Considering this short survey of substantial shortcomings one can only assume that some thesaurus programs (especially those integrated into retrieval software for microcomputers) have been developed for end-users who apply their programs to small-scale if not mini-thesauri and small-scale document collections. Whether a user will -- in the long run -- be satisfied with software of such limitations is doubtful.

However, even thesaurus software which fulfils all basic requirements will, in many cases, not be sufficient. Thesaurus construction still means above all creating an indexing language optimally suited to the specific situation where it is to be used, i.e. to the potential or real size of the document collection, to the user's demands, to the type of documents, to the subject field etc. Thesaurus features like vocabulary size, the relations defined and the display of vocabulary are interdependent but also determined by these specific conditions of a given environment.

In order to meet these needs a good thesaurus program should leave as much flexibility and freedom in adapting the indexing language to one's requirements with respect to size and complexity of the vocabulary, the definition of relations and the display of vocabulary.

Additionally, thesauri must more and more respond to needs and demands from outside the traditional individual working environment. The new tendencies have already been outlined: office automation calling for compatibility or integration of thesauri and subsystems of thesauri, the increasing importance of artificial intelligence calling for additional relations, but above all a trend that will more and more affect the end-user directly, namely the exchange and integration of data from different systems which demand for greater openness of the systems both in a technical and conceptual respect, that is, import and export of thesaurus data must be facilitated by suitable import and export facilities of the software, and the software must support the construction and use of compatible languages (multilingual or monolingual).

Taking a look at the available thesaurus programs with these additional requirements in mind it seems that a considerable portion of these programs do not meet all of the requirements. Of course, it is the end-user who must decide which software is best suited to his purposes. There is no harm in choosing software that fulfils only basic requirements as long as the functions of the software meet the end-user's requirements. It is, however, harmful if the user buys the wrong software product because either he does not exactly know what he must expect from the software or what he wants to achieve with it. The design of a thesaurus should not be dictated by the software but the software should be made subservient to the requirements of thesaurus work.

The actual purpose of this paper has been to provide the end-user with criteria in order to help him select the appropriate software for his specific needs. If, however, one takes into account the present state of the art of thesaurus software, it seems that the checklist might well be useful in more than one way: Like checklists for other types of software, it may also -- hopefully -- stimulate some of the developers of thesaurus software to revise their software conceptions and improve their sometimes insufficient programs.

Dr. Jochen Ganzmann
Lehrinstitut für Dokumentation in der DGD
Westendstr. 19, D-6000 Frankfurt 1

[This was Dr Ganzmann's address when he wrote this paper. The Lehrinstitut für Dokumentation ceased to exist at the end of 1991 and he is now working for the Frankfurter Allgemeine Zeitung. He can be contacted by email.]

Notes

Note 1: Cf. for instance the special issue of Nachrichten für Dokumentation (Ref. 1) on thesauri and the second editions of Aitchison/Gilchrist (Ref. 2) and Lancaster (Ref. 3).

Note 2: For an extensive and comprehensive discussion of these aspects the reader is referred to Van Wyk's (Ref. 22) checklist for retrieval software.

Note 3: MATER (Ref. 23) has not been accepted as an exchange format for thesauri so far and any feasible alternative has not been developed yet.

Note 4: This judgement is based both on the author's experience with various programs and on the literature on this topic (Ref. 11, Ref. 18, Ref. 19).

References

(1) Nachrichten für Dokumentation 39 (1988) No.1.

(2) Aitchison, J., Gilchrist, A.: Thesaurus construction and maintenance: a practical manual, 2nd ed., London 1987.

(3) Lancaster, F. W.: Vocabulary control for information retrieval, 2nd ed. Arlington 1986.

(4) Schöndorf, P.: Nicht-Konventionelle Thesaurusrelationen als Orientierungshilfen für Indexierung und Recherche -- Analysen ausgewählter Beispiele. Nachrichten für Dokumentation 39 (1988) p.231-244.

(5) Burkart, M.: Neue Thesaurusansätze -- frischer Wind in alten Segeln?. Nachrichten für Dokumentation 39 (1988) p.207-208.

(6) Svenonius, F.: Compatibility of retrieval languages: introduction to a forum. International Classification 10 (1983) p.2-4.

(7) Dahlberg, I.: Towards establishment of compatibility between indexing languages. International Classification 8 (1981) p.86-91.

(8) Rada, R., Martin, B.K.: Augmenting thesauri for information systems. ACM Transactions on Office Information Systems 5 (1987) p.378-392.

(9) Rada, R.: Connecting and evaluating thesauri: issues and cases. International Classification 14 (1987) p.63-69.

(10) Panyr, J.: Thesaurus und Wissensbasierte Systeme -- Thesauri und Wissensbasen. Nachrichten für Dokumentation 39 (1988) p.209-215.

(11) Sick, D.: Aufbau und Pflege komplexer natürlichsprachig basierter Dokumentationssprachen (Thesauri): Aktuelle Tendenzen und kritische Analyse einer ausgewählten autonomen Thesaurus-Software für den Personal Computer. Universität Saarbrücken 1989.

(12) Aitchison, J.: Integration of thesauri in the social sciences, International Classification 8 (1981) p.75-85.

(13) Schwantner, M.: Entwicklung des Indexierungswörterbuchs PHYS/PILOT. In: Deutscher Dokumentartag 1987. Ed. H. Strohl-Goebel. Weinheim 1988. p.329-339.

(14) International Organization for Standardization: Documentation: guidelines for the establishment and development of monolingual thesauri: ISO 2788, 2nd ed., 1986.

(15) Deutsches Institut für Normung: Erstellung und Weiterentwicklung von Thesauri: Einsprachige Thesauri: DIN 1463, T.1. Berlin 1987.

(16) Soergel, D.: Indexing languages and thesauri: construction and maintenance. Los Angeles 1974.

(17) Wersig, G.: Thesaurus-Leitfaden: Eine Einführung in das Thesaurus-Prinzip in Theorie und Praxis. 2nd ed., München 1985.

(18) Ritzler, C.: Vergleichende Untersuchung von PC-Thesaurus-pflegeprogrammen. FH Darmstadt 1989.

(19) Rohou, C.: La gestion automatisée des thésaurus: etude comparative de logiciels. Documentaliste 4 (1987) p.103-108.

(20) Deutsches Institut für Normung. Erstellung und Weiterentwicklung von Thesauri: Mehrsprachige Thesauri: DIN 1463, T.2. Berlin 1988.

(21) International Organization for Standardization. Documentation: Guidelines for the Establishment and Development of Multilingual Thesauri: ISO 5964. Genf [=Geneva] l985.

(22) Van Wyk, J., Tönsing, R.E.: The evaluation of interactive information storage and retrieval software. South African J. of Libr. and Inform. Sci. 55 (1987) p.34-50.

(23) Deutsches Institut für Normung. Format für den maschinellen Austausch terminologischer/lexikographischer Daten-Mater: Kriterienkatalog: DIN 2341. Berlin 1986.

--------------------------------------------------------------

This document is at http://willpowerinfo.co.uk/ganzmann.htm
Valid XHTML 1.0! Last modified 2008-11-26 12:23
Copyright © Willpower Information, 2008

Comments and feedback on content or presentation are welcome and should be sent to Leonard Will

--------------------------------------------------------------

'HOME' icon Return to Willpower Information home page

</BODY