TEI MI W 06Migration Case Study Reports

About the Project

Started in 1988, the Brown University Women Writers Project is a long-term research project devoted to early modern women's writing and electronic text encoding. Our goal is to bring texts by pre-Victorian women writers out of the archive and make them accessible to a wide audience of teachers, students, scholars, and the general reader. We support research on women's writing, text encoding, and the role of electronic texts in teaching and scholarship.

Initially funded by the NEH, the WWP is now supported through licensing fees to Women Writers Online, the on-line face to our textbase, and through grant funding from the NEH and the Delmas Foundation. In the past, the WWP has also received grants from the Mellon Foundation and Apple Computer, Inc.

The WWP hires undergraduate and graduate students as encoders. They enter texts using GNU Emacs, Lennart Stafflin's psgml mode, and lots of home-grown macros on a Sun Unix system maintained by the University, and accessed via an SSH client on Macintosh desktop computers.

In addition to transient student encoders, the WWP has a permanent staff that has fluctuated from one to five, but has been steady at three for several years now. The WWP is part of Brown University's Computing and Information Services. The WWP is advised by a board of scholars, a subset of whom serve on an acquisition committee that decides which texts are next to be added to our textbase.

About the Collection

WWO (the on-line system available to subscribers) currently has over 220 texts, of which just over half have additional contextual materials funded by the Mellon Foundation as part of Renaissance Women Online. The total size of the XML source files for WWO is just over 50 million characters (just under 50 MiB). In addition to the texts available to subscribers, there are over a hundred more ‘under construction’, for a total of just over 97 million characters (just under 93 MiB). Currently we are concentrating on printed works, with an eye towards manuscripts in the future.

Nature of the Content

The collection includes works from every genre: political satire, drama, poetry, religious tracts, and even medical reference. All texts are in English (although there are plenty of phrases and quotations in other languages) and either written or translated by women.

Works from the period included (roughly 1400 to 1850) are often in black-letter; never have a nice, solid, scannable baseline; and often have pagination indicated by signatures, if at all. Several texts have tipped-in pages, errors in the printed page numbers, or publishing errors (such as incorrectly ordered sections). Furthermore typographic errors and difficult-to-read or damaged photocopies are common.

The WWP includes a variety of different kinds of texts including monographs, broadsides, and analytic pieces. Typically, in cases where work by a woman is included in a larger work by men, only the front and back matter (e.g., title page and colophon) and those parts written by the woman are included.

While there are obviously no photographs in the works involved, there are many figures, typically woodcuts or engravings. These are currently captured only by a very short description in the encoding ¹ , i.e. no scanned images.

Nature of the Encoding

In our SGML texts all characters were encoded in US-ASCII, with characters outside that set represented by general entity references, often to SDATA entities. In the XML version, all characters are encoded in the US-ASCII subset of ISO 10646 UTF-8, with characters outside that set represented by general entity references, mostly to numeric character references to the appropriate ISO 10646 code point. However, we do have some characters not in Unicode. We do not use the WSD mechanism.

The WWP textbase uses a single DTD that includes the prose, drama, and verse base tag sets, and the additional tagsets for linking, transcription, names & dates, and figures. This DTD, called wwp-store, includes significant extensions, including complex manipulations of element classes.

The WWP also uses a smaller DTD for internal documentation. This DTD, called wwp-doc, uses only the prose base tag set, and the additional tagsets for linking and for figures. This DTD includes simple extensions, mostly to remove elements (like <pb> ) that make no sense in documents we are authoring.

Before migration the WWP files were nearly XML already. We have required all end tags since 1994; we have required that attributes be quoted since 1997; and we have been using case-sensitive encoding since 1999. The DTD extension files, however, were another story. There we made significant use of SGML features not permitted in XML, particularly inclusion & exclusion exceptions, the ampersand connector, and SDATA entities. At least we have been using an or-bar (instead of comma) in all attribute declared value groups since 1998.

Motivations for Migration

Barriers to Migration

The biggest impediment to our migration was inertia, particularly in the area of processing our texts for web delivery. We already had a system for capturing, validating, machine-checking for common errors, printing for proofreading, proofreading, correcting, and transforming for delivery on the web via a commercial tool (DynaWeb) based on SGML texts that worked, why change? Especially that last step — the transformation from our source SGML to that which DynaWeb could read into its ‘book’ — is cumbersome and finicky, and not all those who wrote the software are readily available to us.

Furthermore, I anticipated that while migrating our instances to XML would be easy migrating the wwp-store DTD would be difficult.

About the Migration Samples

The samples available from the Migration Task Force's web site are a convenience sample that we had already given to TEI for its samples page.

Samples include prose, verse, and drama; interesting features include a letter with a postscript, an interesting <figure> , acrostics, and a complicated note.

Notes on the Conversion Process

Summary and Evaluation

For the WWP, migrating our instances to XML syntax was trivial. Although migrating them into an XML environment was a bit more difficult, it was still pretty easy, and well worth the effort.

Migrating a small, clean (i.e., without much element class manipulation) set of DTD extensions also turned out to be very easy, and well worth the effort.

Migrating a massive, complicated set of DTD extensions has turned out to be very difficult. Projects in this category may need to do a more careful analysis of the benefits of migrating to XML, as the cost can be significant. Furthermore, unless the original author of the extensions or other TEI DTD expertise is available in-house, these projects should consider hiring outside expert assistance.

About the Project

I have been saying that migrating the BNC from SGML to XML would be a trivial problem for approximately five years. My bluff was finally called when the Open University asked us to produce a 4 million word subset of the BNC in XML, sampled according to their criteria for use in a new grammar teaching course.

About the Collection

The British National Corpus (BNC) is a 100 million word snapshot of British English taken at the end of the 20th century. It contains 4130 distinct texts, sampled from a very wide range of materials both spoken and written.

Nature of the Content

The BNC includes samples of most written genres, including newspapers, novels, textbooks, ephemera, etc. It also contains transcripts of a very wide variety of spoken English, including informal conversation, radio broadcasts, meetings, lectures, consultations, etc.

Nature of the Encoding

All the material is in English.

In addition to the usual TEI structural tagging, the texts are segmented into sentence like units, and words; each word carries a POS (part of speech) code.

The BNC has its own DTD, using the TEI prose base, the corpus additional tagset, and a number of modifications to the basic TEI model, as further described in the Users reference Guide. The most recent edition of this Guide includes a section on TEI conformance which explains in excruciating detail the TEI Extension files used to define the BNC DTD.

Motivations for Migration

Desire to use new XML-based tools and to add to the way the corpus can be processed. For example, we can now offer an XSLT stylesheet to convert the texts to HTML for display.

Barriers to Migration

Laziness. No, cost, size, and complexity, chiefly.

About the Migration Samples

For the needs of the OU project, we selected texts totalling one million words for each of four subcorpora: demographically-sampled speech, newspapers, academic prose, and fiction.

The texts chosen were entirely typical of the rest of the BNC as far as format and tagging goes. The process of converting them was entirely automatic.

Notes on the Conversion Process

I spent a lot of time in March defining the process of migrating the DTD, and also drafted a brief document (now on the BNC website at http://www.natcorp.ox.ac.uk/migration.html ) on how to do this properly. What this document does not tell you is that in the end I had to fudge the BNC SGML DTD somewhat. After I had completed the work described in the migration document referred to, and delivered the first few sample XML files, I realised that the BNC suffered badly from what I will call the Underspecification Gotcha. The UG comes about when you have an SGML DTD which makes liberal use of default values for attributes, with declarations such as <!ATTLIST foo bar (ptrzbie|farble|wibble|zip) "zip">

Why is this a problem? Because when such a DTD is used to convert a document to XML, every occurrence of <foo> which does not specify a value for its bar attribute will be converted to read <foo bar="zip"> . And why is that a problem? Well, maybe the BNC project was exceptional in having encoders who did not realise that not specifying a value in some circumstances meant something different from not specifying a value in others. (But I doubt it).

In any case, I went back to the nice TEI-conformant SGML DTD, and mercilessly hacked it so that all defaulted attributes were given the value of #IMPLIED, even where their declared content was given explicitly in the DTD. So, for example, <!ATTLIST foo bar (yes|no|maybe) "maybe"> became <!ATTLIST foo bar (yes|no|maybe) #IMPLIED> passim.

The BNC has a typical TEI Corpus structure, with a corpus header that contains codebooks used to validate indivual texts by means of the usual SGML IDREF/ID mechanism. The corpus header also contains declarations for two specific speakers who appear in many of the BNC spoken texts (PS00: the unknown participant, and PS01: the unknown group participant). To avoid the nuisance of having to include the corpus header (and then remove it again) when processing individual texts, I modified the SGML DTD so that all IDREF valued attributes were changed to CDATA. Another, even more shameful, admission: a significant number of texts were found to be invalid against the intended XML DTD, although the application of minimization rules had made them appear to be valid against the SGML DTD. The reasons for this are murky, but reinforce in me the feeling that inclusions and minimization are indeed the lapses of good taste which I have long suspected them to be.

The BNC SGML files contain several thousand named character entity references. As delivered, the corpus provides declarations for these as SDATA entities declared explicitly. I used two sets of entity declarations: the one used in SGML land mapped each entity to a null string; the one used in XML land mapped each entity to the equivalent Unicode character number reference. This meant that I could use the exceptionally cool facility provided by OSX of retaining character entity references unconverted when parsing the SGML version, but also run the results through an XSLT transform which would produce meaningful Unicode character entity references.

The xmlify shell script, and the XSLT prettyprint stylesheet are available on the task force's tools page; they are also available along with the DTD files etc. that were used in a single archive file.

Summary and Evaluation

This work took longer than expected, of course. It is not yet complete — I haven't tried running it over the whole BNC, though I have run it to produce the four million word OU sample. However, it was a satisfactory demonstration of the advantages of sticking to the standard TEI route. Problems that arose were almost all caused by deviation from the path of righteousness.

About the Project

The EU Copernicus MULTEXT-East project (Multilingual Text Tools and Corpora for Central and Eastern European Languages, http://nl.ijs.si/ME/) is a spin-off of the EU MULTEXT project. MULTEXT is working to develop standards, specifications, and tools for the encoding and processing of linguistic corpora in a wide variety of languages. MULTEXT-East, which ran from 1995 to 1997, developed language resources for six Central and Eastern European languages (Bulgarian, Czech, Estonian, Hungarian, Romanian, and Slovene) and for English, the 'hub' language of the project. It also adapted existing tools and standards to these languages. The main results of the project were an annotated multilingual corpus and lexical resources for the seven languages. The most important resource turned out to be the parallel corpus of George Orwell's novel 1984 in the English original and translations and heavily annotated with linguistic information.

MULTEXT-East resources have been used in a number of studies and experiments. In the course of such work, errors and inconsistencies were discovered in the MULTEXT-East specifications and data, most of which were subsequently corrected. But because this work was done at different sites and in different manners, the resources' encoding began to drift apart. The EU Copernicus project Concede (Consortium for Central European Dictionary Encoding), which ran from 1998 to 2000 and comprised many of the same partners as MULTEXT-East, offered the possibility of bringing the versions back on a common footing. Although Concede was primarily devoted to machine-readable dictionaries and lexical databases, one of its workpackages did consider the integration of its dictionary data with the MULTEXT-East corpus. In the scope of this workpackage, the corrected morphosyntactically annotated corpus was normalised and re-encoded. The Concede release of the MULTEXT-East resources contains the revised and expanded morphosyntactic specifications, the revised lexica, and the significantly corrected and re-encoded 1984 corpus. This edition, V.2 is also freely available, under a research license, from the http://nl.ijs.si/ME/V2/.

This report documents the SGML to XML conversion process of the Concede edition of the MULTEXT-East 1984 multilingual corpus.

About the Collection

The MULTEXT-East parallel corpus contains the novel 1984 in the orginal English and in six translations. The novel is approximately 100,000 words in length and is composed of four parts, each consisting of a number of chapters. The complete seven-language corpus contains 46,626 sentences, 618,879 word tokens, and 125,016 punctuation tokens.

The corpus is meant primarily as a dataset for the development and testing of language technology methods and tools. It has already been used to develop and test machine learning techniques for part-of-speech tagging, word alignment and word-sense disambiguation. While this research by computational linguistics and language technology professionals has resulted in a relativelly large bibliography (a probably somewhat outdated list is given here) it has not, as far as I am aware, been used by a more general audience interested in the novel itself.

Nature of the Content

The novel itself is well known, and deals with an anti-utopian vision of a Stalinist society. An interesting feature from the linguistic perspective is that the novel contains a number of ‘Newspeak’ words (such as Miniluv, doublethink, plusgood, etc.) and, in fact, an Appendix giving the introduction to Newspeak. The interest comes in studying the translations of these non-words, and the varying strategies the translators have used to translate them. Also, any systems processing the text are almost guaranteed a certain number of unknown words.

Nature of the Encoding

The DTD is a parametrization of TEI and uses the following tagsets: TEI.prose, TEI.linking, TEI.analysis, TEI.fs and no local extensions. Character encoding is via ISO 8879:1986//ENTITIES.

The corpus is rooted in the <teiCorpus.2> element, which consist of the header and seven <TEI.2> elements, each one containing one translation of the novel. The corpus and component TEI headers are quite detailed, and include editorial and tags declarations and source and revision descriptions. Each translation is divided into parts and chapters (encoded as <div> elements), and these into paragraphs (encoded as <p> elements).

MSD IDs are defined in a feature structure library, <fsLib> , contained in a dedicated <TEI.2> element. Each <fs> defines an MSD, specifies which languages it is appropriate for, and describes its decomposition into features. The features are defined in a feature library, <fLib> , where assigns each feature value an attribute and value name.

In producing the Concede version, special care was taken to make the SGML encoding XML-like, with quoted attributes and camelCased GIs. The XML flavour was, to an extent, enforced by a special SGML declaration for the corpus, but was otherwise created through instructions to the language components contributors.

Motivations for Migration

On the practical level, the aim was to enable XML processing, in particular XSLT. More generally, there is a need to keep the corpus abreast of current encoding practices.

Barriers to Migration

The only barrier was the effort required to implement the migration.

About the Migration Samples

Although the migration was eventually undertaken for the full corpus, we started with a sample for initial tests and distribution in the scope of the working group. This sample contains only the first chapter (37,014 of the novel's 618,879 word tokens), but retains the structure of the full corpus and includes the complete ancillary files, such as the FS libraries that define the word-level syntactic tagset.

The sample, like the complete corpus, was not expected to present any great challenges to migration, except for the marked sections in the corpus document. If the entity %ONETEXT; was set to IGNORE, the corpus was processed as a whole (and hence defined the language and other IDs in the <teiCorpus.2> header). If it was set to INCLUDE, each language was taken to constitute its own SGML document (and hence defined the IDs in its <TEI.2> header).

Notes on the Conversion Process

As mentioned, we did not expect any real problems with the migration and did not experience many. The manner in which the conversion was performed is probably unorthodox: instead of using automatic tools for the conversion (such as osx), the emacs editor's simple search-and-replace function was considered a satisfactory and not too time-consuming method. This, of course, would not be an option with a project with a greater number of files and modifications. Below is a list of steps we had to perform to migrate from P3 SGML to P4 XML:

1. Much to my embarrassment, the SGML sample itself was found to contain one SGML error in the file orwell.sgml: <tagUsage gi="c" occurs=""6117""> . This was corrected.
2. One instance of an attribute value was found to be unquoted (file ohdr-en.tei): <availability status=restricted>. This was quoted, to conform to XML syntax.
3. The empty elements used (actually this is only <sym> ) were converted to XML syntax
4. The marked section discussed above was dropped — it has been assumed IGNORE and commented out
5. The last step of the conversion involved making a new TEI P4 XML DTD with PizzaChef
6. Finaly, the P4 corpus was validatated with the rxp validating parser

Summary and Evaluation

The conversion process turned out to be quite simple, partially due to the initial XML-like data encoding data. Still, it was surprising that the corpus contained mistakes in the application of XML conventions. This leads to the (probably obvious) conclusion that only a validating parser can guarantee syntactic well-formedness.

As noted above, our simple conversion via the emacs editor would not be an option with a project that contains a greater number of files and modifications to be performed. However, in our case it was an acceptable tool.

It would be difficult to say how long the migration process took with the original sample, as effort was made to answer the WG questionnaire, and to explore the various options offered for implementing the conversion. But performing the the subsequent migration on the whole MULTEXT-East corpus took only about two hours, and this includes the updating the corpus headers.

The XML-ized Concede edition of the MULTEXT-East corpus, together with additional resources, should be shortly released on the MULTEXT-East Web site, as V3.

About the Project

This collection of Middle English texts, begun in 1993, was assembled from works contributed by University of Michigan faculty and from texts provided by the Oxford Text Archive, as well as works created specifically for the Corpus by the Humanities Text Initiative (HTI). The HTI is grateful for the permission of all contributors. All texts in the collection are valid SGML documents, tagged in conformance with the TEI Guidelines and converted to the TEI Lite DTD for wider use.

The HTI intends to develop the Corpus of Middle English Prose and Verse into an extensive and reliable collection of Middle English electronic texts, either by converting the texts ourselves or by negotiating access to other collections produced to specified high standards of accuracy. HTI wants the corpus to include all editions of Middle English texts used in the Middle English Dictionary and the more recent scholarly editions, which in some cases may have superseded them.

About the Collection

At present, sixty-one texts are publicly available and more than sixty others will be coming on-line soon. Currently, 30 MiB of data are online; the expanded collection will be 88 MiB. Texts vary in size from 23 KiB to 6 MiB and are encoded in SGML using a P3 TEI Lite-based DTD.

The collection's audience is made up of scholars, students, and general readers interested in Middle English prose and verse. It is part of the Middle English Compendium but can stand on its own. The Corpus is provided with a full array of search mechanisms, so that texts may be searched individually, in user-designated groups, or collectively. In 2002, users from around the world conducted 30,000 searches, 9,000 browses, and 320,000 text views.

Nature of the Content

The content is primarily electronic versions of nineteenth-century print publications, but includes some original translations and transcriptions. The content encompasses poetry, drama, prose fiction and non-fiction. Only five of the texts currently delivered have associated external image files. All of the new texts have page images associated with them through image references encoded in attributes in the page break elements.

The base language of all the texts is Middle English, but other languages are represented, including French, German, Greek, Latin, and Old English.

Nature of the Encoding

The level of markup encoding is level 4, ‘basic content analysis,’ as described in the Digital Library Federation's TEI Text Encoding in Libraries: Guidelines for Best Encoding Practices. This level of encoding aspires to ‘create text that can stand alone as electronic text, identifies hierarchy and typography, specifies function of textual and structural elements, and describes the nature of the content and not merely its appearance. This level is not meant to encode or identify all structural, semantic or bibliographic features of the text.’ However, the texts predate the TEI in Libraries Guidelines and have several inconsistencies, such as unnumbered nested <div> s in front- and backmatter elements and division numbering that begins at <div0> .

The SGML is normalized and so does not make use of tag minimization or unquoted attributes.

Motivations for Migration

The main motivation behind the migration was to be able to take advantage of the many XML and XML-related tools and technologies that are now available, including our own locally-developed digital library middleware, which will provide more functionality than the current online implementation. In addition, we were taking the opportunity to add more material to the collection and to harmonize encoding practices that had changed (intentionally and unintentionally) over time. In addition to the division numbering practices mentioned above, in the earliest days of the HTI we had a number of text encoders who had worked on the Middle English Dictionary and had specialist interests in various aspects of the texts. Some taggers supplied missing text headings or corrected errors in editions with which they were familiar using different encoding practices.

Barriers to Migration

The largest barrier to migration was the encoding harmonization, which added unnecessary complications to an otherwise straighforward task, and the few unusual characters encountered in 19th-century publications of Middle English manuscripts. We did not find Unicode representations for all of the characters, some of which had locally invented chracter entities. These are displayed as [[entname]]. This is a continuing problem in encoding these older texts, where special characters are used and where even specialists are not in agreement about what exactly they mean. A recent example from EEBO is a G with two dots above, two dots below, and a dot on each side. It is clear in context that it is some unit of measurement, but a measure of how much?

About the Migration Samples

We chose the CME as our migration collection because it is one of our oldest locally-created collections; it contains a variety of genres, languages, and special characters; and it is small and publicly available (unlike the EEBO files we sent as examples to the migration team). In addition, we have a new batch of recently produced texts to add to the collection. As we will be adding to this collection after many years without growth, it seemed to be an ideal time to migrate.

Notes on the Conversion Process

We generally followed the procedures laid out in TEI migration guidelines. We used sx for general SGML to XML conversion, a locally developed stylesheet for encoding harmonization (e.g., renumbering nested divisions to start with <div1> ), and the tei2tei.xsl stylesheet to normalize element and attribute names and cases. As much as possible, character entities in the SGML files were replaced by their Unicode numeric values. A handful of locally-created character entities had to be handled with through creative substitution during the conversion process. We now have osx available, and it appears that non-Unicode entity issues could be avoided through use of this tool. This would have been helpful, especially as we are not yet prepared to index and search UTF-8 encoded files.

Summary and Evaluation

Aside from the analysis of variant encoding practice and non-standard character entities, and the creation of strategies to deal with them, it was a straightforward process. We were helped by the fact that we already use sx and XSLT in current practice, and hindered by an over-ambitious plan of work.

About the Project

The Electronic Text Center at the University of Virginia Library was founded in 1992. The Etext Center builds and maintains an on-line archive of tens of thousands of SGML and XML-encoded electronic texts; it is also a library service point that trains faculty, staff and students in the creation and analysis of electronic texts. In 1996, the Etext Center launched its Japanese Text Initiative (JTI). Founded in partnership with the University of Pittsburgh, the JTI uses an in-house team of specialists to create electronic versions of canonical Japanese literary texts.

About the Collection

The JTI currently contains several hundred titles, including Genji Monogatari: , Manyoshu, and other large multi-volume works. This collection serves an international audience, including many visitors from Japan and the Far East; only a tiny percentage of visitors come to the site from the University of Virginia.

Nature of the Content

The JTI collection includes poetry, drama, memoirs, and prose fiction, ranging in date from the medieval period up through the mid-twentieth century. The structure of the texts varies dramatically — the basic divisional unit can be anything from a single line of poetry to an entire chapter of prose. The collection contains many lengthy multi-volume anthologies, which are often are broken out into separate files for each volume (this was done to aid local processing and is something we'd eventually like to fix).

Nature of the Encoding

The JTI files are encoded in EUC, a Unix-based double-byte encoding scheme for Japanese characters. EUC is widely supported by web browsers and is also supported by OpenText, the search tool we currently use to index our data (neither OpenText nor XPat, its current iteration, supports Unicode). Many of the texts are actually keyboarded on Windows machines in Shift-JIS encoding but are converted to EUC when they are transferred the Unix server for delivery.

The level of encoding is consistent with Level 4 as described in the TEI in Libraries Guidelines (TEI Text Encoding in Libraries: Guidelines for Best Encoding Practices). In other words, we have encoded basic hierarchical and typographic features, but not all bibliographical or semantic features. In general, the prose works in the collection feature very light markup, while poetry and prose are encoded more thoroughly, with annotations, attributions, and more intricate structural features identified.

The JTI uses a straight TEI Lite DTD, with no extensions or modifications. In addition to the standard ISO character sets, the files use a lengthy catalog of SDATA entities to represent characters that are not present in the EUC code tables (mostly obsolete kanji appearing in the older texts).

As is generally the case at the Etext Center, the JTI has always produced very strict ‘XML-like’ SGML files — there's no tag minimization, attributes always appear in quotation marks, and elements are always in correct camelCase. None of the non-XML compliant SGML features (like SUBDOCS) are ever used.

Motivations for Migration

The University of Virginia Library is developing an XML-based integrated digital repository and the Etext Center will be expected to migrate all of its collections, including the JTI, into this new system. The new system will rely heavily on servlet-driven XSLT stylesheets for text dissemination, and although the indexing tool for this system has not yet been selected, it will most likely be incompatible with SGML. The migration process is especially useful for the JTI texts because it provides a good opportunity to convert the documents to Unicode, which contains a number of kanji not available in EUC and also allows us to combine Japanese characters with other non-Western character sets.

Barriers to Migration

Because the new digital repository is not yet in place, the transition will be slightly awkward and will probably require us to run SGML and XML systems simultaneously for some time. We are in fact running two parallel systems right now — an XML system for new materials that are being created in XML and a separate system for legacy data. Eventually the entire Etext Center collection of over 70,000 texts will have to be migrated, and it will be a challenge to find the resources to accomplish this without disrupting ongoing operations. The collections employ many different flavors of markup, including non-standard TEI-like materials that will be significantly harder to migrate due to unusual SGML-only features like inclusion exceptions.

In our current processing environment, texts are parsed when they are initially put online but are not often re-parsed after that point. Because we rely on non-validating tools like OpenText and Perl to deliver our texts, errors introduced by subsequent editing or other processes are sometimes not detected immediately. Thus we suspect that we will find a number of validation errors when we begin the conversion process, and because data that is flawed but usable in an SGML environment will break entirely in XML, the migration will probably necessitate substantial data cleanup. Obviously this cleanup process will be beneficial in its own right, but it will increase our migration costs in the short term.

About the Migration Samples

We selected the JTI texts for migration because they form a well-defined, reasonably sized subset of our holdings and they feature fairly typical markup. We also felt that these texts would benefit from a simultaneous conversion from EUC encoding to Unicode. The encoding scheme and the number of SDATA entities, including some entities representing non-Unicode characters, presented the biggest challenge.

Notes on the Conversion Process

We consulted the earlier SGML-XML migration guidelines on the TEI website, and used the batch script provided there as a starting point. We were initially unable to compile osx under AIX or Solaris, but once we finally managed to install it on a Windows machine using cygwin we transferred all our files over to a PC to begin the conversion. We made a few changes to the script (now reflected in MIW03) to take advantage of some newly enhanced osx features, and we had to tweak the tei2tei.xsl stylesheet to get it to remove default attributes properly. Then we began debugging by running files through the script one at a time.

By far, the biggest headache was wrestling with the EUC encoding. The current version of osx handles this encoding unpredictably. The standard switch for EUC, -beuc-jp, produces corrupted output, as does running the program with no -b switch at all. We were finally successful with the -beuc switch, which ironically triggers an error message yet produces correct EUC output. (We also tried running the files through a command-line EUC to UTF-8 converter and feeding the UTF-8 output into osx, and this caused osx to blow up with ‘non-SGML character’ errors. We assume that osx can handle Unicode input but were unable to make this work on our samples.)

Applying the stylesheet caused further encoding problems, as neither of the XSLT processors we tried (Saxon and XT) would accept EUC encoding. We finally added an EUC to UTF-8 conversion step to the batch script, just before the stylesheet process, and were able to get correct Unicode output.

Once we perfected the script for individual files, we modified it further for batch processing. We wrote a Perl wrapper to pop the declaration and internal subset off the SGML file and replace it with a modified version after processing. The Perl wrapper also concatenates the external entities file generated by osx after every run; when the batch is complete, it pipes these entities through sort and uniq to generate a single external file containing all the non-Unicode Japanese character entities. (Prior to converting the files, we identified all Japanese characters available in Unicode but not in EUC, and after the conversion we replaced those entity references with Unicode values. We still have no satisfactory way to render those characters not available in Unicode. We currently use small in-line GIF images but we may explore the possibility of using the Unicode PUA.)

We were unable to resolve the problem of attribute normalization. While we use a camelCase ID scheme to identify our files, osx maps all attribute values to lowercase.

Summary and Evaluation

Altogether, we spent about a day converting this sample of approximately 500 files, from installing the software to making the final modifications to the batch script. Running an XML parser on the output files revealed only a small handful of validation errors, which were easily fixed. We expect that we will have a much easier time converting our English language collections, because the encoding scheme will not be an issue (although our experience with the Japanese texts will come in handy when we convert our Chinese and other non-Western collections). In general, we found the migration process to be reasonably straightforward and well worth the effort.

About the Project

The Thomas MacGreevy Archive was begun in 1996 as a long-term, interdisciplinary research project committed to exploring the intersections between traditional humanities research and digital technologies. It was begun with grants from The Newman Scholarship Fund at University College Dublin, and Enterprise Ireland. It is published by The Institute for Advanced Technology in the Humanities (University of Virginia), and supported by Maryland Institute for Technology in the Humanities (University of Maryland). The General Editor of the Archive is Susan Schreibman.

About the Collection

The Archive makes available the works of Thomas MacGreevy (1893–1967), Irish poet, art and literary critic, and Director of the National Gallery of Ireland (1950–63). To date, it has focused on publishing MacGreevy's published articles and books, all of which are out of print. It has also reprinted some 20 articles by contemporary critics on MacGreevy and his circle.

The Archive currently publishes some 350 texts encoded in SGML which are delivered via DynaWeb. Texts range from about 125 KiB (book-length texts) to around 6 KiB (review articles of a few paragraphs). The audience for the collection is scholars interested in Thomas MacGreevy, the subjects he wrote about (Irish art, literature and culture, modernist literature and art, for example), and the people he associated with (including Samuel Beckett, James Joyce, Wallace Stevens, W.B. and Jack Yeats).

Nature of the Content

To date, the genres of the texts we have encoded are monographs and periodical articles. The base language for a majority of the texts (about 95%) is English, but many other languages are encoded using TEI's <foreign> element, including French, German, Greek, Irish, Italian, Latin, and Spanish. We capture textual features such as original pagination, emphasised words (in bold and italic, for example), and typographic features (such as capitalised words and small caps). About 10% of the texts have images associated with them.

In addition to encoding pre-existing texts, Susan Schreibman has created a bibliography, which does not exist in any other form, of the complete writings by and about Thomas MacGreevy following the MacGreevy Archive DTD. This bibliography forms the basis of the Archive's ‘Browse’ mode.

Nature of the Encoding

According to the TEI Text Encoding in Libraries: Guidelines for Best Encoding Practices, the level of encoding is scholarly. We not only capture the structural features of the texts, but also encode and regularise personal and association names. In addition, names are editorially supplied when appropriate. All the texts in the Archive have been encoded according to documentation developed specifically for the project (the documentation can be made available upon request).

The DTD the Archive uses is based on TEI's prose base tagset, and includes the following auxiliary tagsets: TEI.certainty, TEI.figures, TEI.linking, TEI.names.dates, TEI.textcrit, and TEI.transcr. It includes the following entity sets: ISOgreek, ISOLatin, ISOpub, and ISOpen.

Since project participants have always encoded using an SGML editor, such as XMetaL (and Author/Editor before that), the data was relatively XML-compliant in that we never utilised tag minimization or unquoted attributes.

Barriers to Migration

For us, the barrier to migration was not the actual conversion process. That was relatively easy as our encoding was so uniform. We spent the most time learning to use the tools and understanding what scripts were needed to preserve entity references and convert element names to camelCase, etc. It will require more time, however, to complete the rest of the conversion process and migrate the data from a SGML publishing system (DynaWeb) to one in XML. This will require programming resources for remounting the Archive, designing new stylesheets (in XSLT), search and browse functionality, and adapting the HTML wrapper.

About the Migration Samples

We migrated the entire the MacGreevy Archive, currently encoded in SGML and available through DynaWeb at IATH. There are approximately 100 more texts in various stages of encoding/proofing. These are still being encoded in SGML (so they can go live when they are ready in DynaWeb). When the new XML database is ready, we will perform another batch file to convert the rest of the SGML texts into XML.

We did not face any particular challenges that scripting could not handle.

Notes on the Conversion Process

For the most part, we followed the procedures laid out in the earlier TEI Migration Committee Guidelines. The steps we took are outlined below:

• Commented out the doctype and DTD declaration
• Commented out entity declarations
• Renamed entities by searching and replacing the ‘&’ with ‘||’
• Used the following command to make element names lower case and do the actual conversion to XML: osx -xlower -xcomment -xempty -xndata
• Used the following script to change element names to camelCase: java com.icl.saxon.StyleSheet $$.xml tei2tei.xsl
• Restored entities by replacing the ‘||’ with ‘&’.

We also took the opportunity to rethink our DTD while creating a new XML DTD from the TEI Pizza Chef. We had, for example, modified our <p> element to allow it to appear after the <closer> tag in order to accommodate postscripts in letters. This modification was unnecessary in XML, since XSLT could easily reposition a <p> element with an attribute value of <postscript> to follow the letter's closing information.

Summary and Evaluation

In our case, the actual conversion process was relatively painless. It took about five hours of my and a programmer's time. The vast majority of that time was spent in understanding how osx works and finding and testing the scripts for preserving entities, converting element names, etc. Once we understood the process, the actual conversion took about 45 minutes for 350 texts. However, migrating the documents to XML is only the beginning of the process. As mentioned earlier, the entire Archive must be migrated into a new XML-based document delivery system, which will take additional time and resources.

About the Project

Documenting the American South (DocSouth), an electronic publishing initiative of the University of North Carolina at Chapel Hill Library, was launched on UNC's Sunsite in October of 1996. Started as a modest effort to make publicly available the full text of a dozen highly circulating slave narratives, it grew into a serious library electronic publishing endeavor. As of May 2004, the site comprises around 170,000 pages (ca. 1,300 titles) of digitized primary source materials related to Southern history, literature, and culture from the colonial period through the first decades of the 20th century. In addition, it includes hundreds of pages of supporting materials; over 12,000 images, including 100 World War I posters; and several oral history interviews and songs from the University of North Carolina at Chapel Hill and other libraries collections.

DocSouth supplies teachers, students, and researchers at every educational level with a wide array of titles they can use for reference, study, instruction, and research.

All the digitized texts receive individual full-level MARC catalog records, created through the OCLC system. These records are added to the UNC online catalog for each text as the project progresses and are available through the OCLC database and OCLC's WorldCat service. The Library is committed to the long-term maintenance of all materials digitized for DocSouth and to the unrestricted free access for all users to all parts of the project.

About the Collection

Total size of the collections—over 3,500,000 KiB

We clearly underestimated our potential audience at the beginning of our endeavor. Having initially targeted college campuses, we were pleasantly surprised to discover that we were reaching a much larger universe of readers, ranging from elementary school students and home schoolers to Hollywood screen writers and genealogists.

The DocSouth readership is already world-wide and is still growing. The most meaningful indicators of its present usefulness and its potential future are the reader comments and questions. They reveal a gratifying multiplicity of uses, especially by those who would not otherwise know that such resources even exist. Selected readers' comments and responses to DocSouth materials can be read in Keep Up the Good Work(s). Readers Comment on Documenting the American South , selected and with a preface by Dr. Joe A. Hewitt, Associate Provost for University Libraries. These comments are a powerful testimony of the services that digitized collections provide to millions who would otherwise not be able to use these resources without coming to the libraries, archives, museums, and repositories where they are stored and preserved.

Nature of the Content

DocSouth includes mainly 19th-century and early 20th-century published texts, with a few titles published in the 17th and 18th-centuries. The most common genres include autobiographies, biographies, essays, travel accounts, poetry, diaries, letters, and memoirs.

Text files include all pictorial materials within a text, e.g., covers, spine, frontspiece, title page, verso, and illustrations. In addition, several manuscripts (diaries and letters) have page images associated with and linked to transcriptions.

DocSouth also includes an ‘image-based’ component of 100 posters and several artifacts from the WWI period, as well as a small audio collection of oral history interviews and songs. We plan to expand these sections in the near future.

All the titles are in English. Occasional occurrence of other languages (mostly European) are marked with an appropriate ID in the <language> element.

Nature of the Encoding

All the texts are encoded with 3P TEI Lite (version 1.6)DTD. No extensions have been used.

All the texts in the DocSouth collections followed in-house guidelines written specifically for Author/Editor, a commercially available SGML-aware software ² . These guidelines are based on recommendations for ‘basic content analysis’ (level 4) in the Digital Library Federation's TEI Text Encoding in Libraries: Guidelines for Best Encoding Practices and contain detailed descriptions of all tags used in DocSouth collections, along with several examples from texts. All DocSouth encoded titles can be distributed as stand-alone texts.

In addition, we preserve such textual features as original pagination and emphasized words (e.g., bold, italic).

[Modification of the dtd.] As mentioned above, several manuscript materials (letters and diaries) have page images associated and linked to the transcription. For this purpose, an ENTITY attribute was added to <pb> element (examples provided with Migration Samples). This issue, i.e. linking page images to page transcriptions, was raised many times in the TEI community and requires a close attention on part of the TEI-C.

Motivations for Migration

Barriers to Migration

The conversion per se did not present a major issue, as we had uniform encoding practices implemented since 1997 and a large body of legacy data. Several problems have been encountered with texts encoded before 1997. Those were mainly problems with inconsistency with encoding and with not implementing strict validation tools. Besides that, we needed extra time to learn how to use several new tools, to write scripts and make them work.

That said, I will mention barriers that many institutions encounter when launching any new initiative: time constraints; staffing issues, and a previously approved work plan facing a deadline. In our case, we had to work on several concurrent projects that could not be postponed, as they were funded by federal agencies and had to be finished on time. Finally, we had already in place a well working, approved by time, system that we would need to change, at least partially.

About the Migration Samples

Notes on the Conversion Process

The work on the conversion has been done in two stages.

The second and final conversion took place some time at the end of 2003. It happened that this forced delay in our effort to convert DocSouth collections to xml came to our advantage—by that time, a new improved osx version was in place and the Migration WG finalized its recommendations.

We wrote a shell script cnvrt.bash that included several steps of the conversion process outlined in the WG recommendations. The script can run either in batch mode or on individual files.

Summary and Evaluation

In summary, the conversion process was straightforward in most cases and presented a few already mentioned problems. However, people have to realize that it is not a completely trivial undertaking that might take some time to learn new tools and to make scripts work.

In our case, between 96 and 97% of the texts represent a large collection of standardized texts that have been encoded according to the DocSouth in-house encoding guidelines. The remaining 3–4% (approx. 50 texts) were encoded earlier (1994–1995) and were checked for conformance with existing in-house encoding practices, before they were converted to XML. That process of checking for conformance, validating old files and correcting encountered validation errors took much more time than we have initially planned.

Finally, as was discussed above, since our texts use a standard subset of the TEI Guidelines (3P TEI Lite, version 1.6) we did not have to migrate our DTDs and we could use the prepackaged TEI XML DTD (teixlite.dtd).

About the Project

The Victorian Women Writers Project began in the spring of 1995 to produce highly accurate transcriptions of works by British women writers of the 19th century. The works, selected with the assistance of the Advisory Board, include anthologies, novels, political pamphlets, religious tracts, children's books, and volumes of poetry and verse drama. The project is supported by the Digital Library Program and Library Electronic Text Resource Service (LETRS) at Indiana University. The project's General Editor is Perry Willett, who is currently Head of the Digital Library Production Services at University of Michigan.

About the Collection

The collection comprises over 150 works by more than forty writers, including Mathilde Blind, Mary Elizabeth Braddon, Caroline Norton, and Lady Jane Wilde. The collection contains 187 files, totalling about 50 MiB in size.

The texts are encoded in SGML using the P3 TEI Lite DTD. The audience for the collection is scholars, students, and general readers interested in Victorian literature and culture.

Nature of the Content

The content is varied and represents a number of genres, including poetry, drama, fiction, and non-fiction prose. About forty percent of the texts have external image files associated with them. The base language of all the texts is English, but other languages, including French, German, Greek, Italian, and German, are represented.

Nature of the Encoding

The character encoding is ASCII; non-ASCII characters are represented by entities declared in the internal DTD subset, rather than in external entity files.

The level of markup encoding is level 4, "basic content analysis," as described in the TEI Text Encoding in Libraries: Guidelines for Best Encoding Practices document (on-line at http://www.diglib.org/standards/tei.htm). This level of encoding aspires to "create text that can stand alone as electronic text, identifies hierarchy and typography, specifies function of textual and structural elements, and describes the nature of the content and not merely its appearance. This level is not meant to encode or identify all structural, semantic or bibliographic features of the text."

The SGML is normalized and so does not make use of tag minimization or unquoted attributes.

Motivations for Migration

The main motivation behind the migration was to be able to take advantage of the many XML and XML-related tools and technologies that are now available.

Barriers to Migration

There were no significant barriers to migration.

About the Migration Samples

We chose the Victorian Women Writers project as a migration sample because it represents a moderately sized collection with a fair bit of variation in terms of genres. It is also our oldest self-produced collection.

Although the collection represents a number of different genres, it did not pose any particular challenges.

Notes on the Conversion Process

We followed the procedures laid out in the earlier TEI migration guidelines. We used osx for general SGML to XML conversion, and the tei2tei.xsl stylesheet to convert to camelCase. Internal entity declarations from the SGML files were replaced in the XML files with parameter entities pointing to external entity files with standard entity declarations that expand to the XML-style Unicode character entity. For instance, &trade; expands to &#x2122;.

The conversion was not difficult, and after conversion and the insertion of a new DOCTYPE declaration, with pointers to an XML TEI Lite DTD and the appropriate external entity files, all the files validated. The validation process occasionally revealed erroneous markup, such as the unnecessary use of an entity reference for a common character, such as a colon or hyphen. These few errors were fixed manually.

Summary and Evaluation

The process was very straightforward. It took one afternoon to convert 187 files. A fortunate byproduct of the conversion process was the detection and correction of a few errors or bad markup practice.

About the Project

The Thesaurus musicarum italicarum is a corpus of digitized sources of Italian music theory dating from before the end of the eighteenth century. The emphasis lies on works originating in the sixteenth and early seventeenth century. The TMI was initiated in 1995 by the former Department of Computers and Humanities at Utrecht University (which merged into the Institute of Information and Computing Sciences in 1999). As an ongoing project, the TMI is now also sustained by a group of researchers from several European countries and the US. Due to lack of funding, work on TMI has slowed down since 2001, but we hope to reactivate the project in the not too distant future.

The TMI comes in two flavours, a website (TmiWeb) and a (small) series of CD-ROM publications. In TmiWeb, treatises are available in a straightforward presentation format. Mark-up is relatively simple and optimized for searching and browsing, and there are no facsimiles. Part of the content of TmiWeb is work in progress, or, ideally, dynamic by character. The CD-ROM format is chosen for editions that have more elaborate and stable markup. There are more presentation options and, to allow verification, sources are also available in facsimile.

We employ DynaWeb 4.3 as a server for TmiWeb. The software used for our first CD-ROM was the Panorama Viewer; the second one uses DynaText 4.3.

About the Collection

Thirty treatises have been published in the TMI so far; the total file size of the published treatises is about 18 MiB of SGML. Some 20 more treatises are in various stages of preparation. The audience for the TMI consists primarily of students and researchers of music (history, theory, historical performance), as well as those studying or researching the history of science (especially mathematics and physics), the history of ideas, and languages.

Nature of the Content

Music treatises are generally prose monographs. Some are in verse, in dialogue form, or consist of a collection of letters. The basic language is Italian, with considerable regional influence (Toscan, Venetian, Neapolitan). There are many terms and quotations in Latin and Greek, and some in Hebrew. The size of the treatises ranges from 10–450 pages in TMI; both bigger and smaller examples are known to exist. Interesting features abound, including pagination errors, page headers from another book, wrong page order, errata lists with new mistakes, indices with entries pointing to non-existing pages, etc. More interesting are multiple editions of treatises (there are four editions of Pietro Aaron's Toscanello, encoded as one SGML instance. One of these contains numerous marginal notes and illustrations (!) by a sixteenth-century reader.

Nature of the Encoding

The level of encoding (as described in the Digital Library Federation's TEI Text Encoding in Libraries: Guidelines for Best Encoding Practices) is 3, 4, or 5, depending on the treatise. Several treatises on the website clearly fall in level 3. On the other hand, treatises prepared for CD-ROM publication have extensive text-critical markup (variants, corrections, normalisation) and features such as personal names, cross-references, and citations of books and music.

The texts are XML-like in several respects. Tag minimization techniques are not used, and all attribute values are quoted. Element names are usually not in camelCase: they tend to be either completely in lower case or in upper cases. There are many references to SDATA entities.

Motivations for Migration

Barriers to Migration

About the Migration Samples

The samples I submitted to the samples directory are Toscanello: and Trattato... di tutti gli tuoni: , both by Pietro Aaron, and Gioseffo Zarlino's Le istitutioni harmoniche: . The Trattato... di tutti gli tuoni: is pretty simple. The Toscanello: is very complex, as it contains full textcritical markup, including variants from the four existing editions of the work. Le istitutioni harmoniche: is just big, full of illustrations, music examples, SDATA entity references, personal names, and links to other works; and is generally rich in markup. For testing purposes, I've used only Aaron's Trattato delle mutationi, which is one of the simplest texts we have.

Challenges are SDATA and DTD modifications.

Notes on the Conversion Process

Summary and Evaluation