P5: Recommandations pour l'encodage et l'échange de textes électroniques

Conceptually, the simplest method of disentangling two (or more) conflicting hierarchical views of the same information is to encode it twice (or more), each time capturing a single view.

This method is TEI Conformant. Its advantages are that each way of looking at the information is explicitly represented in the data and that the individual views are simple to process. The disadvantages are that the method requires the maintenance of multiple copies of identical textual content (an invitation to inconsistency) and that there is no explicit indication that the various views, which might be in separate files, are related to each other: it might prove difficult to combine the views or access information from one view while processing the file that contains the encoding of another.⁸⁰

A second method for accommodating non-hierarchical objects in an XML document involves marking the start and end points of the non-nesting material. This prevents textual features that fall outside the privileged hierarchy from invalidating the document while identifying their beginnings and ends for further processing. The disadvantage of this method is that no single XML element represents the non-nesting material and, as a result, processing with XML technologies is significantly more difficult.

The empty elements used at each end are called segment-boundary elements or segment-boundary delimiters. There are several variations on this method of encoding.

The use of these elements is by definition TEI Conformant. Care should be taken, however, that the meaning of the milestone elements is preserved: semantically, for example, lb is used to mark the start of a new (typographical) line. While in much modern poetry, typographical and metrical line divisions correspond, lb does not itself make a metrical claim: in encoding verse from sources, such as Old English manuscripts, where physical line breaks are not used to indicate metrical lineation, the correspondence would break down entirely.

This method is TEI Conformant.

If the custom elements can be replaced by TEI elements and attributes without loss of information, this method is TEI Conformable (see 23.4 Conformance); if the custom elements introduce information or distinctions that cannot be captured using standard TEI elements, the method is an extension.

In each of the above examples (except the last), the relationship between the start and end delimiters (where these exist) of a given feature is implicit: it is assumed that "end" delimiters close the nearest preceding "start" delimiter, or, in the case of milestones, that the milestone marks both the end of the preceding example and the beginning of the next. Complications arise, however, when the non-nesting text overlaps with other non-nesting text of the same type, as, for example, in a grammatical analysis of the various possible interpretations of the noun phrase fast trains and planes. In this case, the adjective fast can be understood as either modifying trains and planes or just trains:

In this encoding, the first interpretation, in which fast modifies the NP trains and planes, the NP trains and planes is opened using an anchor tag with the xml:id value NPInterpretationA and closed with an anchor with the same value on corresp; in the second interpretation, in which fast forms a NP with trains, the NP fast cars is opened using an anchor tag with the xml:id value NPInterpretationB and closed with an anchor tag that has the same value on corresp.

Despite their advantages, segment boundary delimiters incur the disadvantage of cumbersome processing: since the elements of the analysis (e.g., the sentences in the poems, or phrases in the above example) are not uniformly represented by nodes in the document tree, they must be reconstituted by software in an ad hoc fashion, which is likely to be difficult and may be error prone.

Most important for some encoders, the method also disguises the relationship between the beginning and the ending of each logical element. This makes it impossible for standard validation software to provide the same kind of validation possible elsewhere in the encoding. When using grammar-based schema languages it is not possible to define a content model for the range limited by empty elements.⁸¹

A third method involves breaking what might be considered a single logical (but non-nesting) element into multiple smaller structural elements that fit within the dominant hierarchy but can be reconstituted virtually. For example, if a passage of direct discourse begins in the middle of one paragraph and continues for several more paragraphs, one could encode the passage as a series of said elements, each fitting within a p element. The resulting encoding is valid XML, but the text in each said element represents only a portion of the complete passage of direct discourse. For this reason these elements are sometimes called ‘partial elements’.

There are two main problems with this type of encoding. The first is that it invariably means that the encoding will have more elements claiming to represent a feature than there are actual instances of that feature in the text. Thus, for example, the passage from Scorn not the sonnet marks seven spans of text using seg, even though there are only four linguistic sentences in the passage.

The second problem is that it can be semantically misleading. Although they are tagged using the element for sentence, for example, very few of the textual features encoded using seg in this example represent actual linguistic sentences: with this key, for example, is a prepositional phrase, not a sentence; Of this small lute gave ease to Petrarch's wound is a string corresponding to no single grammatical category.

Taken together, these problems can make automatic analysis of the fragmented features difficult. An analysis that intended to count the number of sentences in Wordsworth's poem, for example, would arrive at an inflated figure if it understood the seg elements to represent complete rhetorical sentences; if it wanted to do an analysis of his syntax, it would not be able to assume that seg delimited linguistic sentences.

This use of join is TEI Conformant.

The major advantage of fragmentation and virtual joins is that it allows all the hierarchies in the text to be handled explicitly: both the privileged one directly represented and the alternate hierarchy that has been split up and rejoined. The major disadvantages are that (like most of the other methods described here) it privileges one hierarchy over the others, requires special processing to reconstitute the elements of the other hierarchies, and, except in the case of join, can be semantically misleading.

This is very similar to the use of join discussed above. The main advantages of the stand-off method are that it is possible to specify attributes on the aggregate seg elements, and that there exists off-the-shelf software that will perform appropriate processing. Stand-off markup may be used even when the base text being annotated is plain text, i.e. does not have any XML encoding. In this case, the range of text to be marked up is indicated by character offsets (see 16.2.5 TEI XPointer Schemes, in particular 16.2.5.5 string-range(fragmentIdentifier, offset [, length])). Another distinction concerns the number of files which can serve as link targets. Often, one (dedicated) annotation is used as the link target of all the other annotations. It is also possible to freely interlink several layers.

It has been noted that stand-off markup has several advantages over embedded annotations. In particular, it is possible to produce annotations of a text even when the source document is read-only. Furthermore, annotation files can be distributed without distributing the source text. Further advantages mentioned in the literature are that discontinuous segments of text can be combined in a single annotation, that independent parallel coders can produce independent annotations, and that different annotation files can contain different layers of information. Lastly, it has also been noted that this approach is elegant.

But there are also several drawbacks. First, new stand-off annotated layers require a separate interpretation, and the layers—although separate—depend on each other. Moreover, although all of the information of the multiple hierarchies is included, the information may be difficult to access using generic methods.

Inasmuch as it uses elements not included in the TEI namespace, stand-off markup involves an extension of the TEI.

There exist many non-XML methods of encoding a text that either solve or do not suffer the problem of the inability to encode overlapping hierarchies. These include, but are not limited to, the following proposals.

• Applying the notion of concurrent markup to XML (Hilbert et al. (2005)). This reintroduces the CONCUR feature of SGML, which was omitted from the XML specification.
• Designing a form of document representation in which several trees share all or part of the same frontier, and in which each individual view of the document has the form of a tree (see Dekhtyar and Iacob (2005)).
• The ‘colored XML’ proposal (Jagadish et al. (2004)), which stores a body of information as a set of intertwined XML trees. This approach eliminates unnecessary redundancy and makes the database readily updatable, while allowing the user to exploit different hierarchical access paths.
• The MultiX proposal (Chatti et al. (2007)) , which represents documents as directed graphs. Because XML is used to represent the graph, the document is, at least in principle, manipulable with standard XML tools.
• The Just-In-Time-Trees proposal (Durusau and O'Donnell (2002)), which stores documents using XML, but processes the XML representation in non-standard ways and allows it to be mapped onto data structures that are different from those known from XML.
• The Layered Markup and Annotation Language (LMNL) proposal. This offers alternatives to the basic XML linear form as well as its data and processing models. It uses an alternative notation to XML and a data structure based on Core Range Algebra (Tennison and Piez (2002)).
• Markup Languages for Complex Documents (MLCD). This provides a notation (TexMECS) and a data structure (Goddag) as well as a draft constraint language for the representation of non-hierarchical structures; see Huitfeldt and Sperberg-McQueen (2001).

These approaches are based either on non-standard XML processing or data models, or not based on XML at all. Since TEI is currently based on XML they are not described any further in these Guidelines. Use of these methods with the TEI will certainly involve extensions; in most cases the documents will also be non-conformant.