P5: Linee guida per la codifica e l'interscambio del testo elettronico

In section 3.6 Simple Links and Cross-References we introduced the simplest pointer elements, ptr and ref. Here we introduce additionally the link element, which represents an association between two (or more) locations by specifying each location explicitly. Its own location is irrelevant to the intended linkage. All three elements use the attribute target, provided by the att.pointing class as a means of indicating the location or locations referenced or pointed to.

• att.pointing definisce un insieme di attributi usati da tutti gli elementi che puntano ad altri elementi tramite uno o più URI

  target

  specifica la destinazione di un riferimento, fornendo uno o più riferimenti URI

• link/ definisce tra elementi o porzioni di testo un'associazione o legame ipertestuale non meglio specificabile da altri elementi

The ptr element may be called a ‘pure pointer’, because its primary function is simply to point. A pointer sets up a connection between an element (which, in the case of a pure pointer, is simply a location in a document), and one or more others, known collectively as its target. The ptr and ref elements point, conceptually, at a single target, even if that target may be discontinuous in the document. The link element specifies at least two targets and represents an association between them, independent of its own location.

These three elements also share a common set of attributes, derived from the att.pointing and att.typed classes:

• att.pointing definisce un insieme di attributi usati da tutti gli elementi che puntano ad altri elementi tramite uno o più URI

  evaluate

  indica il significato previsto quando la destinazione di un puntatore è essa stessa un puntatore

• att.typed assegna degli attributi generici utilizzabili per qualsiasi classificazione e sottoclassificazione di elementi

  type	caratterizza l'elemento utilizzando una classificazione o tipologia funzionale
  subtype	stabilisce, se necessario, una sottocategorizzazione dell'elemento

As noted elsewhere, the target attribute may take as value one or more URI reference. In the simplest case, each such reference will indicate an element in the current document (or in some other document), for example by supplying the value used for its global xml:id attribute. It may however carry as value any form of URI, such as a URL pointing to some other document or location on the Internet. Pointing or linking to external documents and pointing and linking where identifiers are not available is described below in section 16.2 Pointing Mechanisms.

As an example of the use of mechanisms which establish connections among elements, consider the practice (common in 18th century English verse and elsewhere) of providing footnotes citing parallel passages from classical authors.

Such footnotes can of course simply be encoded using the note element (see section 3.8 Notes, Annotation, and Indexing) without a target attribute, placed adjacent to the passage to which the note refers:⁵²

<l>(Diff'rent our parties, but with equal grace</l>
<l>The Goddess smiles on Whig and Tory race,</l>
<l>
 <note type="imitation" place="bottom" anchored="false">
  <bibl>Virg. Æn. 10.</bibl>
  <quote>
   <l>Tros Rutulusve fuat; nullo discrimine habebo.</l>
   <l>—— Rex Jupiter omnibus idem.</l>
  </quote>
 </note>'Tis the same rope at sev'ral ends they twist,
</l>
<l>To Dulness, Ridpath is as dear as Mist)</l>

bibliography 

This use of the note element can be called implicit pointing (or implicit linking). It relies on the juxtaposition of the note to the text being commented on for the connection to be understood. If it is felt that the mere juxtaposition of the note to the text does not make it sufficiently clear exactly what text segment is being commented on (for example, is it the immediately preceding line, or the immediately preceding two lines, or what?), or if it is decided to place the note at some distance from the text, then the pointing or the linking must be made explicit. We now consider various methods for doing that.

Clearly, there are many reasons for which an encoder might wish to represent a link or association between different elements. For some of them, specific elements are provided in these Guidelines; some of these are discussed elsewhere in the present chapter. The link element is a general purpose element which may be used for any kind of association. The element linkGrp may be used to group links of a particular type together in a single part of the document; such a collection may be used to represent what is sometimes referred to in the literature of Hypertext as a web, a term introduced by the Brown University FRESS project in 1969, and not to be confused with the World Wide Web.

• linkGrp (gruppo di collegamenti) definisce una raccolta di associazioni o legami ipertestuali

As a member of the class att.pointing.group, this element shares the following attributes with other members of that class:

• att.pointing.group definisce un insieme di attributi comuni a tutti gli elementi che includono gruppi di puntatori

  domains	stabilisce facoltativamente gli identificatori di elementi all'interno dei quali si trovano tutti gli elementi indicati dai contenuti dell'elemento in questione
  targFunc	(funzione obiettivo) descrive la funzione di ognuno dei valori dell'attributo targets dei marcatori corrispondenti link, join, o alt

It is also a member of the att.pointing and att.typed classes, and therefore also carries the attributes specified in section 16.1.1 Pointers and Links above, in particular the type attribute.

The linkGrp element provides a convenient way of establishing a default for the type attribute on a group of links of the same type: by default, the type attribute on a link element has the same value as that given for type on the enclosing linkGrp.

Note that there must be a single parent element for each ‘domain’; if some notes are contained by a section with identifier dunnotes, and others by a section with identifier dunimits, an intermediate pointer must be provided (as described in section 16.1.4 Intermediate Pointers) within the linkGrp and its identifier used instead.

In the preceding examples, we have shown various ways of linking an annotation and a single verse line. However, the example cited in fact requires us to encode an association between the note and a pair of verse lines (lines 284 and 285); we call these two lines a span.

There are a number of possible ways of correcting this error: one could use the target attribute to indicate one end of the span and the special purpose targetEnd attribute on the note element to point to the other. Another possibility might be to create an element which represents the whole span itself and assign that an xml:id attribute, which can then be linked to the note and ref elements. This could be done using for example the lg element defined in section 3.12.1 Core Tags for Verse or the ‘virtual’ join element discussed in section 16.7 Aggregation.

The all value of evaluate is used on the link element to specify that any pointer encountered as a target of that element is itself evaluated. If evaluate had the value none, the link target would be the pointer itself, rather than the objects it points to.

Where a linkGrp element is used to group a collection of link elements, any intermediate pointer elements used by those link elements should be included within the linkGrp.

Like the ubiquitous if misnamed XHTML pointing attribute href, the TEI pointing attributes can point to a document that is not the current document (the one that contains the pointing element) whether it is in the same local filesystem as the current document, or on a different system entirely. In either case, the pointing can be accomplished absolutely (using the entire address of the target document) or relatively (using an address relative to the current base URI in force). The ‘current base URI’ is defined according to Marsh 2001. In general the current base URI in force is the value of the xml:base attribute of the closest ancestor that has one. If there is none, the base URI is that of the current document.

xml:base is a useful way of handling the repeated use of long external URIs. However, it is less convenient when your text contain many references to a variety of different sources in different locations. Even in the case of relative links on the local file system, ref or target attributes may become quite lengthy and make XML code difficult to read. To deal with this problem, the TEI provides a useful method of using abbreviated pointers and documenting a way to dereference them automatically.

Imagine a project which has a large collection of XML documents organized like this:

• anthology
  • poetry
    • poem.xml
  • prose
    • novel.xml
• references
  • people
    • personography.xml

Through this mechanism, any processor which encounters a data.pointer with a protocol unknown to it can check the listPrefixDef in the header to see if there is an available expansion for it, and if there is, it can automatically provide the expansion and generate a full or relative URI.

When creating private URI schemes, it is recommended that you avoid using any existing registered prefix. A list of registered prefixes is maintained by IANA at http://www.iana.org/assignments/uri-schemes.html.

Note that this mechanism can also be used to dereference other abbreviated pointing systems which are based on prefixes, such as Tag URIs.

The matchPattern and replacementPattern attributes are also used in dereferencing canonical reference patterns, and further examples of the use of regular expressions are shown in 16.2.6 Canonical References.

If elements are not directly addressable by means of an identifier, because no identifier was originally given to them and the document cannot be modified to add one, they may still be pointed to by means of their position in the XML element tree. This method of pointing uses the element() scheme defined by the World Wide Web Consortium (Grosso et al, 2003). In this scheme, an element may be identified by stepwise navigation using a slash-separated list of child element numbers. For each step the integer n locates the nth child element of the previously located element. Thus a pointer such as <ptr target="foo.xml#element(/1/4)"/> indicates the fourth child element starting from the root element of the document indicated by the URI foo.xml.

As noted above, we could also point directly to this line if it had an identifier of its own. In another digital edition of Shakespeare, based on the first folio, each line is given an identifier based on its ‘through line number’. Our pointer to this line can now be represented simply as <ptr target="#element(Ham01245)"/>, or even more simply as <ptr target="#Ham01245"/>. The notation <ptr target="#xxx"/> is a convenient abbreviation for <ptr target="#element(xxx)"/>. This method requires, of course, that the ‘Through Line Number’ is supplied as the value of an xml:id attribute on each line, and must therefore be unique within each document. In section 16.2.6 Canonical References we discuss a method of pointing to the line which does not have this requirement.

The pointing scheme described in this chapter is one of a number of such schemes envisaged by the W3C, which together constitute a framework for addressing data within XML documents, known as the XPointer Framework (Grosso et al 2003). This framework permits the definition of many other named addressing methods, each of which is known as an XPointer Scheme. The W3C has predefined a set of such schemes, and maintains a register for their expansion. The element() scheme described above is one such scheme, defined by the W3C, and widely implemented by XML processing systems.

Another important scheme, also defined by the W3C, and recommended by these Guidelines is the xpath1() pointer scheme, which allows for any part of an XML structure to be selected using the syntax defined by the XPath specification. This is further discussed below, 16.2.5.2 xpath1(Expr). These Guidelines also define five other pointer schemes, which provide access to parts of an XML document such as points within data content or stretches of data content. These additional TEI pointer schemes are defined in sections 16.2.5.3 left() and right() to 16.2.5.6 match(fragmentIdentifier, string [, index]) below.

By ‘canonical’ reference we mean any means of pointing into documents, specific to a community or corpus. For example, biblical scholars might understand ‘Matt 5:7’ to mean ‘the book called Matthew, chapter 5, verse 7.’ They might then wish to translate the string ‘Matt 5:7’ into a pointer into a TEI-encoded document, selecting the element which corresponds to the seventh div element within the fifth div element within the div element with the n attribute valued ‘Matt.’

Several elements in the TEI scheme (gloss, ptr, ref, and term) bear a special attribute, cRef, just for this purpose. Using the system described in this section, an encoder may specify references to canonical works in a discipline-familiar format, and expect software to derive a complete URI from it. The value of the cRef attribute is processed as described in this section, and the resulting URI reference is treated as if it were the value of the target attribute. The cRef and target attributes are mutually exclusive: only one or the other may be specified on any given occurrence of an element.

When an application encounters a canonical reference as the value of cRef attribute, it follows a sequence of specific steps to transform it into a URI reference.

1. Ascertain the correct refsDecl following the rules summarized in section 15.3.3 Summary.
2. For each cRefPattern element encountered in the appropriate refsDecl, in the order encountered:
   1. match the value of cRef to the regular expression found as the value of the matchPattern attribute
   2. if the cRef value matches, take the value of the replacementPattern attribute and substitute the back references ($1, $2, etc.) with the corresponding matched substrings
   3. the result is taken as if it were a relative or absolute URI reference specified on the target attribute; i.e., it should be used as is or combined with the current xml:base value as usual
   4. no further processing of this cRef against the refsDecl should take place
   5. if, however, the cRef value does not match the regular expression specified on matchPattern attribute, proceed to the next cRefPattern
3. If all the cRefPattern elements are examined in turn and none matches, the pointer fails.

The regular expression language used as the value of the matchPattern attribute is that used for the pattern facet of the World Wide Web Consortium's XML Schema Language in an Appendix to XML Schema Part 2.⁵⁸ The value of the replacementPattern attribute is simply a string, except that occurences of ‘$1’ through ‘$9’ are replaced by the corresponding substring match. Note that since a maximum of nine substring matches are permitted, the string ‘$18’ means ‘the value of the first matched substring followed by the character ‘8’’ as opposed to ‘the eighteenth matched substring’. If there is a need for an actual string including a dollar sign followed by a digit that is not supposed to be replaced, the dollar sign should be written as %24.

A common requirement in text analysis is to represent correspondences between two or more parts of a single document, or between places in different documents. Provided that explicit elements are available to represent the parts or places to be linked, then the global linking attribute corresp may be used to encode such correspondence, once it has been identified.

• att.global.linking definisce un insieme di attributi per ipertesto o altro legame abilitati per tutti gli elementi quando è selezionato il sottoinsieme di marcatori per i collegamenti

  corresp

  (corrispondente) punta a elementi che hanno una qualche corrispondenza con l'elemento corrente

This is one of the attributes made available by the mechanism described in the introduction to this chapter (16 Linking, Segmentation, and Alignment). Correspondence can also be expressed by means of the link element introduced in section 16.1 Links.

Where the correspondence is between spans, the seg element should be used, if no other element is available. Where the correspondence is between points, the anchor element should be used, if no other element is available.

In the following example, we use the same mechanism to express a correspondence amongst the anchors introduced following the fifth word after English in a text:

English language. Except for not very<anchor xml:id="eng1"/>
<!-- ... -->
English at all at the time<anchor xml:id="eng2"/>
<!-- ... -->
English was still full of flaws<anchor xml:id="eng3"/>
<!-- ... -->
English. This was revised by young<anchor xml:id="eng4"/>
<!-- ... -->
<linkGrp type="five-word collocates">
  <link type="collocates of ENGLISH" target="#eng1 #eng2 #eng3 #eng4"/>
  <!-- ... -->
</linkGrp>

One very important application area for the alignment of parallel texts is multilingual corpora. Consider, for example, the need to align ‘translation pairs’ of sentences drawn from a corpus such as the Canadian Hansard, in which each sentence is given in both English and French. Concerning this problem, Gale and Church write:

The alignment produced by Gale and Church's program can be expressed in four different ways. The encoder must first decide whether to represent the alignment in terms of points within each text (using the anchor element) or in terms of whole stretches of text, using the seg element. To some extent the choice will depend on the process by which the software works out where alignment occurs, and the intention of the encoder. Secondly, the encoder may elect to represent the actual encoding using either corresp attributes attached to the individual anchor or seg elements, or using a free standing linkGrp element.

There is no requirement that the corresp attribute be specified in both English and French texts, since (as noted above) this attribute is defined as representing a mutual association. However, it may simplify processing to do so, and also avoids giving the impression that the English is translating the French, or vice versa. More seriously, this encoding does not make explicit that it is in fact the entire stretch of text between the anchors which is being aligned, not simply the points themselves. If for example one text contained material omitted from the other, this approach would not be appropriate.

The preceding encoding of the alignment of parallel passages from two texts requires that those texts and the alignment all be part of the same document. If the texts are in separate documents, then complete URIs, whether absolute or relative (section 16 Linking, Segmentation, and Alignment), will be required. These external pointers may appear anywhere within the document, but if they are created solely for use in encoding links, they may for convenience be grouped within the linkGrp (or other grouping element that uses them for linking).

To demonstrate this facility, we consider how we might encode the alignments in an extract from Comenius' Orbis Sensualium Pictus, in the English translation of Charles Hoole (1659).

Each topic covered in this work has three parts: a picture, a prose text in Latin describing the topic, and a carefully-aligned translation of the Latin into English, German, or some other vernacular. Key terms in the two texts are typographically distinct, and are linked to the picture by numbers, which appear in the two texts and within the picture as well.

As printed, the Comenius text exhibits three kinds of alignment.

1. The English and Latin portions are printed in two parallel columns, with corresponding phrases, (represented above by seg elements), more or less next to each other.
2. Particular words or phrases are marked as terms in the two languages by a change of rendition: the English text, which otherwise uses black letter type throughout, has the words The Study, a Student, Studies, and Books in a roman font; in the Latin text, which is printed in roman, the corresponding words (Museum, Studiosus, Studiis, and Libros) are all in italic.
3. Numbered labels appear within the text portions, linking keywords to each other and to sections of the picture. These labels, which have been left out of the above encoding, are attached to the first, third, and last segments in each language quoted below, and also appear (rather indistinctly) within the picture itself. Thus, the images of the study, the student, and his books are each aligned with the correct term for them in the two languages.

...
<seg xml:id="e9803">where a <ref n="2" target="#p982">Student</ref>,</seg>
<seg xml:id="l9803">ubi <ref n="2" target="#p982">Studiosus</ref>,</seg>
...

This map, of course, only aligns whole segments and image portions, since these are the only parts of our encoding which bear identifiers and can therefore be pointed to. To add to it the alignment between the typographically distinct words mentioned above, new elements must be defined, either within the text itself or externally by using stand off techniques. Encoding these word pairs as term and gloss, although intuitively obvious, requires a non-trivial decision as to whether the Latin text is glossing the English, or vice-versa. Tagging all the marked words as term avoids the difficult decision, but might be thought by some encoders to convey the wrong information about the words in question. Simply tagging them as additional embedded seg elements with identifiers that can be aligned like the others is also a possibility.

Provided that explicit elements are available to represent the parts or places to be synchronized, then the global linking attribute synch may be used to encode such synchronization, once it has been identified.

• att.global.linking definisce un insieme di attributi per ipertesto o altro legame abilitati per tutti gli elementi quando è selezionato il sottoinsieme di marcatori per i collegamenti

  synch

  (sincrono) punta a elementi sincroni rispetto all'elemento corrente

This is another of the attributes made globally available by the mechanism described in the introduction to this chapter. Alternatively, the link and linkGrp elements may be used to make explicit the fact that the synchronous elements are aligned.

To illustrate the use of these mechanisms for marking synchrony, consider the following representation of a spoken text:

B: The first time in twenty five years, we've cooked Christmas
   (unclear) for a blooming great load of people.
A: So you're [1] (unclear) [2]
B: [1] It will be [2] nice in a way, but, [3] be strange. [4]
A: [3] Yeah [4], yeah, cos it, it's [5] the [6]
B: [5] not [6]

This representation uses numbers in brackets to mark the points at which speakers overlap each other. For example, the [1] in A's first speech is to be understood as coinciding with the [1] in B's second speech.⁶¹

A synchronous alignment specifies which points in a spoken text occur at the same time, and the order in which they occur, but does not say at what time those points actually occur. If that information is available to the encoder it can be represented by means of the when and timeline elements, whose description and attributes are the following:

• when/ fornisce un'indicazione temporale in termini relativi, rispetto ad altri elementi determinati sulla stessa scala temporale, o in termini assoluti

  absolute	indica un valore temporale assoluto
  interval	indica la componente numerica di un intervallo temporale
  unit	specifica l'unità temporale nel quale è espresso l'attributo interval, se questo non è ereditato dal genitore timeline
  since	identifica il punto di riferimento per la determinazione del valore temporale dell'elemento when corrente, ottenuto aggiungendo l'intervallo al punto di riferimento temporale

• timeline (scala temporale) indica una serie di punti ordinati temporalmente i quali possono essere collegati agli elementi di un testo parlato al fine di ottenere un allineamento temporale del testo stesso

  origin	indica l'origine della scala temporale, cioè il momento in cui ha inizio
  interval	indica la componente numerica di un intervallo temporale
  unit	specifica l'unità temporale corrispondente al valore dell'attributo interval della scala temporale o dei suoi punti costitutivi

Each when element indicates a point in time, either directly by means of the absolute attribute, whose value is a string which specifies a particular time, or indirectly by means of the since attribute, which points to another when. If the since is used, then the interval and unit attributes should also be used to indicate the amount of time that has elapsed since the time specified by the element pointed to by the since attribute; the value -1 can be given to indicate that the interval is unknown.

If the when elements are uniformly spaced in time, then the interval and unit values need be given once in the timeline, and not repeated in any of the when elements. If the intervals vary, but the units are all the same, then the unit attribute alone can be given in the timeline element, and the interval attribute given in the when element.

The origin attribute in the timeline element points to a when element which specifies the reference or origin for the timings within the timeline; this must, of course, specify its position in time absolutely. If the origin of a timeline is unknown, then this attribute may be omitted.

Finally, suppose that a digitized audio recording is also available, and an XML file that assigns identifiers to the various temporal spans of sound is available. For example, the following Synchronized Multimedia Integration Language (SMIL, pronounced "smile") fragment:

<audio src="rtsp://soundstage.pi.cnr.it:554/home/az/bncSound/xmas4lots.mp3" 
 xml:id="au1" begin="05.2s" />
<audio src="rtsp://soundstage.pi.cnr.it:554/home/az/bncSound/xmas4lots.mp3" 
 xml:id="au2" begin="05.7s" />
<audio src="rtsp://soundstage.pi.cnr.it:554/home/az/bncSound/xmas4lots.mp3" 
 xml:id="au3" begin="05.9s" />
<audio src="rtsp://soundstage.pi.cnr.it:554/home/az/bncSound/xmas4lots.mp3" 
 xml:id="au4" begin="06.3s" />
<audio src="rtsp://soundstage.pi.cnr.it:554/home/az/bncSound/xmas4lots.mp3" 
 xml:id="au5" begin="06.9s" />
<audio src="rtsp://soundstage.pi.cnr.it:554/home/az/bncSound/xmas4lots.mp3" 
 xml:id="au6" begin="07.4s" />

URIs pointing to the <audio> elements could also be included as a fourth component in each of the above link elements, thus providing a synchronized audio track to complement the transcribed text.

For further discussion of this and related aspects of encoding transcribed speech, refer to chapter 8 Transcriptions of Speech.

Most of the mechanisms defined in this chapter rely to a greater or lesser extent on the fact that tags in a marked-up document can both assert a property for a span of text which they enclose, and assert the existence of an association between themselves and some other span of text elsewhere. In stand-off markup, there is a clear separation of these two behaviours: the markup does not directly contain any part of the text, but instead includes it by reference. One specific mechanism recommended by these Guidelines for this purpose is the standard XInclude mechanism defined by the W3C; another is to use pointers as demonstrated elsewhere in this chapter.

There are many reasons for using stand-off markup: the source text might be read-only so that additional markup cannot be added, or a single text may need to be marked up according to several hierarchically incompatible schemes, or a single scheme may need to accommodate multiple hierarchical ambiguities, so that a single markup tree is not the most faithful representation of the source material.

This section describes a generic mechanism for expressing all kinds of markup externally as stand-off tags, for use whenever it is appropriate.

Throughout this section the following terms will be systematically used in specific senses.

source document

a document to which the stand-off markup refers (a source document can be either XML or plain text); there may be more than one source document.

internal markup

markup that is already present in an XML source document

stand-off markup

markup that is either outside of the source document and points in to it to the data it describes, or alternatively is in another part of the source document and points elsewhere within the document to the data it describes

external document

a document that contains stand-off markup that points to a different, source document

internalize

the action of creating a new XML document with external markup and data integrated with the source document data, and possibly some source document markup as well

externalize

a process applied to markup from a pre-existing XML document, which splits it into two documents, an XML (external) document containing some of the markup of the original document, and another (source) XML document containing whatever text content and markup has not been extracted into the stand-off document; if all markup has been externalized from a document, the new source may be a plain text document

The three major requirements satisfied by this scheme for stand-off markup are:

1. any valid TEI markup can be either internal or external,
2. external markup can be internalized by applying it to the document content by either substituting the existing markup or adding to it, to form a valid TEI document, and
3. the external markup itself specifies whether an internalized document is to be created by substituting the existing internal markup or by adding to it

.

Stand-off markup which relies on the inclusion of virtual content is adequately supported by the W3C XInclude recommendation, which is also recommended for use by these Guidelines.⁶⁶ XInclude defines a namespace (http://www.w3.org/2001/XInclude), which in these Guidelines will be associated with the prefix xi:, and exactly two elements, <xi:include> and <xi:fallback>. XInclude relies on the XPointer framework discussed elsewhere in this chapter to point to the actual fragments of text to be internalized. Although XInclude only requires support for the element() scheme of XPointer, these Guidelines permit the use of any of the pointing schemes discussed in section 16.2 Pointing Mechanisms.

XInclude is a W3C recommendation which specifies a syntax for the inclusion within an XML document of data fragments placed in different resources. Included resources can be either plain text or XML. XInclude instructions within an XML document are meant to be replaced by a resource targetted by a URI, possibly augmented by an XPointer that identifies the exact subresource to be included.

The <xi:include> element uses the href attribute to specify the location of the resource to be included; its value is an URI containing, if necessary, an XPointer. Additionally, it uses the parse attribute (whose only valid values are text and xml) to specify whether the included content is plain text or an XML fragment, and the encoding attribute to provide a hint, when the included fragment is text, of the character encoding of the fragment. An optional <xi:fallback> element is also permitted within an <xi:include>; it specifies alternative content to be used when the external resource cannot be fetched for some reason. Its use is not however recommended for stand-off markup.

XInclude currently only requires support for one XPointer scheme, called element(). As described in 16.2.4 W3C element() Scheme, the element() scheme can use either a bare name (denoting an element with a specific xml:id attribute) or a child sequence (a numerical sequence of slash-separated child numbers specifying a path in the XML tree whose final step selects a specific subtree of XML content) to specify its target. Another scheme, xpointer(), has not yet become a W3C recommendation, although it has been part of the XPointer drafts from the beginning. The xpointer() scheme and the TEI schemes defined earlier (see 16.2.5 TEI XPointer Schemes) add the concepts of points and ranges, which can be used to specify sub-node fragments (e.g., a few words within a longer text node) or trans-node fragments (e.g., a segment of text that spans across different branches of the overall XML tree).

The operations of internalizing and externalizing markup are very useful and practically important. XInclude processing as defined by the W3C is internalization of one or more source documents' content into a stand-off document. TEI use of XInclude for stand-off markup enables use of XInclude-conformant sofware to perform this useful operation. However, internalization is not clearly defined for all stand-off files, because the structure of the internal and external markup trees may overlap. In particular, when an external markup document selects a range that overlaps partial elements in the source document, it is not clear how the semantics of internalization (inclusion) should work, since partial elements are not XML objects.⁶⁷ XInclude defines a semantics for this case that involves only complete elements.

When a range selection partially overlaps a number of elements in a source document, XInclude specifies that the partially overlapping elements should be included as well as all completely overlapping elements and characters (partially overlapping characters are not possible). The effect of this is that elements that straddle the start or end of a selected range will be included as wrappers for those of their children that are completely or partially selected by the range. For example, given the following source document:

  <body>
  <p xml:id="par1">home, <emph>home</emph> on Brokeback Mountain.</p>
  <p xml:id="par2">That was the <emph>song</emph> that I sang</p>
  </body>
  

and the following external document:

  <body xmlns:xi="http://www.w3.org/2001/XInclude">
     <div><xi:include href="example1.xml"
xpointer="range(xpath1(id(par1)//emph),xpath1(id(par2)//emph))"/>
     </div>
 </body>    

the resulting document after XInclude processing of this external document would be:

   <body xmlns:xi="http://www.w3.org/2001/XInclude">
   <div>
<p xml:id="par1"><emph>home</emph> on Brokeback Mountain.</p>
<p xml:id="par2">That was the <emph>song</emph></p>    
   </div>
   </body>
  

The result of the inclusion is two paragraph elements, while the original range designated in the source document overlapped two paragraph fragments. The semantics of XInclude require the creation of well-formed XML results even though the pointing mechanisms it uses do not necessarily respect the hierarchical structure of XML documents, as in this case. While this is a good way to ensure that internalization is always possible, it has implications for the use of XInclude as a notation for the description of overlapping markup structures.

When overlapping hierarchies need to be represented for a single document, each hierarchy must be represented by a separate set of XInclude tags pointing to a common source document. This sort of structure corresponds to common practice in work with linguistic text corpora. In such corpora, each potentially overlapping hierarchy of elements for the text is represented as a separate stream of stand-off markup. Generally the source text contains markup for the smallest significant units of analysis in the corpus, such as words or morphemes, this information and its markup representing a layer of common information that is shared by all the various hierarchies. As a way of organizing the representation of complex data, this technique generally allows a large number of xml:id attributes to be attached to the shared elements, providing robust anchors for links and facilitating adjustments to the source document without breaking external documents that reference it.

Any tag can be externalized by removing its content and replacing it with an <xi:include> element that contains an XPointer pointing to the desired content.

<content>To make a prairie it takes a clover and one bee,\n
One clover, and a bee,\n
And revery.\n
The revery alone will do,\n
If bees are few.\n
</content>

<text xmlns:xi="http://www.w3.org/2001/XInclude">
 <body>
  <head>1755</head>
   <l>
    <xi:include href="Source.xml" parse="xml"
 xpointer="string-range(element(/1),  0, 48)"/>
   </l>
   <l>
    <xi:include href="Source.xml" parse="xml"
 xpointer="string-range(element(/1), 49, 71)"/>
   </l>
   <l>
    <xi:include href="Source.xml" parse="xml"
 xpointer="string-range(element(/1), 72, 83)"/>
   </l>
   <l>
    <xi:include href="Source.xml" parse="xml"
 xpointer="string-range(element(/1), 84,109)"/>
   </l>
   <l>
    <xi:include href="Source.xml" parse="xml"
 xpointer="string-range(element(/1),110,126)"/>
   </l>
 </body>
</text>

Please note that this specification requires that the XInclude namespace declaration is present in all cases. The <xi:fallback> element contains text or XML fragments to be placed in the document if the inclusion fails for any reason (for instance due to inaccessibility of an external resource). The <xi:fallback> element is optional; if it is not present an XInclude processor must signal a fatal error when a resource is not found. This is the preferred behaviour for use with stand-off markup. These Guidelines recommend against the use of <xi:fallback> for stand-off markup.

The whole source fragment identified by an XInclude element, as well as any markup therein contained is inserted in the position specified, and an XInclude processor is required to ensure that the resulting internalized document is well-formed. This has obvious implications when the external document contains XML markup. A plain text source document will always create a well-formed internalized document.

While a TEI customization may permit <xi:include> elements in various places in a TEI document instance, in general these Guidelines suggest that validity be verified after the resolution of all the <xi:include> elements.

When the source text is plain text the overall form of the XPointer pointing to it is of minimal importance. The form of the XPointer matters considerably, on the other hand, when the source document is XML.

In this case, it is rather important to distinguish whether we intend to substitute the source XML with the new one, or just to add new markup to it. The XPointers used in the references can express both cases.

A simple way is to make sure to select only textual data in the XPointer. For instance, given the following document:
Source.xhtml

<html>
    <body>
  <div>To make a prairie it takes a <a href="clover.gif">clover</a>
    and one <a href="bee.gif">bee</a>,</div>
  <div>One <a href="clover.gif">clover</a>, and
    a <a href="bee.gif">bee</a>,</div>
  <div>And revery.</div>
  <div>The revery alone will do,</div>
  <div>If bees are few.</div>
    </body>
</html>

the expression range(element(/1/2/1.0),element(/1/2/11.1)) will select the whole poem, text content and div elements and hypertext links (NB: in XPointer whitespace-only text nodes count).

On the contrary, the expressions xpointer(//text()/range-to(.)) and xpointer(string-range(//text(),"To")/range-to(//text(),"few.") will only select the text of the poem, with no markup inside.

Thus, the following could be a valid stand-off document for the Source.xhtml document:
External2.xml

<text xmlns:xi="http://www.w3.org/2001/XInclude">
 <body>
  <head>1755</head>
  <l>
   <xi:include href="Source.xhtml"
 xpointer='xpointer(string-range(//div[1]/text(),"To")/range-to(//div[1]/text(),"bee,")'/>
  </l>
  <l>
   <xi:include href="Source.xhtml"
 xpointer='xpointer(string-range(//div[2]/text(),"One")/range-to(//div[2]/text(),"bee,")'/>
  </l>
  <l>
   <xi:include href="Source.xhtml"
 xpointer='xpointer(string-range(//div[3]/text(),"And")/range-to(//div[3]/text(),".")'/>
  </l>
  <l>
   <xi:include href="Source.xhtml"
 xpointer='xpointer(string-range(//div[4]/text(),"The")/range-to(//div[4]/text(),",")'/>
  </l>
  <l>
   <xi:include href="Source.xhtml"
 xpointer='xpointer(string-range(//div[5]/text(),"If")/range-to(//div[5]/text(),".")'/>
  </l>
 </body>
</text>