P5: Guidelines for Electronic Text Encoding and Interchange

TEI: Words and Above⚓︎17.1.1 Words and Above

Although different languages have very different rules about what constitutes a ‘word’ or a ‘sentence’, these remain generally useful concepts. In this section we discuss elements provided for marking up linguistic units down to the word level, however defined.

• s (s-unit) contains a sentence-like division of a text.
• cl (clause) represents a grammatical clause.
• phr (phrase) represents a grammatical phrase.
• w (word) represents a grammatical (not necessarily orthographic) word.

As members of the att.segLike class, these elements all share the following attribute:

• att.segLike provides attributes for elements used for arbitrary segmentation.

  function

  (function) characterizes the function of the segment.

They also share attributes from att.typed:

• att.typed provides attributes that can be used to classify or subclassify elements in any way.

  type	characterizes the element in some sense, using any convenient classification scheme or typology.
  subtype	(subtype) provides a sub-categorization of the element, if needed

These elements are also all members of the model.segLike class, which is a subclass of model.phrase. They may thus appear anywhere that text is permitted within a document, when the module defined by this chapter is included in a schema.

The w and pc elements belong to the att.linguistic class, which supplies attributes that may be used for lightweight linguistic annotation (see section 17.4.2 Lightweight Linguistic Annotation below):

• att.linguistic provides a set of attributes concerning linguistic features of tokens, for usage within token-level elements, specifically w and pc in the analysis module.

  lemma	provides a lemma (base form) for the word, typically uninflected and serving both as an identifier (e.g. in dictionary contexts, as a headword), and as a basis for potential inflections.
  lemmaRef	provides a pointer to a definition of the lemma for the word, for example in an online lexicon.
  pos	(part of speech) indicates the part of speech assigned to a token (i.e. information on whether it is a noun, adjective, or verb), usually according to some official reference vocabulary (e.g. for German: STTS, for English: CLAWS, for Polish: NKJP, etc.).
  msd	(morphosyntactic description) supplies morphosyntactic information for a token, usually according to some official reference vocabulary (e.g. for German: STTS-large tagset; for a feature description system designed as (pragmatically) universal, see Universal Features).
  join	when present, provides information on whether the token in question is adjacent to another, and if so, on which side.

Additionally, these elements also have access to the att.lexicographic.normalized class, which supplies the attributes norm and orig: the former for handling normalization/regularization at the word level, the latter providing the original form if the element content is modernized or regularized. Note that these attributes are a local (word-level) alternative to the robust mechanism that uses the choice, orig, and reg elements, discussed in section 3.5.2 Regularization and Normalization and in chapter 12 Critical Apparatus. The choice-based mechanism is the default descriptive device, while the norm and orig attributes are used to handle a subset of normalizations in linguistic contexts where a single sequence of tokens is a priority, for example in historical corpora subject to linguistic analysis. It needs to be stressed that the simplified attribute-based mechanism is not meant to be used for editorial interventions. ⁷⁷

Clauses may be further divided into phr elements in the same way. A text may be segmented directly into clauses, or into phrases, with no need to include segmentation at a higher level as well.

The type attribute on linguistic segment categories can be used to provide additional interpretative information about the category. The function attribute on the cl and phr elements can be used to provide additional information about the function of the category. Legal values for these two attributes are not defined by these Guidelines, but should be documented in the segmentation element of the encodingDesc element within the document's header. A general approach to the encoding of linguistic categories for parts of a text is discussed in section 17.4 Linguistic Annotation below.

This style of markup may introduce spurious new lines and blanks into the text. If the original layout is important, it should be explicitly encoded, using such facilities as the lb element, the global rend or rendition attributes, etc.

The w, m, and c elements are identical in meaning to the seg element with a type attribute of ‘w’, ‘m’, or ‘c’ respectively, and may occur wherever seg is permitted to occur. However, their content is more constrained than seg: for example, the w element should only contain w, m, c elements or pc elements, or plain text; the m element should contain only c or pc elements or plain text; both the c and pc elements should contain only plain text, most often only a single character or a sequence of graphemes to be treated as a single character. Consequently, while these more specific elements can be translated directly into typed seg elements, the reverse is not necessarily the case.

TEI: Linguistic Annotation by Means of Generic TEI Devices⚓︎17.4.1 Linguistic Annotation by Means of Generic TEI Devices

As one example of such types of analysis, consider the following sentence, taken from the Lancaster/IBM Treebank Project (Leech and Garside (1991)).

Our discussion focuses on the way that this sentence might be analysed using the CLAWS system developed at the University of Lancaster but exactly the same principles may be applied to a wide variety of other systems.⁸⁰ Output from the system consists of a segmented and tokenized version of the text, in which word class codes have been associated with each token. CLAWS offers outputs in a variety of non-XML and XML formats: for example, the simplest format for the sample sentence would be:

The_AT0 victim_NN1 's_POS friends_NN2 told_VVD police_NN2 that_CJT Kruger_NP0 
drove_VVD into_PRP the_AT0 quarry_NN1 and_CJC never_AV0 surfaced_VVD⚓

This approach requires the text to be fully segmented, using the linguistic segment elements described in section 17.1 Linguistic Segment Categories, so that the scope of the ana attribute used to point to each interpretation is clearly defined. A further analysis into phrase and clause elements can be superimposed on the word and morpheme tagging in the preceding illustration. For example, CLAWS provides the following constituent analysis of the sample sentence (the word class codes have been deleted):

[N [G The victim's G] friends N] [V told [N police N] [Fn that 
[N Krueger N] [V [V& drove [P into [N the quarry N]P]V&] and 
[V+ never surfaced V+]V]Fn]V]⚓

Each linguistic segment so far discussed has been well-behaved with respect to the basic document hierarchy, having only a single parent. Moreover, the segmentation has been complete, in that each part of the text is accounted for by some segment at each level of analysis, without discontinuities or overlap. This state of affairs does not of course apply in all types of analysis, and these Guidelines provide a number of mechanisms to support the representation of discontinuities or multiple analyses. A brief overview of these facilities is provided in chapter 20 Non-hierarchical Structures; also see 16 Linking, Segmentation, and Alignment. These mechanisms all depend to a greater or lesser degree on the use of pointing elements of various kinds.

TEI: Lightweight Linguistic Annotation⚓︎17.4.2 Lightweight Linguistic Annotation

While these Guidelines offer a variety of means to add linguistic information to textual units and much of that has been presented above, two kinds of use cases and two groups of users call for a dedicated set of specialized attributes to carry linguistic information. One relevant use case is where basic linguistic information gets added to an existing resource, in which generic attributes such as type or ana have already been used to encode other categorizations and analyses. The other group of users and use cases involves corpus linguists and resources built from scratch as lightly annotated language corpora. In the latter kind of projects, energy and person-hours are not devoted to careful literary analysis and hand-encoding of the relevant phenomena, but rather to the analysis of the completed resources, and therefore the phase of resource-building must be quick and relatively effortless, requiring minimal structural markup, well-established containers for grammatical information, and a standardized way of filling them in.

The aims defined above can be realized by means of lightweight linguistic annotation using attributes that belong to the att.linguistic class:

• att.linguistic provides a set of attributes concerning linguistic features of tokens, for usage within token-level elements, specifically w and pc in the analysis module.

  lemma	provides a lemma (base form) for the word, typically uninflected and serving both as an identifier (e.g. in dictionary contexts, as a headword), and as a basis for potential inflections.
  pos	(part of speech) indicates the part of speech assigned to a token (i.e. information on whether it is a noun, adjective, or verb), usually according to some official reference vocabulary (e.g. for German: STTS, for English: CLAWS, for Polish: NKJP, etc.).
  msd	(morphosyntactic description) supplies morphosyntactic information for a token, usually according to some official reference vocabulary (e.g. for German: STTS-large tagset; for a feature description system designed as (pragmatically) universal, see Universal Features).
  join	when present, provides information on whether the token in question is adjacent to another, and if so, on which side.

The essence of lightweight linguistic annotation is that the basic grammatical information is encapsulated at the word level, together with the orthographic shape of the word. This has clear advantages for automatic processing but, on the other hand, this form of data encapsulation also imposes restrictions on the extent of information that can be encoded, essentially limiting it to a single tokenization and lemmatization schema, a single tagset, and a subset of the possible analyses (out from potentially many guesses at the part-of-speech or morphosyntactic descriptions, single values have to fit into the existing attributes). Another important principle that this kind of annotation is sensitive to is the need for (near) homomorphism between the assumed tokenization (division of the text stream into minimal units) and the division into minimal syntactic units (word forms, in the terminology of ISO Morpho-Syntactic Framework, ISO 24611⁸¹), because it is the former that results from the process of tokenization, but the latter that can be lemmatized and meaningfully described by means of grammatical features. Where tokens are only minimally mismatched with word forms, various repair strategies can be used (e.g., recursing w to capture multi-token compounds or using att.fragmentable to point at disjoint tokens). Beyond that, more robust TEI mechanisms, based on standoff principles and feature structures, should replace lightweight annotation.

The basic grammatical information encoded by means of att.linguistic is sufficient for the purpose of enhancing queries or improving the analysis of search results by, for example, making it possible to distinguish between the noun cut and the identically spelled verb cut in English, and further between e.g. the present-tense form of cut and its past-tense or past-participial forms. For the former contrast, the part-of-speech (pos) attribute should be used, whereas the latter may use pos and/or msd attributes, depending on the annotation vocabulary adopted for the project in question. The various grammatical realizations of a single ‘dictionary word’ can be captured by means of the attribute lemma, which provides a common label for them. For example, English verbs are typically lemmatized as the base form (also called bare infinitive), so the value of lemma for the verbal forms write, writes, wrote, written, and writing is typically write.

Together with the span-delimiting elements mentioned in this section, such as s, cl, or phr, lightweight grammatical annotation may be used to build basic syntactic constituency structures, where hierarchical information is expressed through span containment rather than by relations among tree nodes. This is however the limit of this kind of annotation: for the purpose of describing true constituency or dependency syntactic structures, one needs to turn to more robust mechanisms offered by the TEI, which may involve graph description (see chapter 19 Graphs, Networks, and Trees) or standoff techniques (see section 16.9 Stand-off Markup), and where grammatical labels may need to be annotated by means of feature structures (see chapter 18 Feature Structures).

Some of the above-mentioned robust methods will also prove handy in cases where more than one tagset (label inventory) is used to label the words, or where automatic morphological analysis yields multiple possibilities (for example, the form cutting is morphologically ambiguous between verbal, adjectival, and nominal) and needs to be followed by (often also automatic) disambiguation in morphosyntactic contexts, with varying probabilities that may also need to be recorded together with their corresponding part-of-speech and morphosyntactic values.

It should be borne in mind that tokenization, lemmatization, part-of-speech identification, and morphosyntactic labelling, especially when performed automatically, should in most cases be seen as involving pragmatic decisions, dictated by concrete practical goals, economy of description, or the demands of particular analytic and/or visualization tools. It comes therefore as no surprise that numerous alternative (and often conflicting) lemmatization strategies and tagsets exist, in use by various communities and various tools, and that they change with time (a case in point is the CLAWS tagset for English, with several versions that merge the part-of-speech and morphosyntactic information to various degrees). ⁸² The nature and description of these systems is outside the scope of these Guidelines, but it has to be stressed that all the strategies adopted for linguistic annotation, even at the lightweight level of complexity, must be documented in the header of the given electronic resource, not only for the purpose of guaranteeing successful data interpretation and exchange, but also for the sake of sustainability of the results of the given project.

The last of the att.linguistic attributes, join, has the most text-technological flavour. It can be used to amend the loss of whitespace-related information in non-inline markup.

Compare the following two listings. The first difference between them is in the tagset used (CLAWS-5 vs. CLAWS-7) and only serves to exemplify the need to document the choice of descriptive vocabulary in the header, lest the encoded information is unreadable or confusing. The second difference is the difference in the treatment of inter-token whitespace, and it is here that the join attribute proves indispensable.

Note that projects will need to decide whether they want to redundantly encode full information on the adjacency of each token (in which case, the above listing should also have join with the value right on the tokens victim and surfaced, or whether information on a single direction of adjacency is enough. Strategies vary, and it is important to document them in the TEI header.

The final examples lay out a strategy for dealing with e.g. historical corpora where it is on the one hand important to maintain a steady stream of token-level elements (w and pc) for efficient processing, but, on the other hand, it is also important to either record the original spelling (when the corpus text is normalized) or to record the normalized variants (when the element content of the corpus preserves the original spelling). The attribute class att.lexicographic.normalized can be used for that purpose:

• att.lexicographic.normalized provides attributes for usage within word-level elements in the analysis module and within lexicographic microstructure in the dictionaries module.

  norm	(normalized) provides the normalized/standardized form of information present in the source text in a non-normalized form
  orig	(original) gives the original string or is the empty string when the element does not appear in the source text.

TEI: Spoken Text⚓︎17.4.3 Spoken Text