18 Simple Analytic Mechanisms
Table des matières
This chapter describes a module for associating simple analyses and interpretations with text elements. We use the term analysis here to refer to any kind of semantic or syntactic interpretation which an encoder wishes to attach to all or part of a text. Examples discussed in this chapter include familiar linguistic categorizations (such as ‘clause’, ‘morpheme’, ‘part-of-speech’ etc.) and characterizations of narrative structure (such as ‘theme’, ‘reconciliation’ etc.). The mechanisms presented in this chapter are simpler but less powerful than those described in chapter 19 Feature Structures.
Section 18.1 Linguistic Segment Categories introduces elements which can be used to characterize text segments according to the familiar linguistic categories of sentence or s-unit, clause, phrase, word, morpheme, character, and punctuation mark. These elements represent special cases of the generic seg element described in section 17.3 Blocks, Segments, and Anchors.
Section 18.2 Global Attributes for Simple Analyses introduces an additional global attribute which allows passages of text to be associated with specialized elements representing their interpretation. These ‘interpretative’ elements (span and interp) are described in detail in section 18.3 Spans and Interpretations. They allow the encoder to specify an analysis as a series of names and associated values,76 each such pair being linked to one or more stretches of text, either directly, in the case of spans, or indirectly, in the case of interpretations.
Finally section 18.4 Linguistic Annotation revisits the topic of linguistic analysis, and illustrates how these interpretative mechanisms may be used to associate simple linguistic analysis with text segments.
TEI: Linguistic Segment Categories⚓︎18.1 Linguistic Segment Categories
In this section we introduce specialized linguistic segment category elements which may be used to represent the segmentation of a text into the traditional linguistic categories of sentence, clause, phrase, word, morpheme, characters, and punctuation marks.
TEI: Words and Above⚓︎18.1.1 Words and Above
Although different languages have very different rules about what constitutes a ‘word’ or a ‘sentence’, these remain generally useful concepts. In this section we discuss elements provided for marking up linguistic units down to the word level, however defined.
- s (phrase) contient une division textuelle de type phrase
- cl (proposition) représente une proposition grammaticale
- phr (syntagme) représente un syntagme grammatical
- w (mot) représente un mot grammatical (pas nécessairement orthographique)
As members of the att.segLike class, these elements all share the following attribute:
- att.segLike fournit des attributs pour des éléments utilisés pour une segmentation arbitraire.
function (fonction) caractérise la fonction du segment.
They also share attributes from att.typed:
- att.typed fournit des attributs qui peuvent être utilisés pour classer ou interclasser des
éléments de n'importe quelle façon.
type caractérise l'élément en utilisant n'importe quel système ou typologie de classification approprié. subtype (sous-type) fournit une sous-catégorisation de l'élément, si c'est nécessaire.
These elements are also all members of the model.segLike class, which is a subclass of model.phrase. They may thus appear anywhere that text is permitted within a document, when the module defined by this chapter is included in a schema.
The w and pc elements belong to the att.linguistic class, which supplies attributes that may be used for lightweight linguistic annotation (see section 18.4.2 Lightweight Linguistic Annotation below):
- att.linguistic provides a set of attributes concerning linguistic features of tokens, for usage
within token-level elements, specifically w and pc in the analysis module.
lemma fournit le lemme du mot (entrée du dictionnaire) lemmaRef provides a pointer to a definition of the lemma for the word, for example in an online lexicon. pos (part of speech) indicates the part of speech assigned to a token (i.e. information on whether it is a noun, adjective, or verb), usually according to some official reference vocabulary (e.g. for German: STTS, for English: CLAWS, for Polish: NKJP, etc.). msd (morphosyntactic description) supplies morphosyntactic information for a token, usually according to some official reference vocabulary (e.g. for German: STTS-large tagset; for a feature description system designed as (pragmatically) universal, see Universal Features). join when present, provides information on whether the token in question is adjacent to another, and if so, on which side.
Additionally, these elements also have access to the att.lexicographic.normalized class, which supplies the attributes norm and orig: the former for handling normalization/regularization at the word level, the latter providing the original form if the element content is modernized or regularized. Note that these attributes are a local (word-level) alternative to the robust mechanism that uses the choice, orig, and reg elements, discussed in section 3.5.2 Regularization and Normalization and in chapter 13 Critical Apparatus. The choice-based mechanism is the default descriptive device, while the norm and orig attributes are used to handle a subset of normalizations in linguistic contexts where a single sequence of tokens is a priority, for example in historical corpora subject to linguistic analysis. It needs to be stressed that the simplified attribute-based mechanism is not meant to be used for editorial interventions. 77
<s>Nineteen fifty-four, when I was eighteen years old,
is held to be a crucial turning point in the history of
the Afro-American — for the U.S.A. as a whole — the
year segregation was outlawed by the U.S. Supreme Court.</s>
<s>It was also a crucial year for me because on June 18,
1954, I began serving a sentence in state prison for
possession of marijuana.</s>
</p>
<s>
<cl>It was about the beginning of September, 1664,
<cl>that I, among the rest of my neighbours,
heard in ordinary discourse
<cl>that the plague was returned again to Holland; </cl>
</cl>
</cl>
<cl>for it had been very violent there, and particularly at
Amsterdam and Rotterdam, in the year 1663, </cl>
<cl>whither, <cl>they say,</cl> it was brought,
<cl>some said</cl> from Italy, others from the Levant, among some goods
<cl>which were brought home by their Turkey fleet;</cl>
</cl>
<cl>others said it was brought from Candia;
others from Cyprus. </cl>
</s>
<s>
<cl>It mattered not <cl>from whence it came;</cl>
</cl>
<cl>but all agreed <cl>it was come into Holland again.</cl>
</cl>
</s>
</p>
Clauses may be further divided into phr elements in the same way. A text may be segmented directly into clauses, or into phrases, with no need to include segmentation at a higher level as well.
<l>
<cl part="I">Tweedledum and Tweedledee</cl>
</l>
<l>
<cl part="F">Agreed to have a battle;</cl>
</l>
<l>
<cl part="I">For Tweedledum said <cl part="I">Tweedledee</cl>
</cl>
</l>
<l>
<cl part="F">
<cl part="F">Had spoiled his nice new rattle.</cl>
</cl>
</l>
</div>
<div type="stanza">
<l>
<cl part="I">Just then flew down a monstrous crow,</cl>
</l>
<l>
<cl part="F">As black as a tar barrel;</cl>
</l>
<l>
<cl part="I">Which frightened both the heroes so,</cl>
</l>
<l>
<cl part="F">
<cl>They quite forgot their quarrel.</cl>
</cl>
</l>
</div>
<cl next="#c5" xml:id="c3" part="I">For Tweedledum said
<cl next="#c6" xml:id="c4" part="I">Tweedledee</cl>
</cl>
</l>
<l>
<cl prev="#c3" xml:id="c5" part="F">
<cl prev="#c4" xml:id="c6" part="F">Had spoiled his nice new rattle.</cl>
</cl>
</l>
The type attribute on linguistic segment categories can be used to provide additional interpretative information about the category. The function attribute on the cl and phr elements can be used to provide additional information about the function of the category. Legal values for these two attributes are not defined by these Guidelines, but should be documented in the segmentation element of the encodingDesc element within the document's header. A general approach to the encoding of linguistic categories for parts of a text is discussed in section 18.4 Linguistic Annotation below.
<phr>was outlawed</phr>
<phr type="PP"
function="postmodifier-agent">by the U.S. Supreme Court.</phr>
<s>
<cl type="finite-declarative"
function="independent">
<phr type="NP" function="subject">Nineteen fifty-four,
<cl type="finite-relative-declarative"
function="appositive">when <phr type="NP" function="subject">I</phr>
<phr type="VP" function="predicate">was eighteen years old</phr>
</cl>
</phr>,
<phr type="VP" function="predicate">
<phr type="V" function="verb-main">is held</phr>
<phr type="NP" function="complement">
<cl type="nonfinite"
function="predicate-nom.">
<phr type="V" function="copula">to be</phr>
<phr type="NP"
function="predicate-nom.">a crucial turning point
<phr type="PP"
function="postmodifier">in
<phr type="NP" function="prep.obj.">the history
<phr type="PP"
function="postmodifier">of the Afro-American</phr>
</phr>
</phr>
—
<phr type="PP"
function="postmodifier-appositive">for
<phr type="NP" function="prep.obj.">the U.S.A.
<phr type="PP"
function="postmodifier">as a whole</phr>
</phr>
</phr>
</phr>
—
<phr type="NP"
function="appositive-predicate-nom.">the year
<cl type="finite-relative"
function="adjectival">
<phr type="NP" function="subject">segregation</phr>
<phr type="VP" function="predicate">
<phr type="V" function="verb-main">was outlawed</phr>
<phr type="PP"
function="postmodifier">by the U.S. Supreme Court</phr>
</phr>
</cl>
</phr>
</cl>
</phr>
</phr>.</cl>
</s>
<s>
<cl type="finite-declarative"
function="independent">
<phr type="NP" function="subject">It</phr>
<phr type="VP" function="predicate">
<phr type="V" function="verb-main">was</phr>
also
<phr type="NP"
function="predicate-nom.">a crucial year for me</phr>
</phr>
<cl type="declarative-finite"
function="dependent-causative">because
<phr type="PP"
function="sentence_adverb">on June 18, 1954</phr>,
<phr type="NP" function="subject">I</phr>
<phr type="VP" function="predicate">
<phr type="V" function="verb-main">began serving</phr>
<phr type="NP" function="complement">a sentence in state prison
<phr type="PP" function="complement">for possession of marijuana</phr>
</phr>
</phr>
</cl>
</cl>
</s>.
</p>
This style of markup may introduce spurious new lines and blanks into the text. If the original layout is important, it should be explicitly encoded, using such facilities as the lb element, the global rend or rendition attributes, etc.
The w, m, and c elements are identical in meaning to the seg element with a type attribute of ‘w’, ‘m’, or ‘c’ respectively, and may occur wherever seg is permitted to occur. However, their content is more constrained than seg: for example, the w element should only contain w, m, c elements or pc elements, or plain text; the m element should contain only c or pc elements or plain text; both the c and pc elements should contain only plain text, most often only a single character or a sequence of graphemes to be treated as a single character. Consequently, while these more specific elements can be translated directly into typed seg elements, the reverse is not necessarily the case.
<w lemma="timeo">timeo</w>
<w lemma="danaii">Danaos</w>
<w lemma="et">et</w>
<w lemma="donum">dona</w>
<w lemma="fero">ferentes</w>
</s>
http://lexicon.org/latin.xml
, we might for example revise the above example as:
<w lemmaRef="http://lexicon.org/latin.xml#timeo">timeo</w>
<w lemmaRef="http://lexicon.org/latin.xml#danaii">Danaos</w>
<!-- ... -->
</s>
TEI: Below the Word Level⚓︎18.1.2 Below the Word Level
It is sometimes helpful to markup explicitly sub-word components such as morphemes, characters, or punctuation.
- m (morphème) représente un morphème grammatical
- c (caractère) représente un caractère
- pc (punctuation character) contient un caractère ou une chaîne de caractères considérés comme un signe de ponctuation unique.
<m type="base">
<m type="prefix" baseForm="con">com</m>
<m type="root">fort</m>
</m>
<m type="suffix">able</m>
</w>
<seg type="base">
<seg type="prefix">com</seg>
<seg type="morph">fort</seg>
</seg>
<seg type="suffix">able</seg>
</seg>
There is a substantial linguistic difference between characters like letters or diacritics and punctuation marks. The former are used to construct meaningful units like morphemes or words. The latter are functionally independent units acting at the level of syntactic units. A word may consist of a single letter (for example ‘I’ in English), but this does not mean that we should use c instead of w to mark it up.
<c>M</c>
<c>O</c>
<c>A</c>
<c>I</c>
<w>doth</w>
<w>sway</w>
<w>my</w>
<w>life</w>
</phr>
![](Images/punctus.png)
<pc>
<g ref="#pelev">;</g>
</pc> natus est in bethlehem
<pc>.</pc>o <pc>.</pc> mira gratia...
<!-- elsewhere -->
<char xml:id="pelev">
<!-- definition of the punctus elevatus character -->
</char>
The pc element carries special attributes to record analyses of the functional behaviour or classification of the punctuation mark it contains. The unit attribute may be used, as on the milestone element to name the kind of unit which the punctuation mark delimits, for example a paragraph or section. The pre attribute may be used to indicate whether the punctuation precedes or follows the unit it delimits. The force attribute indicates the strength of the association between the punctuation mark and its adjacent word.
This segmentation, crude as it is, succeeds in representing the idea that did occurring as a morphological component of the word didn't has something in common with the word <do>. A further advantage of segmenting the text down to this level is that it becomes relatively simple to associate each such segment with a more detailed formal analysis, for example by providing a baseform, or morphological analysis at whichever level is appropriate. This matter is taken up in detail in section 18.4 Linguistic Annotation.
TEI: Global Attributes for Simple Analyses⚓︎18.2 Global Attributes for Simple Analyses
When the module described by this chapter is selected, an additional attribute is defined for all elements:
- att.global.analytic fournit des attributs globaux complémentaires pour associer des analyses ou des interprétations
spécifiques avec des portions de texte appropriées.
ana (analyse) indique un ou plusieurs éléments contenant des interprétations de l'élément qui porte l'attribut ana.
The ana attribute may be specified for any element. Its effect is to associate the element with one or more others representing an analysis or interpretation of it. Its target should be one of the elements described in the section 18.3 Spans and Interpretations below, or some other interpretative element such as note, on which see section 3.9 Notes, Annotation, and Indexing or fs, on which see chapter 19 Feature Structures. If a hierarchical form of classification is desired then it may point to category element at a suitable level in a taxonomy see 2.3.7 The Classification Declaration.
TEI: Spans and Interpretations⚓︎18.3 Spans and Interpretations
The simplest mechanisms for attaching analytic notes in some structured vocabulary to particular passages of text are provided by the span and interp elements, and their associated grouping elements spanGrp and interpGrp.
- span associe une interprétation sous forme d'annotation directement à un passage donné dans un texte.
- spanGrp (groupement de fragments de texte) regroupe des éléments span
- interp (interprétation) interprétation sous la forme d'une annotation concise, pouvant être liée à un passage dans un texte
- interpGrp (groupe d'interprétations) regroupe un ensemble d'interprétations ayant en commun une mention de responsabilité ou un type
These elements are all members of the class att.interpLike, and thus share the following attribute:
- att.interpLike fournit les attributs pour des éléments qui exposent une analyse ou une interprétation
formelles.
inst (cas) pointe vers les instances de l'analyse ou de l'interprétation représentées par l'élément courant.
They also inherit the following attributes from att.global.responsibility:
- att.global.responsibility provides attributes indicating the agent responsible for some aspect of the text,
the markup or something asserted by the markup, and the degree of certainty associated
with it.
cert (certitude) donne le degré de certitude associée à l'intervention ou à l'interprétation. resp (responsable) indique l'agent responsable de l'intervention ou de l'interprétation, par exemple un éditeur ou un transcripteur.
The type attribute of the span and interp elements may be used to indicate that the annotations are of specific types, for example thematic or structural. The annotation itself is supplied as the content of the span or interp element. In the case of the span element, the span of text being annotated is indicated by values of the from, to or target attributes, used in combination as follows. If only the from attribute is supplied, then the span is coterminous with the element indicated by its value; if both from and to are supplied, the span runs from the start of the element indicated by the from attribute up to the end of the element indicated by the to attribute; if the target attribute is used, the span is defined by aggregating the contents of the (possibly non-contiguous) elements pointed to by its values. It is an error to supply only the to attribute; to supply more than one pointer value for either to or from attributes; or to supply either of these in conjunction with the target attribute. In the case of interp (see below), the span is indicated by a pointer from a link element or some similar mechanism. The resp attribute indicates the annotator responsible for this annotation.
<w>What</w>
<w>did</w>
<w>you</w>
<w xml:id="mk01">make</w>
<w xml:id="up01">up</w>
</s>
<span from="#mk01" to="#up01">phrasal verb "make up"</span>
<w>What</w>
<w>did</w>
<w>you</w>
<w xml:id="mk02">make</w>
<w xml:id="up02">up</w>
</s>
<span target="#mk02 #up02">phrasal verb "make up"</span>
<w>Did</w>
<w>you</w>
<w xml:id="mk03">make</w>
<w>it</w>
<w xml:id="up03">up</w>
</s>
<span target="#mk03 #up03">phrasal verb "make up"</span>
<s xml:id="MaQp1s2p114s1">There was certainly a definite point at which the
thing began.</s>
<s xml:id="MaQp1s2p114s2">It was not; then it was suddenly inescapable,
and nothing could have frightened it away.</s>
<s xml:id="MaQp1s2p114s3">There was a slow integration, during which she,
and the little animals, and the moving grasses, and the sun-warmed
trees, and the slopes of shivering silvery mealies, and the great
dome of blue light overhead, and the stones of earth under her feet,
became one, shuddering together in a dissolution of dancing
atoms.</s>
<s xml:id="MaQp1s2p114s4">She felt the rivers under the ground forcing
themselves painfully along her veins, swelling them out in an
unbearable pressure; her flesh was the earth, and suffered growth
like a ferment; and her eyes stared, fixed like the eye of the
sun.</s>
<s xml:id="MaQp1s2p114s5">Not for one second longer (if the terms for time
apply) could she have borne it; but then, with a sudden movement
forwards and out, the whole process stopped; and <emph rend="italic">that</emph> was <soCalled rend="dquo">the
moment</soCalled> which it was impossible to remember
afterwards.</s>
<span from="#MaQp1s2p114s3"
to="#MaQp1s2p114s5">the moment</span>
<s xml:id="MaQp1s2p114s6">For during that space of time (which was
timeless) she understood quite finally her smallness, the
unimportance of humanity.</s>
</p>
<span from="#MaQp1s2p114s3"
to="#MaQp1s2p114s5">the moment</span>
<!-- other spans identified by DTL here -->
</spanGrp>
Spans may also be used to represent structural divisions within a narrative, particularly when these do not coincide with the structure implied by the element structure. Consider the following narrative:
Sigmund, the son of Volsung, was a king in Frankish country. Sinfiotli was the eldest of his sons, the second was Helgi, the third Hamund. Borghild, Sigmund's wife, had a brother named — But Sinfiotli, her stepson, and — both wooed the same woman and Sinfiotli killed him over it.79 And when he came home, Borghild asked him to go away, but Sigmund offered her weregild, and she was obliged to accept it. At the funeral feast Borghild was serving beer. She took poison, a big drinking horn full, and brought it to Sinfiotli. When Sinfiotli looked into the horn, he saw that poison was in it, and said to Sigmund ‘This drink is cloudy, old man.’ Sigmund took the horn and drank it off. It is said that Sigmund was hardy and that poison did him no harm, inside or out. And all his sons could tolerate poison on their skin. Borghild brought another horn to Sinfiotli, and asked him to drink, and everything happened as before. And a third time she brought him a horn, and reproachful words as well, if he didn't drink from it. He spoke again to Sigmund as before. He said ‘Filter it through your mustache, son!’ Sinfiotli drank it off and at once fell dead.
Sigmund carried him a long way in his arms and came to a long, narrow fjord, and there was a small boat there and a man in it. He offered to ferry Sigmund over the fjord. But when Sigmund carried the body out to the boat, it was fully laden. The man said Sigmund should go around the fjord inland. The man pushed the boat out and then suddenly vanished.
King Sigmund lived a long time in Denmark in the kingdom of Borghild, after he married her. Then he went south to Frankish lands, to the kingdom he had there. Then he married Hiordis, the daughter of King Eylimi. Their son was Sigurd. King Sigmund fell in a battle with the sons of Hunding. And then Hiordis married Alf, the son of King Hialprec. Sigurd grew up there as a boy.
Sigmund and all his sons were tall and outstanding in their strength, their growth, their intelligence, and their accomplishments. But Sigurd was the most outstanding of all, and everyone who knows about the old days says he was the most outstanding of men and the noblest of all the warrior kings.
<s xml:id="S1">Sigmund ... was a king in Frankish country.</s>
<s xml:id="S2">Sinfiotli was the eldest of his sons.</s>
<s xml:id="S3">Borghild, Sigmund's wife, had a brother ...</s>
<s xml:id="S4A">But Sinfiotli ... wooed the same woman</s>
<s xml:id="S4B">and Sinfiotli killed him over it.</s>
<s xml:id="S5">And when he came home, ... she was obliged to accept it.</s>
<s xml:id="S6">At the funeral feast Borghild was serving beer.</s>
<s xml:id="S7">She took poison ... and brought it to Sinfiotli.</s>
<s xml:id="S17">Sinfiotli drank it off and at once fell dead.</s>
<anchor xml:id="EOS17"/>
</p>
<p xml:id="P2">Sigmund carried him a long way in his arms ... </p>
<p xml:id="P3">King Sigmund lived a long time in Denmark ... </p>
<p xml:id="P4">Sigmund and all his sons were tall ... </p>
<spanGrp resp="#TMA"
type="narrative-structure">
<span from="#S1" to="#S3">introduction</span>
<span from="#S4A">conflict</span>
<span from="#S4B">climax</span>
<span from="#S5" to="#S17">revenge</span>
<span from="#EOS17">reconciliation</span>
<span from="#P2" to="#P4">aftermath</span>
</spanGrp>
Note the use of an empty anchor element to provide a target for the ‘reconciliation’ unit which is normally part of the narrative pattern but which is not realized in the text shown.
The same analysis may be expressed with the interp element instead of the span element; this element provides attributes for recording an interpretive category and its value, as well as the identity of the interpreter, but does not itself indicate which passage of text is being interpreted; the same interpretive structures can thus be associated with many passages of the text. The association between text passages and interp elements should be made either by pointing from the text to the interp element with the ana attribute defined in section 18.2 Global Attributes for Simple Analyses, or by pointing at both text and interpretation from a link element, as described in chapter 17 Linking, Segmentation, and Alignment.
<s xml:id="MarQp1s2p114s1">There was certainly a definite point ... </s>
<s xml:id="MarQp1s2p114s2">It was not; then it was suddenly inescapable ... </s>
<seg xml:id="MarQp1s2p114s3-5"
ana="#moment">
<s xml:id="MarQp1s2p114s3">There was a slow integration ... </s>
<s xml:id="MarQp1s2p114s4">She felt the rivers under the ground ... </s>
<s xml:id="MarQp1s2p114s5">Not for one second longer ... </s>
</seg>
<s xml:id="MarQp1s2p114s6">For during that space of time ... </s>
</p>
<interp xml:id="moment">the moment</interp>
type="structuralunit">
<interp xml:id="INTRO">introduction</interp>
<interp xml:id="CONFLICT">conflict</interp>
<interp xml:id="CLIMAX">climax</interp>
<interp xml:id="REVENGE">revenge</interp>
<interp xml:id="RECONCIL">reconciliation</interp>
<interp xml:id="AFTERM">aftermath</interp>
</interpGrp>
<seg xml:id="SS1-SS3" ana="#INTRO">
<s xml:id="SS1">Sigmund ... was a king in Frankish country.</s>
<s xml:id="SS2">Sinfiotli was the eldest of his sons.</s>
<s xml:id="SS3">Borghild, Sigmund's wife, had a brother ... </s>
</seg>
<s xml:id="SS4A" ana="#CONFLICT">But Sinfiotli ... wooed the same woman</s>
<s xml:id="SS4B" ana="#CLIMAX">and Sinfiotli killed him over it.</s>
<seg xml:id="SS5-SS17" ana="#REVENGE">
<s xml:id="SS5">And when he came home, ... she was obliged to accept it.</s>
<s xml:id="SS6">At the funeral feast Borghild was serving beer.</s>
<s xml:id="SS17">Sinfiotli drank it off and at once fell dead.</s>
</seg>
</p>
<anchor xml:id="NIL1" ana="#RECONCIL"/>
<p xml:id="PP2">Sigmund carried him a long way in his arms ... </p>
<p xml:id="PP3">King Sigmund lived a long time in Denmark ... </p>
<p xml:id="PP4">Sigmund and all his sons were tall ... </p>
<join xml:id="PP2-PP4"
target="#PP2 #PP3 #PP4" ana="#AFTERM"/>
<link target="#INTRO #SS1-SS3"/>
<link target="#CONFLICT #SS4A"/>
<link target="#CLIMAX #SS4B"/>
<link target="#REVENGE #SS5-SS17"/>
<link target="#RECONCIL #NIL1"/>
<link target="#AFTERM #PP2-PP4"/>
</linkGrp>
One obvious advantage of using interp rather than span elements for the Sigmund text is that the interp elements can be reused for marking up other texts in the same document, whereas the span elements cannot. On the other hand, the use of interp elements may require the creation of special text elements not otherwise needed (e.g. the seg and the join in the revised encoding of the text), whereas the use of span elements does not.
TEI: Linguistic Annotation⚓︎18.4 Linguistic Annotation
By linguistic annotation we mean here any annotation determined by an analysis of linguistic features of the text, excluding as borderline cases both the formal structural properties of the text (e.g. its division into chapters or paragraphs) and descriptive information about its context (the circumstances of its production, its genre or medium). The structural properties of any TEI-conformant text should be represented using the structural elements discussed elsewhere in this chapter and in chapters 3 Elements Available in All TEI Documents, 4 Default Text Structure, 6 Verse, 7 Performance Texts, 8 Transcriptions of Speech, 10 Dictionaries, and 16 Language Corpora. The contextual properties of a TEI text are fully documented in the TEI header, which is discussed in chapter 2 The TEI Header, and in section 16.2 Contextual Information.
Other forms of linguistic annotation may be applied at a number of levels in a text. A code (such as a word-class or part-of-speech code) may be associated with each word or token, or with groups of such tokens, which may be continuous, discontinuous, or nested. A code may also be associated with relationships (such as cohesion) perceived as existing between distinct parts of a text. The codes themselves may stand for discrete and non-decomposable categories, or they may represent highly articulated bundles of textual features. Their function may be to place the annotated part of the text somewhere within a narrowly linguistic or discoursal domain of analysis, or within a more general semantic field, or any combination drawn from these and other domains.
The manner by which such annotations are generated and attached to the text may be entirely automatic, entirely manual or a mixture. The ease and accuracy with which analysis may be automated may vary with the level at which the annotation is attached. The method employed should be documented in the interpretation element within the encoding description of the TEI header, as described in section 2.3.3 The Editorial Practices Declaration. Where different parts of a language corpus have used different annotation methods, the decls attribute may be used to indicate the fact, as further discussed in section 16.3 Associating Contextual Information with a Text.
TEI: Linguistic Annotation by Means of Generic TEI Devices⚓︎18.4.1 Linguistic Annotation by Means of Generic TEI Devices
As one example of such types of analysis, consider the following sentence, taken from the Lancaster/IBM Treebank Project (Leech and Garside (1991)).
Our discussion focuses on the way that this sentence might be analysed using the CLAWS system developed at the University of Lancaster but exactly the same principles may be applied to a wide variety of other systems.80 Output from the system consists of a segmented and tokenized version of the text, in which word class codes have been associated with each token. CLAWS offers outputs in a variety of non-XML and XML formats: for example, the simplest format for the sample sentence would be:
The_AT0 victim_NN1 's_POS friends_NN2 told_VVD police_NN2 that_CJT Kruger_NP0 drove_VVD into_PRP the_AT0 quarry_NN1 and_CJC never_AV0 surfaced_VVD⚓
<w ana="#AT0">The </w>
<w ana="#NN1">victim</w>
<w ana="#POS">'s</w>
<w ana="#NN2">friends </w>
<w ana="#VVD">told </w>
<w ana="#NN2">police </w>
<w ana="#CJT">that </w>
<w ana="#NP0">Kruger </w>
<w ana="#VVD">drove </w>
<w ana="#PRP">into </w>
<w ana="#AT0">the </w>
<w ana="#NN1">quarry </w>
<w ana="#CJC">and </w>
<w ana="#AV0">never </w>
<w ana="#VVD">surfaced</w>
</s>
<interp xml:id="AT0">Definite article</interp>
<interp xml:id="AV0">Adverb</interp>
<interp xml:id="CJC">Conjunction</interp>
<interp xml:id="CJT">Relative that</interp>
<interp xml:id="NN1">Noun singular</interp>
<interp xml:id="NN2">Noun plural</interp>
<interp xml:id="NP0">Proper noun</interp>
<interp xml:id="POS">Genitive marker</interp>
<interp xml:id="PRP">Preposition</interp>
<interp xml:id="VVD">Verb past tense</interp>
</interpGrp>
This approach requires the text to be fully segmented, using the linguistic segment elements described in section 18.1 Linguistic Segment Categories, so that the scope of the ana attribute used to point to each interpretation is clearly defined. A further analysis into phrase and clause elements can be superimposed on the word and morpheme tagging in the preceding illustration. For example, CLAWS provides the following constituent analysis of the sample sentence (the word class codes have been deleted):
[N [G The victim's G] friends N] [V told [N police N] [Fn that [N Krueger N] [V [V& drove [P into [N the quarry N]P]V&] and [V+ never surfaced V+]V]Fn]V]⚓
<phr ana="#n">
<phr ana="#gn">
<w ana="#AT0">The</w>
<w ana="#NN1">victim</w>
<m ana="#POS">'s</m>
</phr>
<w ana="#NN2">friends</w>
</phr>
<phr ana="#v">
<w ana="#VVD">told</w>
<phr ana="#n">
<w ana="#NN2">police</w>
</phr>
<cl ana="#fn">
<w ana="#CJT">that</w>
<phr ana="#n">
<w ana="#NP0">Krueger</w>
</phr>
<phr ana="#v">
<phr ana="#v1">
<w ana="#VVD">drove</w>
<phr ana="#pr">
<w ana="#PRP">into</w>
<phr ana="#n">
<w ana="#AT0">the</w>
<w ana="#NN1">quarry</w>
</phr>
</phr>
</phr>
<w ana="#CJC">and</w>
<phr ana="#v2">
<w ana="#AV0">never</w>
<w ana="#VVD">surfaced</w>
</phr>
</phr>
</cl>
</phr>
<c ana="#pun">.</c>
</s>
<interp xml:id="v2">coordinate continuation</interp>
<interp xml:id="v">verbal</interp>
<interp xml:id="no">nominal</interp>
<interp xml:id="gn">genitive</interp>
<interp xml:id="fn">finite clause</interp>
<interp xml:id="pr">prepositional</interp>
<interp xml:id="v1">coordinate start</interp>
</interpGrp>
<w xml:id="word-1">The</w>
<w xml:id="word-2">victim</w>
<w xml:id="word-3">'s</w>
<w xml:id="word-4">friends</w>
<w xml:id="word-5">told</w>
<w xml:id="word-6">police</w>
<w xml:id="word-7">that</w>
<w xml:id="word-8">Kruger</w>
<w xml:id="word-9">drove</w>
<w xml:id="word10">into</w>
<w xml:id="word11">the</w>
<w xml:id="word12">quarry</w>
<w xml:id="word13">and</w>
<w xml:id="word14">never</w>
<w xml:id="word15">surfaced</w>
</s>
<link target="#word-1 #AT0"/>
<link target="#word-2 #NN1"/>
<link target="#word-3 #POS"/>
<link target="#word-4 #NN2"/>
<link target="#word-5 #VVD"/>
<link target="#word-6 #NN2"/>
<!--... -->
</linkGrp>
Each linguistic segment so far discussed has been well-behaved with respect to the basic document hierarchy, having only a single parent. Moreover, the segmentation has been complete, in that each part of the text is accounted for by some segment at each level of analysis, without discontinuities or overlap. This state of affairs does not of course apply in all types of analysis, and these Guidelines provide a number of mechanisms to support the representation of discontinuities or multiple analyses. A brief overview of these facilities is provided in chapter 21 Non-hierarchical Structures; also see 17 Linking, Segmentation, and Alignment. These mechanisms all depend to a greater or lesser degree on the use of pointing elements of various kinds.
TEI: Lightweight Linguistic Annotation⚓︎18.4.2 Lightweight Linguistic Annotation
While these Guidelines offer a variety of means to add linguistic information to textual units and much of that has been presented above, two kinds of use cases and two groups of users call for a dedicated set of specialized attributes to carry linguistic information. One relevant use case is where basic linguistic information gets added to an existing resource, in which generic attributes such as type or ana have already been used to encode other categorizations and analyses. The other group of users and use cases involves corpus linguists and resources built from scratch as lightly annotated language corpora. In the latter kind of projects, energy and person-hours are not devoted to careful literary analysis and hand-encoding of the relevant phenomena, but rather to the analysis of the completed resources, and therefore the phase of resource-building must be quick and relatively effortless, requiring minimal structural markup, well-established containers for grammatical information, and a standardized way of filling them in.
The aims defined above can be realized by means of lightweight linguistic annotation using attributes that belong to the att.linguistic class:
- att.linguistic provides a set of attributes concerning linguistic features of tokens, for usage
within token-level elements, specifically w and pc in the analysis module.
lemma fournit le lemme du mot (entrée du dictionnaire) pos (part of speech) indicates the part of speech assigned to a token (i.e. information on whether it is a noun, adjective, or verb), usually according to some official reference vocabulary (e.g. for German: STTS, for English: CLAWS, for Polish: NKJP, etc.). msd (morphosyntactic description) supplies morphosyntactic information for a token, usually according to some official reference vocabulary (e.g. for German: STTS-large tagset; for a feature description system designed as (pragmatically) universal, see Universal Features). join when present, provides information on whether the token in question is adjacent to another, and if so, on which side.
The essence of lightweight linguistic annotation is that the basic grammatical information is encapsulated at the word level, together with the orthographic shape of the word. This has clear advantages for automatic processing but, on the other hand, this form of data encapsulation also imposes restrictions on the extent of information that can be encoded, essentially limiting it to a single tokenization and lemmatization schema, a single tagset, and a subset of the possible analyses (out from potentially many guesses at the part-of-speech or morphosyntactic descriptions, single values have to fit into the existing attributes). Another important principle that this kind of annotation is sensitive to is the need for (near) homomorphism between the assumed tokenization (division of the text stream into minimal units) and the division into minimal syntactic units (word forms, in the terminology of ISO Morpho-Syntactic Framework, ISO 2461181), because it is the former that results from the process of tokenization, but the latter that can be lemmatized and meaningfully described by means of grammatical features. Where tokens are only minimally mismatched with word forms, various repair strategies can be used (e.g., recursing w to capture multi-token compounds or using att.fragmentable to point at disjoint tokens). Beyond that, more robust TEI mechanisms, based on standoff principles and feature structures, should replace lightweight annotation.
The basic grammatical information encoded by means of att.linguistic is sufficient for the purpose of enhancing queries or improving the analysis of search results by, for example, making it possible to distinguish between the noun cut and the identically spelled verb cut in English, and further between e.g. the present-tense form of cut and its past-tense or past-participial forms. For the former contrast, the part-of-speech (pos) attribute should be used, whereas the latter may use pos and/or msd attributes, depending on the annotation vocabulary adopted for the project in question. The various grammatical realizations of a single ‘dictionary word’ can be captured by means of the attribute lemma, which provides a common label for them. For example, English verbs are typically lemmatized as the base form (also called bare infinitive), so the value of lemma for the verbal forms write, writes, wrote, written, and writing is typically write.
Together with the span-delimiting elements mentioned in this section, such as s, cl, or phr, lightweight grammatical annotation may be used to build basic syntactic constituency structures, where hierarchical information is expressed through span containment rather than by relations among tree nodes. This is however the limit of this kind of annotation: for the purpose of describing true constituency or dependency syntactic structures, one needs to turn to more robust mechanisms offered by the TEI, which may involve graph description (see chapter 20 Graphs, Networks, and Trees) or standoff techniques (see section 17.9 Stand-off Markup), and where grammatical labels may need to be annotated by means of feature structures (see chapter 19 Feature Structures).
Some of the above-mentioned robust methods will also prove handy in cases where more than one tagset (label inventory) is used to label the words, or where automatic morphological analysis yields multiple possibilities (for example, the form cutting is morphologically ambiguous between verbal, adjectival, and nominal) and needs to be followed by (often also automatic) disambiguation in morphosyntactic contexts, with varying probabilities that may also need to be recorded together with their corresponding part-of-speech and morphosyntactic values.
It should be borne in mind that tokenization, lemmatization, part-of-speech identification, and morphosyntactic labelling, especially when performed automatically, should in most cases be seen as involving pragmatic decisions, dictated by concrete practical goals, economy of description, or the demands of particular analytic and/or visualization tools. It comes therefore as no surprise that numerous alternative (and often conflicting) lemmatization strategies and tagsets exist, in use by various communities and various tools, and that they change with time (a case in point is the CLAWS tagset for English, with several versions that merge the part-of-speech and morphosyntactic information to various degrees). 82 The nature and description of these systems is outside the scope of these Guidelines, but it has to be stressed that all the strategies adopted for linguistic annotation, even at the lightweight level of complexity, must be documented in the header of the given electronic resource, not only for the purpose of guaranteeing successful data interpretation and exchange, but also for the sake of sustainability of the results of the given project.
The last of the att.linguistic attributes, join, has the most text-technological flavour. It can be used to amend the loss of whitespace-related information in non-inline markup.
Compare the following two listings. The first difference between them is in the tagset used (CLAWS-5 vs. CLAWS-7) and only serves to exemplify the need to document the choice of descriptive vocabulary in the header, lest the encoded information is unreadable or confusing. The second difference is the difference in the treatment of inter-token whitespace, and it is here that the join attribute proves indispensable.
<w pos="AT">The</w>
<w pos="NN1">victim</w>
<w pos="GE" join="left">'s</w>
<w pos="NN2">friends</w>
<w pos="VVD">told</w>
<w pos="NN2">police</w>
<w pos="CST">that</w>
<w pos="NP1">Kruger</w>
<w pos="VVD">drove</w>
<w pos="II">into</w>
<w pos="AT">the</w>
<w pos="NN1">quarry</w>
<w pos="CC">and</w>
<w pos="RR">never</w>
<w pos="VVD">surfaced</w>
<pc pos="." join="left">.</pc>
</s>
Note that projects will need to decide whether they want to redundantly encode full information on the adjacency of each token (in which case, the above listing should also have join with the value right on the tokens victim and surfaced, or whether information on a single direction of adjacency is enough. Strategies vary, and it is important to document them in the TEI header.
<w pos="PPER" lemma="wir"
msd="pers:subst:p1:nom:pl">Wir</w>
<w pos="VVFIN" lemma="fahren"
msd="p1:pl:pres:ind">fahren</w>
<w pos="APPR" lemma="in" msd="--">in</w>
<w pos="ART" lemma="d"
msd="def:acc:sg:masc">den</w>
<w pos="NN" lemma="Urlaub"
msd="acc:sg:masc">Urlaub</w>
<pc pos="$." lemma="." msd="--" join="left">.</pc>
</s>
The final examples lay out a strategy for dealing with e.g. historical corpora where it is on the one hand important to maintain a steady stream of token-level elements (w and pc) for efficient processing, but, on the other hand, it is also important to either record the original spelling (when the corpus text is normalized) or to record the normalized variants (when the element content of the corpus preserves the original spelling). The attribute class att.lexicographic.normalized can be used for that purpose:
- att.lexicographic.normalized provides attributes for usage within word-level elements in the analysis module and
within lexicographic microstructure in the dictionaries module.
norm (normalisé) donne une forme normalisée de l'information fournie par le texte source sous une forme non normalisée. orig (original) indique la chaîne originale ou contient une chaîne vide si l'élément n'apparaît pas dans le texte source.
<w norm="Freundschaft">Freundſchafft</w>
<w norm="angeboten">angebotten</w>
<w lemma="have" pos="vvz">hath</w>
<w lemma="bring" pos="vvn">brought</w>
<w lemma="forth" pos="av" orig="sorth">forth</w>
TEI: Spoken Text⚓︎18.4.3 Spoken Text
<u xml:id="u2">Yes, anything else?</u>
<u xml:id="u3">No thanks.</u>
<u xml:id="u4">That'll be dollar forty.</u>
<u xml:id="u5">Two dollars</u>
<u xml:id="u6">Sixty, eighty, two dollars. Thank you.</u>
<spanGrp type="transactions">
<span from="#u1">sale request</span>
<span from="#u2" to="#u3">sale compliance</span>
<span from="#u4">sale</span>
<span from="#u5">purchase</span>
<span from="#u6">purchase closure</span>
</spanGrp>
TEI: Module for Analysis and Interpretation⚓︎18.5 Module for Analysis and Interpretation
The module described in this chapter makes available the following components:
- Module analysis: Mécanismes analytiques simples
-
- Eléments définis: c cl interp interpGrp m pc phr s span spanGrp w
- Classes définies: att.global.analytic att.lexicographic.normalized att.linguistic
The selection and combination of modules to form a TEI schema is described in 1.2 Defining a TEI Schema.
[cat pronoun, personal true, substituting true, person 1, case nominative, number
plural]
, and has been mapped from a sequence of attribute-value pairs suitable for feature
structure notation, into a compressed form that fits inside a single attribute value.