BioInfer: a corpus for information extraction in the biomedical domain

Entity relationship annotation

The entity relationship annotation, or relationship annotation in short, is based on the notions of an entity and a relationship. An entity denotes a named bioentity or a physical or abstract entity pertaining to a named bioentity. A relationship captures a stated relation between two or more entities or other relationships. Both the entities and the relationships reflect the information explicitly stated in the sentence. They exist in their own right, abstracted from the sentence text, and we first discuss them on this level. Later in this section, we introduce how the entities and relationships are bound to the actual text of the sentences.

The relationships belong to a variety of relationship types and are often structurally complex. The relationship types can be, for example, observed co-occurrence, sequence similarity, or physical binding. For example, the two relationships (bind, promote) in the sentence

mDia binds profilin to promote actin polymerization.

are of distinctly different types. Further, a relationship can be stated on different levels of specificity. In the sentence

Aip1p interacts with cofilin to disassemble actin filaments.

there are two relationships: interact and disassemble. Clearly, disassemble is a more specific statement than interact.

In order to capture the various relationship types as well as their different levels of specificity, we introduce the relationship type ontology that defines relationship types and organizes them in a hierarchical manner, from the most general to the most specific relationship types. The ontology specifies over 60 relationship types at 5 levels of generality. Each annotated relationship is categorized within this ontology at the most specific level applicable.

Apart from belonging to a large variety of types, the relationships are also structurally complex. The prevailing approach in previously published biomedical corpora has been to only annotate pairwise relationships between physical entities. However, it becomes clear that this representation in many cases fails to accurately capture the stated information. Consider, for example, the following sentence.

Four yeast spliceosomal proteins (PRP5, PRP9, PRP11, and PRP21) interact to promote U2 snRNP binding to pre-mRNA.

Using only pairwise relationships, the sentence would most likely be decomposed into all pairwise relationships among the spliceosomal proteins and all pairwise relationships between the spliceosomal proteins and U2 snRNP. However, any such decomposition leads to significant imprecisions. The sentence does not specify which pairwise interactions take place among the four spliceosomal proteins, nor does it assert a pairwise relationship between any of the spliceosomal proteins and, for example, U2 snRNP. Instead of using only pairwise relationships, the information in the sentence can be better captured through structurally more complex relationships: an interact relationship among all the four spliceosomal proteins and a promote relationship between the interact relationship and the pairwise bind relationship. In the BioInfer corpus, the relationships are annotated using logic formulas. Each relationship is expressed through a predicate whose arguments are instantiated with entities or other predicates. The predicate name, arity, and the semantic roles of its arguments are defined in the relationship type ontology together with a description of the relationships intended to be annotated using this predicate. The relationship annotation of a sentence is then a set of formulas that state all the relevant relationships in the sentence. Let us consider the previous example sentence. The formula

INTERACT(PRP5, PRP9, PRP11, PRP21)

expresses the relationship among the spliceosomal proteins. In the usual interpretation, it does not explicitly assert a pairwise relationship between any two of the four proteins. The formula

STIMULATE(INTERACT(PRP5, PRP9, PRP11, PRP21), BIND(U2 snRNP, pre-mRNA))

describes the relationships stated in the sentence, accounting for their complex nature. It is this formula that is used in the corpus to annotate the example sentence.

The formula-based annotation system is more powerful than the simple pairwise relationship annotation. However, practical considerations and the current state-of-the-art in IE systems often call for pairwise relationships, even at the cost of introducing inaccuracies such as those discussed above. It is thus necessary to employ a scheme where the annotation can be decomposed in terms of pairwise relationships. The BioInfer annotation scheme allows such decomposition. Multiple-argument predicates such as INTERACT(PRP5, PRP9, PRP11, PRP21) can be decomposed in the obvious manner and predicates whose arguments are other predicates can be either omitted or recursively decomposed. It depends on the predicate whether better approximation can be achieved by omission or by recursive decomposition. Several examples of the relationship annotation are provided in Appendix I.

Entity annotation

The BioInfer corpus is focused on the development of IE systems for extracting relationships between genes, proteins, and RNAs. This focus influences the entity annotation as currently only entities that are relevant to this focus have been annotated. As a typical IE system extracts relationships between named entities, we require that an entity is named or pertaining to a named entity, more specifically a named gene, protein, or RNA, in order to be relevant to the corpus focus. For example, actin is a named entity, actin expression pertains to a named entity, but a 50 kDa protein as an entity is not considered named and thus not annotated in the current version of the corpus. The typical use in literature and databases such as Swiss-Prot is used as a guide in deciding for borderline cases whether an entity is considered named or not named.

In addition to named entities, extended named entities, defined as terms denoting other physical entities, processes and properties pertaining to named entities, are also annotated when relevant to relationships. Consider the following example:

Deletion of SIR4 enhanced mURA3 and METI5 silencing.

The sentence contains three named entities (SIR4, mURA3, MET15). Further, the entity deletion of SIR4 is a process extended named entity pertaining to SIR4, similarly for mURA3 silencing and METI5 silencing. In the sentence

Finally, both receptors can interact with FADD, TRADD, and RIP.

the annotation identifies three entities: the named entities FADD, TRADD, and RIP. The reference to both receptors does not constitute an entity, because it is not a named entity, nor it is pertaining to a named entity (within the sentence).

While it is common to restrict IE systems to only take into account named bioentities as objects participating in relationships, consideration of extended named entities allows a better resolution of the actual statements in the sentence. Consider the following example:

Abundance of actin is affected by the differential calreticulin expression.

Although it is not incorrect to state that calreticulin affects actin, a more detailed resolution of the entities is desirable. The affect relationship in this sentence is between two property/process entities: abundance of actin and calreticulin expression. Abundance of actin is a property pertaining to actin and calreticulin expression is a process pertaining to calreticulin. Both actin and calreticulin are named entities. As a further illustration, consider the sentence

Characterization of gelsolin truncates that inhibit actin depolymerization by severing activity of gelsolin and cofilin.

where the related entities are gelsolin truncates, actin depolymerization, activity of gelsolin and activity of cofilin rather than the proteins gelsolin, actin, and cofilin themselves.

The need to resolve that, for instance, abundance of actin is a protein amount property pertaining to the protein actin leads to two crucial entity annotation concepts: entity nesting and entity typing. Entity nesting allows annotation of entities within entities with the outer entity typically being a semantic modification, extension or specification of the inner entity. For instance, the outer entity abundance of actin is a specification of the inner entity actin in the sense that it identifies a single property of actin. Each entity is given a type from the hierarchical ontology of entity types developed for this purpose. The ontology captures all relevant entity types, both physical entity types such as gene and protein as well as abstract entity types such as process and property. Similarly to the relationship type ontology, the entity types are organized in a hierarchical manner and in the annotation each entity is given the most specific applicable type from the ontology.

The introduction of entity nesting and typing allows us now to state a general principle regarding the entity annotation. Only named entities of the gene, protein, and RNA types are annotated in each sentence. In addition, those named entities that participate in a relationship are nested inside the broadest entity relevant to the particular relationship. Thus, up to special cases described later, every entity that participates in an annotated relationship has a named gene, protein or RNA at its innermost level of nesting.

This is demonstrated in the example sentence

Dispersal of profilin_A from such sites suggests that profilin_B is involved in reorganization of actin cytoskeleton.

where the entities profilin_A, profilin_B, actin, actin cytoskeleton, and reorganization of actin cytoskeletonwould be annotated. Since the process dispersal of profilin_A does not participate in any annotated relationship, it is not annotated. The protein actin, on the other hand, is nested in the entity actin cytoskeleton, which is in turn nested in the entity reorganization of actin cytoskeleton, the broadest entity relevant to the stated relationship PARTICIPATE(profilin_B, reorganization of actin cytoskeleton).

Relationship and entity type ontologies

Before discussing the BioInfer relationship ontology, we briefly characterize its design goals in relation to those of other ontologies. There are a number of biomedical and top-level ontologies, each of which defines a part of the domain of the entities and relationships identified in BioInfer. For instance, many physical relationships on the molecular level are specified within the molecular function ontology of Gene Ontology (GO) [18]. Similarly, control processes such as upregulation are defined in the biological process ontology of GO. On the other hand, non-process relationships such as sequence similarity fall in scope of BioInfer annotation, but are not defined in GO. Moreover, BioInfer captures generic relationships such as condition whereby unknown molecular dependencies result in a process or event being a necessary condition for another process or event to take place. For instance in the sentence

Alpha-catenin links beta-catenin to the actin-based cytoskeleton

Alpha-catenin causes an underspecified relationship between beta-catenin and the actin-based cytoskeleton. However, the molecular basis for these relationships remain unspecified. There are no terms corresponding to the cause/link relationships at this level of generality in GO.

Another consideration is the aim of the BioInfer corpus at supporting IE systems in the domain. While pathway ontologies such as BioPAX [19] aim at the description of known molecular pathways, BioInfer aims at capturing what was stated in free text, which is usually not definite knowledge. The BioInfer relationship ontology captures the domain of statements about relationships rather than the relationships themselves. Thus, by using the ontology, the annotation avoids inference which refines text statements into pathway knowledge, and captures only the explicitly stated facts. Taking into account the considerations discussed above, we have designed the ontology to capture the relationships present in the BioInfer corpus text at a level of detail suitable for both the annotators and later application of the corpus to IE system development. The ontology was constructed based on the corpus data, to cover statements about the biologically relevant relationships found in the corpus text. It by enumeration defines the set of relationships relevant with respect to BioInfer. In this respect, it is closer to a hierarchical lexicon rather than a complete ontology defining the broad domain of biomedical entity interactions and relationships.

The relationship type ontology (Figure 3) specifies four broad classes of relationships. The observation class captures experimental observations, such as co-occurrence and coprecipitation. The classes part_- of and is_-a describe taxonomical, substructure/superstructure, physical/functional similarity, and equality relationships. The fourth class of relationships specifies causal relationships, where one entity causes a change in the state of another entity. While the class structure of the relationship ontology is fixed, the predicates are introduced into the ontology according to need – the choice of predicates is thus empirical and depends on the coverage of the annotated corpus. For example, of all the possible post-translational modifications only few, such as PHOSPHORYLATE, were sufficiently common in the corpus to merit a separate predicate in the ontology. To illustrate the coverage of the predicates of the annotated relationships and to estimate how often new predicates would need to be added in extending the annotation, we calculated the cumulative total number of different predicates that occur in the annotation when traversing the sentences in an arbitrary order (Figure 4). We find that, for example, 50% of all predicates are, on average, seen already after the first 74 sentences. Further, the total number of predicates seen increases only by one in approximately the last 220 sentences. We may extrapolate that to annotate another 1100 sentences, only approximately five new predicates would need to be introduced into the ontology.

The state of an entity, for the purpose of the BioInfer corpus, is characterized by four properties: amount, location, dynamics, and physical. These properties specify the amount, spatial distribution, degree of activity, and physical state of the entity. In the relationship type ontology, the change node, a specialization of the causal node, has for each of the four properties a subtree that contains predicates for relationships that affect the respective property, thus resulting in a change of the state. For example, an upregulation relationship affects the dynamics of an entity and therefore the UPREGULATE predicate is located in the dynamics subtree.

The Genia event ontology [20] serves a similar purpose as the BioInfer relationship ontology. It will be used in the future event annotation of the Genia corpus and has been designed as a modified subset of 34 terms from GO biological process (32 terms) and molecular function (2 terms) ontologies, with three additional terms. There are similarities in the structure of the BioInfer relationship and Genia event ontologies. For example, the metabolism vs. regulation distinction in the Genia event ontology is comparable to the physical vs. dynamics distinction in the BioInfer relationship ontology. However, the Genia event ontology focuses on the metabolism subtree, while the BioInfer relationship ontology focuses on the dynamics subtree, both ontologies providing a more detailed resolution at their focus point. Additionally, the Genia event ontology does not currently support non-process relationships.

Several groups have undertaken efforts to define and use frames, or conceptual structures, mostly focusing on the definition of predicate-argument structures for verbs [21-24]. Frame recognition has been proposed as a step in information extraction processes. In the biomedical domain a recent effort, PASBio, defined argument structures for 30 verbs known to be used to describe biological events [25]. PASBio takes a verb-oriented approach following the PropBank scheme. The BioInfer relationship annotation differs from efforts to define and use verb frames for IE in both aims and focus. However, we note that in this context the BioInfer approach most resembles FrameNet in that the annotation is centered around relationships instead of verbs and includes relationships stated through words other than verbs. Also, by contrast to PropBank and PASBio, BioInfer defines only the minimal core arguments of the relationship specifying the participants in the relationship.

Before we introduce the entity type ontology, let us consider an important contrast:

(i) entity_A causes entity_B dephosphorylation

(ii) entity_A dephosphorylates entity_B

(iii) entity_A inhibits entity_B dephosphorylation

There are two possible ways to annotate the sentence (i):

(a) CAUSE(entity_A, entity_B dephosphorylation)

(b) DEPHOSPHORYLATE(entity_A, entity_B)

The annotation (b) differs from the annotation (a) by interpreting the verb cause. By contrast, the sentences (ii) and (iii) have each only one possible annotation in the BioInfer corpus: the sentence (ii) would be annotated using (b) and the sentence (iii) would be annotated using (a) with CAUSE replaced by SUPPRESS. The sentences (i) and (iii) have the same surface structure and it would thus be practical if their annotation formulas had the same structure. On the other hand, sentences (i) and (ii) have a very similar (although not fully equal) meaning and it would thus be desirable for them to have the same annotation. Generally, this situation arises with generic statements such as cause, result in, and lead to. We chose not to interpret such generic statements. The sentence (i) would thus be annotated according to (a). To alleviate the impact of the decision on the contrast of (i) with (ii), we design the entity type ontology to mirror parts of the relationship type ontology in a way that allows automatic reinterpretation of the annotation from (a) to (b). We now introduce the entity type ontology.

The entity type ontology (Figure 5) comprises of three main subtrees. The physical entity subtree is, up to minor differences, the Genia ontology of physical entities. For example, we introduce gene as a specialization of domain or region of DNA and fusion protein as a specialization of individual protein. The process subtree mirrors the change subtree in the relationship type ontology. For almost every state change in the change subtree of the relationship ontology (e.g. phosphorylate) there is a corresponding process in the process subtree of the entity type ontology (e.g. phosphorylation). The predicates in the change subtree of the relationship type ontology are used in sentences like (ii) while their corresponding process entity types are used in sentences like (i) and (iii). The last main subtree of the entity type ontology is the property subtree, which defines the main properties associated with entity state, that is, amount, location, function, dynamics, and physical state.

The interrelationship of the ontologies is illustrated in Figure 6. The key relationship between the ontologies is the correspondence of the process and change subtrees. This correspondence is due to the practical focus of the corpus on IE. In isolation, process extended entities are not of relevance, because they do not specify a relationship of several named bioentities. For instance, cofilin phosphorylation describes a biological process which is not of interest as a relationship PHOSPHORYLATE(_, cofilin), because its agent is left unspecified. However, it is of interest as a process if it participates in some other relationship. For this reason, we maintain separate definitions for the same underlying biological phenomenon as a relationship and as a process extended named entity. However, due to the correspondence between the ontologies, it is possible to automatically reinterpret the process extended named entities in terms of underspecified relationships, as exemplified in this paragraph, and thus remove the distinction.

Text binding

We have so far introduced the entities and relationships without discussing their binding to the text in the corpus. For instance, in the sentence

Inhibition of actin_A polymerization by a synthetic dodecapeptide patterned on the sequence around the actin_B-binding site of cofilin.

the desired output of an IE system is the BIND(cofilin, actin) relationship. While the exact words in the sentence which were used to extract this information, in this case actin_A vs. actin_B, are not necessarily of particular interest to the users of the IE system, they are of crucial importance in its development. It is important to specify text binding for the entities and relationships since correct information extracted from incorrect words is usually considered incorrect in IE system evaluation. In the example above, if the IE system used the word actin_A to extract the relationship BIND(cofilin, actin), the output would have to be considered erroneous.

Up to very few exceptions detailed later in the Methods section, each entity and each predicate in the BioInfer corpus annotation is bound to the sentence text. As a rule, an entity is always bound to the minimal text that suffices to resolve the identity of the entity and its type within the ontology. A predicate is bound to the minimal text that suffices to infer the predicate name and thus the type of the corresponding relationship. Typically, assigning the text binding for an entity is straightforward and clear rules can be constructed to guide the annotators.

The text binding for a predicate is, on the other hand, less straightforward and developing a set of rules to guide the annotators is more difficult. Most relationships are stated with a verb and this verb then becomes the text binding for the predicate. There are, however, many other word and phrase categories that appear in a text binding for a predicate, such as also known as stating name equality and recruitment of ... to stating a localization relationship. The choice of the predicate based on the sentence text is context-dependent: there is no one-to-one correspondence between the words and phrases of the text and the predicates used. The relationship type ontology serves as a controlled vocabulary: it defines a set of types, each of which can be stated in many different ways in the text. For example, all of the phrases affinity for, bind to, associate with, cofactor, contact to, contains site for, epitopes on, receptor for, and many other, have been used in the corpus in text binding for the predicate BIND. On the other hand, the phrase associate with is also commonly bound to the predicate RELATE in cases where it does not imply direct physical binding. There is thus a many-to-many correspondence between the actual text phrases and the relationship types.

For the purpose of the dependency annotation, the sentence text is divided into tokens following a simple set of rules that can be easily implemented. However, for the entity and relationship annotation the tokens are not sufficiently detailed. Consider, for example, the token Arp2/3 which contains two entities, Arp2 and Arp3. We aim at capturing such entities and consequently our entity annotation scheme allows the tokens to be divided into sub-tokens. In the example, the token Arp2/3 is divided into the subtokens Arp 2 / 3. These subtokens are then combined into two entities, Arp2 and Arp3. The text correspondence of an entity is thus defined as a set of subtokens that, when textually concatenated, form the string identifying the entity. For nested entities, the subtokens of the inner entity are a proper subset of the subtokens of the outer entity.

Let us now summarize the key concepts in the entity relationship annotation. Relationships hold between entities. Entities can be either physical entities such as proteins or abstract entities such as processes and properties. The entities are nested to capture their inner structure. The type of each entity is annotated using the entity type ontology. The relationships among the entities are stated as logic formulas with predicates defined in the relationship type ontology, which also specifies the meaning of each predicate and the roles of its arguments. The annotation is bound to the text through sets of subtokens assigned to the entities and predicates. The ontologies are interdependent with the connection being the notion of entity state, state properties, and changes in state.

Dependency annotation

Many of the relationship statements are syntactically complex, involving, for example, coordination and long-distance dependencies between the words stating the relationship. To extract such relationships, many recently proposed IE systems employ full parsing [26-30]. To develop and evaluate both the parsing and domain analysis components of such systems, syntactic annotation is necessary.

The BioInfer corpus contains syntactic annotation using the Link Grammar (LG) formalism. LG is a well-documented dependency-type grammatical formalism with the advantage that the Link Grammar Parser is freely available and open source [31]. The LG grammar and its documentation can thus be examined in detail and serve as a reference for both annotators and users of the corpus. LG has further been applied to several BioNLP tasks, including IE targeting protein-protein interactions [27,29,32,33]. In producing the BioInfer corpus annotation, the LG formalism was followed systematically and extended only when necessary to provide annotation for phenomena not covered by the current version of the grammar. Due to the large number of dependency types in the LG formalism (approximately 100 main types and 400 types in total), we use an automatic method to assign the types; the method is presented in the Methods section.

In addition to considering correctness and completeness requirements in creating the annotation, we also take into account practical aspects of IE system development and adopt some of the approximations that are implemented in parsers that help to increase robustness and decrease ambiguity, while not being harmful to IE performance. For example, most parsers do not fully resolve noun phrase (NP) bracketing. As noun phrase internal structure is not a key issue for IE systems, we find the approximation acceptable and do not fully resolve NP bracketing. In contrast, we resolve semantically important ambiguities in the text such as prepositional phrase attachment, coordination, and relative clause attachment.

To extend the applicability of the dependency annotation we introduce macro-dependencies, a special form of annotation that can be expanded in different ways corresponding to the different, yet equally plausible, dependency analyses of a single grammatical phenomenon. Macro-dependencies thus allow systematic differences between dependency annotation schemes to be taken into account. We define NP macro-dependency to address the possibly most frequent systematic difference between dependency formalisms, the parallel vs. serial (chained) attachment of pre-modifiers to a noun. The choice of attachment scheme is largely arbitrary, with, for example, LG and MiniPar [34] attaching the pre-modifiers in parallel, and, for example, Connexor Machinese Syntax [35] attaching the pre-modifiers serially. Each parser systematically follows one of these two schemes instead of attempting to fully resolve the NP structure.

In the annotation, an NP macro-dependency is used to connect the leftmost pre-modifier and the head noun. An NP macro-dependency can be expanded automatically to attach all pre-modifiers spanned by the macro-dependency in either of the two manners introduced above, making the annotation applicable to the two most common analyses (see Figure 7). Further, the macro-dependency is used consistently also in cases where there is only one pre-modifier. This allows the mere presence of the macro-dependency to be used as an indicator of a non-elementary noun phrase.

Macro-dependency annotation can be defined for many other grammatical phenomena where parsers systematically differ. However, annotation with many different types of macro-dependencies would increase the complexity of the annotation process and would require greater efforts from the annotators. For these reasons, we currently only apply NP macro-dependencies.

Quality of the annotation

As also discussed in the Methods section below, several steps were taken to assure the quality of the annotation, including redundant dependency annotation, frequent discussions among annotators to resolve and document ambiguous cases, and repeated verification of the annotated data against a set of written rules. These rules were formulated iteratively, from an initial tentative version to their final form documented in the annotation manual [36]. For the current initial release of the corpus, we did not undertake the effort to measure inter-annotator agreement, a measure of the stability of the annotation scheme. Rather, we have focused our limited resources on annotating as many sentences as possible while mainitaining the quality standard. Nevertheless, quantifying the reliablility of the annotation is an important issue that we intend to address in the future. We discuss below some of the issues relating to the measurement of inter-annotator agreement for the various types of annotation present in the corpus. The standard approach to measuring inter-annotator agreement on categorization tasks is to fully annotate a set of sentences by two separate annotators and calculate the kappa statistic [37,38], defined as $\kappa =\frac{P(a)-P(e)}{1-P(e)}$, where P(a) is the measured probability of agreement between annotators, and P(e) is the probability that agreement is due to chance. As the BioInfer corpus contains several types of annotation, each of which is to some extent independent of the others, it is natural to measure agreement separately for the different annotation types. However, even with this simplification, there are a number of difficulties in calculating the κ statistic for many of the annotation types, including the following:

• If may be difficult or impossible to calculate κ when the set of possible annotations is very large or not clearly defined [39]. For example, the space of possible entity occurrences, potentially discontinuous and overlapping, is huge, causing simple estimates of P(a) to approach one and P(e) to approach zero: annotators will agree that the vast majority of possible entities should not be annotated. Similar issues arise in relationship annotation and, to some extent, also in dependency annotation.

• In calculating κ, the annotation categories are assumed to be discrete and mutually exclusive, so that each categorization is either correct or absolutely incorrect. However, as the entity and relationship types are hierarchical, it would be appropriate to recognize different degrees of correctness in type categorization, for example when one annotator assigns a specific and another a related, but less specific type to an entity.

• The κ statistic essentially measures adjusted accuracy, which is not the favored measure of performance for many of the annotation types in the corpus. For example, for entity recognition, separate measurement of exact and sloppy matches [32] or boundary matches [40] could be more informative.

For these reasons we intend, instead of using κ, to measure agreement separately for the different annotation types, applying the most informative measures for each type.

Sentence simplification

NP macro-dependencies have an important role in the simplification of the sentences. As the attachment of pre-modifiers in noun phrases covered by NP macro-dependencies can be determined with simple rules without parsing, any NP macro-dependency present in the sentences can be truncated so that all the pre-modifiers spanned by the macro-dependency are removed. This procedure reduced the number of words by more than 20% and removed all problems related to the parsing of these noun phrases. The sentences were further simplified by capitalizing common uncapitalized proper names and removing citations.

Parsing

We parsed the simplified sentences with a modified version of the LG parser that overcomes some of the problems of the unmodified LG parser for biomedical text [10]. The modifications include an extended biomedical lexicon [33] as well as support for many of the grammatical phenomena not recognized by the unmodified parser. For each sentence, we consider up to a thousand alternative parses produced by the parser.

Majority types

For each sentence, we determine by comparison against the manually created dependency structure the subset of parses with the maximal number of recovered dependencies. The intuition underlying this selection heuristic is that parses having correct dependency structure also have the correct dependency types. From the set of maximally correct parses, we then determine for each dependency the majority type, that is, the type which most commonly appears for the dependency in the parses.

Finalization

We transfer the majority types from the simplified to the full sentences and resolve the types of the dependencies associated with the truncated noun phrases using straightforward rules.

We found that the quality of dependency types is significantly higher for sentences that receive fully correct parses in the parsing step. We thus mark sentences depending on whether or not they were fully correctly parsed, allowing users of the corpus to choose whether to use all dependency types or only the most reliable ones. To evaluate the quality of the automatically produced dependency types, we manually examined 30 randomly chosen sentences with fully correct parses and 30 randomly chosen sentences for which fully correct parses were not obtained.

For the fully correctly parsed sentences, the heuristic method assigned the correct dependency type to 575 out of 586 dependencies (98%). For the sentences without fully correct parse, 828 out of 962 dependencies (86%) were assigned their correct type. Out of the 1100 sentences, 741 (67%) received a fully correct parse.