Automatic Resolution of Ambiguous Terms Based on Machine Learning and Conceptual Relations in the UMLS

WSD Research Using Relations

Many machine-readable dictionaries contain a rich set of relationships linking senses. For example, all nouns in WordNet,¹² which is a handcrafted online lexicon, are organized into one conceptual network through IS-A relationships. For a term W, we define a term that has relations with a sense S of W in a conceptual network as a conceptual relative of W via an associated sense S. For example, the word summer is a conceptual relative of the word spring with an associated sense the springtime sense of spring, since summer and the springtime sense of spring are seasons of the year and the relation between them is a sibling relation (i.e., they share a common parent—the season of the year). There are two observations for WSD methods that use conceptual relatives. One is that conceptual relatives with certain relations tend to appear in similar contexts. For example, the summer and springtime sense of spring can appear in similar contexts, such as “Spring is my favorite season” and “Summer is my favorite season.” The other observation is that correct senses for the words in a natural language expression will have closer sense relations (in a conceptual network) than incorrect combinations of senses. For instance, in “Spring is my favorite season,” the springtime sense of spring has a IS-A relation with the season of the year, while other combination of senses (e.g., spring as a fountain and season as sports season) have weaker relationships.

There are two different WSD methods using conceptual relatives based on the above two observations. One method uses unambiguous conceptual relatives with certain relations, such as IS-A or synonymy, in a concept-oriented dictionary to derive sense-tagged corpora automatically for use with supervised WSD classifiers. Our previous work,²⁴ as described earlier, belongs to this category. For example, we can build a sense-tagged corpus for CSF by extracting all instances that contain the full forms cerebral spinal fluid or colony-stimulating factor and then replacing the full forms by the ambiguous term CSF along with the corresponding senses. A WSD classifier is then constructed automatically from the sense-tagged corpus by applying supervised machine learning techniques (the detailed information about supervised machine learning will be described later). An experiment demonstrated that classifiers trained on the derived sense-tagged corpora achieved an overall precision of about 97% for terms that had unambiguous synonyms found in the UMLS, with greater than 90% precision for each individual ambiguous term.

The other method consists of looking up the conceptual relatives of an ambiguous term W in the context. The method takes a number of conceptual relatives via a relation and a formula to measure the relatedness of conceptual relatives with each of the senses of W in a concept-oriented dictionary, and then uses them to determine the sense of W in the context. In the general English domain, researchers usually choose WordNet as the concept-oriented dictionary. Sussna²⁶ used several relation types (such as IS-A relation, synonymy) in WordNet and chose a measure that takes account of the shortest path, the number of edges associated with the same type leaving a node, the depth of a given edge in the overall tree, and a weight assignment for each relation type. Sussna evaluated his method on five documents from Time magazine and compared it to human experts using the same evidence, with a precision measure of 52.3%. Agirre and Rigau²³ proposed a method that used conceptual relatives via the relation IS-A and chose a measure which is sensitive to the following parameters: the length of the shortest path that connects the concepts involved, the depth in the hierarchy and the density of concepts in the hierarchy. Agirre and Rigau²⁷ evaluated their method on the noun portion of a document that contained 2,079 words. The overall performance was measured in terms of precision and recall, with 66.4% for precision and 58.8% for recall.

One Sense per Discourse

In their experiments with WSD, Gale, Church, and Yarwosky²⁸ observed a strong relationship between discourse and sense. They proposed a hypothesis: one sense per discourse. When a word occurs more than once in a discourse, all occurrences of that word will share the same meaning. They conducted an experiment using 9 ambiguous words and a total of 82 pairs of concordance lines for those words and showed that 94% occurrences of ambiguous words from the same discourse have the same meaning. However, Krovetz²⁹ reported that 94% is only true for coarse-grained distinctions (i.e., to distinguish between bank as a bank of a river or as a financial bank) among senses; it is only 67% using fine-grained distinction (i.e., a financial bank sense of bank will split into several senses such as a depository financial institution, savings bank, or the funds held by a gambling house) among senses when evaluated on two manually tagged corpora, Semcor¹² and DSO,³⁰ using WordNet as the sense inventory. However, Krovetz’s²⁹ result was based on two manually sense-tagged corpora in which the agreement was 57%²⁰ for the overlapping part. The hypothesis has been used by Yarowsky²² for WSD in the general English domain.

Supervised Machine Learning Techniques on Sense-tagged Corpora

Generally, supervised machine learning techniques³¹ use annotated data to make a decision for un-annotated data. It is widely used to learn classifiers for decision support systems in the biomedical domain. For example, given a set of medical reports each describing a pregnancy and a birth using 200 features (e.g., patient’s weight, height), machine-learning techniques can be used to learn a classifier to categorize patients with high risk of emergency cesarean section. In order to use supervised machine-learning technique, the first step requires transforming each annotated instance to a feature representation, usually a feature set fv = {(f₁, v₁), (f₂, v₂), . . . (f_n, v_n)}, where f_i is a feature and v_i is its corresponding value. For example, each medical report is transformed to a set with 200 elements in the above example. After transforming each instance to a feature representation, a supervised learning algorithm can be used to build classifiers, i.e., to categorize an un-annotated instance to a fixed number of categories. A large number of learning algorithms have been proposed in the literature including Bayesian probabilistic algorithms, artificial neural networks, decision trees, support vector machines, and instance-based algorithms.³¹

When supervised machine-learning techniques are used to resolve the ambiguity of a term W, we first need to transform each sense-annotated instance of W to a feature vector based on the context of W, which is formed by terms surrounding W. An important issue in using supervised learning for generating a WSD classifier is the appropriate choice of features. Intuitively, a good feature should capture sources of knowledge critical for determining the sense of W. Various kinds of features representing different knowledge sources have been presented in WSD research.¹⁷ Features can be the surrounding terms of W in a fixed window size and their values can be Boolean values to indicate their existence in a given instance. For instance in Sentence I below, the features with nonzero values in the corresponding feature vector when considering the ambiguous word CSF and a window size of 3, are the, origin, of, and, the, processes. Features can also be derived from the surrounding terms of W in a fixed window size in the universe through different mappings, such as the orientation and/or distance of the surrounding terms (e.g., the features for Sentence I when considering the orientation and distance in a window size 3 are the/L3, origin/L2, of/L1, and/R1, the/R2, processes/R3, where L is for left, R is for right and the number is for the distance), local collocations (i.e., a short sequence of words near W taking the word order into account), utilization of further linguistic knowledge such as part of speech (POS) tags (i.e., verb or noun) or stemming techniques (discharge, discharged, discharging, and discharges are treated as the same feature discharg), or domain knowledge, such as the semantic categories in a semantic lexicon.

Sentence 1. After a brief summary of current views on the origin of CSF and the processes underlying its elaboration, the author discusses studies of isolated chorid plexus in extracorporeal perfusion.

Several supervised learning algorithms have been implemented in the general English domain including Naïve Bayes algorithm, decision lists, neural network, and instance-based learning. We used Naive Bayes algorithm. For other supervised machine learning algorithm, readers can refer to a survey paper of Marquez.³² A Bayes classifier applies the Bayes decision rule when choosing a sense S from a set of senses {S1, S2, . . . , Sm} given a feature set fv, the rule that minimizes the probability of error³³:

Equation 1.

The Bayes decision rule is optimal because it minimizes the probability of error. For each individual case, it chooses the sense with the highest conditional probability and hence the smallest error rate. The conditional probability P(SK | fv) is computed using Bayes’ Theorem:

Equation 2.

P(Sk) is the prior probability of sense Sk: the probability that we have an instance of Sk if we do not have any knowledge about the context. P(fv | Sk) is the likelihood probability of fv given Sk, and P(fv) is the prior probability of the feature set fv; fv usually can be eliminated (since it is a constant for all senses and hence does not influence what the maximum is). By applying the logarithm on the probabilities, Equation 1 is equivalent to the following equation:

Equation 3.

P(fv | Sk) is usually estimated using the Naïve Bayes assumption, i.e., features are conditionally independent of each other:

Equation 4.

Using the Naive Bayes assumption, we get the following modified decision rule:

Equation 5.

P((f_j, v_j), Sk) and P(Sk) are computed via maximum likelihood estimation from the sense-tagged corpus:

Equation 6.

where occu((f_j, v_j), Sk) is the number of (f_j, v_j) co-occurring with sense Sk in the sense-tagged corpus, occu(Sk) is the number of occurrences of Sk in the sense-tagged corpus, and occu(W) is the total number of occurrence of W.

The power of Naïve Bayes learning is due to its efficiency and its ability to combine evidence from a large number of features that are derived from a large number of instances.

Resources

Below we present information about several resources on which our method and evaluation are based.

MEDLINE ³⁴ is the NLM bibliographic database that contains over 11 million references to journal articles in life sciences with a concentration on biomedicine. Each entry contains the citation information for the corresponding journal article, and also often contains an abstract.

The UMLS integrates various terminologies pertaining to biomedicine.³⁵ The Metathesaurus (META) is one component of the UMLS that contains information about biomedical concepts and terms from many controlled terminologies. The META is organized by concept, where each distinct concept has been assigned a unique concept identifier (CUI). All concept names corresponding to the same concept are assigned the same CUI. For instance, congestive heart failure and biventricular heart failure are two different concept names with the same CUI (C0018802). Each concept name has a term status to indicate whether it is the preferred concept name of the corresponding concept, or if it is suppressed (i.e., abbreviated or problematic). In the 2001 version of the UMLS, the table MRCON lists all concept names with their corresponding CUIs. It has 797,359 English concepts and 1,462,202 different English concept names (in the following, the 2001 version of the UMLS is assumed).

Another table called MRREL lists relationships between UMLS concepts. There are 9,524,132 entries in MRREL. Among them, 9,518,798 were derived directly from the source vocabularies. The remaining 5,334 entries are relationships between different sources that were created during the construction of the UMLS. There are 9 different relationship types, including broader (RB), narrower (RN), other-related (RO), parent (PAR), child (CHD), sibling (SIB), similar (RL), qualifier (AQ), and be-qualified (QB). However, since relations in the UMLS were mostly derived from different source vocabularies, the definition of relationship types may not be consistent. Two concepts may have multiple relationship types defined in the MRREL table. For example, the concepts C0004015 (i.e., aspartic acid) and C0085845 (i.e., aspartate) have a parent relation and a broader relation from source vocabulary AOD99; they have a narrower relation from source vocabulary MSH2001; in source vocabulary LNC10o, they have an other-related relation. A concept may have a relation with itself. For example, the concept C0022709 angiotensin = converting enzyme has RO relation with itself in source vocabularies CSP2000 and LNC10o.

The SPECIALIST Lexicon, an English language lexicon, is another component of the UMLS. The lexicon consists of a set of lexical entries with one entry for each spelling or set of spelling variants with a particular part of speech. The table LRAGR lists all variant forms for each entry in the lexicon. The UMLS also contains a Semantic Network where each CUI in META has been assigned to one or multiple semantic categories.

There are two different kinds of ambiguities presenting in the UMLS: conceptual and semantic. Conceptual ambiguity refers to the ambiguity caused by terms with multiple concepts, whereas semantic ambiguity refers to the ambiguity caused by terms that have multiple semantic categories. For example, concepts that are organic chemicals most likely are also pharmacologic substances.

There are 187,943 (of 797,359) concepts possessing multiple semantic categories in the UMLS. There were 4,547 conceptually ambiguous terms that represented 11,178 concepts in the UMLS ambiguous term table AMBIG.SUI, with an average ambiguity of 2.46. Johnson⁴ investigated the semantic ambiguity of a semantic lexicon that was based on the UMLS and discharge summaries and proposed a set of preference rules to reduce the semantic ambiguity. For example, in the discharge summary domain, chemical concepts occur only under the semantic category chemicals viewed functionally instead of under chemical viewed structurally. After applying his preference rules to the derived semantic lexicon, occurrences of entries with multiple semantic types were reduced from 9.41% to 1.46% in discharge summaries. Rindflesch and Aronson⁹ considered the conceptual ambiguity of the UMLS and proposed to use semantic categories of neighboring concepts to resolve the ambiguity. They conducted a preliminary study and found that a manually crafted set of rules based on semantic categories of neighboring concepts successfully resolved conceptual ambiguity around 80% of the time. Aronson and colleagues³⁷ proposed that machine-learning techniques could be used to derive rules instead of the manual crafting process. However, there is no published study of this approach according to our knowledge.

Below we introduce two programs that are discussed in the Methods section and the Experiment section: MetaMap¹ and the UMLS abbreviation extraction program.³⁸

MetaMap¹ is a highly configurable program that maps biomedical text to concepts in the META. Options control MetaMap’s output as well as its internal behavior, such as how aggressive to be in generation of word variants, whether or not to ignore META strings containing very common words, and whether to consider or to ignore word order. The initial purpose of the MetaMap program was to improve retrieval of bibliographic material such as MEDLINE citations. It has also been applied to several data mining efforts such as to detect clinical findings,³⁹ molecular binding expressions,⁴⁰ or novel relationships between drugs and diseases.⁷

The UMLS abbreviation extraction program³⁸ was developed for extracting (abbreviation, full form) pairs from the UMLS. It utilizes several fixed patterns associated with abbreviations in the META as well as the fact that abbreviations are considered synonyms in the META. The program was executed using the 2000 version of the UMLS. It extracted 163,666 different (abbreviation, full form) pairs with a recall of 96% and precision of 97.5%. The UMLS abbreviations were highly ambiguous: 54% of abbreviations with three characters had multiple full forms; the number of different full forms for all abbreviations with three characters was 3.05, while it was 10.9 for abbreviations with two characters. The UMLS abbreviations covered 66% of unique full forms found in the medical reports, and for frequently occurring abbreviations, the coverage was around 80%.

Our Method and Its Difference from Related Work

The method proposed in this article considers conceptual ambiguity in the UMLS. It utilizes conceptual relations defined in the UMLS to automatically derive sense-tagged corpora for ambiguous terms. WSD classifiers can then be automatically constructed using the derived sense-tagged corpora.

Our methods differ from related work in several ways. The previous investigations for resolving UMLS ambiguity were based on manually crafted rules. Johnson’s method concentrated on reducing semantic ambiguity in a semantic lexicon; it was based on a particular sub-domain and manually crafted rules, but provided no solution for resolving ambiguity in the context. The method suggested by Aronson and colleagues,³⁷ which proposes to apply machine learning techniques to derive a set of WSD rules based on semantic categories of neighboring concepts, requires an annotated corpus, where semantic categories of neighboring concepts are annotated and senses of the corresponding ambiguous terms are also annotated.

The previous WSD work^23,26,27 that utilized conceptual relatives disambiguated all nouns in general English text; the conceptually oriented dictionary used was WordNet. Our method concentrates on automatic construction of a sense-tagged corpus for each individual ambiguous biomedical term in the biomedical domain. Instead of using WordNet, we use the UMLS as our conceptually oriented knowledge source. WordNet and the UMLS are different in the following ways. The goal of WordNet is to provide a database of lexical relations for computational linguistic research, and the power of WordNet lies in a manual handcrafting process and domain-independence. The goal of the UMLS is to overcome retrieval problems caused by differences in terminology by integrating different electronic biomedical terminologies into one concept-oriented knowledge base. Almost all relationships are directly mapped from original source terminologies. WordNet has a strict definition about its relationship types such as IS-A etc. However, in the UMLS, the definition of relationship types in MRREL is vague, and different terminologies have their own definition for the same relationship type. For example, a parent relation may be considered as a child relation in two different sources, causing circular hierarchical relationships.⁴¹

Our previous work,²⁴ which used unambiguous synonyms to derive a sense-tagged corpus, is limited because it can only be used to build a WSD classifier for an ambiguous term W under the following two assumptions: each sense of W has unambiguous synonyms; and there are enough sense-tagged instances for each sense. However, some senses may not have unambiguous synonyms, or may not occur in enough instances. The current method does not require that each sense have unambiguous synonyms. It utilizes conceptual relatives that occur in the context to determine the sense and then to derive a sense-tagged corpus. Our previous method and the current method can be combined to increase our ability to acquire sense-tagged corpora automatically, and then used to train supervised WSD classifiers.

Establishing Conceptual Relatives for Each Sense

For each sense Si of W, we use concepts that have a direct relation with Si (i.e., concepts with CUIs that co-occur with CUI_Si in the MRREL table) to derive conceptual relatives. We consider concepts with relation types that are conceptual such as “Broader”, “Narrower,” and “Parent.” We exclude concepts with qualifier relation types (i.e., “Qualifier” or “Be-Qualified”) since they have high frequency, and provide little sense disambiguation information. Each CUI is added to the relative CUI set of W (RCUI(W)) with its associated sense Si and the relations. We consider that each concept has a synonymy relation with itself, and add each CUI_Si of W in the relative CUI set of W with its associated sense Si and a relationship type synonymy, but disregard relations among different senses of W in the MRREL table. For example, C0020255 (i.e., hydrocephalus) and C0007806 (i.e. CUI_S1 of CSF) have an “Other” relation in MRREL; therefore C0020255 is added to the relative CUI set of CSF with its associated sense CSF₁.

For each CUI in the relative CUI set of W, we gather all unambiguous English concept names. Because concept names with a term status “suppressed” are problematic, we exclude them; in addition, because abbreviations are highly ambiguous, we exclude those identified as abbreviations by the UMLS abbreviation extraction program. Each concept name is normalized by changing it to lower-case, removing symbols such as nos in cerebrospinal fluid, nos or unspecified in hydrocephalus, unspecified, removing some patterns such as parenthetical expressions (ck) in creatine kinase (ck), or ck – in ck – creatine kinase, and substituting punctuations by blanks. The resulting strings with length greater than 4 are considered as conceptual relatives of W with corresponding associated senses. Note that strings with length less than or equal to 4 are excluded because they usually have a high level of ambiguity. In addition, they may be problematic because many of them are abbreviations in the context but are not identified as abbreviations by the UMLS abbreviation extraction program. For example, FOX is not an abbreviation in the UMLS; but in the text Lipid hydroperoxides were measured by the FOX assay, it abbreviates ferrous oxidation with xylenol orange.

Identifying Conceptual Relatives in Each Abstract

CRMap identifies conceptual relatives in an abstract using the following phases: preprocessing, exact-string matching, UMLS-SPECIALIST normalization matching, and stem normalization matching.

In the preprocessing phase, we remove parenthetical expressions that contain a capitalized term with fewer than six characters. This is based on the observation that parenthetical expressions containing a short capitalized term inside are usually abbreviation-type parenthetical expressions. The punctuation is replaced by blank and the text is changed to lower case. As an example, the text The influence of prednisone on S-angiotensin-converting enzyme (S-ACE) activity was examined is changed to the influence of prednisone on s angiotensin converting enzyme activity was examined.

The three matching phases are processed subsequently. All three matching phases match conceptual relatives of the longest possible length. The matching phases differ in whether they require normalization or not, and if so, the normalization method used. In the exact-matching phase, conceptual relatives are used without normalization, while in the UMLS-SPECIALIST normalization matching phase, CRMap normalizes each word in the conceptual relative set and abstracts and maps it to its base-form in accordance with the SPECIALIST Lexicon LRAGR table, if applicable. In the stem-normalization matching phase, CRMap uses the Porter-stemmer⁴² to normalize each word to its stem.

For example, in Abstract 1, which contains the abbreviation CSF, three conceptual relatives are identified, each associated with the sense CSF₁ (i.e., cerebrospinal fluid): hydrocephalus, spinal cord, and brain).

Abstract 1. The brain (CSF₁_SIB) from an infant with a cystic occipital mass present at birth is examined in serial section. The occipital mass proved to be a rhombic roof ventriculocele. Within the posterior fossa, it was bound to an occipital lobe encephalocele which issued as a diverticulum of the left lateral ventricle through a microgyric cortical defect in the territory of the left posterior cerebral artery. The posterior medial aspects of both cerebral hemispheres were herniated downward into the widened tentorial gap. Craniolacunae were prominent on the inner aspect of the skull. The aqueduct and central canal of the spinal cord (CSF₁_SIB) were widely dilated, although the lateral ventricles were collapsed. It is suggested that hydrocephalus (CSF₁_RO) secondary to obstruction to flow of CSF through the rhombic roof entrained a sequence of events giving rise to the rhombic roof ventriculocele and causing occlusion of the posterior cerebral artery and subsequent diverticulation of the lateral ventricle through an infarcted region of the posterior-medial hemisphere.

Besides CRMap, the MetaMap program can also be used to identify concepts that have relations with senses of W.

Deriving a Sense-Tagged Corpus

For an abstract that contains W, since all occurrences of W in the abstract hold the same sense S based on the one sense per abstract assumption, we call S the sense of W in that abstract. Note that not every abstract contains conceptual relatives that can be identified using CRMap. For an abstract that has conceptual relatives identified by CRMap, if all conceptual relatives are associated with one sense, the sense of W in the abstract is assigned with that sense without any doubt. For example, the sense of CSF in Abstract 1 is assigned to CSF₁ (i.e., cerebrospinal fluid). For abstracts that contain conceptual relatives with multiple associated senses, a sense assignment scheme is needed in order to assign senses to those abstracts. For simplicity, we use the following sense assignment scheme while leaving other possible sense assignment schemes to discussion: assign the majority vote of the associated senses of the identified conceptual relatives; if there is a tie, randomly choose one of the tied senses.

The collection of abstracts in which the sense of W can be determined using conceptual relatives is the resulting sense-tagged corpus for W (STC(W)). More sense-tagged abstracts can be derived by constructing a supervised WSD classifier of W that is trained on the derived sense-tagged corpus and then by performing the disambiguation task on the whole abstract collection of W. A large sense-tagged corpus for W is formed by combining instances in STC(W) and instances that are sense-tagged by the constructed WSD classifier of W.

Derivation of Ambiguous Abbreviations

The UMLS (2001 version) was processed using the UMLS extraction program to obtain a list of (abbreviation, full form) pairs. We chose three-letter abbreviations since they were moderately ambiguous in the UMLS, and appeared frequently in writing. We kept only those pairs where the abbreviation was listed as an ambiguous term in AMBIG.SUI, had multiple full forms, appeared more than 100 times in the collection of medical reports, and full form was a UMLS concept name.

Derivation of the Gold Standard

For each abbreviation A, conceptual relatives for each sense of A were then established using MRREL. We extracted a collection of MEDLINE abstracts that contained A in upper case. The collection was divided into two sets I and II : Set I consisted of abstracts with an occurrence of A inside a parenthetical expression, and Set II consisted of all others. The gold standard sense of A for each abstract in Set I was then automatically derived using synonyms of A (i.e., conceptual relatives with concept identifiers from the sense representation set of A).

For each abstract in Set I, if a synonym (SYN) of A occurred in the abstract with a pattern SYN (A), we considered the gold standard for the correct sense of A in the abstract to be the associated sense of SYN. After determining the correct sense of A, the abstract was automatically modified by replacing the pattern SYN (A) with A, and then added to the gold standard set of A (GSS(A)). For instance, Abstract 2 was modified to Abstract 2′ with the gold standard sense attached at the beginning (separated using the sign “|”). Only modified abstracts were used for further processing. The sense at the beginning of each abstract was used for evaluation purposes to determine correctness, but not used by the disambiguation method itself.

Abstract 2. After a brief summary of current views on the origin of cerebrospinal fluid (CSF) and the processes underlying its elaboration, the author discusses studies of isolated chorid plexus in extracorporeal perfusion. . . .

Abstract 2′. CSF₁ | After a brief summary of current views on the origin of CSF and the processes underlying its elaboration, the author discusses studies of isolated chorid plexus in extracorporeal perfusion. . . .

Evaluation

We identified conceptual relatives in each abstract from the MEDLINE abstracts collection of A using CRMap and subsequently derived a sense-tagged corpus for A (STC(A)). Additional sense-tagged abstracts were derived by constructing a supervised WSD classifier that was trained on STC(A) and then performing the disambiguation task on the entire abstract collection of A. A large sense-tagged corpus for A was formed by combining instances in STC(A) and instances that were sense-tagged by the WSD classifier of A.

We evaluated STC(A) using the gold standard set of A (i.e., GSS(A)) with two measures: recall, i.e. the ratio of the number of abstracts with correctly identified sense to the total number of abstracts in GSS(A), and precision, the ratio of the number of abstracts with correctly identified sense to the number of abstracts that were sense-tagged using conceptual relatives.

We constructed a WSD classifier for A by applying the Naive Bayes algorithm on the derived sense-tagged corpus for A. We transformed each abstract in the corpus to a feature set where features were words after stemming (A was excluded for consideration as a feature, and the Porter Stemmer⁴² was used) and values were Boolean values that indicate their existence in a context. For example, features with the value 1 in the feature set for Abstract 2′ are after, a, brief, summari ,…, in, extracorpor, and perfus. Note that in the Naive Bayes algorithm, the inclusion of stop words (i.e., words that occur frequently disregarding different instances and terms) as features does not affect the decision process, because they contribute almost the same conditional probability to all senses.

The constructed WSD classifier of A was executed to assign senses for the MEDLINE abstracts that contained A but were not in STC(A). A larger sense-tagged corpus for A was then derived by combining instances in STC(A) and instances that were sense-assigned by the WSD classifier of A and consisted of all abstracts that contained A. The quality of the corpus was related to the quality of STC(A) and the performance of the WSD classifier. The performance of the WSD classifier was evaluated for precision on abstracts that were in the gold standard set of A but were not in STC(A). Note that there is no recall measure since a Naive Bayes WSD classifier determines the sense for each instance. We also compared the quality of sense-tagged corpora using two different mapping programs (i.e., CRMap and MetaMap, for several abbreviations).