Python NLP Library: Spacy

With Spacy, a sophisticated NLP library, differently trained models for a variety of NLP tasks can be used. From tokenization to part-of-speech tagging to entity recognition, Spacy produces well-designed Python data structures and powerful visualizations too. On top of that, different language models can be loaded and fine-tuned to accommodate NLP tasks in specific domains. Finally, Spacy provides a powerful pipeline object, facilitating mixing built-in and custom tokenizer, parser, tagger and other components to create language models that support all desired NLP tasks.

This article introduces Spacy. You will learn how to install the library, load models, and apply text processing and text semantics tasks, and finally how to customize a spacy model.

The technical context of this article is Python v3.11 and Spacy v3.5.3. All examples should work with newer versions too.

This article originally appeared at my blog admantium.com.

Spacy Library Installation

The Spacy library can be installed via pip:

python3 -m pip install spacy

All NLP tasks require a model to be loaded first. Spacy provides models build on different corpora and for different languages as well. See the full list of models. In general, the models can be distinguished into the size of its corpora, which leads to different results during NLP tasks, and the technology used to build the model, which is an unknow internal format or transformer-based models, such as Berta.

To load a specific model, following snippet can be used:

python -m spacy download en_core_web_lg

NLP Tasks

Spacy supports the following tasks:

• Text processing
  • Tokenization
  • Lemmatization
• Text Syntax
  • Part-of-speech tagging
• Text Semantics
  • Dependency Parsing
  • Named Entity Recognition
• Document Semantics
  • Classification

Furthermore, Spacy supports these additional features:

• Corpus Management
• Word Vectors
• Custom NLP Pipeline
• Model Training

Text Processing

Spacy applies text processing essentials automatically when one of the pretrained language model is used. Technically, text processing happens around a configurable pipeline object, an abstraction similar to the SciKit Learn pipeline object. This processing always begins with tokenization, and then adds additional data structures that enrich the information of parsed text. All of these tasks can be customized too, e.g. swapping the tagger component. The following descriptions focus only on the built-in features when using pre-trained models.

Tokenization

Tokenization is the first step and straightforward to apply: Apply a loaded model to a text, and tokens emerge.

import spacy
nlp = spacy.load('en_core_web_lg')

# Source: Wikipedia, Artificial Intelligence, https://en.wikipedia.org/wiki/Artificial_intelligence
paragraph = '''Artificial intelligence was founded as an academic discipline in 1956, and in the years since it has experienced several waves of optimism, followed by disappointment and the loss of funding (known as an "AI winter"), followed by new approaches, success, and renewed funding. AI research has tried and discarded many different approaches, including simulating the brain, modeling human problem solving, formal logic, large databases of knowledge, and imitating animal behavior. In the first decades of the 21st century, highly mathematical and statistical machine learning has dominated the field, and this technique has proved highly successful, helping to solve many challenging problems throughout industry and academia.'''

doc = nlp(paragraph)
tokens = [token for token in doc]

print(tokens)
# [Artificial, intelligence, was, founded, as, an, academic, discipline

Lematization

Lemmas are generated automatically; they are properties of the tokens.

doc = nlp(paragraph)

lemmas = [token.lemma_ for token in doc]

print(lemmas)
# ['artificial', 'intelligence', 'be', 'found', 'as', 'an', 'academic',

The configurable component applies lemmatization either on rules or lookup. To see which mode the built-in model uses, execute the following code:

lemmatizer = nlp.get_pipe("lemmatizer")

print(lemmatizer.mode)
# 'rule'

There is no stemming in Spacy.

Text Syntax

Part-of-Speech Tagging

In Spacy, part-of-speech tags come in two flavors. The POS property is an absolute category to which a token belongs, characterized as a universal POS tag. The TAG property is a more nuanced category that builds upon the dependency parsing and named entity recognition.

Following table lists the POS classes.

For the TAG classes, I could not find a definitive explanation in the documentation. However, this Stackoverflow thread hints that the TAG refer to classes used in academic papers about dependency parsing.

To see the POS and TAG associated with the tokens, run the following code:

doc = nlp(paragraph)
for token in doc:
    print(f'{token.text:<20}{token.pos_:>5}{token.tag_:>5}')

#Artificial            ADJ   JJ
#intelligence         NOUN   NN
#was                   AUX  VBD
#founded              VERB  VBN
#as                    ADP   IN
#an                    DET   DT
#academic              ADJ   JJ
#discipline           NOUN   NN

Text Semantics

Dependency Parsing

Dependency parsing checks the contextual relationships of words and chunks of words. This step greatly enhances the machine readably semantic information from a text.

Spacy provides both a textual representation as well as a graphical representation of the dependencies.

doc = nlp(paragraph)
for token in doc:
    print(f'{token.text:<20}{token.dep_:<15}{token.head.text:<20}')

# Artificial          amod           intelligence
# intelligence        nsubjpass      founded
# was                 auxpass        founded
# founded             ROOT           founded
# as                  prep           founded
# an                  det            discipline
# academic            amod           discipline
# discipline          pobj           as
# in                  prep           founded

To render those relationships graphically, run the following commands.

from spacy import displacy

nlp = spacy.load("en_core_web_lg")

displacy.serve(doc, style="dep", options={"fine_grained": True, "compact": True})

It will output a structure as shown here:

Note that the capabilities of this processing step are limited to the language models original training corpus. Spacy offers two ways to enhance the parsing. First, models can be training from scratch. Second, recent releases of Spacy provide transformer models, and these can be fine-tuned to work with a more domain specific corpora as well.

Named Entity Recognition

Entities inside a text refer to persons, organizations or objects and can be detected by Spacy for a processed document.

Recognized entities are part of a parsed document and can be access by the ents property.

doc = nlp(paragraph)
for token in doc.ents:
    print(f'{token.text:<40}{token.label_:<15}')

# 1956                                    DATE
# the years                               DATE
# AI                                      ORG
# the first decades of the 21st century   DATE

Alternatively, they can be visualized.

Similar to the dependency parsing, the result of this step is very dependent on its training data. For example, if used on book texts, it might fail to identify the name of its characters. To help with this case, a custom KnowledgeBase object can be created, which will be used to identify possible candidates for named entities during the text processing.

Document Semantics

Classification

Spacy does not include classification or categorization algorithms by itself, but other open-source projects extend Spacy for executing machine learning tasks.

Just to show one example: The Berttopic extension is an out-of-the-box document classification projects that even provides a visual representation.

This project is installed by running pip install "bertopic[spacy]". Applying this project to a set of 200 articles gives the following result:

import numpy as np
import pandas as pd
from bertopic import BERTopic

X = pd.read_pickle('ml29_01.pkl')
docs = X['preprocessed'].values

topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)

print(topic_model.get_topic_info())
# Topic Count Name
# -1 30 -1_artificial_intelligence_machine
# 1 22 49_space_lunar_mission

Additional Features

Corpus Management

Spacy defines a Corpus object, but it is used to read JSON or plaintext files for training custom Spacy language models.

The only property of all processed texts that I could find in the documentation is vocab, a lookup table for all words encountered in the processed text.

Text Vectors

For all built-in model of the category md or lg, word vectors are included. In Spacy, individual tokens, spans (user-defined slices of a documents), or complete documents can be represented as vectors.

Here is an example:

nlp = spacy.load("en_core_web_lg")
vectors = [(token.text, token.vector_norm) for token in doc if token.has_vector]

print(vectors)
# [('Artificial', 8.92717), ('intelligence', 6.9436903), ('was', 10.1967945), ('founded', 8.210244), ('as', 7.7554812), ('an', 8.042635), ('academic', 8.340115), ('discipline', 6.620854),

span = doc[0:10]

print(span)
# Artificial intelligence was founded as an academic discipline in 1956

print(span.vector_norm)
# 3.0066288

print(doc.vector_norm)
# 2.037331438809547

The documentation does not disclose which specific tokenization method is being used. The non-normalized tokens have 300 dimensions, which could hint at that the FastText tokenization method is being used:

token = doc[0]

print(token.vector.dtype, token.vector.shape)
# float32 (300,)

print((token.text, token.vector))
#'Artificial',
# array([-1.6952  , -1.5868  ,  2.6415  ,  1.4848  ,  2.3921  , -1.8911  ,
#         1.0618  ,  1.4815  , -2.4829  , -0.6737  ,  4.7181  ,  0.92018 ,
#        -3.1759  , -1.7126  ,  1.8738  ,  3.9971  ,  4.8884  ,  1.2651  ,
#         0.067348, -2.0842  , -0.91348 ,  2.5103  , -2.8926  ,  0.92028 ,
#         0.24271 ,  0.65422 ,  0.98157 , -2.7082  ,  0.055832,  2.2011  ,
#        -1.8091  ,  0.10762 ,  0.58432 ,  0.18175 ,  0.8636  , -2.9986  ,
#         4.1576  ,  0.69078 , -1.641   , -0.9626  ,  2.6582  ,  1.2442  ,
#        -1.7863  ,  2.621   , -5.8022  ,  3.4996  ,  2.2065  , -0.6505  ,
#         0.87368 , -4.4462  , -0.47228 ,  1.7362  , -2.1957  , -1.4855  ,
#        -3.2305  ,  4.9904  , -0.99718 ,  0.52584 ,  1.0741  , -0.53208 ,
#         3.2444  ,  1.8493  ,  0.22784 ,  0.67526 ,  2.5435  , -0.54488 ,
#        -1.3659  , -4.7399  ,  1.8076  , -1.4879  , -1.1604  ,  0.82441 ,

Finally, Spacy offers an option to provide user-defined word vectors for a pipeline.

Custom NLP Pipeline

The reference model for pipelines is included in the pretrained language models. They encompass the following:

• Tokenizer
• Tagger
• Dependency Parser
• Entity Recognizer
• Lemmatizer

The pipeline steps need to be defined in a project-specific configuration file. The full pipeline for all of these steps is defined as follows:

[nlp]
pipeline = ["tok2vec", "tagger", "parser", "ner", "lemmatizer"]

This pipeline can be extended. For example, to add a text categorization step, simply append the pipeline declaration with textcat, and provide an implementation.

There are several pipeline components that can be used, as well as the option to add custom components that will interact with and enrich the document representation achieved by Spacy.

Also, external custom models or model components can be incorporated into Spacy, for example exchanging the word-vector representation or using any transformer-based models which is based on PyTorch or the external transformers library.

Model Training

Based on the pipeline object and an extensive configuration file, new models can be trained. The Quickstart guide includes a UI widget in which the desired pipeline steps and local file paths can be created interactively. The training data needs to have the form of Python data objects that include the desired properties that should be trained. Here is an example from the official documentation for training named entity recognition.

# Source: Spacy, Training Pipelines & Models, https://spacy.io/usage/training#training-data

nlp = spacy.blank("en")
training_data = [
  ("Tokyo Tower is 333m tall.", [(0, 11, "BUILDING")]),
]

For the training data itself, several converters are supported, such as JSON, Universal Dependencies, or Named Entity Recognition formats like IOB or BILUO.

To facilitate training, a Corpus object can be used to iterate over JSON or PlainText documents.

Summary

Spacy is a state-of-the art NLP library. By using one of the pre-trained models, all basic text processing tasks (tokenization, lemmatization, part-of-speech tagging) and text semantics tasks (dependency parsing, named entity recognition) are applied. All models are created through a pipeline object, and this object can be used to customize any of these steps, for example providing custom tokenizer or swap the word-vector component. Additionally, you can use other transformer-based models, and extend the pipeline with tasks for text classification.