Understanding NLP Algorithms in Python

Posted on In , , Disqus: Word count in article: 4.2k Reading time ≈ 4 mins.

Natural Language Processing (NLP) is a field of artificial intelligence that deals with the interaction between computers and human language. Python is a popular programming language for NLP tasks due to its simplicity and ease of use. In this article, we will delve into the various NLP algorithms available in Python and how they can be applied to solve real-world problems.

Text Pre-processing

Text pre-processing is an important step in NLP, as raw text data is often messy and needs to be cleaned and transformed before it can be used in NLP algorithms. Some of the common text pre-processing techniques include:

  • Tokenization: This is the process of breaking down text into individual words or phrases, also known as tokens.
1
2
3
4
5
6
7
8
import nltk
nltk.download('punkt')
  
from nltk.tokenize import word_tokenize
  
text = "This is an example sentence for tokenization."
tokens = word_tokenize(text)
print(tokens)

Output: ['This', 'is', 'an', 'example', 'sentence', 'for', 'tokenization', '.']

  • Stop word removal: Stop words are common words in a language that do not add much meaning to the text, such as “a”, “an”, “the”, etc. These words can be removed to reduce the size of the text data.
1
2
3
4
5
6
nltk.download('stopwords')
from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))
tokens = [token for token in tokens if token not in stop_words]
print(tokens)

Output: ['example', 'sentence', 'tokenization', '.']

  • Stemming: This is the process of reducing words to their root form, also known as the stem. For example, “running” and “runner” would both be reduced to the stem “run”.
1
2
3
4
5
6
nltk.download('rslp')
from nltk.stem import RSLPStemmer

stemmer = RSLPStemmer()
tokens = [stemmer.stem(token) for token in tokens]
print(tokens)

Output: ['exampl', 'sentenc', 'token', '.']

Text Classification

Text classification is a process of categorizing text data into one or more predefined categories based on its content. One of the most popular NLP algorithms for text classification is the Naive Bayes algorithm.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from nltk.classify import NaiveBayesClassifier
from nltk.sentiment import SentimentIntensityAnalyzer

nltk.download('vader_lexicon')

sentiment_analyzer = SentimentIntensityAnalyzer()
def sentiment(text):
    return sentiment_analyzer.polarity_scores(text)['compound']

featuresets = [(sentiment(tweet), label)
               for (tweet, label) in tweets]

train_set, test_set = featuresets[:100], featuresets[100:]
classifier = NaiveBayesClassifier.train(train_set)

print("Accuracy:", nltk.classify.util.accuracy

Named Entity Recognition

Named Entity Recognition (NER) is a process of identifying and classifying named entities in a text, such as people, organizations, locations, and dates. The goal of NER is to extract structured information from unstructured text data.

1
2
3
4
5
6
7
8
9
import spacy

nlp = spacy.load("en_core_web_sm")

text = "Steve Jobs, the co-founder of Apple Inc., was born on February 24, 1955 in San Francisco."
doc = nlp(text)

for entity in doc.ents:
    print(entity.text, entity.label_)

Output:Steve Jobs PERSON Apple Inc. ORG February 24, 1955 DATE San Francisco GPE

Part-of-Speech Tagging

Part-of-Speech (POS) tagging is the process of marking each word in a text with its corresponding part of speech, such as noun, verb, adjective, etc. This information is useful for a variety of NLP tasks, such as parsing and text classification.

1
2
3
4
5
6
7
nltk.download('averaged_perceptron_tagger')

from nltk import pos_tag

tokens = nltk.word_tokenize(text)
pos_tags = pos_tag(tokens)
print(pos_tags)

Output: [('Steve', 'NNP'), ('Jobs', 'NNP'), (',', ','), ('the', 'DT'), ('co-founder', 'JJ'), ('of', 'IN'), ('Apple', 'NNP'), ('Inc.', 'NNP'), (',', ','), ('was', 'VBD'), ('born', 'VBN'), ('on', 'IN'), ('February', 'NNP'), ('24', 'CD'), (',', ','), ('1955', 'CD'), ('in', 'IN'), ('San', 'NNP'), ('Francisco', 'NNP'), ('.', '.')]

Conclusion

In this article, we have explored some of the common NLP algorithms available in Python and how they can be applied to solve real-world problems. From text pre-processing, text classification, Named Entity Recognition, to Part-of-Speech tagging, these algorithms form the foundation of NLP and provide a good starting point for those who are new to the field.