Lesson · 16 min

Words in text

Text is data too

Not all data is a table. Comments, reviews, emails and documents are text. A program cannot filter or count a text until it cuts the text into pieces. This lesson shows the first steps. Search tools and AI apps start with the same steps.

From a text to words

split() cuts a text at the spaces. The pieces are not clean words yet.

text = "The dock was full. Really full!"
print(text.split())   # ['The', 'dock', 'was', 'full.', 'Really', 'full!']

For a person, full. and full! are the same word. For Python, they are two different texts. The and the are also different. Make the text lower case, and remove the punctuation at the ends of each piece. strip can take the characters that it removes.

print("full!".strip(".,!?;:"))   # full
print("Really".lower())          # really

A piece that has only these characters, such as ..., becomes an empty text. Leave it out.

Count the words

A Counter can count the words of many texts. Its method update adds a list of words to the counts.

from collections import Counter

counts = Counter()
for comment in comments:
    counts.update(words(comment))

Words that say nothing

The most common words of almost every English text are the, a, and, and is. They tell you nothing about the subject of the text. Such words are stop words. Leave them out before you look at the top of the counts.

content = Counter()
for word, count in counts.items():
    if word not in STOP_WORDS:
        content[word] = count

Whole words, not pieces of words

"art" in "The start was hard" is True, because the letters art are inside start. To find a word, look in the list of words, and not in the text.

print("art" in words("The start was hard"))   # False

Chunks

Programs often cut a long text into parts that have a fixed number of words. The parts are chunks. An AI app does this, because a language model can read only a limited amount of text at one time. range with a step gives the start of each chunk.

items = ["a", "b", "c", "d", "e"]
for start in range(0, len(items), 2):
    print(items[start:start + 2])
# ['a', 'b']
# ['c', 'd']
# ['e']

The last chunk can be shorter than the others.

Your turn

comments.txt has the comments of 11 riders. main.py reads them into the list comments, and it has a set of stop words. Do the tasks in order.

Tasks

5 to do

Task 1, not checked yet. Write a function words(text). It returns the list of the words of the text, in lower case. It removes the characters . , ! ? ; : from both ends of each word, and it leaves out a piece that is then empty.
Task 2, not checked yet. Make a variable counts with the number of times that each word is in the comments. Use your function words and a Counter.
Task 3, not checked yet. Make a list top3 with the 3 most common words that are not in STOP_WORDS. The most common word is first.
Task 4, not checked yet. Write a function chunks(items, size). It cuts the list items into lists of size items, and it returns the list of these lists. The last list can be shorter.
Task 5, not checked yet. Write a function mentions(comments, word). It returns the list of the comments that have word as a whole word. Capital letters make no difference.

Task 1 of 5

Write a function words(text). It returns the list of the words of the text, in lower case. It removes the characters . , ! ? ; : from both ends of each word, and it leaves out a piece that is then empty.

All tasks