Classic Text Preprocessing

Before models learned to read raw text, engineers cleaned it by hand: tokenizing, lowercasing, removing stop words, and stemming words to their roots.

Before a machine can count words or feed them to a model, someone has to decide what a “word” even is. That decision used to be an entire pipeline of hand-written rules, and understanding it explains why modern tokenizers look the way they do.

Classic preprocessing had one goal: turn messy human text into a clean, uniform list of units a statistical model could work with. Every step traded away some information to make what remained easier to count.

Tokenization: splitting text into units

The first move is cutting a string into tokens. The naive version splits on whitespace:

"The cats aren't hungry." → ["The", "cats", "aren't", "hungry."]

Notice the problems immediately. hungry. carries a period. aren't is really two ideas, are and not. Classic tokenizers layered on rules: strip punctuation, split contractions, handle hyphens. Every language broke the rules differently. Chinese has no spaces between words at all, so it needed a separate segmentation algorithm.

This brittleness is exactly why modern systems abandoned word-level splitting for subword tokenization, which learns its units from data instead of hard-coding them.

Normalization: collapsing variants

Once you have tokens, you make them uniform so that trivially different spellings count as the same thing.

• Lowercasing. The and the become one token. Cheap and usually helpful, but it destroys the distinction between Apple the company and apple the fruit.
• Removing accents and punctuation. café becomes cafe.
• Handling numbers. Often every number is replaced with a single <NUM> placeholder, since the exact value rarely matters for topic classification.

Each normalization step shrinks the vocabulary and groups related forms together. It also throws information away, and whether that trade is worth it depends entirely on the task. Sentiment analysis may not care about case; a code search engine cares enormously.

Stop word removal

Some words appear everywhere and carry almost no topical signal: the, is, at, and, of. These are stop words, and classic pipelines deleted them from a fixed list.

"the cat sat on the mat" → ["cat", "sat", "mat"]

The logic was practical. In a bag-of-words model, the might be the most frequent token in every document, drowning out words that actually distinguish one document from another. Removing it sharpened the signal and shrank the data.

The catch: stop words carry more than people assumed. “To be or not to be” is almost entirely stop words. Negation words like not flip meaning completely, yet many stop lists removed them. Modern models keep every token precisely because these small words do real grammatical work.

Stemming and lemmatization

run, running, ran, and runs are the same idea in four costumes. Treating them as four separate tokens splits the evidence for a concept across four counts. Two techniques collapsed them.

Stemming chops off endings with crude rules. The Porter stemmer, from 1980, is the classic. It is fast and dumb:

running → run, studies → studi, argument → argument

Note studi is not a real word. Stemming does not care; it only needs consistency, so that every form of “study” maps to the same stem.

Lemmatization is the careful version. It uses a dictionary and part-of-speech information to map each word to its true base form, or lemma:

studies → study, better → good, was → be

Lemmatization is slower and needs linguistic resources, but it produces real words and handles irregular forms that stemming mangles.

Why this matters now

Every step above was a human deciding, in advance, what information a model was allowed to see. That is the defining trait of classic NLP: feature engineering done by hand. Preprocessing was where domain expertise lived, and a better stop list or stemmer could measurably improve a system.

Modern language models inverted this. They keep case, punctuation, and stop words, because the model itself learns which distinctions matter for which context. The preprocessing collapsed down to one learned step: subword tokenization. What used to be a dozen hand-tuned rules became a single data-driven algorithm.

Understanding the old pipeline is not nostalgia. It shows you what each design choice was for, so that when a modern tokenizer keeps The distinct from the, you know exactly which old assumption it is rejecting and why.

What to remember

• Classic preprocessing turned raw text into clean, countable units through a sequence of hand-written rules.
• Tokenization splits text; normalization (lowercasing, stripping punctuation) collapses trivial variants.
• Stop word removal deletes high-frequency, low-signal words, but risks removing meaningful ones like not.
• Stemming chops endings crudely; lemmatization maps to true dictionary base forms.
• Every step trades information for uniformity, and the trade is task-dependent. Modern models mostly abandoned it because they learn the distinctions themselves.

Next: Bag of Words and N-grams — the first way anyone turned these cleaned tokens into numbers.