NLP (Natural Language Processing)

Definition

The field of AI focused on understanding and generating human language.

Why It Matters

NLP is the older umbrella term for everything LLMs now dominate, tokenisation, parsing, sentiment, named-entity recognition, translation, summarisation. Pre-2022 NLP was a stack of specialised models, each fine-tuned for one task. LLMs collapsed most of them into one general-purpose layer.

Key Points

• Pre-LLM NLP stack: tokeniser → POS tagger → NER model → dependency parser → task classifier. Each was a separately trained model.
• spaCy, NLTK and Stanford CoreNLP were the dominant frameworks until ~2021, replaced by transformer fine-tuning for most tasks.
• BERT (2018) showed bidirectional pretraining + task-specific fine-tuning beat dedicated architectures on 11 of 11 NLP benchmarks.
• Classic NLP subtasks (NER, sentiment analysis, coreference resolution, relation extraction) are now routine prompt instructions to a general LLM.
• Structured prediction tasks (POS tagging, constituency parsing) still sometimes use specialised models where exact annotation format matters more than raw accuracy.

Example

Named-entity recognition (NER), part-of-speech tagging, and sentiment analysis used to be distinct models with their own training pipelines. Today a single LLM with the right prompt covers all three plus tasks that didn't have dedicated NLP models a decade ago.

Related Terms

• LLM (Large Language Model)A neural network trained on massive text datasets that can generate, understand and manipulate human language. Examples: GPT-4, Qwen, Claude.
• EmbeddingA numerical representation of text, images, or other data that AI models can process and compare.
• TransformerThe neural network architecture behind modern AI models. Introduced in the 2017 paper "Attention Is All You Need."

Quick Facts