Question about the <unk> in the subsample function of chapter_natural-language-processing-pretraining

[IMMORTAALLL]

Why is it that in the subsample function of this section, the token is discarded when calculating the word count, but is not added to sentences later?
It may do harm to the ability of model to learn the rear token.

[muzammil5539]

Clarification: Why <unk> is Discarded in Subsampling and Not Added Back to Sentences

Related to: chapter_natural-language-processing-pretraining → subsample function

---

Summary

In the current implementation, the <unk> token is excluded from word counts during subsampling and not added back to sentences. This is intentional and desirable because:

• <unk> is a placeholder, not a stable semantic unit.
• Counting/training on <unk> blurs distinctions between many unrelated rare words.
• Excluding <unk> reduces noise and improves the quality of learned word co-occurrences and embeddings.

Below is the conceptual explanation, a concrete example, and the math intuition behind this design choice.

---

Conceptual Rationale

During training, the vocabulary is capped (e.g., to the top frequent words). Words not in the vocabulary are mapped to <unk>. The <unk> symbol aggregates many different rare or unseen words into one token with no consistent meaning.

Subsampling aims to downweight high-frequency, low-information tokens to reveal more informative contexts. Treating <unk> as a real word would:

1. Inflate its frequency with heterogeneous content.
2. Distort co-occurrence statistics.
3. Lead embeddings to learn a "spurious average" of unrelated concepts.

Therefore:

• We do not count <unk> when estimating token frequencies for subsampling.
• We do not insert <unk> back into sentences post-subsampling; we keep sentences composed of known, meaningful tokens.

---

Concrete Example

Assume a vocabulary of size 10,000. The following words are Out-of-Vocabulary (OOV) at training time:

"electromechanical", "bioinformatics", "nanolithography" <unk>

If we treated <unk> like a normal token, the model would see unrelated contexts such as:

• <unk> is advancing rapidly
• <unk> microscopes are expensive

The model would then try to merge these into a single embedding. This muddies the representation space. By excluding <unk>, we preserve cleaner word co-occurrences among known tokens.

---

Math Intuition for Subsampling

Let:

• be the empirical frequency (probability) of token in the corpus.
• be the subsampling threshold (typical values: to ).

A common keep-probability used in Word2Vec-style subsampling is:

Intuition:

• For very frequent tokens (e.g., "the", "is"): . These are downsampled aggressively.
• For rare in-vocabulary tokens: is small, so , keeping them to enrich informative contexts.

Where <unk> fits in

If we included <unk>, its empirical frequency could be artificially high. Subsampling would then remove <unk> instances aggressively, but the ones that remain would still be semantically incoherent, injecting noise into the training signal for neighboring words.

---

Training vs. Inference

Phase	Handling of <unk>
Training	Build sentences from known vocabulary items only; drop <unk> to keep co-occurrences meaningful.
Inference	Map unseen words to <unk> to ensure robustness so the sequence can still be processed.

Tip

Modern Note: Tokenization methods like BPE or WordPiece reduce reliance on <unk> by decomposing rare words into subwords, preserving information without collapsing it into a single "unknown" bucket.