This LLM has a Context window of 1

Introduction

An n-gram model analyzes training data to determine how frequently specific word sequences occur. For example, given the phrase “my house is,” the model examines all words that follow “my” in the dataset. If “house” is the most common continuation, the model predicts “house.”

In this Post, I will cover the mathematical basics of modern LMs and train my own N-Gram Model based on my lecture notes. I’ll take a mathematical dive into different token selection functions and prove why the maximum likelihood assumption leads to periodic repetition.

Mathematical Basics

This section is largely based on the book Speech and Language Processing by Jurafsky & Martin (2026, I.3)

Most language models use context consisting of $n-1$ tokens $C:= w_1w_2\cdots w_{n-1}$ to predict the nth token $w_n$ and describe the probability for every next token with $P(w_n|w_{1:n-1}) = P(w_n|C)$. N-Gram models limit this context by assuming $\approx P(w_n| w_{n-N+1} \cdots w_{n-1})$. In the extreme case of a bigram model ($N=2$) this assumption simplifies to $P(w_n|C)\approx P(w_n|w_{n-1})$, we call this the markov assumption. The probability of outputting a sentence $s=w_{1}w_{2}\dots w_{m}$ with token-length $m$ based on context token $c$ is $P(s)=\prod_{w\in s}P(w)$. I define $S_{N}$ as the set of all sentences with length N. And $s_{w_n}$ We define $T(a | B)$ as the number of occurrences of $a$ with the sentence $B$ preceeding. The next token selection is decided by $Sel: C \longrightarrow w_n$.

In our case, we define $P(w_{n}|C)=\frac{ T(w_{n}| w_{n-N+1 \cdots w_{n-1}})}{ T(w_{n})}$, simplifying to $P(w_{n}|C) = \frac{T(w_n|w_{n-1})}{T(w_n)}$ with bigrams.

Greedy decoding always ends in a loop

Let’s take the maximum likelihood assumption of picking with $Sel(C) := \max_{w_{i}} (P(w_{i}|C))$.

Before we improve $Sel$, let’s prove why choosing greedily will fail.

Proof

Adapted from Knuth (1997)

We define a greedy $Sel$ as:

$$f: V \longrightarrow V, \quad f(w) := \operatorname*{arg\,max}_{w' \in V} P(w' \mid w),$$

Given vocabulary $V$ with $|V| = v < \infty$: Generation from a seed $w_{0}$ is then nothing but iterating: $w_{t} = f(w_{t-1}) = f^{t}(w_{0}).$

Theorem. For every $w_{0} \in V$ the greedy sequence $(w_{t})_{t \ge 0}$ is eventually periodic: there are $i,j \in \mathbb{R}$ where $i < j \le v$ such that $w_{i} = w_{j}$, and the sequence repeats with an arbitrary period $p$: $f^{p+1}(w_{i})=w_j$

Proof. Look at the first $v+1$ tokens $w_{0}, w_{1}, \dots, w_{v}$. They are $v+1$ elements drawn from a set of only $v$ tokens, so by the pigeonhole principle two of them must coincide: $w_{i} = w_{j}$ for some $0 \le i < j \le v$. Because $f$ is a function:

$$\forall k \ge 0:\quad w_{i+k} = f^{k}(w_{i}) = f^{k}(w_{j}) = w_{j+k}$$

So the sequence is periodic with period $p = j - i$ from index $i$ on. $\blacksquare$

What do we learn from that?

1. Repetition is structural, not a bug of the dataset. Any finitely large dataset doesn’t fix the underlying problem.
2. The model can produce at most $|V|$ distinct texts
3. Termination is luck. We can’t guarantee the loop contains a EOS.

Markov

To prevent this, we sample from $P(\cdot|C)$. Thus, $f$ is no longer a function, but a stochastic kernel of a markov chain. This leads to the key insight that a bigram model with this $Sel$ is equivalent to a markov chain, where nodes are tokens and edge weights are $P(w_n|w_{n-1})$.

We can also model higher-order N-Grams as markov chains, where the number of nodes increases exponentially with $V^N$.

The better selection Funct

This section is based on Jurafsky & Martin (2026) too

The maximum likelihood assumption fails in the real world. No matter how big the training set, some sentences will never be able to be generated. That leaves the model locked on a subset $\subset$ of real language.

One option is Laplace Smoothing:

$$P_\text{Laplace}(w_{i})=\frac{ T(w_{i} + 1}{ T(w_{n})}$$

This leads to this Prediction function in bigram models:

$$P_\text{Laplace} (w_{n}|w_{n-1})=\frac{T(w_{n-1}w_n)+1}{\sum\limits_{w}(T(w_{n-1}w)+1)}\overset{\sum1=V}{=}\frac{T(w_{n-1}w_{n})+1}{T(w_{n-1})+V}$$

Laplace charges a proportional tax on evidence

What count would have produced this probability without smoothing? Solving $P_\text{Laplace}=\frac{T^{*}}{T(w_{n-1})}$ for $T^{*}$ gives the effective count

$$T^{*}(w_{n-1}w_{n}) = \bigl(T(w_{n-1}w_{n})+1\bigr)\cdot\underbrace{\frac{T(w_{n-1})}{T(w_{n-1})+V}}_{=:d(w_{n-1})}$$

That factor is brutal at the vocabulary sizes we are dealing with. My vault has $V\approx 2\cdot10^{4}$ types, and a decently frequent context like Gradient might have $T(w_{n-1})=200$. Then $d \approx \frac{200}{20200}\approx 0.0099$, and the bigram Gradient Descent, observed 50 times, ends up with an effective count of roughly $0.5$. We spent a whole corpus collecting that evidence and then threw away $99\%$ of it to fund $V$ continuations that never happened.

Lets add more!

Add-$k$ smoothing, $P_{\text{add-}k}(w_{n}|w_{n-1})=\frac{T(w_{n-1}w_{n})+k}{T(w_{n-1})+kV}$, only changes the tax rate. The structural problem survives: the amount of mass taken from observed events scales with $V$ and with nothing about the context except its total count. A single $k$ tuned on held-out data is always a compromise between contexts we know a lot about and contexts we saw three times.

Discount, don’t rescale

Both scale multiplicatively. But held-out counts show the MLE error is additive: bigrams seen $r$ times recur about $r-0.75$ times (2026). So subtract a fixed $\delta$ instead. Normalization then forces the freed mass to be

$$\lambda_{w_{n-1}}=\frac{\delta\,N_{1+}(w_{n-1}\bullet)}{T(w_{n-1})},\qquad N_{1+}(w_{n-1}\bullet)=\bigl|\{w: T(w_{n-1}w)>0\}\bigr|$$

so diverse contexts leak more. It is only consulted in unseen contexts, so count how many distinct contexts $w$ appears in, not how often. Descent is frequent, yet almost always follows Gradient.

Knesner-Ney

$$P_{\text{KneserNey}}(w_{i}|w_{i-1})=\underbrace{\frac{\max (T(w_{i-1}w_{i})-\delta,0)}{\sum_{w}T(w_{i-1}w)}}_{\text{measured evidence, deficient by}\ 1-\lambda_{w_{i-1}}} +\lambda_{w_{i-1}}P_{KN}(w_{i})$$

We arrive at Knesner-Ney. The right term spends exactly the deficit the left one creates. $\lambda$ is normalized, so $P_{KN}$ is freely chosen.

Perplexity

For LM evaluation, we use Perplexity. It is the exponentiated average negative log-likelihood the model assigns to held-out text:

$$\text{PP}(w_{1:M}) = \exp\left(-\frac{1}{M}\sum_{i=1}^{M}\log P(w_{i}\mid w_{i-N+1:i-1})\right)$$

Read it as an effective branching factor: how many words is the model effectively choosing between at each step? A uniform model over my vocabulary has a perplexity of exactly 15,622, because it is choosing uniformly between 15,622 options. Lower is better (Jurafsky & Martin, 2026).

Building an N-Gram Model

I was able to extract 585 .md files, skipping the following:

forbidden = [
    r"*/.claude/*",
    r"*/.obsidian/*",
    r"*.excalidraw.md",
    r"*/node_modules/*",
]

Preprocessing: Markdown is not prose, since markup is far more repetitive than text, so a model trained on the raw vault would spend most of its probability mass learning that [ follows ]. Twenty-one ordered regex substitutions normalise this, anything carrying display text is reduced to it (aliased wikilinks, markdown links, callout headers), anything else is deleted rather than replaced (frontmatter, code, math, embeds, URLs, HTML, hashtags, rules). Pure markers (headings, bullets, blockquotes, checkboxes, emphasis) are stripped. Because a lecture note is a stack of grammatically unrelated bullet fragments, each line becomes its own training sequence. All exact duplicate notes are dropped by content hash, since the same content being in training and validation set would defeat the point.

277k tokens is very small. For scale, GPT-3 was trained on roughly 300 billion.

How much context is worth paying for?

So I trained one model per order, from $N=0$ (uniform) up to $N=19$, and measured each on the same held-out documents:

Improvement seems to be lorarithmic. The “context window of 1” divides perplexity by almost six, and the trigram cuts it by another two thirds. After $N=4$ perplexity doesn’t seem to improve.

Later it surprisingly goes back up. Past $N=12$, more context makes the model measurably worse. This seemingly is because it has fewer training data for longer lines. At $N=3$, 15% of held-out tokens arrive with a context the model has never seen; by $N=12$ it is 35%.

What it actually writes

Perplexity doesn’t paint the whole Picture. So a test run: same seed, same prompt, one model per order:

--- N = 1 ---
the main
 of
: ,
:
.
 . 1
 the 3 the , b
die

--- N = 2 ---
the main process
:
for all
x lecture 03 .
removed and the
in order to a basis of b ,
fixed a 4 , then there is true .

--- N = 3 ---
the main process before processes are launched when using a kernel , the
removed property from the other files : , email 42 , concat name , :
the trainer no longer randomly selects a seed no longer returns a triplet 18390
: there is a subset of examples for pytorch 2.4 .
if is even .

--- N = 5 ---
the main academic risks are : unclosed gaps in securityprivacy weeks 14 15 , the rsa calculation error , and the missing .
you can decompress drawing data with the command palette : decompress current excalidraw file .

--- N = 10 ---
the main academic risks are : unclosed gaps in securityprivacy weeks 14 15 , the rsa calculation error , and the missing buffer-overflow payload practice .
a graph is planar if it can be drawn on a paper such that no edges cross each other

Every line break is the model emitting </s> and starting a fresh sequence, so the number of lines is itself a signal: the higher the order, the longer it keeps going before it decides EOS.

$N=1$ is function words and punctuation — the model has no context, so it produces the most common tokens in the language, which are the ones carrying no meaning. $N=3$ is the interesting one: locally grammatical, globally nonsense. “the main process before processes are launched when using a kernel” actually quite good, you’d have to secondguess to figure out it is nonsense.

And then $N=5$ and $N=10$ stop writing and start quoting. “A graph is planar if it can be drawn on a paper such that no edges cross each other” is a line from my notes on discrete modelling, reproduced word for word.

The minimum is a lie

That is just an impression though, so let’s measure it. For each order I sampled a few hundred lines, cut them into 8-grams, and checked how many of those 8-grams appear word for word in the training split and, separately, how many of the generated lines are entire lines of the vault reproduced exactly:

The rows move together. Every point of perplexity I bought past $N=3$ was paid for in originality: at the perplexity minimum, two out of three 8-word spans the model produces are copied straight out of my notes, and half of the lines it writes are completely out of my notes.

At $N=10$, 62% of the contexts the model knows occur exactly once in the entire corpus. The model isn’t choosing. It is looking up the rest of the line.

Which brings us back to its own proof. Greedy decoding fails because it is a function on a finite set, so it must eventually cycle. Sampling fixes that, but at high $N$ it stops repeating itself and starts repeating me. Both are a model with nothing of its own to say, and only the first one is visible in the loss.

If you want the most original model, I suggest $N=3$ or $4$: still recombining, just before the counts get sparse enough to overfit. → The best-scoring model and the best model are not the same.

One model, two languages

My notes are bilingual, and both go into one model with one vocabulary. So it code-switches mid-line, as I do in my notes:

fixed an issue to avoid validation loop : : set aller möglichen hypothesen nach der hypothesis
der lokale daemon muss als eigenständige for any

Improvement opportunities

• Estimate the discount instead of guessing it. $\delta$ is hard-coded to 0.75 throughout. Let’s change that and use a separate delta for N=1,2,3+: $\delta = \frac{n_1}{n_1 + 2n_2}$ (2026).
• Top-k sampling. We could store the k most next $w_i$ and $P(w_i| C)$, normalized such that $\sum\limits_{i=1}^{k}P(w_{i}|C)=1$ in a key-value store and run the Markov kernel only on those fixed number of tokens. This would mean Inference goes from $\Theta(N\cdot V)$ per token to $\Theta(1)$, independent of vocabulary and order. Note that this isn’t a lossless operation (Fan et al., 2018).

Limitation

There is a ceiling, and it is not an engineering problem. An n-gram model stores a table indexed by context, so its parameter count grows as $V^{N}$. With my modest 15,622-token vocabulary, a 10-gram table has $15622^{10} \approx 10^{42}$ cells. I observed 277,208 of them. Every additional token of context multiplies the space by 15,622 and adds zero new evidence.

That is the wall the whole field hit, and the reason the next idea had to be structurally different rather than merely bigger. This ceiling was already visible in the smoothing literature by 2001 (Goodman, 2001), and still standing in 2007 when Google trained 5-grams on two trillion tokens of web text and watched the gains go logarithmic.

Conclusion

So: what does a context window of 1 get you? $\rightarrow$ A perplexity of 137.8, sentences that are grammatical for about four words at a time, and a model small enough an Arduino would be able to run it easily.

It doesn’t get you anywhere near the level of competence modern LLMs have and finds no relation between similar tokens. You can’t embed tokens or find any semantically valuable information in the models.

References

1. Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical Neural Story Generation. https://arxiv.org/abs/1805.04833
2. Goodman, J. T. (2001). A bit of progress in language modeling. Computer Speech & Language, 15(4), 403–434. https://doi.org/10.1006/csla.2001.0174
3. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models (3rd ed., I.3). https://web.stanford.edu/~jurafsky/slp3/
4. Knuth, D. E. (1997). The Art of Computer Programming, Volume 2: Seminumerical Algorithms (3rd ed.). Addison-Wesley.
5. Ney, H., Essen, U., & Kneser, R. (1994). On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language, 8(1), 1–38. https://doi.org/https://doi.org/10.1006/csla.1994.1001