From Cosine to the Edge: Semantic Search in Production

1. A Confession and a Project

Over thirteen articles, this series has built mathematical machinery: vectors that represent words, matrix products that transform them, attention mechanisms that contextualize them. All with the implicit promise that this mathematics is good for something. This article keeps that promise. We are going to build, from the inner product to deployment, the semantic search running at this very moment in this blog’s menu, which the attentive reader can question at /busca/ before, during, and after reading. For the first time in the series, this article’s complete application is in production. The laboratories appear beside the mathematics they make manipulable: cosine in Section 4, InfoNCE in Section 7, and approximate search in Section 9.

First, the confession. Until recently, this blog’s search was powered by a JavaScript library called lunr.js, running entirely in the browser. Because a static site has no server to process queries, client-side search must carry the indexable documents to the client to build the index. The inherited mechanism serialized in full the pages and articles marked access: open; for all other articles, it carried the first 80 words. In the current build of that old implementation, the generated file is about 70 kB, plus approximately 85 kB for the Lunr library, both measured before HTTP compression. They were not megabytes, nor the full contents of every article. The engineering flaw was elsewhere: every visitor paid for transferring and building an index whose size grew with the collection, even if they never made a single query.

The poor author discovered this while auditing his own site, in the classic circumstance in which such things are discovered: while looking for another problem. We must forgive the innocent man. He inherited the template from another project and had never worried about its structure or chosen technology.

There were, therefore, two flaws to correct, one of engineering and one of quality. We have just described the engineering flaw.

The quality flaw is subtler and more interesting: lexical search, even when well implemented, does not understand what it is looking for. That flaw will occupy most of this article because correcting it requires exactly the mathematics of the previous articles: embeddings, attention, softmax, and matrix products, plus one new ingredient that we will derive calmly, a formal notion of proximity between meanings. Along the way, we will keep the series’ commitment: no object will be used before it is defined, no claim will lack a mechanism, number, or example, and the attentive reader will see the calculations done by hand before seeing them executed by a GPU in Cloudflare’s infrastructure.

The word poor matters in this context. Like everything, everything, everything on this blog, for now, it is free and maintained on free systems. The solution has to be free. So the marvelous algorithms used by artificial intelligence assistants are out of the question.

The article’s plan mirrors the path of a query. Sections 2 and 3 establish the problem: why word search fails and what it means to search by meaning. Sections 4 through 7 build the mathematics: cosine similarity, the encoder that turns text into a vector, and the contrastive training that gives those vectors geometric meaning. Sections 8 through 10 build the system: text chunking, the vector index, and the architecture at Cloudflare’s edge, with the production scars documented where they hurt. Section 11 settles the bill: what it costs, spoiler, the fixed cost is zero, and where the limits lie.

2. What Lexical Search Sees

To criticize word search fairly, we must first describe it precisely. The classical information retrieval model represents each document as an unordered collection of terms, a bag of words. This model measures the relevance of a document $d$ to a query $q$ by summing, over the query terms $t$, a weight that grows with the term’s frequency in the document and decreases with the term’s popularity in the collection. The best-known version of this weight is TF-IDF, from term frequency–inverse document frequency:

where $\mathrm{tf}_{t,d}$ is the number of occurrences of term $t$ in document $d$, $N$ is the number of documents in the collection, and $\mathrm{df}_t$ is the number of documents containing $t$. The logarithm compresses the scale: a term present in all $N$ documents receives weight $\log(N/N) = 0$. Ubiquitous words such as “the” and “of” distinguish nothing, while a term present in only one document receives the maximum weight $\log N$. Refinements such as BM25, which saturates the contribution of $\mathrm{tf}$ and normalizes by document length, have dominated lexical search for three decades, and lunr.js implements a close relative of this family.

The weak point is not the weight; it is the dictionary. To the bag of words model, each term is an opaque symbol, and two different symbols are simply different. The query “computing” does not find a document that only says “computational” unless a stemmer, the component that reduces words to a stem, knows they share “comput”. lunr.js uses the Porter stemmer for English by default; when applied to Portuguese, it treats “computação” and “computacional” like strangers who have never met. Even a perfect Portuguese stemmer would not save the query “how does attention work in neural networks” when faced with an article about “mecanismos de atenção em Transformers”: there is no shared stem between “attention” and “atenção”, only shared meaning. Lexical search operates on spelling; we want to operate on semantics. Meaning is everything. Semantics is meaning.

The persistent reader who has followed the series recognizes the diagnosis: it is the same problem as the one-hot representation in the basic vectorization article, where each word became a vector with a single 1 and the inner product between different words was always zero. The solution, there as here, is to replace opaque symbols with dense vectors that encode use and context. The difference is scale: there we vectorized words; now we need to vectorize entire paragraphs and measure distances between them.

3. From Meaning to Geometry

The hypothesis supporting everything that follows has appeared in this series since the article on distributed embeddings: words occurring in similar contexts have similar meanings. This is the distributional hypothesis of Harris and Firth. word2vec, dissected in three articles in the series, materializes the hypothesis: it trains vectors to assign larger inner products to pairs frequently observed together and smaller ones to sampled negative pairs. The result is a space in which “king” and “queen” lie close together and certain directions encode relationships.

I know! Nobody can stand this metaphor anymore. But poor me: it is Sunday, Spain has just won the 2026 World Cup, and I am tired.

It would be tempting to vectorize a paragraph by adding or averaging the vectors of its words. The temptation is instructive because it fails for reasons the series has already explained. First, the average destroys order: “the teacher failed the student” and “the student failed the teacher” have exactly the same average vector and opposite meanings. This is the sequence modeling problem from the article on sequences. Second, word2vec assigns a single vector to each spelling: the “bank” where we sit and the “bank” where we keep money share the same point in space, a crushed polysemy that contaminates any average. The remedy for both ailments has been known to the reader since the article on attention: compute each word’s vector while looking at the sentence’s other words. We need a contextual encoder, also known in life’s bars as a Transformer, that reads the whole paragraph and produces a single vector representing it.

Let us, then, fix the contract that the rest of the article will fulfill. We want a function

that maps a passage of text, whether a seven-word query or a three-hundred-word paragraph, to a vector of 1024 real numbers, with the property that texts with similar meaning map to nearby vectors. Sections 5 through 7 will build $E$; Section 4, humbler and more fundamental, defines what “nearby” means. The number 1024 was not chosen by this pretentious scribe: it is the dimension of the model we will use, and Section 5 will show where it comes from.

4. Cosine Similarity

We have already discussed cosine similarity in one or another article in this series. But we will not shy away from revisiting it.

Let $u, v \in \mathbb{R}^n$ be two vectors. The inner product between them is the sum of the componentwise products, and the norm of a vector is the square root of its inner product with itself:

The cosine similarity between nonzero vectors $u$ and $v$ is the inner product normalized by their norms:

The name is not decorative: in $\mathbb{R}^2$ and $\mathbb{R}^3$, this quantity is exactly the cosine of the angle between the two vectors, and in any dimension it is defined as such. The guarantee that the definition makes sense, that the value always falls within $[-1, 1]$, like a respectable cosine, is the Cauchy–Schwarz inequality, $\lvert \langle u, v \rangle \rvert \le \lVert u \rVert \lVert v \rVert$, which holds in every inner product space.

Let us do the calculation by hand once, in $\mathbb{R}^3$, with numbers that will reappear in the laboratory at the end of this section. Let $u = (2, 1, 0)$ and $v = (1, 2, 0)$:

Two vectors pointing in nearby directions: high similarity. Now take $w = (-1, 2, 0)$ against the same $u$: the inner product is $2\cdot(-1) + 1\cdot2 + 0 = 0$, and the similarity is $0$, so the vectors are orthogonal. In the learned space, this indicates the absence of directional alignment; it does not prove that the texts are semantically independent. And what if we double $v$ to $v’ = (2, 4, 0)$? The inner product doubles to $8$, but the norm of $v’$ also doubles to $2\sqrt{5}$, and the similarity remains $8/(\sqrt{5}\cdot 2\sqrt{5}) = 8/10 = 0{,}8$. This is the property that justifies choosing cosine to compare vectors: it is invariant under multiplication of a vector by a positive scalar. The property removes the vector’s magnitude from the comparison; by itself, it does not claim that a long paragraph and a short sentence about the same subject will receive the same representation.

The reader may object: why not use the more familiar Euclidean distance $\lVert u - v \rVert$? For unit-norm vectors, the objection evaporates, and it is worth seeing why. Expanding the squared distance using the bilinearity of the inner product:

where the final equality uses $\lVert u \rVert = \lVert v \rVert = 1$. On the unit sphere, Euclidean distance and cosine similarity are monotonic functions of one another: ranking documents by smaller distance or larger cosine produces exactly the same order. That is why BGE-M3’s dense mode normalizes vectors before comparison. Our Vectorize index was explicitly configured with the cosine metric; mathematically, its high-precision results could also be ranked by inner product or Euclidean distance between these normalized vectors. The inner product is the operation that the series spent an entire article learning to perform quickly. Speed matters in the modern world of technology.

4.1 Laboratory: Measure Angles

Now that the calculation and the equivalence are visible, we can make them manipulable. The laboratory below places vectors $u$ and $v$ in the plane and recalculates, with every drag, the inner product, the norms, the cosine, and the Euclidean distance after normalization. Start with $u=(2,1)$ and $v=(1,2)$, the example that produced $0{,}8$; double the length of $v$ without changing its direction and confirm that the cosine does not move; then look for the orthogonal position where similarity becomes zero. The buttons reconstruct these three cases with one click, but it is worth dragging the endpoints and forcing geometry to confess with your own hands.

5. The Encoder: Attention, Now Complete

The function $E$ from Section 3 will be BGE-M3, an open model from BAAI, the Beijing Academy of Artificial Intelligence, published by Chen and colleagues in 2024. Before the details, its genealogy, because it explains the architecture: BGE-M3 inherits the architecture and tokenizer of XLM-RoBERTa-large, a multilingual, 24-layer Transformer encoder. Its subsequent training incorporated data from 194 languages, Portuguese among them. The dimensions we will use throughout the article come from the model’s published configuration: dimension $d = 1024$, $h = 16$ attention heads, and a vocabulary of 250,002 subwords.

And, as I said before, being poor means I have to choose things that are efficient yet cheap. Preferably free.

The path of a text through the encoder has four stages: tokenization, embedding, the 24 attention layers, and, in Section 6, the pooling that collapses everything into a single vector. The diligent reader should prepare herself, because we are going to walk through each of these stages. It will take a while. You may stop, drink some water, and say an Our Father. There is no hurry. I will wait.

5.1 Tokenization and the Embedding Layer

The input text is broken into subwords by the SentencePiece tokenizer inherited from XLM-RoBERTa: units that can be whole, frequent words (“de”, “atenção”) or fragments (“comput”, “acional”), the same subword idea discussed in the article on language probability. The shared vocabulary places all languages in the same set of identifiers and allows fragments to be reused when there is orthographic overlap. That is not enough to make “computação” and “computation” semantically close, nor does it require the two words to share fragments. Equivalence across languages is learned by the encoder from multilingual contexts and parallel pairs, as we will see in Section 7.

In another article in the series I called subwords tokens. But today the lexeme token is acquiring a new meaning. Almost an economic one. So we will stick with subword.

Each subword is replaced by one row from an embedding matrix $W_E \in \mathbb{R}^{\vert V\vert \times d}$, where $\vert V\vert = 250\,002$ and $d = 1024$.

Here we should pause for arithmetic sanity, in the spirit of the series: this table alone has $250\,002 \times 1024 = 256\,002\,048$ parameters. About 45% of the model’s approximately 568 million parameters. In other words, almost half of BGE-M3 is dictionary; the other half, which does the computing, is the 24 layers we will examine next. A position vector is also added to each subword vector so the next layer knows the order. Without it, we would return to the bag of words from Section 2. Finally, the tokenizer places the special marker <s> at the start of the sequence. The BGE-M3 paper calls it [CLS], notation we will adopt from now on; it represents no word, and Section 6 will reveal its occupation.

The result of this stage is a matrix $X \in \mathbb{R}^{L \times d}$: one row per token, with $L$ tokens in total. The original BGE-M3 checkpoint documents sequences of up to 8,192 tokens, an extension built during the RetroMAE stage; its configuration file records 8,194 maximum positions. The Cloudflare endpoint used by this blog currently advertises a window of 60,000 tokens, a limit of the hosted service. Our passages of approximately 300 words remain far below both.

5.2 One Attention Layer, Symbol by Symbol

The article on attention built the mechanism calmly; here we will derive it in full, all at once, because we promised depth and because the enlightened reader already has the mathematical instruments she needs. So I can be direct.

Each of the 24 layers performs two operations: multi-head attention and a feed-forward network, both wrapped in residual connections and normalization.

Attention begins by projecting $X$ into three roles. With weight matrices $W_Q, W_K, W_V \in \mathbb{R}^{d \times d_k}$ learned during training, we compute

where $Q$ contains the queries, what each token asks; $K$ contains the keys, what each token offers as an identifier; and $V$ contains the values, what each token delivers if selected.

With $h = 16$ heads, each head works in dimension $d_k = d / h = 1024 / 16 = 64$.

The compatibility score between every pair of tokens in a head is a matrix multiplication, $Q K^\top \in \mathbb{R}^{L \times L}$: entry $(i, j)$ is the inner product between the question from token $i$ and the offer from token $j$, the same operation from Section 4, now measuring affinity between tokens instead of affinity between texts. The GEMM that article 13 taught us to accelerate is the engine behind all of this, repeated across the 16 heads in each of the 24 layers.

The scores are divided by $\sqrt{d_k} = 8$ before softmax, and the series owes the reader a derivation of that constant. Suppose the components of $q, k \in \mathbb{R}^{d_k}$ are independent random variables with mean $0$ and variance $1$, approximately the state of projected activations at the beginning of training. The inner product $\langle q, k \rangle = \sum_{i=0}^{d_k-1} q_i k_i$ sums $d_k$ independent terms, each with mean $0$ and variance $1$; the variance of the sum is the sum of the variances, $d_k$, and the standard deviation is $\sqrt{d_k}$. Without the division, the scores of a head with $d_k = 64$ would have standard deviation $8$, and softmax, whose derivative becomes very small at the extremes, as we saw in the article on dreaded mathematics, would tend to saturate: a distribution that becomes almost one-hot through statistical accident, with little gradient flowing through the attention probabilities. The division returns the scores to variance $1$, keeping softmax in a region more favorable to learning.

With the scores scaled, we must still prevent positions used only for padding to equal lengths from participating in attention. To do so, we add a mask $M \in \mathbb{R}^{L \times L}$, with $M_{ij}=0$ when key $j$ is valid and $M_{ij}=-\infty$ when it is padding. Each row then passes through softmax, which transforms a score vector $s \in \mathbb{R}^L$ into a probability distribution,

Valid positions receive positive weights, masked positions receive zero in the limit, and all weights sum to one. The result weights the values:

Small numbers, as always, before large abstractions. Take a sequence of $L = 3$ valid tokens, therefore with $M=0$, in a toy head with $d_k = 2$, and suppose the projections have produced

where row $i$ is the vector of token $i$. Then

and the reader should check one entry by hand: position $(2, 2)$ is $\langle (1,1), (1,1) \rangle = 2$. Token 2, which points along the diagonal, is most similar to itself. Dividing by $\sqrt{d_k} = \sqrt{2} \approx 1{,}414$ and applying softmax to row 0 turns the scores $(0{,}707;\; 0;\; 0{,}707)$ into the weights $(0{,}401;\; 0{,}198;\; 0{,}401)$: token 0 attends equally strongly to itself and to token 2, which shares its first component, and half as strongly to token 1, which is orthogonal to it. The output of token 0 is the weighted average of the rows of $V$ using these weights: $0{,}401\,(1,0) + 0{,}198\,(0,1) + 0{,}401\,(1,1) = (0{,}802;\; 0{,}599)$. Token 0 entered as $(1, 0)$ and left contaminated by the neighbors it resembles. The contamination is the mechanism, and the entire example fits into a five-minute pencil-and-paper check.

Each token leaves this operation as a weighted average of the values of all valid tokens in the sequence, with weights that sum to one. The token “bank” in a sentence about interest may leave carrying information from “interest” and “account”, while the same “bank” in a sentence about parks may carry information from “park” and “sit”. This is the polysemy from Section 3 being handled by six matrix multiplications: the three projections $XW_Q$, $XW_K$, and $XW_V$; the two products $QK^\top$ and $AV$, where $A$ contains the attention weights; and the final projection by $W_O$. The 16 heads do this in parallel, each with its own weights and able to learn different combinations; their results are concatenated back to dimension $d = 1024$ and projected by $W_O \in \mathbb{R}^{d \times d}$.

Next comes the feed-forward network, applied independently to each token: two linear transformations with the GELU nonlinearity between them, $\operatorname{FFN}(x) = \operatorname{GELU}(x W_1 + b_1) W_2 + b_2$, with $W_1 \in \mathbb{R}^{1024 \times 4096}$ and $W_2 \in \mathbb{R}^{4096 \times 1024}$. The layer expands the dimension by a factor of 4 and then projects it back down. Each of the two sublayers, attention and FFN, is wrapped in a residual connection followed by layer normalization: the output is $\operatorname{LayerNorm}_{\gamma,\beta}(x + \operatorname{Sublayer}(x))$. Normalization first transforms the $d$ components to approximately mean $0$ and variance $1$, using an $\varepsilon$ for stability, and then applies the learned gain $\gamma$ and learned bias $\beta$. Because of those two parameters, the final output need not preserve mean $0$ or variance $1$. The residual gives the gradient a direct path through the 24 layers; normalization controls the scale of the activations along them.

Let us close the subsection with the parameter count, which is the test that we understand the architecture. Per layer: the four attention projections ($W_Q, W_K, W_V, W_O$, each $1024 \times 1024$ plus bias) total about $4{,}2$ million; the two FFN matrices ($1024 \times 4096$ and $4096 \times 1024$, with biases) total about $8{,}4$ million; the gains and biases of the two layer normalizations are small change. Total: $\approx 12{,}6$ million per layer, $\approx 302$ million across the 24 layers. Adding the 256 million in the vocabulary table, about $8{,}4$ million positional embeddings, and the additional retrieval heads brings us to the model’s approximately 568 million documented parameters. The numbers close; the architecture is exactly what I said it would be.

Read this way, it seems easy, but it gave me a headache like no other, almost all of Saturday and the whole of Sunday correcting, changing, and writing code. Even with artificial intelligence assistants helping with the code, it was a mad and completely unexpected amount of work. I swear! When I started, I thought one prompt would solve everything. Vain hope!

6. From Token to Text: Pooling

The 24 layers produce a matrix $H \in \mathbb{R}^{L \times d}$: one contextualized vector per token. The contract from Section 3, however, asks for a single vector per text. The operation that collapses $L$ vectors into one is called pooling, and BGE-M3, in the mode we use, solves it elegantly: it takes only the first row of $H$, the final vector of the [CLS] token, and normalizes it:

Why would the first row carry the entire text? The attention from Section 5.2 distinguishes positions thanks to positional embeddings, but allows [CLS] to query every valid token in every layer. In dense mode, the score used by the loss is calculated from the final state of this marker, and the gradient sculpts it, layer after layer, into an aggregator. It summarizes the text because it was trained to do so, not because the first position has any special geometric property.

[CLS] is a reporter seated in the front row whose sole job, learned under pressure, is to leave the room with a summary of the meeting. Final normalization is not a detail: with $\lVert E(\cdot) \rVert = 1$, the identity from Section 4 holds, and comparing texts by inner product, cosine, or Euclidean distance becomes the same decision.

Two important notes complete the portrait of the model:

First: the original BGE-M3 checkpoint extends the XLM-RoBERTa limit to 8,192 tokens and undergoes RetroMAE pretraining. In it, the encoder receives a masked version of the text and produces the sentence vector; a shallow decoder receives this vector and another, more aggressively masked version, and tries to reconstruct the hidden tokens. The reconstruction therefore does not start from the vector alone, but it forces the vector to carry information useful for retrieval.

Second: the “M3” in BGE-M3 announces multi-lingual, multi-functionality, multi-granularity, with little modesty. The multifunctionality deserves a paragraph because it concerns a choice I made. On this infernal Sunday afternoon.

Besides the dense [CLS] vector, the one we use, BGE-M3 produces two other representations in the same pass through the encoder. One is sparse: weights learned only for the subwords present in the text, implicitly forming a vector the size of the vocabulary in which most positions are zero. It is lexical search rebuilt inside the Transformer, useful for matching rare terms and proper names that the dense vector may dilute. The other is a multi-vector representation in the ColBERT lineage: one projected vector per token, compared through late interaction between query and document. This mode can improve retrieval quality, as BGE-M3 experiments show, but it is proportionally more expensive to store and query. Our system uses only dense mode, and the reason is engineering: the index in Section 9 stores one 1024-dimensional vector per passage, while the alternatives would require structures and operations a blog cannot amortize. The reader is left with the map, however: when dense search fails on a rare proper name, she will know which mode of the same model may help.

Which reminds me that the BERT articles stayed in draft. I did not solve the exercises or make the laboratory artifacts. One of these days I have to publish those articles. The course stopped being mine and, consequently, I stopped teaching natural language processing. That is, I published only the first part of the course.

7. Training: Geometry by Contrast

The most important conceptual step is still missing. The architecture in Sections 5 and 6 produces some vector for each text. But nothing in it guarantees that similar texts receive similar vectors. Here is the crux: this property does not come from the architecture; it comes from contrastive training, and the loss function that governs it deserves a complete derivation.

The training datum is a pair $(q, p^{+})$: a query and a passage known to be relevant, such as title and body, instruction and answer, or parallel sentences. During the unsupervised stage, the authors collected 1.2 billion textual pairs from 194 languages; later stages added supervised and synthetic data and hard negatives. Within a batch containing other passages $p_1, \dots, p_m$, the negatives, the basic form of the InfoNCE loss for dense mode is:

where $\tau > 0$ is the temperature, a small scalar, typically between $0{,}01$ and $0{,}1$, that expands similarity differences before softmax. The reader recognizes the structure: it is the same cross-entropy from the article on language probability, applied to a “classification” whose correct class is the true passage and whose incorrect classes are the negatives in the batch. Minimizing $\mathcal{L}$ pushes $\cos(E(q), E(p^{+}))$ upward and every $\cos(E(q), E(p_j))$ downward: it brings true pairs closer and drives false pairs apart. Repeated across many batches and training stages, the pressure sculpts the entire space: the only way for the model to pay a low loss on all pairs simultaneously is to organize vectors by meaning. The geometry was not programmed; it was extorted.

Temperature controls the severity of the extortion, and it is worth deriving how. With $\tau = 1$, a similarity difference of $0{,}1$ between the correct passage and a negative becomes a ratio of $e^{0{,}1} \approx 1{,}11$ between exponentials, almost indifferent. With $\tau = 0{,}02$, the same difference becomes $e^{5} \approx 148$: softmax starts severely punishing negatives that come close. Low temperatures concentrate the gradient on hard negatives, passages that resemble the correct one without being correct, precisely the cases that separate a good search from a decorative one. And this search is going to be good!

The multilingualism promised in Section 5.1 comes together here: the data include parallel sentences and correspondences across languages. The same contrastive pressure that brings “how does attention work” close to a relevant paragraph also brings semantically corresponding texts written in Portuguese and English together, organizing the languages in a shared space. This does not guarantee that every translation receives identical vectors, but it enables the cross-language retrieval observed in the model’s tests. For this blog, which I still write mainly in Portuguese, about technical literature in English, with terms from both languages mixed in the same paragraph, that property is not a luxury. Again, it is the concern of someone with his feet on the ground. That was the requirement that decided the model choice. Remember? Did I say something about a poor scribe?

Finally, BGE-M3’s particular touch: self-distillation (self-knowledge distillation).

The three modes from Section 6, dense, sparse, and multi-vector, are trained together, and their combined scores serve as a teacher for each mode individually: each mode also learns to imitate the committee to which it belongs. The ablations published by the authors show that this regime improves dense mode relative to training without self-distillation. As an intuition, the reader can think of some of the sparse mode’s lexical sensitivity influencing the dense vector, the TF-IDF from Section 2 beneficially haunting the vector that retired it.

7.1 Laboratory: The Temperature of Extortion

The laboratory below gives substance to the loss from Section 7. Query $q$ remains fixed, while positive passage $p^+$ and three negatives can be dragged around the unit circle. Every movement changes the cosine, the logit divided by $\tau$, the probability produced by softmax, and the loss $-\log P(p^+\mid q)$. The final column shows $\partial\mathcal{L}/\partial\cos$: the negative sign for the positive passage commands an increase in its similarity; the positive signs for the negatives command decreases in theirs.

Start with the two presets having margin $0{,}1$. At $\tau=1$, the ratio between exponentials is only $e^{0{,}1}\approx1{,}11$; at $\tau=0{,}02$, the same geometry produces $e^5\approx148$. Then drag negative n1 until it nearly ties $p^+$ and observe where the training pressure concentrates. The geometry remains the same; the thermostat decides how much it hurts.

8. Engineering I: Chunking the Collection

The mathematics is complete; engineering begins, along with decisions that have no theorem, only trade-offs. The first: what exactly should we vectorize? Vectorizing each entire article is tempting and wrong for two measurable reasons.

First, resolution: the single vector of a 10,000-word article that devotes three paragraphs to softmax must also represent the other 97% of its content, and a query about softmax may match that global representation poorly.

Second, limit and cost: the original checkpoint accepts 8,192 tokens, but this series’ article on GEMM has 10,759 words even before subword tokenization. Cloudflare’s hosted endpoint currently advertises 60,000 tokens and can receive the entire article, but depending on that extension would tie the indexer to the service and make attention process a much longer sequence. Chunking still reduces inference cost and preserves compatibility with the original checkpoint.

The adopted solution is chunking: split each article into passages of controlled size and vectorize each passage. Our indexer targets passages of approximately 300 words, based on previous experiments and a comfortable margin below both token limits. We apply two rules of civility.

The first: passage boundaries respect paragraph boundaries, and gigantic paragraphs are divided by sentences; a passage that begins in the middle of a derivation poisons its own vector.

The second: the article title is prefixed to every passage before vectorization, because a loose paragraph saying “division by $\sqrt{d_k}$ prevents saturation” gains considerable identity when it reaches the encoder accompanied by “Paying Attention: division by…”. The text is also cleaned before splitting: code blocks, LaTeX formulas, and markup are removed because this index prioritizes prose and because those structures would consume tokens without a previously measured retrieval benefit. A CUDA block in the middle of a passage is, in this version, noise that smells like a signature.

The next version will find a way to include LaTeX and mathematics. I have had one or two ideas about that.

The numbers from the real collection, measured again by the indexer, are 62 published articles split into 935 passages. That represents an average of 15 passages per article, with the GEMM article contributing 42. Each passage becomes a 1024-dimensional vector, giving $935 \times 1024 = 957\,440$ dimensions, or 3.83 MB of 32-bit floating-point coordinates. This is not the complete index size: it excludes metadata and the search structure maintained by Vectorize. It is also larger, not smaller, than the roughly 155 kB of the two JavaScript files in the Lunr solution before HTTP compression. The proper advantage lies in where the cost lives: the vectors remain on the server, and the visitor neither downloads the collection nor builds the index in the browser. The coordinates contain no readable prose, but the metadata stores the URL, title, and summary used to compose the response.

Mental note: revisit the GEMM article when I finish the competitive programming book.

Updates are incremental, and the mechanism fits in one sentence containing a hash: for each article, the indexer stores the SHA-256 of the URL, title, access classification, and cleaned text. On every deployment, it reprocesses only articles whose hash has changed. A new, revised, or removed article, or one whose relevant metadata changed, costs API calls; the others cost a local string comparison.

Publishing a new post on this blog adds it to the index without further intervention when credentials are configured: after building the site, the deployment script invokes the indexer. This step is deliberately nonfatal; if the API fails, the site is still published, and the index remains one revision behind until the next successful run.

The code comes after the mathematics, as the house rules require. Breaking the series’ custom of writing C++, the passage below is Python because it is the actual production code for this blog’s indexer, not an illustration.

That is because I had the brilliant idea of inheriting code from the web to save time. It never works; I never learn. The second version will be in C++, perhaps during the year-end holidays.

The heart of chunking and incremental updates appears in the following fragment:

def chunk_text(text, target=300, hard_max=450):
    """Groups paragraphs; very long ones are divided by sentences."""
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, current, current_words = [], [], 0
    for para in paragraphs:
        words = para.split()
        if len(words) > hard_max:
            sentences = re.split(r"(?<=[.!?])\s+", para)
            para_parts, buf, buf_w = [], [], 0
            for sent in sentences:
                sw = len(sent.split())
                if buf_w + sw > target and buf:
                    para_parts.append(" ".join(buf))
                    buf, buf_w = [], 0
                buf.append(sent)
                buf_w += sw
            if buf:
                para_parts.append(" ".join(buf))
        else:
            para_parts = [para]
        for part in para_parts:
            pw = len(part.split())
            if current_words + pw > target and current:
                chunks.append("\n\n".join(current))
                current, current_words = [], 0
            current.append(part)
            current_words += pw
    if current:
        chunks.append("\n\n".join(current))
    return chunks

# Increment: URL, title, access, and text participate in the hash.
digest = hashlib.sha256(
    ("|".join([post["url"], post["title"],
               post.get("access", "closed")]) + "\n" + text)
    .encode("utf-8")
).hexdigest()
if state["posts"].get(source, {}).get("hash") == digest:
    continue
embed_inputs = [post["title"] + "\n\n" + c for c in chunks]

The attentive reader should notice the title prefix on the last line and the four components of the hash. The didactic decision from the previous paragraph became a concatenation, and there is no network call until the 64 hexadecimal digits of SHA-256 disagree with the saved state. The savings from incremental indexing do not come from a clever algorithm; they come from comparing the digest before spending money.

9. Engineering II: The Vector Index

With the collection vectorized, searching is trivial in principle: given the query vector $E(q)$, calculate the inner product against each of the index’s $N$ vectors and return the largest values. Brute force costs $N \cdot d$ multiply-accumulate operations per query. With $N = 935$ and $d = 1024$, that is $957\,440$ FMAs, about 1.91 million FLOPs under the convention that counts one multiplication and one addition per FMA. The coordinate set occupies 3.83 MB; a vectorized, contiguous CPU implementation can scan it in a few milliseconds, although complete latency depends on memory, language, and result ordering. Let us therefore confess without drama: exact search would be perfectly viable for this blog’s current collection.

The system nevertheless uses a real vector index: Cloudflare Vectorize. This decision deserves an honest defense instead of marketing.

From a practical standpoint, besides search, Vectorize provides durable storage, incremental updates through an API, metadata per vector, and integration with the rest of the architecture in Section 10. Without it, I would have to build everything by hand around our brute force.

Besides practicality, there are our principles. Vector search at scale is a rich problem, and the reader should know the map even while using a managed version. When $N$ reaches the millions, brute force $O(N \cdot d)$ stops being pocket change in a ping, and the classical answer is approximate nearest neighbor (ANN) indexes, which trade a fraction of exactness for orders of magnitude in speed. The two dominant families are partitioning indexes such as IVF, which groups vectors into Voronoi cells through k-means and examines only the cells nearest to the query, and navigable graphs such as HNSW, which builds a neighborhood graph in hierarchical layers and answers queries by descending the graph through greedy search, with approximately logarithmic empirical cost on many datasets. Cloudflare does not publicly document which structure Vectorize uses under the hood. Its query documentation does state, however, that approximate scoring is the default and that returnValues: true activates high-precision scoring over the original values. Our code asks for metadata, not values; the results are therefore approximate even with only 935 vectors. The small collection reduces the practical need for approximation, but it does not turn it into exact search.

Cloudflare did not fall from the sky. Besides being the CDN I use for the blog, Vectorize was passionately defended in a thread on X by a developer who was hired by Anthropic this week. Vectorize stayed in my head, and today I had no doubts.

Each vector enters the index with metadata: URL, title, and a short article summary; and with a deterministic identifier derived from the URL and passage position. This makes each reindexing an idempotent replacement instead of a duplication. The query asks for the $k = 8$ nearest neighbors; because a good article on the subject tends to place several passages among the eight, the system deduplicates by URL, keeps the best passage from each article, and returns up to five articles. It is the distinction, inherited from any decent retrieval system, between retrieving passages and presenting documents.

9.1 Laboratory: The Price of Approximation

The next laboratory reduces the index to 32 unit vectors and fixes $k=3$, small enough for every comparison to fit on the screen. Brute force visits all 32 points and supplies the exact answer. IVF divides the circle into four cells and lets us choose how many to examine; the didactic HNSW mode enters through a sparse layer and traverses a neighborhood graph with a limited budget. This two-layer visualization preserves the central HNSW mechanism, but it does not aim to reproduce every detail of construction, pruning, and search in a production library.

Drag the query to an IVF boundary and reduce the number of cells examined. Then do the same in the graph with few visits. Red circles mark exact neighbors that approximation left behind; recall@3 measures the recovered fraction. Increasing the budget tends to restore the answer at the price of more comparisons. It is the contract of ANN indexes turned into a calculation that can fail in front of the reader, as every good engineering calculation should be able to do.

10. Engineering III: The Edge and Its Scars

The stage is still missing. This blog is a static site: Markdown compiled by Jekyll into HTML, served by Cloudflare Pages from a repository. There is no application server, and that absence is a feature, a design choice, not a limitation. Semantic search, however, must execute code for each query: vectorize the question and query the index. The solution is to run that code at the edge, in Cloudflare’s distributed infrastructure, as a function only a few dozen lines long. This function receives the query, validates its length, calls Workers AI as a managed service to obtain $E(q)$ from the same BGE-M3 used for indexing (symmetry is mandatory: query and collection must inhabit the same vector space and be vectorized by the same model), queries Vectorize, deduplicates, and returns JSON containing titles, URLs, and summaries. The platform decides where to serve the inference, so there is no guarantee that the GPU is in the same physical location from which the HTML was served. The core, written in the JavaScript used in production, can be seen here:

const ai = await env.AI.run("@cf/baai/bge-m3", { text: [q] });
const embedding = ai.data[0];                    // E(q), 1024 dimensions
const res = await env.SEARCH_INDEX.query(embedding, {
  topK: 8, returnMetadata: "all",
});
// Several passages may point to the same article:
// deduplicate by URL, keeping the best score
const byUrl = new Map();
for (const m of res.matches) {
  const prev = byUrl.get(m.metadata.url);
  if (!prev || m.score > prev.score) byUrl.set(m.metadata.url, m);
}

The end-to-end latency measured from my chair: about 2.8 seconds for the first query and 90 milliseconds for repeated queries, served by a five-minute cache at the edge itself. Because the cache avoids both inference and the index query, this difference measures the full path and does not allow us to assign an exact share of latency to the encoder. And I really did test it.

We promised scars, and an article that reported only successes would be advertising. Two production failures are worth recording because both teach us something.

The first: the Workers AI backend occasionally returned HTTP 408 during our runs, and the first version of the indexer treated that status as a definitive client error, aborting a stage after nineteen minutes of work. Status 408 means the server did not complete the request within the expected time; the observed load explains the episode’s context, not the service’s internal cause. Teeth grinding in anger. It took me half an afternoon to notice.

The diagnosis teaches the taxonomy that HTTP manuals rarely underline: among 4xx errors, which generally mean “the fault is yours, do not insist”, 408 and 429 are exceptions that can be transient. These two bandits may mean “not now”, and the adopted response is to try again, with exponential backoff between attempts and smaller batches per call. After the correction, indexing crosses 408 responses with up to five attempts per batch and finishes; the incremental state from Section 8 ensures that, in the worst case, an interruption resumes where it stopped without repeating articles already persisted. It takes time: the observed full load consumed almost two hours. An entire World Cup final.

The second scar is subtler. In the first live tests, the endpoint responded. Quickly, confidently, and with exactly the same results for every query.

There I was, prostrate before an oracle that had decided the answer before hearing the question.

It stole the other half of my Sunday afternoon. It took me a while to remember why I use Cloudflare.

The cause: a CDN cache rule predating the project, configured to store everything for two hours while ignoring the query string. For static HTML pages, a legitimate optimization; for an endpoint where the query string IS the question, deadly poison.

The first query after each expiration was truly answered and stored; all others, for two hours, received the first query’s answer. The correction had the two ends every cache correction requires: an exception rule in the CDN for the search path, and the endpoint hardened to declare no-store to the outside world while internally retaining a per-query cache whose key includes what the outside key ignored. The lesson, which transcends CDNs: a shared cache with the wrong key is not a slow cache or a cold cache. It is a system that answers someone else’s question, with the worst property a system can possess in the modern world: conviction.

11. The Zero-Cost Arithmetic

We promised zero fixed cost in the first paragraph; an article in this series does not end without checking the promise against current numbers. Three resources are consumed, each with a free allowance. In Vectorize, the $957\,440$ stored vector dimensions occupy 19.1% of the 5 million limit, so the collection can grow about 5.2 times while maintaining the same average number of passages per article. The monthly query allowance is 30 million queried vector dimensions. Under Cloudflare’s published formula, the 935 stored vectors and the vectors sent in queries enter this calculation; after the current load, approximately 28,361 queries of 1024 dimensions remain per month, about 945 per day in a 30-day month. Cache hits from Section 10 do not query Vectorize and consume none of this allowance.

The encoder inference uses another unit. Workers AI grants 10,000 neurons per day and currently charges 1,075 neurons per million input tokens for @cf/baai/bge-m3. The allowance therefore equals approximately 9.3 million input tokens per day; an ordinary query, with tens of tokens, consumes a small fraction of a neuron. The collection’s initial load fit comfortably within the observed budget, and Section 8’s incremental indexing exists precisely to avoid repeating it with every publication. These limits are current commercial properties of the service, not mathematical constants, but today no new bill arrives because of search.

12. Where We Stand

Before the synthesis, the curious reader can open /busca/ and question the complete application of this mathematics. Start with “why do used-car dealers deceive” and see which article answers, noting that none of the query’s words need appear in it. The laboratories in Sections 4, 7, and 9 showed the path at successive scales: measure an angle, sculpt the angles through training, and avoid measuring all of them during retrieval. Production search repeats that path in 1024 dimensions across the collection’s 935 passages.

Thirteen articles of mathematics and one of production later, the synthesis fits into a sentence the series will carry forward: a language model turns meaning into geometry, and searching means measuring angles. The attention in Section 5 consists of inner products between tokens; the training in Section 7 is a discipline of angles between texts; the search in Section 9 uses inner products again, now served by Cloudflare’s distributed infrastructure. The diligent reader who has reached this point has learned more than how to use semantic search. She has learned why it works, the difference between operating a machine and owning it. In the next article, the series will return to the decoder and generation; search will remain in the menu, working, as mathematics should when it grows up and gets a job.

References

BA, J. L.; KIROS, J. R.; HINTON, G. E. Layer Normalization. arXiv preprint arXiv:1607.06450, 2016. Available at: https://arxiv.org/abs/1607.06450.

CHEN, J.; XIAO, S.; ZHANG, P.; LUO, K.; LIAN, D.; LIU, Z. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv preprint arXiv:2402.03216, 2024. Available at: https://arxiv.org/abs/2402.03216.

CLOUDFLARE. bge-m3 (BAAI). Workers AI Docs, 2026. Available at: https://developers.cloudflare.com/workers-ai/models/bge-m3/.

CLOUDFLARE. Pricing: Cloudflare Vectorize. Cloudflare Docs, 2026. Available at: https://developers.cloudflare.com/vectorize/platform/pricing/.

CLOUDFLARE. Pricing: Workers AI. Cloudflare Docs, 2026. Available at: https://developers.cloudflare.com/workers-ai/platform/pricing/.

CLOUDFLARE. Query Vectors. Cloudflare Vectorize Docs, 2026. Available at: https://developers.cloudflare.com/vectorize/best-practices/query-vectors/.

CONNEAU, A. et al. Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 8440–8451, 2020. Available at: https://aclanthology.org/2020.acl-main.747/.

DEVLIN, J.; CHANG, M.-W.; LEE, K.; TOUTANOVA, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, p. 4171–4186, 2019. Available at: https://aclanthology.org/N19-1423/.

FIRTH, J. R. A Synopsis of Linguistic Theory, 1930–1955. In: Studies in Linguistic Analysis. Oxford: The Philological Society, p. 1–32, 1957.

HARRIS, Z. S. Distributional Structure. Word, v. 10, n. 2–3, p. 146–162, 1954. Available at: https://doi.org/10.1080/00437956.1954.11659520.

HENDRYCKS, D.; GIMPEL, K. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415, 2016. Available at: https://arxiv.org/abs/1606.08415.

JÉGOU, H.; DOUZE, M.; SCHMID, C. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, v. 33, n. 1, p. 117–128, 2011. Available at: https://doi.org/10.1109/TPAMI.2010.57.

KARPUKHIN, V. et al. Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of EMNLP 2020, p. 6769–6781, 2020. Available at: https://aclanthology.org/2020.emnlp-main.550/.

KHATTAB, O.; ZAHARIA, M. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Proceedings of SIGIR 2020, p. 39–48, 2020. Available at: https://doi.org/10.1145/3397271.3401075.

KUDO, T.; RICHARDSON, J. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 66–71, 2018. Available at: https://doi.org/10.18653/v1/D18-2012.

MACQUEEN, J. Some Methods for Classification and Analysis of Multivariate Observations. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, v. 1, p. 281–297, 1967. Available at: https://digicoll.lib.berkeley.edu/record/113015?v=pdf.

MALKOV, Y. A.; YASHUNIN, D. A. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, v. 42, n. 4, p. 824–836, 2020. Available at: https://doi.org/10.1109/TPAMI.2018.2889473.

MIKOLOV, T.; CHEN, K.; CORRADO, G.; DEAN, J. Efficient Estimation of Word Representations in Vector Space. Proceedings of the International Conference on Learning Representations, 2013. Available at: https://arxiv.org/abs/1301.3781.

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY. Secure Hash Standard (SHS). FIPS PUB 180-4, 2015. Available at: https://doi.org/10.6028/NIST.FIPS.180-4.

OORD, A. van den; LI, Y.; VINYALS, O. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748, 2018. Available at: https://arxiv.org/abs/1807.03748.

PORTER, M. F. An Algorithm for Suffix Stripping. Program, v. 14, n. 3, p. 130–137, 1980. Available at: https://doi.org/10.1108/eb046814.

REIMERS, N.; GUREVYCH, I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP-IJCNLP 2019, p. 3982–3992, 2019. Available at: https://aclanthology.org/D19-1410/.

ROBERTSON, S.; ZARAGOZA, H. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, v. 3, n. 4, p. 333–389, 2009. Available at: https://doi.org/10.1561/1500000019.

SALTON, G.; BUCKLEY, C. Term-weighting Approaches in Automatic Text Retrieval. Information Processing & Management, v. 24, n. 5, p. 513–523, 1988. Available at: https://doi.org/10.1016/0306-4573(88)90021-0.

STEELE, J. M. The Cauchy-Schwarz Master Class: An Introduction to the Art of Mathematical Inequalities. Cambridge: Cambridge University Press, 2004. Available at: https://doi.org/10.1017/CBO9780511817106.

VASWANI, A. et al. Attention Is All You Need. Advances in Neural Information Processing Systems, v. 30, p. 5998–6008, 2017. Available at: https://arxiv.org/abs/1706.03762.

XIAO, S.; LIU, Z.; SHAO, Y.; CAO, Z. RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder. Proceedings of EMNLP 2022, p. 538–548, 2022. Available at: https://aclanthology.org/2022.emnlp-main.35/.