Hybrid memory search: lexical, dense, and the fusion that actually ranks

An agent that writes memories and cannot retrieve them is a library without a catalog. The files are there. Nothing comes back. This piece is about the catalog: what lexical search is, what dense search is, why you run both, and how you combine the two lists without lying to yourself about the numbers.

Glorics is the concrete case at the end. The retrieval idea does not belong to us. BM25 is Robertson and Zaragoza. Cover density is Clarke, Cormack and Tudhope. Reciprocal Rank Fusion is Cormack, Clarke and Büttcher, SIGIR 2009. BEIR is Thakur, Reimers and Gurevych, 2021. Local embeddings are a model choice. Fusion is the part most systems get wrong.

01What memory search is

Memory, here, is text you kept on purpose: observations, notes, decisions, fragments of a brief. You do not load all of it into the next prompt. You ask a question of the store and you take back a short list. That question is a retrieval problem. Information retrieval has a century of this. Agents did not invent it. They made the cost of a bad catalog obvious, because a wrong passage goes straight into the next action.

A retriever has one job: given a query, return the passages most likely to matter, fast enough to use. There are two classical ways to do that job, and they fail on different queries.

The store underneath can be markdown on disk, rows in Postgres, a SQLite file. What matters for search is the index: a projection of that text you can query. The source of truth stays readable. If you cannot open the memory as text, you should not trust the hit.

02Lexical search

grep finds a string and stops. BM25 ranks. This is lexical search: the thing people call keyword search. It is the default in Lucene, Elasticsearch, OpenSearch, and SQLite FTS5. Two signals: how often the term appears in this document, and how rare that term is in the whole collection. A rare term that appears is a strong vote. A long document is not automatically a better document. Term frequency saturates, so stuffing the same word does not win.

score(D, Q) = Σ IDF(qi) × [ f(qi, D) × (k1 + 1) ]
                         / [ f(qi, D) + k1 × (1 - b + b × |D|/avgdl) ]

k1 = 1.2    term-frequency saturation
b  = 0.75   length normalisation

SQLite FTS5 implements this. The hidden rank column is BM25, signed negative: lower is better. ORDER BY rank is ascending. Do not treat that number as a 0–1 quality. It is not.

CREATE VIRTUAL TABLE chunks_fts USING fts5(
    text,
    content='chunks',
    content_rowid='id',
    tokenize='porter unicode61'
);

SELECT c.id, c.text, rank
FROM chunks_fts
JOIN chunks c ON chunks_fts.rowid = c.id
WHERE chunks_fts MATCH 'implant score SEO'
ORDER BY rank;

Postgres is a different machine. tsvector plus ts_rank / ts_rank_cd will find terms and order them. ts_rank_cd is not BM25. It is cover-density ranking (Clarke, Cormack, Tudhope, 1999). It sees term frequency and proximity. It does not see inverse document frequency. Native Postgres has no global term statistics, so it cannot run BM25 without an extension (pg_search, ParadeDB, pg_textsearch). Calling the Postgres path “BM25” is a category error. It is still a lexical retriever. It is a weaker one.

03Semantic search

Dense retrieval does not look for identical words. It looks for similar meaning. Each passage becomes a vector. The query becomes a vector, with the same model used at index time. Passages whose vectors sit close to the query come back first.

"How do I speed up my app?"
        →  [ 0.023, -0.158,  0.847, ... ]

"Tips for improving app performance"
        →  [ 0.019, -0.162,  0.831, ... ]     close

"Best restaurants near me"
        →  [ 0.742,  0.534, -0.128, ... ]     far

On Glorics the model is sentence-transformers all-mpnet-base-v2: 768 dimensions, about 110 million parameters, mean pooling, L2-normalised. Cosine similarity is then a dot product. In pgvector, <=> is cosine distance. Similarity is 1 - (embedding <=> query). That conversion is correct for this operator.

Two limits the model card states, and that a memory index must live with:

• The model is English. Hugging Face tags it en. French notes, Corsican briefs, mixed client language: the dense list is weaker than the lexical list on those queries. A multilingual encoder (bge-m3, multilingual-e5) is the actual fix. We have not made that switch.
• Maximum sequence length is 384 tokens. A long observation is truncated before it is embedded. The index of a 2,000-word note is not the note. It is the first window the tokenizer kept.

The Python service sits on localhost:4001. Nothing in the default path leaves the machine. If the service is down we do not invent meaning with a hash of the string. Cosine is switched off. Ranking becomes lexical only.

04Why you run both

Lexical finds “SEO score 91” and misses “the piece did well”. Dense understands that those two sentences are cousins, and cannot be trusted on the number, the date, or the client name. Run one retriever and you have a blind spot. BEIR (Thakur, Reimers, Gurevych, 2021) still has BM25 as a brutal zero-shot baseline; dense models often lose out of domain. Hybrid is the production default, not a law that dense always wins. The open question is how you fuse.

                 query
                   │
           ┌───────┴───────┐
           ▼               ▼
        lexical          dense
        (words)         (vectors)
           │               │
           └───────┬───────┘
                   ▼
                fusion
                   │
                   ▼
            ranked list

Both searches run in parallel. Each side is asked for more rows than you will return (we use ×4). The surplus is not waste. Fusion can promote a passage that appears in both lists. Then you cut.

05Why you must not add the scores

BM25 is unbounded. It moves with collection statistics, query length, and term rarity. Cosine on a normalised embedding lives in a narrow band, often 0.2 to 0.9 for related English sentences. They do not share a unit. A weighted sum of the raw numbers is not a blend. It is whichever scale is louder.

A common attempt: turn the lexical list into 1 / (1 + rank) and add it to cosine. That is still two different objects. Cosine is a magnitude. 1 / (1 + rank) is a position. And rank from ROW_NUMBER() is 1-based. The best lexical row is rank 1, not rank 0.

final = 0.70 × cosine_similarity + 0.30 × 1/(1 + lexical_rank)

ROW_NUMBER() starts at 1
  best lexical row:  1/(1+1) = 0.50
  fused, lexical only: 0.30 × 0.50 = 0.15

A cutoff at 0.35 therefore drops every lexical-only hit.
An identifier, an error code, a dotted token, a French name
that the English encoder did not neighbour: gone.

That table is an illustration of the arithmetic, not a measured run. Cosine values are typical, not logged. The only hard fact in it: with this formula and a 0.35 cutoff, a perfect lexical-only hit cannot survive. We shipped that formula. Then we stopped using the cutoff on the engine.

06Reciprocal Rank Fusion

RRF throws the scores away and keeps the positions. Each list votes. Rank 1 votes more than rank 20. The votes add. There is one parameter, k, fixed at 60 in a pilot on TREC and left alone in the validation (Cormack, Clarke, Büttcher, SIGIR 2009). That is why Elasticsearch, Azure AI Search, Weaviate, OpenSearch and the Supabase hybrid recipe all start at 60.

RRF(d) = Σ  1 / (k + rank_i(d))     over each list
k = 60
rank is 1-based

Rank 1 contributes  1/61 ≈ 0.0164
Rank 10 contributes 1/70 ≈ 0.0143

A hit in both lists beats a hit in one,
without asking BM25 and cosine to share a scale.

Weighted RRF multiplies each list’s contribution by a weight. On the Glorics engine that is still 70 / 30, with a local rescale (k+1) / (k + rank) so that rank 1 contributes 1.0 before the weight. Ranking is the same as 1/(k+rank) up to a constant. A lexical-only hit at rank 1 still surfaces: weight 0.30. That is the point of leaving the cutoff.

RRF is blind to the size of the gap between rank 1 and rank 2. That is the trade. If you have labeled queries and calibrated scores, a normalised convex combination can beat it. Most teams do not have that set. Rank fusion is the default that does not require you to.

07How Glorics runs it

Two surfaces, one idea. The monitor is SQLite: FTS5 (real BM25) and sqlite-vec, score blend 70 / 30, cutoff 0.35. The engine is Postgres: ts_rank_cd (not BM25) and pgvector HNSW, weighted RRF, k = 60, no cutoff. Same 70 / 30. Same ×4 surplus. Different fusion, different lexical algorithm. The article used to describe one path as if it were both. It is not.

-- Engine sketch. Lexical side is ts_rank_cd, not BM25.

WITH
  lexical AS (
    SELECT id,
           ROW_NUMBER() OVER (
             ORDER BY ts_rank_cd(content_tsv, q) DESC
           ) AS rank
    FROM chunks, plainto_tsquery('english', $1) q
    WHERE project_id = $2 AND content_tsv @@ q
    LIMIT top_k * 4
  ),
  dense AS (
    SELECT id,
           ROW_NUMBER() OVER (
             ORDER BY embedding <=> $3::vector
           ) AS rank
    FROM chunks
    WHERE project_id = $2
    ORDER BY embedding <=> $3::vector
    LIMIT top_k * 4
  )
SELECT COALESCE(d.id, l.id) AS id,
       COALESCE(1.0 / (60 + d.rank), 0) * 0.70
     + COALESCE(1.0 / (60 + l.rank), 0) * 0.30 AS rrf
FROM dense d
FULL OUTER JOIN lexical l ON d.id = l.id
ORDER BY rrf DESC
LIMIT $4;

Lexical match on the engine uses english and simple plainto_tsquery. English stemming alone drops French tokens and dotted identifiers (JSON-LD, GPTBot). That is a parser problem, not a ranking problem. plainto_tsquery is also AND of terms: every token must match or the row is absent from the lexical list. Dense retrieval is what still finds the row when the wording moved.

What we index

The source is text. We project it into chunks: content, a tsvector, a 768-dimension embedding, a SHA-256 so unchanged text is never re-embedded. Dedup is (content_hash, project_id). A small HTTP embedder on localhost:4001 serves the model. Observations are buffered (ten items, or five seconds) before insert. When a collection grows past a threshold we summarise the cold tail with an explicit contract: do not drop numbers, do not merge distinct facts, do not invent a cleaner story than the log. That summary is another memory, not a licence to forget the archive.

Search, then read

Discovery and reading are two calls. Search returns a short preview and a score. The next call opens the passage. That is how a person uses an index: check the catalog, then open the page. The prompt stays small. The store stays the store.

query
  │
  ▼
search     →  hits: id, score, short preview
  │
  ▼
get        →  the passage, not the whole store
  │
  ▼
the next prompt, with only what was asked for

08What this piece does not claim

• We do not ship a cross-encoder re-ranker on the fused list. Retrieve-then-rerank is the academic default for precision. It is also another model, another latency, another checkpoint. Fusion is the judgement layer we run.
• We do not have BM25 on Postgres. We have ts_rank_cd. If lexical quality on the engine becomes the bottleneck, the fix is an extension that actually implements BM25, not a rename in the article.
• We do not have a multilingual dense index. French is handled on the lexical side. The encoder is still English.
• 70 / 30 is not a finding. k = 60 is.
• A hit is a candidate. A human is responsible for what is published from it.

What to remember

Lexical precision for identifiers. Dense recall for meaning. Fusion that does not pretend the two scores share a unit. An index a person can still open as text. Local embeddings, so the default path does not leave the machine. On Glorics, that is FTS5 plus sqlite-vec on the monitor, and ts_rank_cd plus pgvector on the engine, with RRF on the path that has to survive an identifier.

If you cannot read the memory, you do not get to trust the hit.