Speed search with stop words and champion lists

Reduce noisy postings and limit scoring work with domain-aware stop words, champion lists, and safe recall tradeoffs.

Stop words and champion lists

A scoring function cannot rescue a document that was removed before scoring began. We can reduce search work by omitting common words or retaining only selected candidates for each term, but each shortcut discards information. The useful question is which queries lose an answer when that information disappears.

We'll compare removing words with capping candidates, then follow how each changes the results.

Removing words from the representation

wiki/bm25-production-ranking The BM25 lesson kept every analyzed token and let document frequency influence its weight. Another approach removes selected words entirely. A configured list of words to omit is a stop-word list. Commonness may motivate the choice, but the actual removal is a product and language policy rather than a consequence of the score formula.

Our experiment chooses exactly a , and and to . They are test inputs, not a recommended English stop list. The unchanged corpus retains them; the replay creates a separate pruned copy so later lessons continue to use the same source texts and statistics.

Deleting tokens can change more than whether a term is searchable. If the remaining tokens are packed together, their positions change. The original d5 text includes keys to rows . A packed pruned representation contains keys rows , so a request for that exact adjacent phrase can match even though the original document did not contain it.

The posting-list lesson's phrase helper checks adjacent analyzed tokens. In this experiment the stop policy is applied to both document and query text. That lets keys to rows continue to match after both sides lose to , while the distinct literal request keys rows also becomes a match. Preserving the original position gaps would avoid that particular adjacency error, though queries requiring a removed word would still need a representation that retained it.

Which common words are expendable?

Inspect both how widely a term occurs ( df / N ) and how often it repeats. Define the averaging population: total occurrences divided by all documents differs from the average among matching documents. These are diagnostics, not a universal rule for deleting terms.

In a database-only catalog, database may occur almost everywhere yet still matter in an exact title. A required word, negation or direction can be essential even when it has little ranking weight. Review representative title and phrase queries before choosing a stop list. Lower weighting retains evidence that deletion removes.

A configured language list is only a starting policy. Elasticsearch's stop filter allows explicit words, language lists and case handling. Its trailing-token option also matters for completion: removing the word currently being typed can change the offered suggestions. Our experiment does not emulate that analyzer; it deliberately repacks tokens to expose the phrase error.

Keeping selected candidates

There is another way to reduce work without deleting a word: retain a small preselected list of documents for it. Such a retained subset is called a champion list. The subset is chosen before the full query is known. It can be built for any term, not only common words; list sizes can differ by term.

Our chosen cap is one document per term. Build each list by scoring that single term with our BM25 variant and retaining the highest result, using identifier order for ties. At query time, take the union of the retained lists, then apply the full requested matching condition to those candidates. This is a candidate restriction, not a complete implementation of arbitrary top-result search.

For database , the equal-length single-term tie retains d2 ahead of d5 . For index , the shorter document d8 wins over d5 . The combined query database index therefore begins with only d2 and d8 available, even though d5 contains both terms. A document that is second for each individual term can be the best combined answer.