In a lucene based search engine, BM25 rewards shorter snippets of text. I’ve mentioned that term frequency might not matter, it’s still good to bias towards shorter snippets of text with fewer terms.
Consider the case a user searches for angularjs
Which is more relevant?
- A book title mentions “angularJS” but also “web design” and “javascript”
- A book title JUST mentioning angularJS, and nothing else
The latter will be more “about” the concept than the one mentioning many topics. It’s a safer bet.
-Doug
This is part of Doug’s Daily Search tips - subscribe here
Upcoming course: Build your own vector database
Want to understand what makes embedding retrieval fast, relevant, and useful in real AI systems? Join Build your own vector database and build the core pieces yourself, from embeddings and indexing to search and retrieval.
Doug Turnbull
More from DougTwitter | LinkedIn | Newsletter | Bsky