Bayesian BM25 is cool

Look at this math and grasp at its majesty:

P(R) = P(R | BM25) * P(R | Emb) # Prob(Relevance) = lexical * embedding

OK what’s so special about that? That’s an AND. A probabilistic way of combining scores so that when BOTH “things happen”, the final result becomes true. Here when BOTH lexical AND embedding scores are high, this will be high.

What Bayesian BM25 does, as explained in my blog article, is calibrate BM25 scores so they become meaningful probabilities.

For your labeled dataset:

• A “meh result” BM25 → map to P=0.5
• What’s a “good results” BM25 score → map that to a 1.0

Once calibrated, you can pass unscaled, whacky BM25 into BB25’s formula and get a properly calibrated BM25 probability for your dataset. Final probability of relevance can use our probabilistic and - making hybrid search about combining probabilities, and moving away from imprecise RRF solutions

Check out the paper here

-Doug

​

​

This is part of Doug’s Daily Search tips - subscribe here