Skip to content

Conversation

@maximedb
Copy link
Contributor

@maximedb maximedb commented Feb 9, 2022

Hi,

This pull request fixes two issues with the sparse search:

  1. Because of the step in trange, some queries are never evaluated (only the first one of each batch is evaluated) as reported in Error in beir/beir/retrieval/search/sparse/sparse_search.py #60.
  2. Dense search skips some results (see here), this was not implemented in the sparse search. It impacts the results on ArguAna and Quora (Can't understand a certain line #37)

Let me know if you have any questions.

Maxime.

@maximedb maximedb mentioned this pull request Feb 9, 2022
@thakur-nandan
Copy link
Member

Hi @maximedb and @jin530,

Thanks for the pull request. Indeed as mentioned in #60, this is an error that sadly I wasn't able to find earlier myself.
Nevertheless, the scores for SPARTA reported in the BEIR paper are accurate. It used an old script that did not have the issue. I will soon have a look and merge the PR.

Kind Regards,
Nandan Thakur

@nreimers nreimers merged commit a55552d into beir-cellar:main Feb 16, 2022
wisebayes pushed a commit to wisebayes/beir that referenced this pull request May 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants