opensearch-neural-sparse-encoding

Select the model

The model should be selected considering search relevance, model inference and retrieval efficiency(FLOPS). We benchmark models' zero-shot performance on a subset of BEIR benchmark: TrecCovid,NFCorpus,NQ,HotpotQA,FiQA,ArguAna,Touche,DBPedia,SCIDOCS,FEVER,Climate FEVER,SciFact,Quora.

Overall, the v2 series of models have better search relevance, efficiency and inference speed than the v1 series. The specific advantages and disadvantages may vary across different datasets.

Overview

• Paper: Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers
• Fine-tuning sample: opensearch-sparse-model-tuning-sample

This is a learned sparse retrieval model. It encodes the queries and documents to 30522 dimensional sparse vectors. The non-zero dimension index means the corresponding token in the vocabulary, and the weight means the importance of the token.

The training datasets includes MS MARCO, eli5_question_answer, squad_pairs, WikiAnswers, yahoo_answers_title_question, gooaq_pairs, stackexchange_duplicate_questions_body_body, wikihow, S2ORC_title_abstract, stackexchange_duplicate_questions_title-body_title-body, yahoo_answers_question_answer, searchQA_top5_snippets, stackexchange_duplicate_questions_title_title, yahoo_answers_title_answer.

OpenSearch neural sparse feature supports learned sparse retrieval with lucene inverted index. Link: https://opensearch.org/docs/latest/query-dsl/specialized/neural-sparse/. The indexing and search can be performed with OpenSearch high-level API.

Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers.sparse_encoder import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("opensearch-project/opensearch-neural-sparse-encoding-v2-distill")

query = "What's the weather in ny now?"
document = "Currently New York is rainy."

query_embed = model.encode_query(query)
document_embed = model.encode_document(document)

sim = model.similarity(query_embed, document_embed)
print(f"Similarity: {sim}")
# Similarity: tensor([[38.6113]])

decoded_query = model.decode(query_embed)
decoded_document = model.decode(document_embed)

for i in range(len(decoded_query)):
    query_token, query_score = decoded_query[i]
    doc_score = next((score for token, score in decoded_document if token == query_token), 0)
    if doc_score != 0:
        print(f"Token: {query_token}, Query score: {query_score:.4f}, Document score: {doc_score:.4f}")

# Token: york, Query score: 2.7273, Document score: 2.9088
# Token: now, Query score: 2.5734, Document score: 0.9208
# Token: ny, Query score: 2.3895, Document score: 1.7237
# Token: weather, Query score: 2.2184, Document score: 1.2368
# Token: current, Query score: 1.8693, Document score: 1.4146
# Token: today, Query score: 1.5888, Document score: 0.7450
# Token: sunny, Query score: 1.4704, Document score: 0.9247
# Token: nyc, Query score: 1.4374, Document score: 1.9737
# Token: currently, Query score: 1.4347, Document score: 1.6019
# Token: climate, Query score: 1.1605, Document score: 0.9794
# Token: upstate, Query score: 1.0944, Document score: 0.7141
# Token: forecast, Query score: 1.0471, Document score: 0.5519
# Token: verve, Query score: 0.9268, Document score: 0.6692
# Token: huh, Query score: 0.9126, Document score: 0.4486
# Token: greene, Query score: 0.8960, Document score: 0.7706
# Token: picturesque, Query score: 0.8779, Document score: 0.7120
# Token: pleasantly, Query score: 0.8471, Document score: 0.4183
# Token: windy, Query score: 0.8079, Document score: 0.2140
# Token: favorable, Query score: 0.7537, Document score: 0.4925
# Token: rain, Query score: 0.7519, Document score: 2.1456
# Token: skies, Query score: 0.7277, Document score: 0.3818
# Token: lena, Query score: 0.6995, Document score: 0.8593
# Token: sunshine, Query score: 0.6895, Document score: 0.2410
# Token: johnny, Query score: 0.6621, Document score: 0.3016
# Token: skyline, Query score: 0.6604, Document score: 0.1933
# Token: sasha, Query score: 0.6117, Document score: 0.2197
# Token: vibe, Query score: 0.5962, Document score: 0.0414
# Token: hardly, Query score: 0.5381, Document score: 0.7560
# Token: prevailing, Query score: 0.4583, Document score: 0.4243
# Token: unpredictable, Query score: 0.4539, Document score: 0.5073
# Token: presently, Query score: 0.4350, Document score: 0.8463
# Token: hail, Query score: 0.3674, Document score