metadata

license: cc-by-nc-nd-4.0
language:
  - zh
datasets:
  - MIRACL
tags:
  - miniMiracle
  - passage-retrieval
  - knowledge-distillation
  - middle-training
pretty_name: >-
  miniMiracle is a family of High-quality, Light Weight and Easy deploy
  multilingual embedders / retrievers, primarily focussed on Indo-Aryan and
  Indo-Dravidin Languages.
library_name: transformers
pipeline_tag: sentence-similarity

Table 1: Chinese retrieval performance on the MIRACL dev set (measured by nDCG@10)

License and Terms:
Detailed comparison & Our Contribution:
ONNX & GGUF Variants:
Usage:
Notes on Reproducing:
Reference:
Note on model bias

License and Terms:

Detailed comparison & Our Contribution:

English language famously have all-minilm series models which were great for quick experimentations and for certain production workloads. The Idea is to have same for the other popular langauges, starting with Indo-Aryan and Indo-Dravidian languages. Our innovation is in bringing high quality models which easy to serve and embeddings are cheaper to store without ANY pretraining or expensive finetuning. For instance, all-minilm are finetuned on 1-Billion pairs. We offer a very lean model but with a huge vocabulary - around 250K. We will add more details here.

Table 2: Detailed Chinese retrieval performance on the MIRACL dev set (measured by nDCG@10)

Full set of evaluation numbers for our model

{'NDCG@1': 0.43511, 'NDCG@3': 0.42434, 'NDCG@5': 0.45298, 'NDCG@10': 0.50914, 'NDCG@100': 0.5815, 'NDCG@1000': 0.59392}
{'MAP@1': 0.21342, 'MAP@3': 0.32967, 'MAP@5': 0.36798, 'MAP@10': 0.39908, 'MAP@100': 0.42592, 'MAP@1000': 0.42686}
{'Recall@10': 0.63258, 'Recall@50': 0.85, 'Recall@100': 0.91595, 'Recall@200': 0.942, 'Recall@500': 0.96924, 'Recall@1000': 0.9857}
{'P@1': 0.43511, 'P@3': 0.29177, 'P@5': 0.22545, 'P@10': 0.14758, 'P@100': 0.02252, 'P@1000': 0.00249}
{'MRR@10': 0.55448, 'MRR@100': 0.56288, 'MRR@1000': 0.56294}

Usage:

With Sentence Transformers:

from sentence_transformers import SentenceTransformer
import scipy.spatial


model = SentenceTransformer('prithivida/miniMiracle_zh_v1')

corpus = [
    '一个男人正在吃东西',
    '人们正在吃一块面包',
    '女孩抱着婴儿',
    '一个男人正在骑马',
    '一个女人正在弹吉他',
    '两个人推着马车穿过树林',
    '一个人骑着一匹白马在一个封闭的田野里',
    '一只猴子在打鼓',
    '一只猎豹正在猎物后面奔跑',
    '他们享受了一顿美味的盛宴'
]

queries = [
    '一个人在吃意大利面',
    '一个穿着大猩猩服装的人在打鼓'
]


corpus_embeddings = model.encode(corpus)
query_embeddings = model.encode(queries)

# Find the closest 3 sentences of the corpus for each query sentence based on cosine similarity
closest_n = 3
for query, query_embedding in zip(queries, query_embeddings):
    distances = scipy.spatial.distance.cdist([query_embedding], corpus_embeddings, "cosine")[0]

    results = zip(range(len(distances)), distances)
    results = sorted(results, key=lambda x: x[1])

    print("\n======================\n")
    print("Query:", query)
    print("\nTop 3 most similar sentences in corpus:\n")

    for idx, distance in results[0:closest_n]:
        print(corpus[idx].strip(), "(Score: %.4f)" % (1-distance))

# Optional: How to quantize the embeddings
# binary_embeddings = quantize_embeddings(embeddings, precision="ubinary")

With Huggingface Transformers:

T.B.A

How do I optimise vector index cost ?

Use Binary and Scalar Quantisation

How do I offer hybrid search to address Vocabulary Mismatch Problem?

MIRACL paper shows simply combining BM25 is a good starting point for a Hybrid option: The below numbers are with mDPR model, but miniMiracle_zh_v1 should give a even better hybrid performance.

Language	ISO	nDCG@10 BM25	nDCG@10 mDPR	nDCG@10 Hybrid
Chinese	zh	0.175	0.512	0.526

Note: MIRACL paper shows a different (higher) value for BM25 Chinese, So we are taking that value from BGE-M3 paper, rest all are form the MIRACL paper.

Notes on reproducing:

We welcome anyone to reproduce our results. Here are some tips and observations:

Use CLS Pooling and Inner Product.
There may be minor differences in the numbers when reproducing, for instance BGE-M3 reports a nDCG@10 of 59.3 for MIRACL hindi and we Observed only 58.9.

Here are our numbers for the full hindi run on BGE-M3

{'NDCG@1': 0.49714, 'NDCG@3': 0.5115, 'NDCG@5': 0.53908, 'NDCG@10': 0.58936, 'NDCG@100': 0.6457, 'NDCG@1000': 0.65336}
{'MAP@1': 0.28845, 'MAP@3': 0.42424, 'MAP@5': 0.46455, 'MAP@10': 0.49955, 'MAP@100': 0.51886, 'MAP@1000': 0.51933}
{'Recall@10': 0.73032, 'Recall@50': 0.8987, 'Recall@100': 0.93974, 'Recall@200': 0.95763, 'Recall@500': 0.97813, 'Recall@1000': 0.9902}
{'P@1': 0.49714, 'P@3': 0.33048, 'P@5': 0.24629, 'P@10': 0.15543, 'P@100': 0.0202, 'P@1000': 0.00212}
{'MRR@10': 0.60893, 'MRR@100': 0.615, 'MRR@1000': 0.6151}

Fair warning BGE-M3 is $ expensive to evaluate, probably that's why it's not part of any of the MTEB benchmarks.

Reference:

Note on model bias:

Like any model this model might carry inherent biases from the base models and the datasets it was pretrained and finetuned on. Please use responsibly.

prithivida
/

miniDense_chinese_v1

Table Of Contents