dmrau/bow-bert

This model was trained with the Rankers Library please check https://github.com/davidmrau/rankers#Evaluation to learn how to use the model.

This Cross Encoder Ranking Model is invariant to the input order by removing the position embeddings (by being set to zero) during fine-tuning. It was trained on the official MS MARCO training triples using the following training parameters:

• batch size: 128

• learning rate 3e-6

• Optimizer: Adam

Example on how to load and use BOW-BERT:

from transformers import AutoModelForSequenceClassification, AutoTokenizer

# load model
model = AutoModelForSequenceClassification.from_pretrained('dmrau/bow-bert')
# load tokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# tokenize query and passage and concatenate them
inp = tokenizer(['this is a query','query a is this'], ['this is a passage', 'passage a is this'], return_tensors='pt')
# get estimated score
print('score', model(**inp).logits[:, 1])

### outputs identical scores for different 
### word orders as the model is order invariant:
# scores: [-2.9463, -2.9463]

Cite us:

@article{rau2022role,
  title={The Role of Complex NLP in Transformers for Text Ranking?},
  author={Rau, David and Kamps, Jaap},
  journal={arXiv preprint arXiv:2207.02522},
  year={2022}
}