Model
Cross-Encoder for sentence-similarity
This model was trained using sentence-transformers Cross-Encoder class.
Training Data
This model was trained on the STS benchmark dataset. The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.
Usage (Sentence-Transformers)
Using this model becomes easy when you have sentence-transformers installed:
pip install -U sentence-transformers
Then you can use the model like this:
from sentence_transformers import CrossEncoder
model = CrossEncoder('dangvantuan/CrossEncoder-camembert-large', max_length=128)
scores = model.predict([('Un avion est en train de décoller.', "Un homme joue d'une grande flûte."), ("Un homme étale du fromage râpé sur une pizza.", "Une personne jette un chat au plafond") ])
Evaluation
The model can be evaluated as follows on the French test data of stsb.
from sentence_transformers.readers import InputExample
from sentence_transformers.cross_encoder.evaluation import CECorrelationEvaluator
from datasets import load_dataset
def convert_dataset(dataset):
dataset_samples=[]
for df in dataset:
score = float(df['similarity_score'])/5.0 # Normalize score to range 0 ... 1
inp_example = InputExample(texts=[df['sentence1'],
df['sentence2']], label=score)
dataset_samples.append(inp_example)
return dataset_samples
# Loading the dataset for evaluation
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev")
df_test = load_dataset("stsb_multi_mt", name="fr", split="test")
# Convert the dataset for evaluation
# For Dev set:
dev_samples = convert_dataset(df_dev)
val_evaluator = CECorrelationEvaluator.from_input_examples(dev_samples, name='sts-dev')
val_evaluator(model, output_path="./")
# For Test set
test_samples = convert_dataset(df_test)
test_evaluator = CECorrelationEvaluator.from_input_examples(test_samples, name='sts-test')
test_evaluator(models, output_path="./")
Test Result: The performance is measured using Pearson and Spearman correlation:
On dev
Model Pearson correlation Spearman correlation #params dangvantuan/CrossEncoder-camembert-large 90.11 90.01 336M On test
Model Pearson correlation Spearman correlation dangvantuan/CrossEncoder-camembert-large 88.16 87.57
- Downloads last month
- 468
Inference API (serverless) does not yet support transformers models for this pipeline type.
Dataset used to train dangvantuan/CrossEncoder-camembert-large
Evaluation results
- Test Pearson correlation coefficient on Text Similarity frself-reportedxx.xx