Upload folder using huggingface_hub

1_Pooling/config.json

@@ -0,0 +1,9 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false
+}

README.md

@@ -1,3 +1,113 @@
 ---
+pipeline_tag: sentence-similarity
+language: fr
 license: apache-2.0
+datasets:
+- maastrichtlawtech/lleqa
+metrics:
+- recall
+tags:
+- feature-extraction
+- sentence-similarity
+library_name: sentence-transformers
 ---
+
+# distilcamembert-lleqa
+
+This is a [sentence-transformers](https://www.SBERT.net) model: it maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. The model was trained on the [LLeQA](https://huggingface.co/datasets/maastrichtlawtech/lleqa) dataset for legal information retrieval in **French**.
+
+## Usage
+***
+
+#### Sentence-Transformers
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["This is an example sentence", "Each sentence is converted"]
+
+model = SentenceTransformer('maastrichtlawtech/distilcamembert-lleqa')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+#### 🤗 Transformers
+
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+
+#Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+# Sentences we want sentence embeddings for
+sentences = ['This is an example sentence', 'Each sentence is converted']
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('maastrichtlawtech/distilcamembert-lleqa')
+model = AutoModel.from_pretrained('maastrichtlawtech/distilcamembert-lleqa')
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling. In this case, mean pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+print(sentence_embeddings)
+```
+
+## Evaluation
+***
+
+We evaluate the model on the test set of LLeQA, which consists of 195 legal questions with a knowlegde corpus of 27.9K candidate articles. We report the mean reciprocal rank (MRR), normalized discounted cumulative gainand (NDCG), mean average precision (MAP), and recall at various cut-offs (R@k).
+
+|   MRR@10 |   NDCG@10 |   MAP@10 |   R@10 |   R@100 |   R@500 |
+|---------:|----------:|---------:|-------:|--------:|--------:|
+|    36.67 |     37.24 |    29.26 |  52.95 |   78.07 |   90.17 |
+
+## Training
+***
+
+#### Background
+
+We utilize the [distilcamembert-base](https://huggingface.co/cmarkea/distilcamembert-base) model and fine-tuned it on 9.3K question-article pairs in French. We used a contrastive learning objective: given a short legal question, the model should predict which out of a set of sampled legal articles, was actually paired with it in the dataset. Formally, we compute the cosine similarity from each possible pairs from the batch. We then apply the cross entropy loss with a temperature of 0.05 by comparing with true pairs.
+
+#### Hyperparameters
+
+We trained the model on a single Tesla V100 GPU with 32GBs of memory during 20 epochs (i.e., 5.4k steps) using a batch size of 32. We used the AdamW optimizer with an initial learning rate of 2e-05, weight decay of 0.01, learning rate warmup over the first 50 steps, and linear decay of the learning rate. The sequence length was limited to 384 tokens.
+
+#### Data
+
+We use the [Long-form Legal Question Answering (LLeQA)](https://huggingface.co/datasets/maastrichtlawtech/lleqa) dataset to fine-tune the model. LLeQA is a French native dataset for studying legal information retrieval and question answering. It consists of a knowledge corpus of 27,941 statutory articles collected from the Belgian legislation, and 1,868 legal questions posed by Belgian citizens and labeled by experienced jurists with a comprehensive answer rooted in relevant articles from the corpus.
+
+## Citation
+
+```bibtex
+@article{louis2023interpretable,
+  author = {Louis, Antoine and van Dijck, Gijs and Spanakis, Gerasimos},
+  title = {Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models},
+  journal = {CoRR},
+  volume = {abs/2309.xxxxx},
+  year = {2023},
+  url = {https://doi.org/},
+  doi = {},
+  eprinttype = {arXiv},
+  eprint = {2309.xxxxx},
+}
+```

config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "antoinelouis/biencoder-distilcamembert-base-mmarcoFR",
+  "architectures": [
+    "CamembertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "camembert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.30.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 32005
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.2",
+    "transformers": "4.30.0.dev0",
+    "pytorch": "2.1.0.dev20230321+cu117"
+  }
+}

dev_scores.csv

@@ -0,0 +1,2 @@
+MRR@10,NDCG@10,MAP@10,Recall@10,Recall@100,Recall@500,model
+16.11,12.02,9.05,12.73,32.62,52.32,biencoder-antoinelouis-biencoder-distilcamembert-base-mmarcoFRFR-bsard

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:863bd24ca329d25b2082ce3505a7c0265a833818bb14390c1f04513c003397f0
+size 272417581

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 384,
+  "do_lower_case": false
+}

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:988bc5a00281c6d210a5d34bd143d0363741a432fefe741bf71e61b1869d4314
+size 810912

special_tokens_map.json

@@ -0,0 +1,19 @@
+{
+  "additional_special_tokens": [
+    "<s>NOTUSED",
+    "</s>NOTUSED"
+  ],
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "additional_special_tokens": [
+    "<s>NOTUSED",
+    "</s>NOTUSED"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "__type": "AddedToken",
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "model_max_length": 384,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "CamembertTokenizer",
+  "unk_token": "<unk>"
+}