SentenceTransformer based on jhgan/ko-sroberta-multitask
This is a sentence-transformers model finetuned from jhgan/ko-sroberta-multitask. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: jhgan/ko-sroberta-multitask
- Maximum Sequence Length: 128 tokens
- Output Dimensionality: 768 tokens
- Similarity Function: Cosine Similarity
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: RobertaModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
'남자가 노래를 부르는 동안 두 남자가 악기를 연주한다.',
'3분의 1이 노래하는 동안 두 남자가 악기를 연주한다.',
'세 번째 남자가 악기를 연주하는 동안 두 남자가 노래를 부른다.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
Evaluation
Metrics
Semantic Similarity
- Dataset:
sts-dev
- Evaluated with
EmbeddingSimilarityEvaluator
Metric | Value |
---|---|
pearson_cosine | 0.8668 |
spearman_cosine | 0.8703 |
pearson_manhattan | 0.862 |
spearman_manhattan | 0.8684 |
pearson_euclidean | 0.8623 |
spearman_euclidean | 0.8686 |
pearson_dot | 0.8474 |
spearman_dot | 0.8469 |
pearson_max | 0.8668 |
spearman_max | 0.8703 |
Training Details
Training Datasets
Unnamed Dataset
- Size: 568,640 training samples
- Columns:
sentence_0
,sentence_1
, andsentence_2
- Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1 sentence_2 type string string string details - min: 4 tokens
- mean: 19.21 tokens
- max: 128 tokens
- min: 3 tokens
- mean: 18.31 tokens
- max: 93 tokens
- min: 4 tokens
- mean: 14.57 tokens
- max: 54 tokens
- Samples:
sentence_0 sentence_1 sentence_2 발생 부하가 함께 5% 적습니다.
발생 부하의 5% 감소와 함께 11.
발생 부하가 5% 증가합니다.
어떤 행사를 위해 음식과 옷을 배급하는 여성들.
여성들은 음식과 옷을 나눠줌으로써 난민들을 돕고 있다.
여자들이 사막에서 오토바이를 운전하고 있다.
어린 아이들은 그 지식을 얻을 필요가 있다.
응, 우리 젊은이들 중 많은 사람들이 그걸 배워야 할 것 같아.
젊은 사람들은 배울 필요가 없다.
- Loss:
MultipleNegativesRankingLoss
with these parameters:{ "scale": 20.0, "similarity_fct": "cos_sim" }
Unnamed Dataset
- Size: 5,777 training samples
- Columns:
sentence_0
,sentence_1
, andlabel
- Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1 label type string string float details - min: 3 tokens
- mean: 17.16 tokens
- max: 66 tokens
- min: 3 tokens
- mean: 17.11 tokens
- max: 64 tokens
- min: 0.0
- mean: 0.54
- max: 1.0
- Samples:
sentence_0 sentence_1 label 시는 드램 시장이 2003년에 2.9% 성장하여 157억 달러, 2004년에는 43% 성장하여 225억 달러가 될 것으로 예상하고 있다고 말했습니다.
미국 시장은 2003년에 2.1퍼센트가 감소한 30.6억 달러로, 그리고 나서 2004년에 15.7퍼센트가 증가하여 354억 달러로 성장할 것이다.
0.24
오사마 빈 라덴 부인들 수감
인도에서 촬영될 오사마 빈 라덴 영화
0.16
파키스탄 전투기, '탈리반 은신처' 폭탄 터뜨리기
파키스탄은 시리아 측에 무기 공급을 중단하기를 원한다.
0.32
- Loss:
CosineSimilarityLoss
with these parameters:{ "loss_fct": "torch.nn.modules.loss.MSELoss" }
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy
: stepsnum_train_epochs
: 5batch_sampler
: no_duplicatesmulti_dataset_batch_sampler
: round_robin
All Hyperparameters
Click to expand
overwrite_output_dir
: Falsedo_predict
: Falseeval_strategy
: stepsprediction_loss_only
: Trueper_device_train_batch_size
: 8per_device_eval_batch_size
: 8per_gpu_train_batch_size
: Noneper_gpu_eval_batch_size
: Nonegradient_accumulation_steps
: 1eval_accumulation_steps
: Nonelearning_rate
: 5e-05weight_decay
: 0.0adam_beta1
: 0.9adam_beta2
: 0.999adam_epsilon
: 1e-08max_grad_norm
: 1num_train_epochs
: 5max_steps
: -1lr_scheduler_type
: linearlr_scheduler_kwargs
: {}warmup_ratio
: 0.0warmup_steps
: 0log_level
: passivelog_level_replica
: warninglog_on_each_node
: Truelogging_nan_inf_filter
: Truesave_safetensors
: Truesave_on_each_node
: Falsesave_only_model
: Falserestore_callback_states_from_checkpoint
: Falseno_cuda
: Falseuse_cpu
: Falseuse_mps_device
: Falseseed
: 42data_seed
: Nonejit_mode_eval
: Falseuse_ipex
: Falsebf16
: Falsefp16
: Falsefp16_opt_level
: O1half_precision_backend
: autobf16_full_eval
: Falsefp16_full_eval
: Falsetf32
: Nonelocal_rank
: 0ddp_backend
: Nonetpu_num_cores
: Nonetpu_metrics_debug
: Falsedebug
: []dataloader_drop_last
: Falsedataloader_num_workers
: 0dataloader_prefetch_factor
: Nonepast_index
: -1disable_tqdm
: Falseremove_unused_columns
: Truelabel_names
: Noneload_best_model_at_end
: Falseignore_data_skip
: Falsefsdp
: []fsdp_min_num_params
: 0fsdp_config
: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap
: Noneaccelerator_config
: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed
: Nonelabel_smoothing_factor
: 0.0optim
: adamw_torchoptim_args
: Noneadafactor
: Falsegroup_by_length
: Falselength_column_name
: lengthddp_find_unused_parameters
: Noneddp_bucket_cap_mb
: Noneddp_broadcast_buffers
: Falsedataloader_pin_memory
: Truedataloader_persistent_workers
: Falseskip_memory_metrics
: Trueuse_legacy_prediction_loop
: Falsepush_to_hub
: Falseresume_from_checkpoint
: Nonehub_model_id
: Nonehub_strategy
: every_savehub_private_repo
: Falsehub_always_push
: Falsegradient_checkpointing
: Falsegradient_checkpointing_kwargs
: Noneinclude_inputs_for_metrics
: Falseeval_do_concat_batches
: Truefp16_backend
: autopush_to_hub_model_id
: Nonepush_to_hub_organization
: Nonemp_parameters
:auto_find_batch_size
: Falsefull_determinism
: Falsetorchdynamo
: Noneray_scope
: lastddp_timeout
: 1800torch_compile
: Falsetorch_compile_backend
: Nonetorch_compile_mode
: Nonedispatch_batches
: Nonesplit_batches
: Noneinclude_tokens_per_second
: Falseinclude_num_input_tokens_seen
: Falseneftune_noise_alpha
: Noneoptim_target_modules
: Nonebatch_eval_metrics
: Falsebatch_sampler
: no_duplicatesmulti_dataset_batch_sampler
: round_robin
Training Logs
Epoch | Step | Training Loss | sts-dev_spearman_max |
---|---|---|---|
0.3458 | 500 | 0.1504 | - |
0.6916 | 1000 | 0.1662 | 0.8660 |
1.0007 | 1447 | - | 0.8678 |
1.0367 | 1500 | 0.1575 | - |
1.3824 | 2000 | 0.0539 | 0.8590 |
1.7282 | 2500 | 0.0406 | - |
2.0007 | 2894 | - | 0.8703 |
Framework Versions
- Python: 3.11.9
- Sentence Transformers: 3.0.1
- Transformers: 4.41.2
- PyTorch: 2.2.2+cu121
- Accelerate: 0.31.0
- Datasets: 2.20.0
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
MultipleNegativesRankingLoss
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
- Downloads last month
- 2
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for KYUNGHYUN9/test_model
Base model
jhgan/ko-sroberta-multitask
Finetuned
this model
Evaluation results
- Pearson Cosine on sts devself-reported0.867
- Spearman Cosine on sts devself-reported0.870
- Pearson Manhattan on sts devself-reported0.862
- Spearman Manhattan on sts devself-reported0.868
- Pearson Euclidean on sts devself-reported0.862
- Spearman Euclidean on sts devself-reported0.869
- Pearson Dot on sts devself-reported0.847
- Spearman Dot on sts devself-reported0.847
- Pearson Max on sts devself-reported0.867
- Spearman Max on sts devself-reported0.870