Upload TFDistilBertForQuestionAnswering

README.md

@@ -0,0 +1,72 @@
+---
+license: apache-2.0
+base_model: distilbert-base-multilingual-cased
+tags:
+- generated_from_keras_callback
+model-index:
+- name: transformers-qa-kaggle-tpu
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information Keras had access to. You should
+probably proofread and complete it, then remove this comment. -->
+
+# transformers-qa-kaggle-tpu
+
+This model is a fine-tuned version of [distilbert-base-multilingual-cased](https://huggingface.co/distilbert-base-multilingual-cased) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Train Loss: 0.2138
+- Train End Logits Accuracy: 0.9272
+- Train Start Logits Accuracy: 0.9265
+- Validation Loss: 4.0782
+- Validation End Logits Accuracy: 0.4815
+- Validation Start Logits Accuracy: 0.4488
+- Epoch: 14
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- optimizer: {'name': 'Adam', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': False, 'is_legacy_optimizer': False, 'learning_rate': {'class_name': 'PolynomialDecay', 'config': {'initial_learning_rate': 2e-05, 'decay_steps': 122160, 'end_learning_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}}, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
+- training_precision: float32
+
+### Training results
+
+| Train Loss | Train End Logits Accuracy | Train Start Logits Accuracy | Validation Loss | Validation End Logits Accuracy | Validation Start Logits Accuracy | Epoch |
+|:----------:|:-------------------------:|:---------------------------:|:---------------:|:------------------------------:|:--------------------------------:|:-----:|
+| 2.2397     | 0.4602                    | 0.4288                      | 2.0601          | 0.4944                         | 0.4670                           | 0     |
+| 1.7080     | 0.5695                    | 0.5404                      | 2.0597          | 0.5098                         | 0.4782                           | 1     |
+| 1.4423     | 0.6244                    | 0.5975                      | 2.0398          | 0.5124                         | 0.4779                           | 2     |
+| 1.2221     | 0.6718                    | 0.6455                      | 2.1899          | 0.5121                         | 0.4832                           | 3     |
+| 1.0321     | 0.7138                    | 0.6920                      | 2.2860          | 0.5067                         | 0.4742                           | 4     |
+| 0.8740     | 0.7502                    | 0.7307                      | 2.4042          | 0.4981                         | 0.4677                           | 5     |
+| 0.7420     | 0.7809                    | 0.7666                      | 2.6543          | 0.4954                         | 0.4595                           | 6     |
+| 0.6254     | 0.8107                    | 0.7988                      | 2.7405          | 0.4938                         | 0.4607                           | 7     |
+| 0.5325     | 0.8337                    | 0.8255                      | 3.0218          | 0.4911                         | 0.4613                           | 8     |
+| 0.4537     | 0.8550                    | 0.8491                      | 3.1804          | 0.4917                         | 0.4550                           | 9     |
+| 0.3860     | 0.8751                    | 0.8707                      | 3.4298          | 0.4880                         | 0.4524                           | 10    |
+| 0.3284     | 0.8914                    | 0.8890                      | 3.5952          | 0.4826                         | 0.4480                           | 11    |
+| 0.2844     | 0.9053                    | 0.9033                      | 3.6105          | 0.4856                         | 0.4510                           | 12    |
+| 0.2462     | 0.9176                    | 0.9151                      | 3.8751          | 0.4785                         | 0.4504                           | 13    |
+| 0.2138     | 0.9272                    | 0.9265                      | 4.0782          | 0.4815                         | 0.4488                           | 14    |
+
+
+### Framework versions
+
+- Transformers 4.31.0.dev0
+- TensorFlow 2.12.0
+- Datasets 2.13.1
+- Tokenizers 0.13.3

config.json

@@ -0,0 +1,24 @@
+{
+  "_name_or_path": "distilbert-base-multilingual-cased",
+  "activation": "gelu",
+  "architectures": [
+    "DistilBertForQuestionAnswering"
+  ],
+  "attention_dropout": 0.1,
+  "dim": 768,
+  "dropout": 0.1,
+  "hidden_dim": 3072,
+  "initializer_range": 0.02,
+  "max_position_embeddings": 512,
+  "model_type": "distilbert",
+  "n_heads": 12,
+  "n_layers": 6,
+  "output_past": true,
+  "pad_token_id": 0,
+  "qa_dropout": 0.1,
+  "seq_classif_dropout": 0.2,
+  "sinusoidal_pos_embds": false,
+  "tie_weights_": true,
+  "transformers_version": "4.31.0.dev0",
+  "vocab_size": 119547
+}

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d827e5849b70dfbafd600c27dfa28e32ba28b189f450df6999657812ae07fed
+size 539068392