Update README.md

README.md

@@ -1,53 +1,96 @@
 ---
-license: mit
+language:
+- en
+license: apache-2.0
+library_name: transformers
 base_model: gpt2
 tags:
+- law
+- legal
+- australia
 - generated_from_trainer
-model-index:
-- name: open-australian-legal-gpt2
-  results: []
+datasets:
+- umarbutler/open-australian-legal-corpus
+widget:
+- text: "Section 51 of the Constitution provides "
+  example_title: "Text completion"
+- text: "# Question\nWhat is a restraint of trade?\n\nAnswer: "
+  example_title: "Question answering"
+inference:
+  parameters:
+    temperature: 0
+    seed: 42
+    best_of: 4
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# Open Australian Legal GPT2 ‍⚖️
+Open Australian Legal GPT2 is the first open source language model trained on Australian law.
 
-# open-australian-legal-gpt2
+Naturally, as a finetune of [GPT2](https://huggingface.co/gpt2), the model may be used for any of the tasks for which [GPT2](https://huggingface.co/gpt2) is suitable, including text generation, text completion and question answering.
 
-This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on an unknown dataset.
+Trained on 37,560 laws and regulations, comprising 635,482,112 tokens, taken from the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus), the model is intended specifically to be finetuned for downstream natural language processing tasks applied to the Australian legal domain.
 
-## Model description
+To ensure its accessibility to as wide an audience as possible, the model is issued under the [Apache Licence 2.0](https://www.apache.org/licenses/LICENSE-2.0.html).
 
-More information needed
+Those interested in learning more about the model are encouraged to read Umar Butler's accompanying article, [How I built the first open LLM for Australian law](https://umarbutler.com/how-i-built-the-first-open-llm-for-australian-law/).
 
-## Intended uses & limitations
+## Usage 👩‍💻
+The code snippet below demonstrates just one of the many ways in which the model may be accessed:
+```python
+>>> from transformers import pipeline, set_seed
 
-More information needed
+>>> set_seed(42) # We set a seed for reproducibility.
+>>> generator = pipeline('text-generation', model='umarbutler/open-australian-legal-gpt2')
+>>> generator('Under the Crimes Act 1914')
+[{'generated_text': 'Under the Crimes Act 1914, a person who is liable to a payment of a benefit under the Act is also liable to pay'}]
+```
 
-## Training and evaluation data
+## Creation 🧪
+37,560 documents were sampled from the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus) by filtering for primary and secondary legislation that, when stripped of whitespace, was not empty. Such documents were then randomly shuffled and added to blocks 1,024-tokens-long, with GPT2's end-of-sequence token ('<|endoftext|>') being used as a delimiter as well as to pad the end of the final block, resulting in a training dataset of 620,588 blocks, or 635,482,112 tokens.
 
-More information needed
+The training dataset was subsequently fed to [GPT2](https://huggingface.co/gpt2) via [`transformers.Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) with the following hyperparameters:
+| Hyperparameter | Value |
+| --- | --- |
+| Sequence length | 1,024 |
+| Epochs | 3 |
+| Optimiser | AdamW |
+| Learning rate | 1e-5 |
+| Learning rate scheduler | Linear with warmup |
+| Batch size per device | 4 |
+| Weight decay | 0.01 |
+| Warmup ratio | 0.06 |
+| Gradient accumulation steps | 1 |
 
-## Training procedure
+After training for 3 epochs, or 465,441 steps, over a period of ~25 hours on two GeForce RTX 4090s, the model achieved a loss of XX.
 
-### Training hyperparameters
+## Limitations 🚧
+Although the model has not been tested for bias, one would expect it to exhibit much of the same, if not all, the biases of [GPT2](https://huggingface.co/gpt2).
 
-The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 8
-- eval_batch_size: 16
-- seed: 42
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_ratio: 0.06
-- num_epochs: 3
+One might also expect the model to exhibit a bias towards the type of language employed in legislation and regulations (its source materials) as well as towards Commonwealth law (the largest source of legislation in [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus) at the time of the model's creation).
 
-### Training results
+Finally, it is worth noting that the model may lack knowledge of Victorian, Northern Territory and Australian Capital Territory law as licensing restrictions had prevented their inclusion in the training data.
 
+## Licence 📜
+The model is issued under the [Apache Licence 2.0](https://www.apache.org/licenses/LICENSE-2.0.html).
 
+## Citation 🔖
+If you've relied on the model for your work, please cite:
+```bibtex
+@misc{butler-2023-open-australian-legal-gpt2,
+    author = {Butler, Umar},
+    year = {2023},
+    title = {Open Australian Legal GPT2},
+    publisher = {Hugging Face},
+    version = {1.0.0},
+    url = {https://huggingface.co/datasets/umarbutler/open-australian-legal-gpt2}
+}
+```
 
-### Framework versions
+## Acknowledgements 🙏
+In the spirit of reconciliation, the author acknowledges the Traditional Custodians of Country throughout Australia and their connections to land, sea and community. He pays his respect to their Elders past and present and extends that respect to all Aboriginal and Torres Strait Islander peoples today.
 
-- Transformers 4.35.2
-- Pytorch 2.0.1
-- Datasets 2.15.0
-- Tokenizers 0.15.0
+The author thanks the sources of the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus) for making their data available under open licences.
+
+The author also acknowledges the developers of the many Python libraries relied upon in the training of the model, as well as the makers of [GPT2](https://huggingface.co/gpt2), which the model was built atop.
+
+Finally, the author is eternally grateful for the endless support of his wife and her willingness to put up with many a late night spent writing code and quashing bugs.