File size: 3,690 Bytes
5160660 698fef2 c0453e5 5160660 698fef2 5160660 c3aa0fd 5160660 c3aa0fd 5160660 c3aa0fd 5160660 c0453e5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 |
---
library_name: transformers
datasets:
- Svngoku/xP3x-Kongo
language:
- kg
metrics:
- bleu
pipeline_tag: text-generation
tags:
- africa
- languages
---
# Kongo Llama Experiment
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
- `Tokenizer`
```py
from transformers import PreTrainedTokenizerFast
# Assuming your custom tokenizer is `tokenizer`
wrapped_tokenizer = PreTrainedTokenizerFast(
tokenizer_object=tokenizer,
bos_token="[BOS]", # Replace with your special tokens
eos_token="[EOS]", # Replace with your special tokens
unk_token="[UNK]",
pad_token="[PAD]"
)
# Ensure padding is applied to the right side (used in causal language modeling)
wrapped_tokenizer.padding_side = "right"
```
- `Model`
```py
from transformers import LlamaConfig, LlamaForCausalLM
config = LlamaConfig(
vocab_size=len(wrapped_tokenizer), # Get vocab size from the wrapped tokenizer
hidden_size=512, # Adjust model size as needed
intermediate_size=1024,
num_hidden_layers=8, # Set number of layers and heads
num_attention_heads=8,
max_position_embeddings=512,
rms_norm_eps=1e-6,
initializer_range=0.02,
use_cache=True,
pad_token_id=wrapped_tokenizer.pad_token_id,
bos_token_id=wrapped_tokenizer.bos_token_id,
eos_token_id=wrapped_tokenizer.eos_token_id,
)
model = LlamaForCausalLM(config)
```
- `Trainer`
```py
from transformers import TrainingArguments, Trainer
# Define training arguments
training_args = TrainingArguments(
output_dir="kongo-llama", # Output directory for model and checkpoints
num_train_epochs=1,
per_device_train_batch_size=8,
learning_rate=5e-5,
warmup_steps=500,
weight_decay=0.01,
logging_dir="./logs",
logging_steps=10,
save_steps=1000,
)
trainer = Trainer(
model=model, # Your model instance
args=training_args, # Training arguments
train_dataset=dataset, # Tokenized dataset with input_ids and labels
tokenizer=wrapped_tokenizer, # Wrapped tokenizer
data_collator=data_collator, # Data collator for causal language modeling
)
````
### Model Description
<!-- Provide a longer summary of what this model is. -->
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
```py
# Load the model
model = LlamaForCausalLM.from_pretrained('/content/kongo-llama/checkpoint-9000')
# Prepare input text
text = "Nzambi "
inputs = wrapped_tokenizer(text, return_tensors="pt")
# Generate text
generated_ids = model.generate(
max_length=150, # Increased length
num_beams=5, # Use beam search
temperature=0.7, # Adjust temperature for creativity
do_sample=True,
top_k=50, # Limit vocabulary for next token
top_p=0.95 # Nucleus sampling
)
# Decode and print the generated text
generated_text = wrapped_tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_text)
```
|