Update README.md

README.md

@@ -1,5 +1,5 @@
 ---
-license: mit
+license: apache-2.0
 language:
 - vi
 pipeline_tag: token-classification
@@ -27,9 +27,9 @@ For more details on the training process, please refer to this
 
 ## How to use this model
 There are just a few steps: 
-- Step 1: Load the model as a token classification model (*AutoModelForTokenClassification*).
+- Step 1: Load the model as a token classification model (`AutoModelForTokenClassification`).
 - Step 2: Run the input through the model to obtain the tag index for each input token.
-- Step 3: Use the tags' index to retreive the actual tags in the file *selected_tags_names.txt*. Then,
+- Step 3: Use the tags' index to retreive the actual tags in the file `selected_tags_names.txt`. Then,
   apply the conversion indicated by the tag to each token to obtain accented tokens.
 
 ### Step 1: Load model
@@ -95,7 +95,8 @@ tokens, predictions = insert_accents(text, model, tokenizer)
 
 ### Step3: Obtain the accented words 
 
-3.1 Download the tags set file from this repo. Then load it 
+3.1 Download the tags set file (`selected_tags_names.txt`) from this repo. 
+Suppose that it's put int the current dir, we can then load it: 
 ```python
 def _load_tags_set(fpath):
     labels = []
@@ -107,7 +108,7 @@ def _load_tags_set(fpath):
 
     return labels
     
-label_list = _load_tags_set("/content/training_data/vnaccent/corpus-title.train.selected_tags_names.txt")
+label_list = _load_tags_set("./selected_tags_names.txt")
 assert len(label_list) == 528, f"Expect {len(label_list)} tags"
 ```
 
@@ -116,12 +117,101 @@ assert len(label_list) == 528, f"Expect {len(label_list)} tags"
 print(tokens)
 print(list(f"{pred} ({label_list[pred]})" for pred in predictions))
 ```
+
 Obtained 
 ```python
 ['▁Nhi', 'n', '▁nhu', 'ng', '▁mua', '▁thu', '▁di', ',', '▁em', '▁nghe', '▁sau', '▁len', '▁trong', '▁nang', '.']
 ['217 (i-ì)', '217 (i-ì)', '388 (u-ữ)', '388 (u-ữ)', '407 (ua-ùa)', '378 (u-u)', '120 (di-đi)', '0 (-)', '185 (e-e)', '185 (e-e)', '41 (au-âu)', '188 (e-ê)', '302 (o-o)', '14 (a-ắ)', '0 (-)']
 ```
 
+We can see here that our original words have been further split into smaller tokens by the model. But we know the first token of each word 
+starts with the special char "▁".  
+
+Here, we'd need to merge these tokens (and similarly, the corresponding tags) into our original Vietnamese words. 
+Then, for each word, we'd apply the first tag (if it's associated with more than 1 tags) that change the word.  
+
+This can be done as follows: 
+
+```python
+TOKENIZER_WORD_PREFIX = "▁"
+def merge_tokens_and_preds(tokens, predictions): 
+    merged_tokens_preds = []
+    i = 0
+    while i < len(tokens):
+        tok = tokens[i]
+        label_indexes = set([predictions[i]])
+        if tok.startswith(TOKENIZER_WORD_PREFIX): # start a new word
+            tok_no_prefix = tok[len(TOKENIZER_WORD_PREFIX):]
+            cur_word_toks = [tok_no_prefix]
+            # check if subsequent toks are part of this word
+            j = i + 1
+            while j < len(tokens):
+                if not tokens[j].startswith(TOKENIZER_WORD_PREFIX):
+                    cur_word_toks.append(tokens[j])
+                    label_indexes.add(predictions[j])
+                    j += 1
+                else:
+                    break
+            cur_word = ''.join(cur_word_toks)
+            merged_tokens_preds.append((cur_word, label_indexes))
+            i = j
+        else:
+            merged_tokens_preds.append((tok, label_indexes))
+            i += 1
+
+    return merged_tokens_preds
+
+
+merged_tokens_preds = merge_tokens_and_preds(tokens, predictions)
+print(merged_tokens_preds)
+```
+
+Obtained: 
+```python
+[('Nhin', {217}), ('nhung', {388}), ('mua', {407}), ('thu', {378}), ('di,', {120, 0}), ('em', {185}), ('nghe', {185}), ('sau', {41}), ('len', {188}), ('trong', {302}), ('nang.', {0, 14})]
+```
+
+Then our final part: 
+
+```python
+def get_accented_words(merged_tokens_preds, label_list):
+    accented_words = []
+    for word_raw, label_indexes in merged_tokens_preds:
+        # use the first label that changes word_raw
+        for label_index in label_indexes:
+            tag_name = label_list[int(label_index)]
+            raw, vowel = tag_name.split("-")
+            if raw and raw in word_raw:
+                word_accented = word_raw.replace(raw, vowel)
+                break
+        else:
+            word_accented = word_raw
+
+        accented_words.append(word_accented)
+
+    return accented_words
+
+
+accented_words = get_accented_words(merged_tokens_preds, label_list)
+print(accented_words)
+```
+
+Obtained: 
+```python
+['Nhìn', 'những', 'mùa', 'thu', 'đi,', 'em', 'nghe', 'sâu', 'lên', 'trong', 'nắng.']
+```
+
+In this example, the model made 1 mistake with the word "sầu" (but predicted "sâu"). 
+
+
 
 ## Limitations 
-- This model will accept a maximum of 512 tokens, which is a limitation inherited from the base pretrained XLM-Roberta model.
+- This model will accept a maximum of 512 tokens, which is a limitation inherited from the base pretrained XLM-Roberta model.
+- It has a higher accuracy (97%) than an HMM version (91%).
+More info can be found <a href="https://peterhung.org/tech/insert-vietnamese-accent-transformer-model/#vs-hmm" target="_blank">here</a>.
+
+
+## Live Demo
+There is a live demo of this model available <a href="https://ai.vietnameseaccent.com/" target="_blank">here</a>. 
+
+This demo is run on CPU so the speed is quite slow.