Upload 7 files

README.md

@@ -1,3 +1,51 @@
----
-license: mit
----
+---
+language: en
+thumbnail: https://github.com/jackaduma
+tags:
+- exbert
+- security
+- cybersecurity
+- cyber security
+- threat hunting
+- threat intelligence
+license: apache-2.0
+datasets:
+- APTnotes
+- Stucco-Data
+- CASIE
+---
+
+
+# SecBERT
+
+This is the pretrained model presented in [SecBERT: A Pretrained Language Model for Cyber Security Text](https://github.com/jackaduma/SecBERT/), which is a BERT model trained on cyber security text.
+
+The training corpus was papers taken from 
+ * [APTnotes](https://github.com/kbandla/APTnotes)
+ * [Stucco-Data: Cyber security data sources](https://stucco.github.io/data/)
+ * [CASIE: Extracting Cybersecurity Event Information from Text](https://ebiquity.umbc.edu/_file_directory_/papers/943.pdf)
+ * [SemEval-2018 Task 8: Semantic Extraction from CybersecUrity REports using Natural Language Processing (SecureNLP)](https://competitions.codalab.org/competitions/17262). 
+
+SecBERT has its own wordpiece vocabulary (secvocab) that's built to best match the training corpus. 
+
+We trained [SecBERT](https://huggingface.co/jackaduma/SecBERT) and [SecRoBERTa](https://huggingface.co/jackaduma/SecRoBERTa) versions. 
+
+Available models include:
+* [`SecBERT`](https://huggingface.co/jackaduma/SecBERT)
+* [`SecRoBERTa`](https://huggingface.co/jackaduma/SecRoBERTa)
+
+---
+## **Fill Mask**
+
+We proposed to build language model which work on cyber security text, as result, it can improve downstream tasks (NER, Text Classification, Semantic Understand, Q&A) in Cyber Security Domain.
+
+First, as below shows Fill-Mask pipeline in [Google Bert](), [AllenAI SciBert](https://github.com/allenai/scibert) and our [SecBERT](https://github.com/jackaduma/SecBERT) .
+
+
+<!-- <img src="./fill-mask-result.png" width="150%" height="150%"> -->
+
+![fill-mask-result](https://github.com/jackaduma/SecBERT/blob/main/fill-mask-result.png?raw=true)
+
+---
+
+The original repo can be found [here](https://github.com/jackaduma/SecBERT).

config.json

@@ -0,0 +1,20 @@
+{
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 514,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 0,
+  "type_vocab_size": 1,
+  "vocab_size": 52000
+}

gitattributes

@@ -0,0 +1,28 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text 
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cbc89dbadf58a8b0c3cbf574126c6ac71b9f8bbc1ad1bc0260636bf1051df06e
+size 336396808

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08a1be6db1475e8cbcf7ffbc2f80a3b64b82cc62438260bf5cfdb1d385407a76
+size 336411364