Edit model card

Cat Embeddings

A set of embedding model trained for study embedding quality vs model architecture (width/depth) given a size constraint (12M params).

  • cat-emb-2-128: 2 layers/hidden size 128/4.4m
  • cat-emb-4-128: 4 layers/H 128/4.8m
  • cat-emb-8-128: 8 layers/H 128/5.6m
  • cat-emb-12-128: 12 layers/H 128/6.4m
  • cat-emb-2-256: 2 layers/H 256/9.7m
  • cat-emb-4-256: 4 layers/H 256/11.3m

Training

  • stage 1: seq 192, batch size 2048, 50k steps, sentence pairs.
  • stage 2: seq 512, batch size 64, 5k steps, sentence triplets.

Perf

Downloads last month
5
Inference API
Unable to determine this model’s pipeline type. Check the docs .