vicuna / docs /System-requirements.md
albertsoma's picture
Upload 123 files
16c358c

These are the VRAM and RAM requirements (in MiB) to run some examples of models in 16-bit (default) precision:

model VRAM (GPU) RAM
arxiv_ai_gpt2 1512.37 5824.2
blenderbot-1B-distill 2441.75 4425.91
opt-1.3b 2509.61 4427.79
gpt-neo-1.3b 2605.27 5851.58
opt-2.7b 5058.05 4863.95
gpt4chan_model_float16 11653.7 4437.71
gpt-j-6B 11653.7 5633.79
galactica-6.7b 12697.9 4429.89
opt-6.7b 12700 4368.66
bloomz-7b1-p3 13483.1 4470.34

GPU mode with 8-bit precision

Allows you to load models that would not normally fit into your GPU. Enabled by default for 13b and 20b models in this web UI.

model VRAM (GPU) RAM
opt-13b 12528.1 1152.39
gpt-neox-20b 20384 2291.7

CPU mode (32-bit precision)

A lot slower, but does not require a GPU.

On my i5-12400F, 6B models take around 10-20 seconds to respond in chat mode, and around 5 minutes to generate a 200 tokens completion.

model RAM
arxiv_ai_gpt2 4430.82
gpt-neo-1.3b 6089.31
opt-1.3b 8411.12
blenderbot-1B-distill 8508.16
opt-2.7b 14969.3
bloomz-7b1-p3 21371.2
gpt-j-6B 24200.3
gpt4chan_model 24246.3
galactica-6.7b 26561.4
opt-6.7b 29596.6