Text Generation
Transformers
Safetensors
llama
conversational
text-generation-inference
Inference Endpoints

What is the system compatibility to run ise-uiuc/Magicoder-S-DS-6.7B model in our local machine?

#7
by aravindasagarvg - opened

I would like to run ise-uiuc/Magicoder-S-DS-6.7B Model in local machine. Do anyone have know the system compatibility to run the model locally?

Intellligent Software Engineering (iSE) org

During inference, the raw model will roughly occupy <= 20GB GPU memory with bfloat16. Quantization can further relax these constraints. I recommend checking https://huggingface.co/TheBloke/Magicoder-S-DS-6.7B-GGUF. In general, this can run on any system with enough memory.

Sign up or log in to comment