Question
#1
by
streamerbtw1002
- opened
I have a question: How did you combine parts of other pre-trained models into MoE? I'm trying to find a way how.
using merge kit,
and it is just the first or weak MOE, further new and strong MOEs are coming.
shuvom
changed discussion status to
closed
@shuvom Thanks, I am gonna train a 3b AI model and make them into 4 experts.
streamerbtw1002
changed discussion status to
open
great!