maywell

/

PiVoT-MoE

like 8

Hi, Thank you for your interest for my work!
Unfortunately, Solar 10.7B has not been used in this model.
but i am going to use Solar 10.7B in my next work or so.
Since Solar is now on llama architecture it was hard to merge with other mistral based models.

안녕하세요. 작업에 관심 가져주셔서 감사합니다.
안타깝게도, 업스테이지 솔라 모델은 이번 모델에 사용되지 않았습니다 ㅠㅠ
물론 다음 모델에 훌륭한 성능을 지닌 솔라 모델을 사용할 계획은 있습니다.
솔라 모델이 현재 라마 아키텍쳐로만 존재하기에 다른 미스트랄 기반 모델들과 결합하는데에 어려움이 있었습니다.

모델에 관심 가져주셔서 감사합니다 😊

Oh I see, congratulations on your great work!
May I ask what weights were used when initializing the 10.7B experts, as I was unaware of other pretrained/fine-tuned models with 10.7B parameters.

Based on my PiVoT-10.7B-Mistral-v0.2-RP, other 10.7B models with RP finetune on huggingface used.

but i am going to use Solar 10.7B in my next work or so.

@maywell We would love to see the Solar model along with your MOE. Thank you in advance.