Fuzhao Xue

National University of Singapore

Email: f.xue at u.nus.edu

Twitter: @XueFz

Joint work with: Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, Yang You

Aug 21, 2023

GitHub:

GitHub - XueFuzhao/OpenMoE: A family of open-sourced Mixture-of-Experts (MoE) Large Language Models

Introduction

OpenMoE is a project aimed at Igniting the Open-Source MoE Community! Our mission is releasing a series of open-sourced MoE Language Models.

With a small team and a monumental project, we're taking a strategic route. We're initiating the journey by sharing intermediate checkpoints. This invites contributors to join us in shaping the open-source MoE landscape together.

Approach

Data

• 50% The RedPajama + 50% The Stack Dedup.
• A high ratio of coding data is used to improve reasoning ability.

Tokenizer

• umt5 Tokenizer, which can be downloaded on Huggingface or Google Cloud
• The umT5 tokenizer is used to support multi-lingual continue learning in the future.