ByteDance discusses training a model with over 5 trillion parameters; Zhang Yiming says no distillation, don't be swayed by short-term trends like coding

ByteDance is reportedly in discussions about training a state-of-the-art AI model that would have more than 5 trillion parameters, according to internal sources. The company's founder, Zhang Yiming, has stated that they will not use distillation, a technique where a smaller model learns from a larger one, and advised against being distracted by short-term trends in the field, such as the current focus on coding. This move signals ByteDance's ambition to compete at the forefront of AI development, even as it faces competition from other tech giants in China. The news was reported by Sina Finance, citing conversations within the company.