National University of Singapore, Joining Google DeepMind
Email: xuefuzhao at outlook.com
Twitter: @XueFz
Nov 16, 2024
It’s time to say goodbye to school. So I decided to write down the takeaways I had in my phd as a conclude of these three wonderful years.
It seems that engineering is underrated by most research students, and also maybe many professors. I would say solid engineering is the “foundation” of doing good cutting-edge AI research.
More and more good research ideas are inspired by engineering now. For instance, when starting my research career, many papers design models or algorithms to make the model more similar to human (we thought such model is smarter). But now, more impactful works (e.g. Scaling Laws, Mixture of Experts, Grouped Query Attention, Flash Attention) is inspired from taking closer look on experimental results. Human-like or fancy is weak, be helpful and efficient. If you are working on LLMs or Foundation models, you are most likely to contribute to a smaller part of an huge efforts. Our goal is usually committing our innovation to upstream models. Then the key is keeping the model simple and clean, but having amount of solid ablation studies, as what we usually do in developing a good codebase, so good coding and documentation are highly desirable.
Before PhD. It
Research Taste
Time-Management and Prioritization