Sjoerd de Haan, DataDrivenLab · February 2024 · Working with ML in R&D organisations

Why small teams can build foundation models

Seven months ago, Sam Altman was asked this question

“Can a small team of smart engineers build a foundational model?”

Altman answered that “it’s totally hopeless to compete with us on training foundation models”, and that “it’s your job to try anyway” (Christopher 2023).

Seven months later, a small team at Shinoji research beats GTP-4 on the emotional intelligence benchmark. Fine-tuning Mistral, an open source LLM developed by startup Mistral AI, they beated chatGTP spending just $400 on compute.

Dynamics of machine learning innovation

The non-verbal message of Sam Altman in the video, seems to indicate that does not believe what he is saying.

At least I did not believe this 7 months ago.

In order to understand where I come from, you do not need intricate knowledge on LLMs. Rather you need to play close attention to the dynamics around innovation in machine learning.

The dynamics around innovations in machine learning differ in some ways from previous eras of discovery and innovation.

This is both exiting and significant in several ways:

1. From the perspective of the history of science, there are differences in how discovery and technological development unrolls compared to other episodes in history
2. If you are working in the field of machine learning, it is useful to understand where the field is going
3. If you are not working in machine learning, it can be helpful to place the developments in perspective
4. If you are investing in technology as a leader, policy maker, stock investor, venture capitalist, public funding organisation, or in in any other capacity, it can be beneficial to understand the dynamics of research and innovation in machine learning.

Shaking the emotional intelligence benchmark

The Emotional Intelligence Benchmark for Large Language Models (Paech 2023) measures LLMs abilities to understand complex emotions.

Unexpectedly, a small company with no more than a few people forced it’s way to the top of the benchmark.

Shinoji research is a small research company in finance. The website talks about Crypto, FTX, statistics and topic like financial independence. The website looks more like the homepage of a developer than as a company website. Also it does not seem to specialize in LLMs but in it’s application in trade.

Dynamics of machine learning innovation

Ensemble learning

When it comes to practical problems, no no-one beets the crowd. At least not in machine learning. To support this, let’s look at Kaggle. For problems that are not highly specialized, it’s always a surprise who is going to turn up on top of the leader board. Good progress contributions are not necessarily made by highly specialized research teams, but can come from individuals.

I like to refer this to ensemble learning. If you put many teams to work, you can expect outliers or unexpected results.

Diffusion of knowledge

Expert knowledge in machine learning gets mainstream through papers, tutorials, libraries, notebooks, code and courses.

Whole areas that were specialized domains in academics until a few years ago are now part of mainstream machine. Think forecasting, neural networks, natural language processing, Gaussian processes, neural differential equations, reinforcement learning, causal machine learning. It’s all under our hands with open source packages and material.

The myth of compute power

Compute power can greatly accelerate research in machine learning, but it’s importance should not be overstated.

In 2018 I listened to Jeremy Howard. He argued against the trend for huge compute in machine learning research. It was not just a theoretical talk. With a small group of students he entered a competition to train imagenet from scratch as fast as possible. With modest equipment and clever tricks they were Fastest on publicly available infrastructure, fastest on GPUs, and fastest on a single machine. Their secret: do not rely on massive compute, but be creative.

The post is available on https://www.fast.ai/posts/2018-04-30-dawnbench-fastai.html

From Kaggle to the real world

How does this apply to LLMs?

1. The world of LLMs is like a large ensemble; larger than any Kaggle competition. So many players are contributing openly to the space, that we can expect quick developments.
2. LLMs is no rocket science. The basic building blocks of LLMs are are actively used by a large group of academics and practitioners for many years. There are lots of tools, knowledge and experience around.
3. The availability of massive compute might have allowed Open AI make the initial move; clever engineering is quickly catching up. Many players are incentivised to find solution that require less compute, and they are actively pursuing this; not just as a scientific pursuit, but as a serious business.

The Mistral model enabled the developers from Shioji to appear on top of the benchmark for a short time. It was developed by Mistral AI, a French AI startup that is back with at leas half a billion dollars to date.

Although the real world is more complex than a Kaggle competition, we see that the same principles apply; even when we add economic competition.

Conclusion

Large players, startups, and some practitioners of machine learning have incentives to mystify machine learning and AI.

Instead of sketching a world full of chances, they portray machine learning an exclusive world. A world were excellence and merit prevail and where the winner takes all.

Looking the dynamics in machine learning over 10 years, I witnessed another side of machine learning. I saw an episode unfolding where new technology became available to a large group of people; empowering them to build all sorts of applications.

Christopher, Nilesh. 2023. “Sam Altman’s India comments spark a heated backlash.” Rest of World, 13 June. https://restofworld.org/2023/sam-altmans-india/.

Paech, Samuel J. 2023. “EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models.” arXiv.org. https://arxiv.org/abs/2312.06281v2.