Scaling an AI platform: larger models or more people?

Why two directions?

When people talk about “more AI”, they often mean two different things:

	Larger model	More people
What changes	better answers for difficult tasks, such as long specialist texts	more questions at the same moment
What it needs	more memory for the model itself	more memory and computing time for ongoing conversations
Depends on	size of the model	number of people asking at the same time, and length of the texts
Noticeable as	a model does not fit on the hardware	answers start later at peak times

The number of accounts hardly matters here. What matters is how many people expect an answer at the same moment and how much text they work with.

What does a larger model need?

A language model sits completely in the memory of the GPUs. As the model grows, the memory requirement grows with it, even if only one person is asking. Compressed versions of a model need less memory but cost some accuracy. A larger model therefore means more or larger GPUs that carry one model together.

Whether a larger model is needed shows in practice: are the answers good enough for the tasks the platform is there for? For drafts, summaries and information from documents, a medium-sized model is often enough; for long legal or medical texts, a larger one can be better.

What do more people need?

Every question that is being answered takes up additional memory, all the more the longer the text: a short chat little, a long document a lot. When space runs short, further questions wait briefly. More people therefore means more memory for ongoing conversations, that is, additional GPUs on which the same model runs once more and the questions are distributed.

As a guide: depending on the class, the servers from basebox answer 100, 150 or more than 200 questions at the same moment. For around 200 to 2,000 people this is enough, because never everyone asks at the same time.

What helps in which case?

Observation	What helps
Answers are good, but start later at peak times	more GPUs for the same model
Answers for difficult tasks are not good enough	a larger model, and for it more memory on the GPUs
Both	first decide on the model, then the number of GPUs for the usage
Long documents slow everyone down	store documents in knowledge collections instead of uploading them in full each time

How does this work with basebox?

basebox grows without anything changing for staff: apps, permissions, connections and logs stay the same; only the computing power grows. In the cloud we take care of this. With the server from basebox, you choose the class by the number of people; for more than about 2,000 people, basebox runs on larger hardware that you procure yourself, for example from HPE or Dell, and the licence is based on the GPUs. basebox can also connect further servers with GPUs through their API. We clarify with you beforehand which hardware suits your model and your usage; the technical details are in the basebox documentation.