Hybrid operation: when does an AI platform on site with an external model pay off?

What does hybrid operation mean for an AI platform?

This guide is about operation: platform and model in different locations. If you only want to enable external models for selected groups, see Hybrid AI: external models for selected groups.

An AI platform consists of two parts: the platform (sign-in, permissions, apps, documents, connectors, log) and the AI model, which computes the answers. Both can run in the same place, in the cloud or on a server at your site. In hybrid operation they are separate:

• The platform runs on hardware in your own network. Documents, permissions, apps and log stay there.
• The AI model computes elsewhere, for example in the data centre of a university or a federal state, on an existing computing cluster or at a provider of computing power.

For each question, the platform sends the instruction, the question and the relevant text passages from the documents to the model and receives the answer back.

When does hybrid make sense?

• Computing power already exists, but not on site: for example a computing cluster of the operator, the federal state or the university.
• No GPUs in your own data centre: because of space, power, cooling or procurement.
• The data is to be stored on site: documents, log and identity management stay in your own network; only the computing task goes outside.
• Transition: an organisation starts with an external model and brings the computing power on site later.

Hybrid makes less sense if the organisation has to work without an internet connection or if no data may leave the network. Then platform and model belong together on a server at your site. Running AI offline →

What does hybrid mean for data protection?

What matters is where the model computes, because questions and text excerpts go there:

Model computes …	What needs to be clarified
in your own data centre or at the operator	internal rules, network connection, encryption of the link
at a provider in the EU	data processing agreement (Art. 28 GDPR), storage and logging of inputs, training
at a provider based in the USA	in addition, transfer and access rights under US law (US CLOUD Act)

For health data in Germany, § 393 SGB V (German Social Code, Book V) also applies if a cloud service processes it. Check with your data protection officer whether the location of the model suits the data processed in your apps.

What should IT watch?

• Link: The connection between platform and model must be encrypted and stable. If it fails, there are no answers.
• Response time: Every route over the network adds a little to the waiting time.
• Capacity: How many questions can the model at the other location answer at the same moment, and do others share this capacity?
• Interface: Many models can be reached through an OpenAI-compatible interface. This is a widespread format for requests to AI models, independent of the provider.
• Switching: Can the model be replaced later without apps and permissions changing?

How does this work with basebox?

basebox offers hybrid, and customers already use it: basebox runs on your hardware in your network, with sign-in, permissions, apps, connectors and log. The AI models come from another location, for example from your data centre or from a provider you connect. With a third-party provider, we point out the consequences for data protection. You will find the price in your personal quote.

The other options need no hybrid: in the cloud, the AI model in basebox runs on our servers in the Munich-East data centre; with the server, at your site. Your own models with an OpenAI-compatible interface can be connected according to our checklist; your administrator enables them. Your data does not train any model.