Local language models in the office: performance and cost

Your own AI server in the office can summarise texts, answer questions about your documents and produce drafts of minutes and letters without the data leaving the house. It does not reach the model size of large cloud services, though, and it causes ongoing effort. In the example calculation below it costs a good five times as much as the cloud in the first year, and that is mainly due to the purchase. It pays off only in particular cases.

The four layers of a local set-up

A local set-up has four layers. At the bottom is the hardware, above all a graphics card with plenty of memory. On it runs the language model, whose possible size depends on that memory. Above it sits the interface: chat window, login, permissions, logs. At the top are your documents, which the model is meant to access.

The size of the model determines what hardware is needed. A rule of thumb, which I give as my own calculation: with 4-bit quantisation a model needs about half a byte per parameter. A model with 27 billion parameters then takes up around 14 gigabytes, plus memory for the context. A model with 120 billion parameters such as gpt-oss-120b needs around 60 gigabytes. The rented entry-level model GEX45 from Hetzner has 24 gigabytes of graphics memory and costs 214 euros a month plus a 209 euro set-up fee (retrieved 13 November 2026, excluding VAT). According to Hetzner’s page it is at location HEL 1 in Helsinki. For models from about 100 billion parameters, Hetzner offers servers with 96 gigabytes; the cheapest costs 999 euros a month plus a 999 euro set-up fee. A rented server is an external service. As with any cloud, it needs a contract under Art. 28 GDPR; more on that in the article Running AI in the office in line with the GDPR.

What works well locally and where the limits lie

Models of around 30 billion parameters fit locally, and smaller ones on small cards. Large cloud providers in the EU offer bigger ones: IONOS’s price list includes Qwen3.8-27B as well as gpt-oss-120b and Qwen3.5-397B. How large the quality gap is for your tasks, for example technical texts in German or reading invoices, I have not measured reliably, and I found no study on it. Test with your own documents before buying anything.

On load: the 12.5 million output tokens a month in the example calculation below work out, spread over 22 working days of 8 hours, at around 20 tokens per second on average (my conversion). For the busiest hour I assume three times that, about 60 tokens per second for all 15 people together (assumption). How many people a server can serve therefore depends less on the number of accounts than on how many ask at the same time. A third-party test gives around 39 tokens per second for a single request on a card of this class with a model of about 30 billion parameters; with requests running in parallel I reckon on around 130 tokens per second in total (extrapolated, not measured). By my capacity model the server in the example is thus about two thirds loaded in the busiest hour, and I draw the line at 70 per cent. So one card carries about 15 people at this level of use. Considerably more people need further cards with load balancing; with less frequent use it serves correspondingly more. A load test with your own documents belongs in the planning. The cloud has practically no such limit: it bills by use, not per person, and the provider supplies the computing power.

On security: local software is not automatically secure. As an example of the runtime I read up on Ollama. The vendor says it does not see inputs or data when run locally. The documentation says the server by default listens only on 127.0.0.1. Anyone who makes it reachable on the network has to protect access themselves, because according to the NVD entry the programming interface in the original version has no authentication for two key functions. In May 2026 the vulnerability CVE-2026-7482 became known (9.1 out of 10 under CVSS 3.1): versions before 0.17.1 had a memory-reading flaw that an attacker could trigger remotely without logging in. According to the estimate in The Hacker News report, more than 300,000 servers worldwide are likely to have been affected. Maintenance therefore means: apply updates, restrict network access, check logs.

From a data protection point of view, an own server can count as one of the closed systems that the German Data Protection Conference (DSK) prefers. What that covers is in the article on running AI in line with the GDPR.

What it costs: an example calculation

The example is fictitious: a construction company with 15 employees running three solutions, namely a search of project documents, incoming invoices and a managed AI access. All figures except those expressly sourced are assumptions of my cost model, as of 13 November 2026, all amounts net. I calculate with round assumptions and round the totals to 100 euros. For usage I assume that all 15 people use the tool every day: about 40 requests per person and working day, which is about 12,500 requests a month, with 6,000 input tokens per request (a question with extracts from documents) and 1,000 output tokens. Together that is 75 million input tokens and 12.5 million output tokens a month. Only costs that differ between the options are compared. The applications themselves and their upkeep arise on every route alike and are therefore not in the calculation. The calculation covers 12 months, that is the first year including the purchase. I assume no residual value for the hardware.

Own server (purchase)Cloud on pay-per-use

• One-offAssumptions

  Own server (purchase) 14,500 €
  Cloud on pay-per-use 1,200 €

• Per monthMaintenance 300 € for the server, 75 € for the cloud

  Own server (purchase) 330 €
  Cloud on pay-per-use 175 €

• Over 12 monthsSum of both, rounded

  Own server (purchase) 18,500 €
  Cloud on pay-per-use 3,300 €

Source: Author’s example calculation with a fictitious office, net. Assumptions and workings in the text.

Text version of the graphic

• One-off: Own server (purchase) 14,500 €, Cloud on pay-per-use 1,200 € (Assumptions)
• Per month: Own server (purchase) 330 €, Cloud on pay-per-use 175 € (Maintenance 300 € for the server, 75 € for the cloud)
• Over 12 months: Own server (purchase) 18,500 €, Cloud on pay-per-use 3,300 € (Sum of both, rounded)
• Source: Author’s example calculation with a fictitious office, net. Assumptions and workings in the text.

Option	one-off	per month	12 months
Own server (purchase)	€14,500	€330	about €18,500
Rented server (Hetzner GEX45)	about €3,200	about €515	about €9,400
Cloud on pay-per-use (usage based on the IONOS price list, fixed items estimated)	€1,200	€175	€3,300

How the table is calculated, all assumptions unless a source is named:

• Own server, one-off: graphics card 8,000 euros net, rest of the computer 2,500 euros, power protection against outages 500 euros, backup 500 euros, set-up 2.5 days at 1,200 euros (3,000 euros), together 14,500 euros. The card price is an assumption of the author for November 2026. PC Games Hardware gave at least 5,249 euros in its price comparison on 10 September 2026 (the source does not say whether that includes VAT); prices have risen since, and I have not backed today’s price with a source. Per month: electricity 30 euros (about 100 kilowatt hours at 0.30 euros) and maintenance 300 euros (four hours at 75 euros), together 330 euros. Over 12 months that is 18,460 euros, rounded 18,500.
• Rented server, one-off: 209 euros set-up fee (Hetzner) and 3,000 euros for set-up by a service provider. Per month: 214 euros rent (Hetzner) and 300 euros maintenance, together 514 euros. Over 12 months that is 9,377 euros, rounded 9,400. A backup is not included in this line. Caveat: by my capacity model the card in this model does not carry the load of the example (its throughput has not been measured; I use a cautious substitute value). The next larger model, the GEX63, would carry it and would cost about 19,600 euros in the first year.
• Cloud, one-off: 1,200 euros for the connection. Per month: 80 euros fixed items for operating the connection and a search service (an estimate, not from the IONOS price list), 75 euros for one hour of upkeep (access, keys, model changes, cost control; assumption) and about 20 euros usage, together 175 euros. The provider runs the platform itself.
• The usage follows from the IONOS price list for gpt-oss-120b: 75 million input tokens at 0.15 euros per million (11.25 euros) and 12.5 million output tokens at 0.65 euros per million (about 8.13 euros), together about 19.40 euros, calculated as 20 euros. The list does not mention VAT; I treat the amounts as net. Even with daily use by everyone, usage remains the smallest item.
• The maintenance time of four hours a month is a working assumption of the author for an encapsulated server that can be reached only from the office network (updates, backup, logs, faults). At three hours the own server costs about 17,600 euros in the first year, at five hours about 19,400 euros, and the rented server about 8,500 and 10,300 euros respectively. If the server can be reached from the internet, the effort rises.
• The hourly rate of 75 euros is also an assumption, for example for an in-house IT person or a framework contract; I found no independent survey on it. At 150 euros for an external service provider the own server would cost about 22,100 euros in the first year and the cloud 4,200 euros.

Three limits of the calculation: the cloud line uses a larger model (120 billion parameters) than the 32 gigabytes of the gaming card can carry. The card price is for a gaming card (GeForce RTX 5090); other cards come into question for continuous office use, and I have not checked their prices. With the September price from the source (about 4,400 euros net) the total would be about 14,900 euros. And that a single card serves 12,500 requests a month with short waiting times follows from my capacity model with figures from third-party tests, not from a measurement of my own under load. With no maintenance at all, the own server would also cost about 14,900 euros in the first year, still a good four times the cloud. In the first year the cloud remains cheaper in all these cases.

Own server (purchase)Cloud on pay-per-use

• Hardware and set-upone-off

  Own server (purchase) 14,500 €
  Cloud on pay-per-use 1,200 €

• Maintenance and upkeepServer 4 hours, cloud 1 hour a month at 75 € (assumption)

  Own server (purchase) 3,600 €
  Cloud on pay-per-use 900 €

• Electricity or usageElectricity 30 €, usage 20 € a month

  Own server (purchase) 360 €
  Cloud on pay-per-use 240 €

• Fixed cloud items80 € a month (estimate)

  Own server (purchase) 0 €
  Cloud on pay-per-use 960 €

Source: Author’s example calculation with a fictitious office, net, over 12 months. Assumptions in the text.

Text version of the graphic

• Hardware and set-up: Own server (purchase) 14,500 €, Cloud on pay-per-use 1,200 € (one-off)
• Maintenance and upkeep: Own server (purchase) 3,600 €, Cloud on pay-per-use 900 € (Server 4 hours, cloud 1 hour a month at 75 € (assumption))
• Electricity or usage: Own server (purchase) 360 €, Cloud on pay-per-use 240 € (Electricity 30 €, usage 20 € a month)
• Fixed cloud items: Own server (purchase) 0 €, Cloud on pay-per-use 960 € (80 € a month (estimate))
• Source: Author’s example calculation with a fictitious office, net, over 12 months. Assumptions in the text.

What decides it is the purchase. The extra cost of the own server in the first year is about 15,200 euros, of which 13,300 euros is hardware and set-up. In running costs the server costs 155 euros a month more (330 against 175 euros), so at this usage it does not catch up with the cloud later either. It looks different at much higher load: at ten times the usage (scaled up linearly, an assumption), the cloud’s usage costs would be about 200 euros a month and the cloud in total about 355 euros, so just above the running costs of the server. Whether one card carries that load is a separate question. Control and data custody speak for the own server, and in this calculation the price speaks against it.

Who your own server suits

It fits if several of these points apply: your data or contracts forbid passing it to third parties, you have IT support with time for updates and backup, and control is worth the extra price to you. In all other cases a cloud with a data processing agreement in the EU is the simpler solution for a small office. Whether an option holds for your data is decided in the individual case by your data protection officer or a lawyer.

What you can do now

1. Estimate your load: how many people ask at the same time, and how long are the documents? That lets you estimate model size and memory.
2. Get a quote for support before you buy hardware. Maintenance and security updates fall due every month as long as the server runs.
3. Test a model with your own documents, first with a provider under contract or on a rented server, before you buy hardware.
4. Clarify with your data protection officer whether running it in house is necessary for your data. Choosing the model, including its licence, is part of the set-up and not something you have to solve alone.