Self-hosting Falcon and Jais on your own hardware with vLLM

Those are the three durable reasons. Everything else tends to be enthusiasm, and enthusiasm does not survive the first GPU driver incident at two in the morning.

The version stability point is underrated. A managed endpoint can change behaviour on the provider's schedule, which is genuinely difficult when you have validated a clinical or legal workflow against specific outputs. Pinned open weights do not move unless you move them.

This matters more than it used to. Arabic performance in general-purpose open models has historically been weak, and for a UAE deployment that is not a detail you can work around with prompting.

Falcon-H1 comes from TII in Abu Dhabi and performs strongly on Arabic evaluation across its size range, with smaller variants competitive against considerably larger models. Jais is Arabic-first by design, and Jais 2 was built specifically to handle Arabic and English in parallel rather than treating Arabic as a secondary language.

Check the licence before you get attached to a model. Jais 30B is released under Apache 2.0, which is straightforward for commercial deployment. Licences differ across families and across sizes within a family, and this is worth confirming rather than assuming.

PagedAttention borrows the idea of virtual memory paging. Instead of reserving one contiguous block of KV cache per sequence, sized for the worst case, it allocates fixed-size blocks on demand. The waste from over-reservation largely disappears, which means more concurrent sequences on the same card.

Continuous batching addresses the other inefficiency. With static batching, a batch runs until every sequence in it finishes, so short requests sit idle waiting for the longest one. Continuous batching schedules at the iteration level, admitting new requests as slots free up.

Together these are the reason a self-hosted deployment can reach throughput that feels comparable to a managed service. Both are on by default in current versions, so you get them without configuration.

Weights are simple arithmetic. A model served at 16-bit precision needs roughly two bytes per parameter, so a 30 billion parameter model occupies about 60 GB before anything else. Quantising to 8-bit roughly halves that, and to 4-bit roughly quarters it, at some quality cost.

The cache is where teams get caught. Its size scales with the number of layers, the key and value head dimensions, the precision, the sequence length and the number of concurrent sequences. Double your context window and you double the cache for every concurrent user.

The practical consequence: a configuration that benchmarks beautifully with one user can fall over at twenty, because the weights fit comfortably and the cache does not. Always load-test at your target concurrency and your real context length.

Tensor parallelism splits a model across GPUs when it does not fit on one, set with --tensor-parallel-size and the number of cards. It adds inter-GPU communication, so it buys you capacity rather than raw speed.

The memory utilisation flag defaults to 0.90, meaning ninety percent of available VRAM is given to the KV cache pool. On dedicated bare-metal hardware you can push this higher, into the 0.92 to 0.95 range, because nothing else is competing for the card.

The context length cap is the one that causes the most confusion. Serving will refuse to start if the model's default maximum context cannot be accommodated in available memory, and the fix is to set --max-model-len to a value your workload actually needs rather than the theoretical maximum.

One deployment detail worth knowing: running vLLM in Docker requires --ipc=host, because it uses shared memory for inter-process communication and the default container limit is too small.

This is the step where most self-hosting projects stall. The endpoint comes up, the demo is impressive, and then the security review asks who can query it and what happens to the logs.

Permission-aware retrieval deserves particular attention. Naive retrieval flattens permissions: it indexes everything and returns whatever matches, regardless of whether the requesting user was entitled to see the source document. In a regulated environment that is an access control failure, and a silent one.

Logging is the other half. You need per-query records tied to an authenticated identity, with the retrieved sources and the model version recorded, retained long enough to investigate an incident. Under ADHICS the notification window is 24 hours, which is not enough time to build that capability after the fact.

None of these are exotic engineering problems. They are the consequence of a pilot being promoted to production without passing back through the controls a production system would normally face.

The fix is unglamorous and reliable: treat the inference stack as an information asset from day one, with an owner, an inventory entry, a retention policy and a change process. Doing that at the start costs a week. Doing it after an audit costs a quarter.

It is also worth being clear about what self-hosting replaces. You are not removing a dependency, you are exchanging one managed contract for a set of internal boundaries that nobody has written down, and the failures tend to appear between the parts rather than inside them.