What Is a Private LLM Gateway?
A Private LLM Gateway lets you use a language model running on your own machine, through Ollama, LM Studio, vLLM or llama.cpp, from anywhere on the internet. You get one secure HTTPS address and an API key, and your apps and tools call it like a cloud AI service.
How it works
- You install the small CDNShark connector on the machine that runs your model, or on another machine on the same network.
- The connector dials out to CDNShark and keeps an encrypted tunnel open. Nothing connects in to your machine.
- Your apps call your gateway address,
https://<id>.llm.cdnshark.com. The request reaches the nearest CDNShark edge, which checks your API key and passes it through the tunnel to your model. - The answer streams back token by token, the same way it does from a cloud AI service.
What you get
| Feature | Details |
|---|---|
| One address for every tool | Works with OpenAI SDKs and OpenAI-compatible tools, and with Ollama tools. Adding /v1 to the address works too. |
| API key | One key per gateway. Every request needs it, and you can replace it at any time. |
| Streaming | Responses arrive token by token, not all at once at the end. |
| No open ports | Your machine needs no public IP and no firewall rule. It only makes outbound connections. |
| Free TLS | The address uses HTTPS with a certificate we manage. |
What it is not
- We don't host models. The model runs on your hardware, so its speed depends on your machine and your upload bandwidth.
- We don't store your prompts or answers. Requests are encrypted in transit and passed straight through.
- It is not a VPN. The connector only reaches the one model server you set. It can't be used to reach anything else on your network.
Ready to start? See Create a Gateway and Install the Connector.