Editing Openai/694057b6-101c-8007-9a65-c40578c7252d (section)

==== Separately from Python bindings, llama.cpp itself ships <code>llama-server</code>, a lightweight HTTP server exposing OpenAI-compatible endpoints like /v1/chat/completions, and it supports concurrent requests/parallel decoding. GitHub<ref>{{cite web|title=GitHub|url=https://github.com/ggml-org/llama.cpp|publisher=github.com|access-date=2025-12-16}}</ref> ====

Run:

<syntaxhighlight lang="bash">llama-server -m model.gguf --port 8080

</syntaxhighlight>

GitHub<ref>{{cite web|title=GitHub|url=https://github.com/ggml-org/llama.cpp|publisher=github.com|access-date=2025-12-16}}</ref>

Then you can call it from Python with requests (or an OpenAI-compatible client pointed at http://localhost:8080/v1).

If you tell me whether you prefer in-process (bindings) or server-based (HTTP), and what GPUs/build you’re using (CUDA/Vulkan/CPU), I can suggest the cleanest setup + example code for your exact workflow.