Online Service

First, start the xllm service according to the xllm launch documentation. Below are examples of client calls for LLM and VLM. Please modify the parameters according to your actual situation.

For SDK setup, streaming, tool use, and compatibility details, see the OpenAI-compatible API guide and Anthropic-compatible API guide. Dedicated guides cover vision inputs and embeddings.

LLM Client Calls

HTTP Call

Chat mode:

curl http://localhost:9977/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2-7B-Instruct",
    "max_tokens": 10,
    "temperature": 0,
    "stream": true,
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "hello xllm"
      }
    ]
  }'

Completions mode:

curl http://127.0.0.1:9977/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2-7B-Instruct",
    "prompt": "hello xllm",
    "max_tokens": 10,
    "temperature": 0,
    "stream": true
  }'

Sample mode:

curl http://127.0.0.1:9977/v1/sample \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2-7B-Instruct",
    "prompt": "Question: <emb_0> matched or not. Conclusion: <emb_0>",
    "selector": {
      "type": "literal",
      "value": "<emb_0>"
    },
    "logprobs": 5,
    "request_id": "sample-demo-001"
  }'

Typical response:

{
  "id": "sample-demo-001",
  "object": "sample_completion",
  "created": 1773369600,
  "model": "Qwen2-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "text": "True",
      "logprobs": {
        "tokens": ["True", "False"],
        "token_ids": [3456, 7890],
        "token_logprobs": [-0.12, -2.31]
      },
      "finish_reason": "selector_match"
    },
    {
      "index": 1,
      "text": "",
      "logprobs": {
        "tokens": [],
        "token_ids": [],
        "token_logprobs": []
      },
      "finish_reason": "empty_logprobs"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 2,
    "total_tokens": 22
  }
}

/v1/sample notes:

• Only --backend=llm is supported. VLM/DiT are not supported yet.
• selector.type is currently fixed to literal. selector.value is matched against prompt text in full and in order.
• logprobs defaults to 5, with an allowed range of [1, 5].
• choices[i].index is the matched sample_id, corresponding one-to-one with the matched order in prompt.
• If no selector match is found, the service returns 200 with choices=[]. If a matched position has no available logprobs, it returns finish_reason="empty_logprobs".
• Service logs only summary fields such as request_id, sample_id, match_count, and model, and do not log the full prompt.

/v1/sample common error semantics:

• Missing model/prompt/selector/selector.value, selector.type != literal, or out-of-range logprobs returns INVALID_ARGUMENT.
• If the model does not exist or the backend is not llm, it returns UNKNOWN.
• When concurrency reaches the upper limit, it returns RESOURCE_EXHAUSTED.
• When the model is in sleep state, it returns UNAVAILABLE.

Python Call

import requests
import json

url = f"http://localhost:9977/v1/chat/completions"
messages = [
    {'role': 'user', 'content': "List three countries and their capitals."}
]

request_data = {
    "model": "Qwen2-7B-Instruct",
    "messages": messages,
    "stream": False,
    "temperature": 0.6,
    "max_tokens": 2048,
}

response = requests.post(url, json=request_data)
if response.status_code != 200:
    print(response.status_code, response.text)
else:
    choices = json.loads(response.text)["choices"]
    print(choices[0]['message'])

VLM Client Calls

HTTP API

import base64
import requests

api_url = "http://localhost:12345/v1/chat/completions"
image_url = ""

def encode_image(url: str) -> str:
    with requests.get(url) as response:
        response.raise_for_status()
        result = base64.b64encode(response.content).decode("utf-8")

    return result

image_base64 = encode_image(image_url)
payload = {
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
                },
            ],
        }
    ],
    "model": "Qwen2.5-VL-7B-Instruct",
    "max_completion_tokens": 128,
}

response = requests.post(
    api_url,
    json=payload,
    headers={"Content-Type": "application/json"}
)
print(response.json())

OpenAI API

from openai import OpenAI
import base64
import requests

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:12345/v1"
image_url = ""

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

def encode_image(url: str) -> str:
    with requests.get(url) as response:
        response.raise_for_status()
        result = base64.b64encode(response.content).decode("utf-8")

    return result

image_base64 = encode_image(image_url)
chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},
                },
            ],
        }
    ],
    model="Qwen2.5-VL-7B-Instruct",
    max_completion_tokens=128,
)

result = chat_completion.choices[0].message.content
print("Chat completion output:", result)