Build on a governed AI gateway

Introduction#

Sophy is an OpenAI-compatible gateway and operator console. Applications send input in familiar wire formats; Sophy authenticates the key, resolves its centrally managed configuration, calls the bound model through that key’s project-owned Vercel AI Gateway credential, and records project-attributed usage.

Compatibility is intentionally scoped. Chat Completions and Responses cover text, streaming, multimodal input, and function-tool loops. Dedicated routes cover audio transcription, embeddings, image generation, short-lived file uploads, and the primary model bound to a key. The sections below call out differences from the full OpenAI platform.

Supported API surfaces#

Authentication#

Send the Sophy key as a Bearer token. Keys look like mw_live_… and are issued inside a project in the Sophy console. Sign in with any verified email to create a renameable My Project, or join an existing project through an Admin invitation.

Authorization: Bearer mw_live_…

Missing Bearer authentication returns 401 missing_api_key. A malformed, unknown, revoked, expired, or inactive-project key returns 401 invalid_api_key. Keys are checked against the database on every request, so edits and revocation apply immediately.

Quickstart#

Change the base URL and API key, then call one of the supported methods. The placeholder model is required by some OpenAI SDK methods but Sophy uses the model bound to the key.

from openai import OpenAI

client = OpenAI(
    base_url="https://sophy.in/v1",
    api_key="mw_live_…",
)

resp = client.chat.completions.create(
    model="sophy",  # ignored — the key owns the model
    messages=[{"role": "user", "content": "Hello, Sophy!"}],
)
print(resp.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://sophy.in/v1",
  apiKey: "mw_live_…",
});

const resp = await client.chat.completions.create({
  model: "sophy",  // ignored — the key owns the model
  messages: [{ role: "user", content: "Hello, Sophy!" }],
});
console.log(resp.choices[0].message.content);

curl https://sophy.in/v1/chat/completions \
  -H "Authorization: Bearer mw_live_…" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sophy",
    "messages": [{"role": "user", "content": "Hello, Sophy!"}]
  }'

Chat Completions#

Key-owned or ignored: model, temperature, top_p, max_tokens, max_completion_tokens, response_format, n, and user. Sophy applies the key’s configured model and generation parameters instead.

Buffered response

{
  "id": "chatcmpl-…",
  "object": "chat.completion",
  "created": 1735689600,
  "model": "anthropic/claude-sonnet-4.6",
  "choices": [{
    "index": 0,
    "message": { "role": "assistant", "content": "Hello! How can I help?" },
    "finish_reason": "stop"
  }],
  "usage": { "prompt_tokens": 9, "completion_tokens": 7, "total_tokens": 16 }
}

Streaming

A stream begins with an assistant-role chunk, emits text and/or tool-call deltas, finishes with a reason chunk, optionally emits usage, and ends with data: [DONE].

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1735689600,"model":"anthropic/claude-sonnet-4.6","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1735689600,"model":"anthropic/claude-sonnet-4.6","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1735689600,"model":"anthropic/claude-sonnet-4.6","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Responses API#

Use client.responses.create(...) for a Responses-compatible text, multimodal, streaming, or function-tool workflow. Sophy is stateless: send the full conversation and any prior function calls/results in input on every request.

Key-owned or ignored: model, temperature, top_p, max_output_tokens, and text.format.

from openai import OpenAI

client = OpenAI(base_url="https://sophy.in/v1", api_key="mw_live_…")

resp = client.responses.create(
    model="sophy",
    input="Write a haiku about gateways.",
)
print(resp.output_text)

curl https://sophy.in/v1/responses \
  -H "Authorization: Bearer mw_live_…" \
  -H "Content-Type: application/json" \
  -d '{"model":"sophy","input":"Write a haiku about gateways."}'

{
  "id": "resp_…",
  "object": "response",
  "created_at": 1735689600,
  "status": "completed",
  "model": "anthropic/claude-sonnet-4.6",
  "store": false,
  "output": [{
    "id": "msg_…",
    "type": "message",
    "status": "completed",
    "role": "assistant",
    "content": [{ "type": "output_text", "text": "Silent routes converge…", "annotations": [] }]
  }],
  "usage": {
    "input_tokens": 11,
    "input_tokens_details": { "cached_tokens": 0 },
    "output_tokens": 9,
    "output_tokens_details": { "reasoning_tokens": 0 },
    "total_tokens": 20
  }
}

Streaming uses events such as response.created, response.output_text.delta, response.function_call_arguments.delta, and response.completed, each with an increasing sequence_number. There is no [DONE] sentinel.

Multimodal input#

Language keys can send public image/file URLs or URLs returned by POST /v1/files, provided the bound model supports the content type. Sophy ownership-checks its own uploads; public external URLs pass through to the model provider.

uploaded_url = "https://…/uploads/<project>/<key>/invoice-….pdf"
uploaded_filename = "invoice.pdf"

resp = client.chat.completions.create(
    model="sophy",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Summarize this document."},
            {
                "type": "file",
                "file": {
                    "file_url": uploaded_url,
                    "filename": uploaded_filename,
                },
            },
        ],
    }],
)

uploaded_url = "https://…/uploads/<project>/<key>/photo-….png"

resp = client.responses.create(
    model="sophy",
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "What is shown here?"},
            {"type": "input_image", "image_url": uploaded_url},
        ],
    }],
)

Tool / function calling#

Sophy passes function definitions to the bound language model and maps its requested calls back to the selected OpenAI wire format. Sophy never executes a tool. Your application runs it, then sends the result and full conversation state on the next call.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

first = client.chat.completions.create(
    model="sophy",
    tools=tools,
    messages=[{"role": "user", "content": "Weather in Paris?"}],
)
call = first.choices[0].message.tool_calls[0]

second = client.chat.completions.create(model="sophy", tools=tools, messages=[
    {"role": "user", "content": "Weather in Paris?"},
    first.choices[0].message,
    {"role": "tool", "tool_call_id": call.id, "content": "18°C, sunny"},
])

tools = [{
    "type": "function",
    "name": "get_weather",
    "description": "Get weather for a city",
    "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"],
    },
}]

first = client.responses.create(
    model="sophy",
    tools=tools,
    input="Weather in Paris?",
)
call = next(item for item in first.output if item.type == "function_call")

second = client.responses.create(model="sophy", tools=tools, input=[
    {"role": "user", "content": "Weather in Paris?"},
    {
        "type": "function_call",
        "call_id": call.call_id,
        "name": call.name,
        "arguments": call.arguments,
    },
    {
        "type": "function_call_output",
        "call_id": call.call_id,
        "output": "18°C, sunny",
    },
])

Only function tools are supported; other Responses tool types are ignored. Chat uses the nested { type: "function", function: { ... } } shape; Responses uses the flat { type: "function", name, parameters } shape. Legacy Chat functions returns 400 functions_unsupported.

Structured output#

An operator can attach a JSON Schema to a language key. Sophy normalizes the schema across supported model providers. Buffered calls use constrained generation, tolerant JSON extraction and retry, then validate the completed object; streaming uses schema-constrained generation with post-hoc validation.

The schema lives on the key. Client-sent Chat response_format and Responses text.format do not replace it. Successful JSON is returned as the assistant text, so parse choices[0].message.content or response.output_text.

Audio transcription#

Bind the key’s primary model to a transcription model and send audio as multipart/form-data. Sophy returns raw text when no processing policy is set, or applies centrally managed instructions in a second language-model step and returns only that processed text.

Accepted audio formats are MP3/MPEG/MPGA, MP4/M4A, WAV, WebM, FLAC, and OGG. Sophy identifies the format from the file bytes instead of trusting its name or client-declared MIME type. Files outside that allowlist return 400 unsupported_audio_format; files over the limit return 413 file_too_large.

from openai import OpenAI

client = OpenAI(base_url="https://sophy.in/v1", api_key="mw_live_…")

with open("meeting.m4a", "rb") as audio:
    resp = client.audio.transcriptions.create(
        model="sophy",
        file=audio,
        language="en",
        response_format="json",
    )

print(resp.text)

curl https://sophy.in/v1/audio/transcriptions \
  -H "Authorization: Bearer mw_live_…" \
  -F "file=@meeting.m4a" \
  -F "model=sophy" \
  -F "language=en" \
  -F "response_format=json"

{
  "text": "Decisions: launch on Friday and send the checklist today.",
  "processed": true,
  "language": "en",
  "duration": 8.4
}

One uploaded clip consumes one RPM slot and counts as one client request. Sophy attributes transcription and processing to their actual models, while the processor component does not add a second client request. If content logging is enabled on the key, Sophy stores the filename, byte count, language hint, raw transcript, and final text—but never the audio bytes. Turn content logging off when unprocessed text must not be retained.

Embeddings#

Bind the key to an embedding model and use the standard OpenAI embeddings method. The key-owned model is returned in the response, and usage records input tokens and estimated cost.

Pre-tokenized integer arrays are not portable across a gateway-bound model and return 400 token_input_unsupported.

from openai import OpenAI

client = OpenAI(base_url="https://sophy.in/v1", api_key="mw_live_…")

resp = client.embeddings.create(
    model="sophy",
    input=["First document", "Second document"],
    encoding_format="float",
)
print(resp.data[0].embedding)

curl https://sophy.in/v1/embeddings \
  -H "Authorization: Bearer mw_live_…" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sophy",
    "input": ["First document", "Second document"],
    "encoding_format": "float"
  }'

{
  "object": "list",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.012, -0.044, 0.008] },
    { "object": "embedding", "index": 1, "embedding": [-0.031, 0.017, 0.052] }
  ],
  "model": "openai/text-embedding-3-small",
  "usage": { "prompt_tokens": 5, "total_tokens": 5 }
}

Image generation#

Bind the key to a supported image model. Generated images are returned inline as base64 and are not stored by Sophy.

from openai import OpenAI
import base64

client = OpenAI(base_url="https://sophy.in/v1", api_key="mw_live_…")

resp = client.images.generate(
    model="sophy",
    prompt="A red panda coding at a desk, watercolor",
    n=1,
    size="1024x1024",
)
with open("out.png", "wb") as f:
    f.write(base64.b64decode(resp.data[0].b64_json))

curl https://sophy.in/v1/images/generations \
  -H "Authorization: Bearer mw_live_…" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"A red panda coding at a desk, watercolor","size":"1024x1024"}'

{
  "created": 1735689600,
  "data": [{ "b64_json": "iVBORw0KGgoAAAANSUhEUgAA…" }]
}

Files#

Upload a file as multipart/form-data using the field name file. Success returns 201. Use the returned URL in a Chat file/image part or a Responses input_file/input_image part.

Accepted content types are PDF, PNG, JPEG, WebP, GIF, plain text, CSV, and JSON. The maximum size is 4 MiB (4,194,304 bytes). Uploads become eligible for batched cleanup after 24 hours, so they are request staging—not durable file storage or a precise expiry service.

curl https://sophy.in/v1/files \
  -H "Authorization: Bearer mw_live_…" \
  -F "file=@./invoice.pdf"

{
  "id": "https://…/uploads/<project>/<key>/invoice-….pdf",
  "url": "https://…/uploads/<project>/<key>/invoice-….pdf",
  "pathname": "uploads/<project>/<key>/invoice-….pdf",
  "filename": "invoice.pdf",
  "bytes": 48213,
  "contentType": "application/pdf"
}

Upload URLs are public but unguessable so model providers can fetch them; treat each URL as a bearer URL, not a secret store. When a Sophy URL is submitted back through the API, Sophy checks its issuing key, and cross-key reuse returns 403 file_access_denied.

Models#

Returns exactly the primary model currently bound to the authenticated key in OpenAI list shape. This route does not return the full console catalog or an optional transcript processor, and it does not select a model for a later request.

{
  "object": "list",
  "data": [
    {
      "id": "anthropic/claude-sonnet-4.6",
      "object": "model",
      "created": 0,
      "owned_by": "sophy"
    }
  ]
}

Key-owned policy#

The key is Sophy’s unit of configuration. Operators can change its policy without changing or redeploying the client. The next request reads the latest values.

Client values for model and generation parameters are silently ignored. Chat and Responses prompt behavior is controlled separately by Agent mode. Audio transcription rejects a client prompt; only the key-owned system prompt can request post-processing.

When a language or transcript-processing call has an effective system prompt, Sophy prepends a fixed platform security preamble before the key and any applicable Agent-mode client instructions. Eligible multi-turn Anthropic requests also receive an ephemeral prompt-cache breakpoint; cache reads appear in Responses usage as cached_tokens.

Agent mode#

Standard keys are governed mode: Sophy drops client system and developer messages plus Responses instructions, then uses the key-owned prompt.

For a trusted server-side agent whose instructions must vary per request, an operator can enable Agent mode on the key. Sophy keeps the key prompt ahead of client instructions (after Sophy’s fixed security preamble), then appends:

• Chat: consecutive leading system/developer messages, up to the first non-system turn.
• Responses: instructions, followed by leading system/developer input items.

Later system-role items in the conversation are still dropped. Model, generation parameters, schema, knowledge, budgets, and rate limits remain key-owned.

Knowledgebases#

Project Admins can create project-scoped knowledgebases from PDF, DOCX, Markdown, plain text, CSV, and JSON documents up to 4 MiB each. Sophy extracts, chunks, and embeds sources in the background, then lets keys in that project attach to the same collection.

On Chat or Responses requests with usable user text, Sophy embeds the latest user text (capped at its first 8,000 characters), retrieves the six closest chunks, and adds them to the effective prompt. Image- or file-only turns have no retrieval query. A transient retrieval timeout can continue without grounding. Missing, invalid, or billing-blocked project Gateway credentials fail closed before the model call; they never fall back to Sophy’s deployment credentials or another project.

Knowledgebase ingestion and query-embedding spend is tracked separately from client proxy traffic in the console.

Operator console#

One passwordless identity can belong to multiple projects with an independent Admin or Editor role in each. The project in the URL is the tenant boundary, and every read, mutation, key, knowledgebase, log, evaluation, and audit event stays inside it.

Errors and retries#

Before a response stream starts, Sophy-generated failures use the OpenAI error envelope below with Cache-Control: no-store. Field validation errors may use code: null and identify the field in param.

{
  "error": {
    "message": "Invalid API key.",
    "type": "authentication_error",
    "param": null,
    "code": "invalid_api_key"
  }
}

Upstream client-input rejections (400/413/422) become 400 upstream_invalid_request with a safe, capped message. Upstream rate limits preserve 429 and forward Retry-After when present. A valid Sophy key whose project credential is missing, disconnected, invalid, billing-blocked, or undecryptable receives 503 project_gateway_unavailable; Sophy makes no fallback upstream call. Transient network and server failures are hidden behind 502 upstream_error.

Rate limits and monthly quota#

Chat, Responses, audio transcription, embeddings, and image generation enforce the key’s optional requests-per-minute limit and monthly cost cap before the paid model call. /files and /models do not consume those request counters.

• RPM uses a fixed 60-second window. Exceeding it returns 429 rate_limit_exceeded with Retry-After in seconds.
• Monthly quota uses the current UTC calendar month and counts client-attributable spend, including transcript processing. Reaching it returns 402 quota_exceeded.
• Evaluation and knowledgebase spend is recorded under separate sources for visibility and does not consume a key’s proxy-traffic cap.

Console cost values are gateway estimates for operational decisions, not a billing-grade ledger.