serving-api

Frontend and backend API proxy for SwissAI LLM serving.

Live at:

• Prod: serving.swissai.svc.cscs.ch
• Dev: servingdev.swissai.svc.cscs.ch
• Local: with docker compose up

Architecture

                              o
        ┌─────────────────┐  /|\   curl / python SDK
        │    OpenWebUI    │  / \ 
        └────────┬────────┘   |
                 │            │
                 │  POST /v1/chat/completions
                 │            │
                 ▼            ▼
        ┌─────────────────────────┐
        │       serving-api       │  auth + proxy (this repo)
        └─────────────────────────┘
                 │
                 │
                 ▼
        ┌─────────────────┐
        │       OCF       │  OpenTela P2P routing → model=apertus-...
        │                 │  
        └────────┬────────┘
                 │
                 ▼
        ┌─────────────────┐
        │   vllm/sglang   │  model inference (GPU)
        └─────────────────┘

Repo Structure

backend/         # Python API proxy (FastAPI) — auth, caching, routing
frontend/        # web UI (Astro + Svelte)
meta/            # Dockerfiles, k8s manifests, build scripts
tests/           # integration tests
tools/           # metrics & monitoring utilities

OCF (Open Compute Framework) now renamed to OpenTela upstream is maintained at eth-easl/OpenTela. We maintain a fork at swiss-ai/OpenTela to control deployments to dev+prod.

Quick Start

Docker

docker compose up

Local Development

make install      # install backend dependencies
make run          # start backend on :8080

# frontend
cd frontend
npm install && npm run dev