Retrieval-augmented systems rarely serve a single model. A typical pipeline embeds documents and queries, reranks the top candidates with a cross-encoder, detects or redacts personal data before text crosses a trust boundary, and routes content with a zero-shot classifier. The conventional way to serve those four workloads is four Python deployments: sentence-transformers for the vectors, a cross-encoder service for the reranker, a token-classification script per NER model, and an NLI model behind a custom route. Each brings its own export step, its own HTTP contract, and its own dependency tree — and each must be kept in sync when a model is swapped.
rs-infer takes a different design point: one Rust binary that serves all four families behind one HTTP API, built on ONNX Runtime through the ort crate [2] [1]. Models are declared in a YAML config and resolved from the Hugging Face Hub at startup, or read from a local directory — there is no per-model export step and no Python runtime in the serving path.
The goal is predictable self-hosting. An operator should be able to add a model by pasting a repo id, choose the execution backend at compile time, and get OpenAI/vLLM-compatible embeddings, Cohere/Jina-style rerank routes, PII detect/redact, and zero-shot classification from the same process. This article describes the engine internals, the four pipelines, cross-request batching, execution-provider selection, and the tradeoffs that shaped them.
One API, four model families
Every model entry in the config declares a kind, and each kind owns a set of routes. Rather than inventing a schema per family, the API mirrors the contracts clients already speak.
| Family | Routes | Compatible with |
|---|
| Embedding | POST /v1/embeddings · POST /embed | OpenAI, vLLM, TEI |
| Reranker / cross-encoder | POST /v1/rerank · POST /rerank · POST /v1/score | Cohere, Jina |
| PII / NER | POST /pii/detect · POST /pii/redact | — |
| Zero-shot / true-false | POST /classify/zero-shot · POST /classify/true-false | — |
| Ops | GET /health · GET /v1/models · GET /metrics | Prometheus |
Startup is a build, a config, and a binary:
make cpu
./target/release/rsinfer-server --config config.yaml
curl -s localhost:8080/v1/embeddings \
-d '{"model": "e5-small", "input": "hello world"}'
A model is one YAML block. The source is either hf (a Hub repo id) or path (a local directory holding model.onnx + tokenizer.json); execution providers are a priority list, and CPU is always the final fallback.
models:
- name: e5-small
kind: embedding
hf: Xenova/multilingual-e5-small
max_len: 512
replicas: 2
eps: [cpu]
dtype: fp32
batching:
max_rows: 64
max_tokens: 4096
queue_rows: 1024
System architecture
The workspace splits into an engine crate and an HTTP binary. crates/core owns everything model-related; crates/server is a thin axum layer that never touches ONNX directly.
Rendering diagram…
| Layer | Technology |
|---|
| HTTP | axum, tower-http (body limit, timeout), Prometheus client |
| Engine | ort (ONNX Runtime), tokenizers, hf-hub |
| Config | YAML via serde, clap CLI |
| Batching | tokio mpsc queues, length bucketing |
| Tooling | Python helpers (export, quantize, load test), Docker, Make |
| Docs | Fumadocs site with OpenAPI-generated endpoint pages |
A Registry resolves every configured model, builds its session pool, and records per-kind defaults. Loading runs in parallel across models on blocking threads, and the server refuses to start if any model fails — a partially configured process is treated as a configuration error rather than a degraded mode.
Model resolution
The Hub layer exists so that "add a model" is a config edit, not a conversion pipeline. hub.rs resolves each entry into a local set of files and remembers where they came from, so /v1/models can report the actual source.
For a hf entry the resolver lists the repo siblings and picks a graph by preference: model.onnx, then model_fp16.onnx, then model_quantized.onnx and the other quantized variants. An explicit dtype (fp32, fp16, int8) reorders that list, and file pins an exact path when a repo ships only one variant. Large exports keep their weights in a sibling .onnx_data file; the resolver detects it and downloads it alongside the graph, because ONNX Runtime resolves external data relative to model.onnx [1].
Auxiliary files follow the same logic. tokenizer.json, config.json, and 1_Pooling/config.json are looked up in the repo root, in subfolder, and in the conventional onnx/ directory. For model-only ONNX exports, tokenizer_hf points at a second repo that carries the tokenizer — the pattern used for onnx-community/DeBERTa-v3-base-mnli-ONNX, whose tokenizer lives in the original MoritzLaurer repo.
Two paths serve the same resolver:
- Startup download. On first boot the model files land in the HF cache (
HF_HOME, server.hf_cache_dir, or the hf-hub default), honoring HF_TOKEN for gated repos [3].
- Offline materialization.
rsinfer-server download <repo> <out> copies the files into a self-contained folder whose layout matches what path: expects, so a subsequent boot performs no network access at all.
./target/release/rsinfer-server download Xenova/multilingual-e5-small models/e5-small
# options: --revision, --file onnx/model_fp16.onnx, --subfolder onnx, --tokenizer-hf <repo>
The inference engine
The concurrency model follows from an ONNX Runtime constraint: ort::Session is Send but not Sync, and a forward pass takes &mut self. A single session therefore cannot serve concurrent requests, and concurrency comes from replicas — independent sessions of the same graph, sized by the replicas key.
Each model owns a SessionPool: a vector of sessions behind a tokio semaphore, plus a waiter budget (server.max_queue, default 256) and a queue timeout. A request that finds every permit taken and the waiter budget exhausted is rejected immediately with 429; a request that waits longer than queue_timeout_ms gets 503. The error bodies are typed (rate_limit_error, timeout_error) so callers can distinguish shedding from slowness.
Two details keep the async runtime clean:
- Blocking work is pushed off the reactor. Tokenization and inference run inside
spawn_blocking, so axum workers stay available for request handling even when every replica is busy.
- Only one output is materialized. Forwards select a single named output with
OutputSelector instead of reading the default output set. For decoder-style exports this matters: the present.* KV-cache outputs are never copied out of the graph.
Tokenization uses the Hugging Face tokenizers crate [4] with a deliberate tweak: rayon parallelism is disabled by default, because the tokenizer pool would oversubscribe against ONNX Runtime's intra-op threads.
Input tensors are built by intersecting what the tokenizer produced with what the graph declares. input_ids, attention_mask, and token_type_ids are mapped directly; position_ids is derived with the HF convention cumsum(attention_mask) - 1; and past_key_values.* inputs of decoder-only exports receive an empty cache of shape [batch, kv_heads, 0, head_dim], which is what makes single-pass embedding forwards work on graphs like Qwen3-Embedding. An input the engine does not recognize produces an error that lists the model's required inputs instead of a silent shape mismatch. Note that zero-element tensors are rejected by the CoreML execution provider, so these models are CPU-bound by construction.
Thread counts default to intra_threads: 0, meaning each replica may use all logical cores. On an M5 Pro with a 0.6B embedding model at concurrency 8, four replicas × fourteen threads beat every narrower split (4×3, 4×7, 2×7, 1×14) on both throughput and p50: these GEMMs are memory-latency bound, and overlapping full-size ORT thread pools hide stalls better than partitioning cores between replicas. intra_threads exists for hosts where the opposite holds, such as NUMA servers.
Pipelines
Embeddings
An embedding model is the only kind with sub-batching semantics, and it exposes the most compatibility surface. /v1/embeddings accepts strings or pre-tokenized id arrays, honors per-request dimensions, and can return vectors as base64 — the encodings expected by OpenAI-compatible clients [5]. /embed speaks the TEI shape.
Pooling is auto-detected from 1_Pooling/config.json when the export ships one (CLS, mean, or last-token), falling back to masked mean. Truncation is applied before normalization so Matryoshka-style models can serve short vectors [7]: the first d dimensions are kept and then L2-normalized.
v^=∥v[0:d]∥2v[0:d]
Because E5-family models expect task prefixes, the served text is passed through unchanged — prefixing stays a client concern, which keeps one model entry usable for both queries and documents [6].
Rerankers
Reranking scores (query, document) pairs with a cross-encoder. The interesting part is not the forward pass but the score extraction, because "reranker" covers several export families. The engine reads the output shape and applies the matching reduction:
| Output shape | Automatic scoring |
|---|
[B, 1] | sigmoid of the single logit → relevance in (0,1) |
[B, 2] | softmax over (negative, positive) → P(positive) |
[B, S, V] | vocabulary logits at the last non-pad token → P("yes") vs P("no") |
The first two cases cover the standard ms-marco-style classifiers; the third covers Qwen3-Reranker-style exports that emit token distributions. scoring: logit opts into raw unbounded scores for callers that want them. softmax on a two-output model can be replaced by a sigmoid on the logit difference, and yes_no requires the tokenizer to map yes and no as single tokens, which is validated at load time rather than at request time. The routes are shaped after the hosted rerank APIs [8], so a client that speaks Cohere or Jina can point at rs-infer with a base-URL change.
PII detection
PII is token classification with a decoding layer. The model supplies id2label through config.json; the engine computes per-token softmax probabilities, maps labels to BIOES tags, and walks each row maintaining an open span. A token whose probability falls below threshold (default 0.5) is treated as outside any entity, which is what makes confidence a first-class control rather than a fixed argmax.
Spans are reported with character offsets obtained from encode_batch_char_offsets, a mean token probability, and the decoded entity text with surrounding whitespace trimmed. For [B, S, V] outputs the label count is cross-checked against id2label and a mismatch is logged rather than silently truncated. Redaction consumes the same spans: mask replaces each character with a mask character, remove deletes the span, and overlapping entities are skipped once an earlier span has consumed their text.
Zero-shot and true/false
Zero-shot classification is NLI over an encoder-only MNLI model. For every candidate label the engine builds a (text, hypothesis) pair from a template (The text is about {label}. by default) and scores the pair once. A label's score is the entailment logit minus the contradiction logit; single-label mode softmaxes those scores across labels, while multi-label mode applies an independent sigmoid per label — matching the behavior of the reference pipeline.
The true/false route reduces NLI to a boolean judgement. Given a yes/no question, the engine first rephrases it as an affirmative assertion with a small heuristic (Is this email important? → This email is important.), then returns P(entailment) for that assertion. The probability contrasts entailment against contradiction only: MNLI models place most of their mass on the neutral class for unrelated pairs, so including neutral in the softmax would swamp the signal. The assertion actually judged is echoed in the response for auditability, and non-invertible questions must supply an explicit assertion — there is deliberately no best-of-N rephrasing, because garbled splits happen to score high under MNLI entailment. DeBERTa-v3 NLI exports follow this recipe noticeably better than DistilBERT ones [9].
Cross-request dynamic batching
Without batching, every HTTP request tokenizes, pads, and forwards its own batch. When many small requests arrive at once, that wastes both padding and fixed per-forward overhead. With batching: configured, requests enqueue token rows (one per text) into a shared queue, and a gatherer task assembles rows from different requests into larger forwards:
Rendering diagram…
Three design choices matter:
- A row arriving to an empty queue is dispatched immediately. The gatherer drains the backlog without waiting for future rows, so batching never adds latency; it only amortizes cost when requests already overlap.
- Rows are length-bucketed. Sorting the staged rows by token count groups similar lengths into the same forward, minimizing padding to the batch maximum.
- The backlog is spread over idle replicas. The gatherer stages up to
max_rows × idle rows, splits them into up to one chunk per free session, and caps each chunk by the soft max_tokens limit. A full queue (queue_rows, default 1024) sheds with 429.
Batching is opt-in per model, and it is embedding-only. Rows are independent for embeddings, but rerank, zero-shot, and PII outputs are pair- or token-aligned, so those pipelines batch only within a single request. This is a deliberately conservative version of the technique used by high-throughput LLM servers: vLLM inserts new sequences inside a running forward and arrives at much higher utilization [10], and Triton's dynamic batcher offers a more aggressive queueing model [11]. rs-infer's batcher instead accepts that batches are fixed once assembled, which keeps the reply plumbing trivial and the latency guarantees simple.
The gains are backend-dependent, and the project documents them as such. On an M5 Pro, batching improved embedding throughput by roughly 12% for one-document requests and was neutral at sixteen documents per request — CPU forwards scale close to linearly with rows, so packing mainly helps traffic that is many-small-request. On GPUs, where fixed per-forward cost dominates, the same mechanism pays off more.
Execution providers
ONNX Runtime's prebuilt binaries do not combine every execution provider into one build, so rs-infer makes EPs compile-time cargo features and ships Make targets per profile.
| Profile | Command | Provider |
|---|
| CPU (everywhere) | make cpu | CPU |
| macOS + ANE | make mac-coreml | CoreML |
| Linux + CUDA | make gpu-cuda | CUDA |
| Linux datacenter GPUs | make gpu-trt | TensorRT + CUDA |
| Linux consumer RTX | make gpu-rtx | TensorRT-RTX |
At load time, each model's eps list is resolved in order; entries that were not compiled in, or that the linked runtime does not expose, are skipped with a warning, and CPU is appended as the universal fallback. The resolution result is reported by GET /v1/models, so "which provider is actually running this model?" is a query, not a guess.
Two provider-specific behaviors are encoded in the config surface. TensorRT sessions cache their engines (trt_engine_cache), which turns the first forward's compilation cost into a one-time expense. CoreML ships with a caveat instead of an endorsement: for small BERT-size encoders, ONNX Runtime partitions the graph into many CPU↔CoreML segments, and the measured result for e5-small on Apple Silicon was about 3× slower than pure CPU [12]. The project's guidance is therefore to default small encoders to eps: [cpu] and benchmark before enabling an accelerator. Provider configuration like device_id, coreml_compute_units, and per-model ORT log levels are exposed, because provider chatter is exactly the kind of noise operators need to silence.
Operations and observability
The binary behaves like a service, not a demo. Request bodies are capped (max_body_mb, 413 beyond), requests have a wall-clock timeout (request_timeout_ms, 408), queues are bounded per model, and shutdown responds to SIGTERM and Ctrl-C with a graceful drain. Logging is tracing-based: info shows startup, model loading (including file size, session build time, RSS delta, and the resolved EP list), and one line per HTTP request; RUST_LOG=debug adds hub resolution and session construction detail.
Metrics are Prometheus-format on /metrics: rsinfer_http_requests_total by route and status, and rsinfer_http_request_duration_seconds, a histogram with exponential buckets from 5 ms. The docs site also ships an OpenAPI specification rendered with an interactive playground, so request and response shapes can be exercised without writing a client.
The Docker image is a multi-stage build ending in a slim Debian runtime running as a non-root user, with the EP feature set chosen by a FEATURES build argument:
docker build --build-arg FEATURES= -t rsinfer:cpu .
docker run --rm -p 8080:8080 \
-v "$PWD/config.yaml:/etc/rsinfer/config.yaml:ro" \
-v rsinfer-hf-cache:/home/rsinfer/.cache/huggingface \
rsinfer:cpu
There is no built-in auth or TLS. The deployment model is a reverse proxy in front of a private network listener, which keeps certificate and identity concerns out of the inference process.
Design tradeoffs
Compile-time execution providers. Runtime EP selection would produce a friendlier single artifact, but ONNX Runtime's prebuilt distributions make it impractical to link every provider at once. Feature flags keep binaries small and deterministic, at the cost of rebuilding for a different accelerator.
Opt-in, embedding-only batching. Always-on cross-request batching would improve average throughput while growing p99 tails, because a request's rows can span batches. Making it opt-in lets latency-sensitive deployments keep per-request forwards, and making it embedding-only avoids inventing alignment semantics for token- and pair-level outputs. The cost is configuration surface.
Model-agnostic by name, not by signature. The engine reads conventional output names (logits, sentence_embedding, last_hidden_state, token_embeddings, output0) and intersects tokenizer outputs with declared inputs rather than requiring a fixed graph signature. This is what lets optimum-cli and Xenova/onnx-community exports run without a conversion step; the tradeoff is that exotic graphs fail at load time with an explicit error instead of being adapted to.
Encoder-only NLI. Zero-shot and true/false support encoder-only MNLI exports; seq2seq BART-MNLI exports are not supported, because running them as a single classification forward is not what their graph describes.
No auth, no TLS, no Windows. The server targets the pattern "inference process behind a proxy on a trusted network", and macOS/Linux are the supported platforms.
INT8 quantization. dtype: int8 selects dynamically quantized graphs, where activation scales are per-tensor. Results shift slightly with batch padding, so the docs recommend benchmarking quality before serving a quantized model [13].
Limitations
The current scope is deliberate, and a few consequences follow from it.
Hybrid decoders are not ported. Some PII models — notably the LFM2 detector — rely on a hybrid regex layer for format-bound entities like IBANs and phone numbers. The engine's span decoder is pure BIOES, so those entities fragment without the model-specific post-processing.
Decoder-style rerankers are not yet runnable. Qwen3-Reranker exports ship a causal-LM graph requiring position_ids and populated KV-cache inputs along with quantized-only files; the yes_no scoring path is implemented and validated, but the graph plumbing is not complete.
Batch composition is opportunistic. Rows arriving after a batch is dispatched wait for the next one; there is no mid-forward insertion. Under bursty load this leaves some utilization on the table compared with continuous-batching servers.
Pool sizing is static. replicas is fixed at startup. Autoscaling session pools by load would smooth over latency spikes at the cost of memory, which the current design leaves to operators.
None of these are accidents; they are the price of a small binary, a stateless single process, and an engine that stays close to the ONNX Runtime execution model.
Conclusion
rs-infer packages four retrieval-era model families into one Rust service: Hub-declared configs, OpenAI/vLLM-compatible embeddings, Cohere/Jina-style reranking, BIOES PII redaction, and NLI-based zero-shot and true/false classification, all sharing one session pool, one batching mechanism, and one observability surface [2] [1]. The interesting engineering is not any single pipeline but the substrate beneath them — execution-provider selection at compile time, replica pools around a !Sync runtime, tokenizer-to-graph input intersection, and a batcher that amortizes forwards without adding queueing latency. The result is a self-hostable inference process that behaves like infrastructure: configured in YAML, observable in Prometheus, and replaceable one model at a time.
