Latency Budgets for Real-Time AI Orchestration

Practical context

A proposed method for measuring the stages that determine interactive response time.

Assess whether a target Apple workstation can use the optional Metal DOT path.

Implementation and research boundary

Separate context selection, tool transport, model execution and rendering when designing a latency study. Record native versus fallback execution and define the workload before assigning a budget.

The inspected implementation supplies bounded retrieval and explicit backend reporting. It does not supply a published end-to-end latency result for this application, and this note sets no numeric performance promise.

A measurement model with explicit boundaries

For a strictly sequential request, elapsed time can be decomposed into transport, retrieval, model execution, policy checks and presentation. Overlapping operations instead follow a critical path; summing every component would double-count concurrent work. Define whether timing ends at first visible output, the final token or a durable receipt. The simple sequential equation is a measurement model, not a claim that this runtime schedules every stage this way.

Proposed experiment

Pin source and configuration, identify machine and backend status, and separate warm from cold execution. Use a fixed corpus and request set with both empty and substantial evidence. Run each path repeatedly, retain failures as observations and report sample count, median and tail distributions. Measure native and CPU fallback separately; a Metal-capable build may still execute a fallback. Keep network and provider work labeled because local template latency cannot stand in for remote model generation.

Interpretation and acceptance

Set a pilot-specific response-time requirement before collecting results. Report stage timings alongside end-to-end elapsed time, with uncertainty and the chosen concurrency level. Inspect cancellation, timeout and saturation behavior as well as successful requests. Do not multiply an assumed concurrency by GPU or shard counts to infer throughput: the inspected reference system does not establish sharding or linear scaling. A partner can use this method to compare two versions on its workload; this note provides no measured latency, capacity or comparative advantage.

Evaluation before adoption

Inspect backend status on the intended machine and compare native and fallback results. Publish latency or throughput only with a reproducible harness, workload, hardware and methodology.