Faithful Surrogates for Production
A configuration, placement, or policy is only as good as the testbed it is evaluated in, yet production, the only fully faithful testbed, is too costly to measure and too private to share. We build cheap, faithful surrogates for both what is deployed (hardware, software, and models) and what it is deployed into (traffic and co-located tenants), from data we can actually get, such as operation profiles and resource-usage traces, and through high-fidelity simulation.
Our Work
What is deployed. LLM serving platforms choose configurations, place instances, and route requests by predicting latency from per-operation latency models. Building these models means profiling every GPU operation across token counts and batch sizes, from scratch for every new GPU, engine, or model, at a cost of tens of GPU-hours. Dooly (NeurIPS’26) observes that every input dimension of an operation comes either from the model configuration or from the request, so operations recur across configurations: 36 configurations need only 100 unique operations. It labels each dimension with taint propagation in a single inference pass, runs stateful operations such as attention inside the engine’s own setup, and profiles each operation once, cutting profiling GPU-hours by 56%. Its simulator, DoolySim, predicts serving latency within 5% error for time to first token and 8% for time per output token, accurate enough to pick the same best configuration as real measurements.
What it is deployed into. Cloud applications share servers with noisy neighbors that can slow them by 30–70%, yet other tenants’ workloads are private and contention keeps changing. Mimesys (OSDI’26) turns resource-usage traces, which operators already collect, into executable workloads that reproduce realistic contention without exposing tenant code or data. It represents a workload as a composition of resource stressors, learns the mapping from traces to compositions with a diffusion model conditioned on prior state, and aligns the model with real applications by executing generated workloads on hardware. Mimesys matches real slowdowns within 8.3 points on average, vs. 19.4 for the next best approach. Production traffic is similarly hard to obtain: TraceLLM (EMNLP’25) fine-tunes LLMs to generate realistic microservice call-graph traces that can replace real data in microservice management tasks.
Lessons. Three lessons recur across these systems. First, compose surrogates from primitives: GPU operations in Dooly, parameterized resource stressors in Mimesys. Second, identity and state are what make composition hard, so both systems track them explicitly: Dooly with taint labels and engine-native setup, Mimesys by conditioning each window on the previous one. Third, keep the real system in the loop: Dooly reuses the serving engine’s own setup and scheduler, while Mimesys executes generated workloads on real hardware and rewards matching traces. We learn only where the mapping cannot be recovered directly; Dooly’s structure is exact, while Mimesys’s is nonlinear.
Ongoing Directions
- Surrogates that carry over to new hardware. Today Dooly profiles again for each GPU type, and Mimesys workloads trained on one CPU generation lose accuracy on another.
- Configuration under realistic contention. Combining both halves: evaluating serving configurations next to realistic co-located workloads.
- High-fidelity simulators. Surrogates for hardware and deployments that do not exist yet, such as scale-up fabric designs and edge computing.
- Profile-driven model serving. Using efficient profiles to drive serving decisions such as configuration selection, instance placement, and request routing.
Publications
Selecting optimal LLM inference configuration requires evaluation across hardware, serving engines, attention backends, and model architectures. Profile-based simulators are standard tools but hardcode operations to specific configurations and re-profile from scratch, making exploration expensive. Dooly exploits structural understanding that input dimensions are fixed by model configuration or determined by requests. The system performs a single inference pass, labels input dimensions via taint propagation, and selectively profiles only missing operations from its latency database. Stateful operations like attention are isolated using the serving engine’s initialization code. Latency regression models built from the database serve as a drop-in backend for existing simulators. Across two GPU platforms, three attention backends, and diverse model architectures, Dooly achieves simulation accuracy within 5% MAPE for TTFT and 8% for TPOT while reducing profiling GPU-hours by 56.4% across 12 models compared to existing approaches.
Testing applications under realistic resource contention is challenging because production workloads are often inaccessible due to privacy and proprietary concerns. Existing approaches either use simplistic resource stressors that fail to capture temporal dynamics and multi-resource interactions, rely on limited benchmark suites, or require exhaustive per-application profiling. This paper explores an alternative direction: Synthesizing executable workloads from resource usage traces to reproduce realistic colocation scenarios. We present Mimesys, a system that transforms time-series resource usage traces into executable workloads that emulate resource contention patterns. Mimesys represents emulated workloads as compositions of resource stressors and employs a diffusion-based generative model to learn the inverse mapping from traces to stressor compositions. We introduce two key ideas: state-aware conditioning that conditions generation on both target traces and prior system state to capture temporal dependencies, and execution-driven alignment that adapts the model to real application patterns using direct execution feedback without requiring ground-truth labels. Our evaluation shows that Mimesys achieves up to 5.5× higher trace similarity and reproduces application performance under contention 2.6× more accurately than baselines.
Workload traces are essential to understand complex computer systems’ behavior and manage processing and memory resources. Since real-world traces are hard to obtain, synthetic trace generation is a promising alternative. This paper proposes a first-of-a-kind approach that relies on training a large language model (LLM) to generate synthetic workload traces, specifically microservice call graphs. To capture complex and arbitrary hierarchical structures and implicit constraints in such traces, we show how to fine-tune LLMs to generate recursively, making call graph generation a sequence of easier steps. To further enforce learning constraints in traces and generate uncommon situations, we argue for applying additional instruction tuning steps to align our model with the desired trace features. Our evaluation results show that we can generate diverse realistic traces under various conditions and outperform existing methods in accuracy and validity. We demonstrate that our synthetically generated traces can effectively replace real data to optimize important microservice management tasks. Additionally, our model adapts to downstream trace-related tasks, such as predicting key trace features and infilling missing data.
Funding
- Transcraft: An End-to-End Framework for Network Transport Design and Implementation for Emerging Applications. NSF CNS, Award 2403026.
- LDOS: Learning Directed Operating System. NSF Expeditions in Computing, Award 2326576.