Programmable and Resilient Datacenter Infrastructure for AI and Cloud

Modern NICs have evolved into shared, multi-tenant computers, integrating fixed-function and programmable accelerators over common on-chip memory, compute, DMA engines, and interconnect. As cloud providers increasingly offload stateful data-plane functions onto these accelerators, they also introduce a new class of runtime resource-contention problems. Our measurements on NVIDIA ConnectX-7 show that contention over shared on-chip memory inflates a co-located tenant’s packet-processing tail latency by up to 2.2×, even when the interfering workload executes on a different accelerator. We argue that the root cause is architectural: existing NIC control planes provision accelerator state but lack an OS-like resource-management layer that observes, arbitrates, and manages shared resources at runtime. We propose NicOS, a NIC-native resource-management architecture that separates reusable hardware mechanisms from operator-defined policies. To make the vision concrete, we instantiate NicOS for on-chip memory with Accelerator Virtual Memory (AVM), which models memory as a shared pool of named consumers and exposes a common observe-and-act interface across accelerators. A software prototype on NVIDIA BlueField-3 demonstrates feasibility; even emulated in software, NicOS mitigates the resulting tail-latency inflation. We conclude by discussing the hardware primitives and research challenges this vision requires.

An AI cluster now runs two networks. Alongside the Ethernet scale-out fabric, a scale-up fabric such as NVLink or UALink carries accelerators’ most bandwidth-intensive traffic, and it has grown from a few GPUs in a server into a rack- and cluster-scale network of hundreds of accelerators. Yet unlike the Ethernet fabric beside it, it is a network software cannot program. Accelerators inject load/store transactions that traverse links and switches with no software interposition, leaving offload, observability, and resilience beyond software’s reach. We argue that scale-up fabrics should become software-defined, as programmable switches did for Ethernet over the past decade, and that the switch is the natural place to begin. The packet match-action model of P4 and RMT does not carry over, because a scale-up switch must operate on load/store transactions rather than independent packets, stay expressive for emerging GPU communication yet bounded for line rate, and never corrupt GPU memory. We take a first step with MemOps, a transaction-aware programming model that composes bounded, verifiable operators over load/store transactions to express offload, routing, and telemetry.

Hardware-operable failures (HOFs) interrupt large language model (LLM) training but permit recovery on the same hardware without reset, repair, or replacement. Existing recovery systems nevertheless reload checkpoints, recompute lost progress, and rebuild process state, idling GPUs that could otherwise continue training. We present Leto, a fault-tolerant training system that leverages surviving hardware to enable efficient in-place recovery. Our key insight is that the state needed to resume training can be retained or prepared outside the active training process while remaining on the same hardware. Leto retains the working model state and the reusable process state, and preinitializes the remaining state in a shadow trainer. We devise two-tier erasure protection and chunk-level transactional updates to keep the retained model state recoverable and consistent, and reclaim the shadow state when active training needs its GPU memory. Evaluation on 6- and 72-GPU NVIDIA A100 clusters shows that Leto recovers 3.6–6.5× faster than the best-performing checkpointing baselines and improves productive training time by up to 13.7 percentage points. Large-scale simulation shows over 95% productive training time on a 131,072-GPU cluster.

Network bandwidth continues to scale rapidly, yet Internet data transmission performance remains constrained by the legacy 1500 B MTU. This small MTU translates high bandwidth into high packet rates that strain CPU processing at middleboxes and end hosts. While increasing the MTU could substantially improve performance, coordinating upgrades across arbitrary Internet paths is impractical. This paper presents PacketExpress, a packet processing architecture that enables networks to leverage large MTUs for Internet traffic without requiring modifications to neighboring networks. MTU-translating gateways at network borders dynamically aggregate incoming small packets into larger packets for efficient processing, then segment them back when forwarding externally. We present PXIO, a packet processing stack that leverages NIC offload capabilities to achieve high throughput. We introduce F-PMTUD, which determines the path MTU within a single round-trip time without relying on ICMP. For UDP, we present PX-caravan, a tunneling mechanism that encapsulates multiple packets to benefit from large MTUs while preserving packet boundaries. Our prototype achieves 1.47 Tbps throughput with 8 CPU cores while converting over 90% of 1500 B packets into 9000 B packets, improving middlebox performance by up to 4.8× and end-host performance by up to 3.3×.

Graph Neural Networks (GNNs) are widely used for generating expressive node representations in graph datasets. Serving GNNs on large graphs is challenging due to high communication, computation, and memory costs from constructing and executing computation graphs across large neighborhoods. Training-time approximation techniques can reduce these overheads, but they still result in high latency or accuracy loss during serving. We propose Reforge, a system designed to enable low-latency GNN serving for large graphs with minimal accuracy loss through two core ideas. First, Reforge leverages selective recomputation of precomputed embeddings to reuse computation subgraphs, selectively updating only a minority to preserve accuracy. Second, Reforge introduces computation graph parallelism, parallelizing the creation and execution of computation graphs across machines to reduce communication overhead. Evaluations on large graph datasets and GNN models demonstrate that Reforge significantly outperforms current state-of-the-art methods.

This paper proposes a systematic approach to incrementally enabling large MTUs in the Internet. We demonstrate that increasing the MTU size significantly enhances the performance of both middleboxes and end hosts. To bridge MTU mismatches at network borders, we introduce PacketExpress gateway (PXGW), an MTU-translating gateway that dynamically adjusts packet sizes for cross-traffic. PXGW merges and splits TCP payloads on the fly and tunnels UDP packets, ensuring seamless adaptation. We propose F-PMTUD, a new path MTU discovery algorithm that determines the path MTU within a single round-trip without relying on ICMP. Our preliminary evaluation shows that the PXGW prototype achieves 1.46 Tbps of packet forwarding throughput using only 8 CPU cores. After dynamic conversion, 95% of transmitted TCP packets are 9000 B jumbo frames, indicating that most flows were effectively converted into large segments, thereby demonstrating the system’s efficiency and scalability. We also find that large-MTU packets made available via PXGW enhance end-host performance by up to 2.5×.

Emerging SmartNICs, from different vendors or generations, exhibit substantial differences in hardware parallelism and memory interconnects. These variations make porting programs across NICs complex and time-consuming, requiring programmers to refactor code for performance per NIC’s characteristics. We argue that an ideal SmartNIC compilation framework should allow developers to write target-independent programs, with the compiler managing cross-NIC porting and performance optimization. We present Alkali, which achieves this by proposing a new intermediate representation for building flexible compiler infrastructure for multiple NIC targets and developing a new iterative parallelism optimization algorithm that automatically ports and parallelizes the input programs based on the target NIC’s hardware characteristics. Experiments across NIC applications show that Alkali enables developers to write portable, high-performance NIC programs. Our compiler optimization passes can automatically port these programs and make them run efficiently across all targets, achieving performance within 9.8% of hand-tuned expert implementations.

SmartNICs provide in-network computing capabilities in today’s data centers, benefiting a range of applications. Efficient SmartNIC-assisted solutions require programmers to understand SmartNIC architecture, refactor application logic, and relate executions to traffic. A high-level SmartNIC performance model can decouple hardware from its offloaded software, simplifying development. However, prior models cannot dissect the complexity of SmartNIC-offloaded programs, capture the nondeterministic computation and I/O overlap, or diverse traffic profiles. LogNIC systematically analyzes the performance of a SmartNIC-offloaded program using a packet-centric approach, examining packet traversal over heterogeneous domains, interconnects, and memory subsystems. It abstracts device details, represents programs as execution graphs, retains configurable parameters, and generates latency/throughput estimates for a given traffic profile. Extensions handle multi-tenancy, traffic interleaving, and accelerator peculiarity. Our evaluations show LogNIC estimates performance bounds, explores software optimization strategies, and provides guidelines for hardware designs.

The promise of in-network computing remains unrealized in practice, as serving concurrent stateful applications on a programmable switch is challenging due to limited on-chip resources. We argue for an on-rack switch resource augmentation architecture that augments a switch with other programmable network hardware, such as smart NICs, on the same rack. We design and implement ExoPlane, an operating system supporting concurrent applications via pragmatic resource augmentation. ExoPlane includes a practical runtime operating model and state abstraction to manage application state correctly across devices with minimal performance and resource overheads. Our evaluation shows ExoPlane provides low latency, scalable throughput, and fast failover, achieving these with small overhead and minimal application changes.

Network operators need to run diverse measurement tasks on switches for traffic engineering or anomaly detection. Prior work considers only running a single sketch, lacking efficient support for ensembles. This work presents the design and implementation of Sketchovsky, a cross-sketch optimization and composition framework. We identify five cross-sketch optimization building blocks to reduce critical hardware resources. Efficient heuristics select and apply these blocks for arbitrary ensembles. Sketchovsky automatically generates composed code for the hardware compiler. Evaluation shows Sketchovsky makes ensembles up to 18 sketches feasible, reducing up to 45% of critical hardware resources.

Designing networked systems that exploit heterogeneous dataplanes—such as splitting processing between a PISA switch and x86 CPUs—can improve performance and efficiency. Programming multiple hardware targets is challenging because of platform-specific languages. Existing write-once, run-anywhere compilers cannot fully tune NFs for performance across different targets. We explore compiler ideas to exhaustively search for different hardware mappings, tunable for objectives like minimizing memory or maximizing throughput. Our prototype SyNAPSE uses component-based synthesis, supporting x86 and Tofino platforms. Compared to a baseline compiler, SyNAPSE finds thousands of deployment choices, including options reducing controller traffic by an order of magnitude, or halving memory use.

Sketching algorithms provide accurate network measurement with low resources. Programmable switches are attractive for such sketches, but current hardware implementations are inefficient or infeasible. Our contributions: (1) systematic analysis of bottlenecks of hardware sketches, (2) practical, correct-by-construction optimization techniques, (3) SketchLib, an easy-to-use library to help developers efficiently implement sketches in hardware. Evaluation on state-of-the-art sketches shows SketchLib reduces hardware resource footprint up to 96% without impacting fidelity.

SwiSh is a distributed shared state management layer for data-plane P4 programs. It enables running scalable stateful distributed network functions entirely in the data-plane. Several shared variable schemes differ in consistency, performance, and in-switch implementation complexity. We introduce the Strong Delayed-Writes (SDW) protocol, which offers consistent snapshots of shared data-plane objects with strong linearizability, enabling distributed concurrent sketches with precise error bounds. We implement strong, eventual, and SDW consistency protocols in Tofino switches, and compare them in microbenchmarks and network functions (NAT, DDoS detector, and rate limiter). Distributed state management in the data plane is practical and outperforms centralized solutions by up to four orders of magnitude in update throughput and replication latency.

Running datacenter functions (e.g., NATs, load balancers, monitoring) on programmable switches has shown performance benefit, but key fault tolerance is missing. Since networks are no longer stateless, endpoint-only recovery does not suffice. We design and implement RedPlane, a fault-tolerant state store for in-switch applications, providing consistent state access even after switch failure or rerouting. We address challenges for a practical, provably correct replication protocol and implementation in the switch data plane. Evaluation shows RedPlane adds negligible overhead and enables rapid recovery.

Sketching algorithms are attractive for telemetry on programmable hardware switches due to accuracy guarantees and compact data structures. In practice, their implementations can have large (up to 94x) accuracy drops versus theory. Delays from pulling and resetting data plane state induce degradation. We design solutions to reduce these delays and almost eliminate inaccuracy in existing workflows.

Programmable switches are attractive for network functions (NFs), but memory limits stymie support for state-intensive NFs (cloud-scale NATs or load balancers with millions of entries). We introduce TEA (Table Extension Architecture), leveraging DRAM on servers to provide a virtual table abstraction for NFs on switches. This allows switch ASICs to access external DRAM directly from the data plane, without CPUs. We address design and implementation challenges and show with a Tofino-based implementation that TEA achieves low and predictable latency (1.8-2.2 μs) for table lookups, and throughput scaling with more servers (138 M lookups/second with 8 servers).

Reconfigurable datacenter networks (RDCNs) have traditional packet switches augmented with high-bandwidth reconfigurable circuits, assigned to rack pairs per schedule. TCP flows must ramp up sending rates fast when circuits become available. Past TCP variants perform well for ms-scale reconfiguration, but modern RDCNs reconfigure in 20 μs and have short configuration epochs. Existing TCP variants cannot ramp quickly enough. We address this by (1) in-network output queue resizing to prebuffer packets and (2) endpoint adjustment of the congestion window (cwnd) using explicit circuit feedback. Using our RDCN emulator Etalon, we show that combining these increases circuit utilization by 1.91x with only a 1.20x latency increase.

Large-scale cloud applications are using containerization for efficiency and isolation. Data-intensive apps increasingly adopt RDMA for high networking performance. There is an inevitable collision between these. We present FreeFlow, a software-based RDMA virtualization framework for container clouds, using commodity RDMA NICs. FreeFlow meets cloud requirements (isolation, portability, controllability) and is transparent to applications. It provides performance close to bare-metal RDMA with low CPU overhead. With TensorFlow and Spark, FreeFlow gives nearly the same application performance as bare-metal.

Switches are attractive for networking applications (e.g., load balancing, virtual switching) because of their location and high processing rate. Advances in programmable switch ASICs open opportunities for offloading application logic, but limited switch memory remains a challenge. We envision enabling switches to access remote memory from data planes, improving performance for many applications. We design three remote memory primitives using RDMA operations and demonstrate feasibility using a prototype.

Data center storage systems perform replicated transactions for availability and integrity, but still face high tail latency. The root problem is CPU use in the critical path. HyperLoop removes CPU from the critical path by offloading replicated transactions to RDMA NICs, using non-volatile memory for storage. We develop NIC offloading primitives that provide memory operations and guarantee ACID properties, all without CPU. Popular storage applications are easily optimized using the primitives; evaluation shows HyperLoop can reduce 99th percentile latency by about 800x, with near 0% CPU consumption on replicas.