Paper List for Machine Learning Systems

A curated list of machine learning systems papers published in major CS conferences (plus some workshops and journals).

Survey papers are annotated with [Survey 🔍].

arXiv Papers

For arXiv preprints, please see README_arxiv.md.

Table of Contents

• Paper List for Machine Learning Systems
  • arXiv Papers
  • Table of Contents
  • Data Processing
  • Training System
    • ML job analysis on GPU clusters
    • Resource scheduling
    • Distributed training
    • RL post-training
    • AutoML
    • GNN training
  • Inference System
  • Attention Optimization
  • Mixture of Experts (MoE)
  • Communication Optimization & Network Infrastructure for Distributed ML
  • Fault tolerance & Straggler mitigation
  • GPU Memory Management & Optimization
  • GPU Sharing
  • Compiler
  • GPU Kernel Optimization
  • LLM Long Context
  • Model Compression
  • Federated Learning
  • Privacy-Preserving ML
  • ML APIs & Application-Side Optimization
  • ML for Systems
  • Energy Efficiency
  • Retrieval-Augmented Generation (RAG)
  • Simulation
  • Systems for Agentic AI
  • Multimodal
  • Hybrid LLMs
  • Others

Data Processing

Data pipeline optimization

• [VLDB'25] cedar: Composable and Optimized Machine Learning Input Data Pipelines
• [HotInfra'24] Lotus: Characterize Architecture Level CPU-based Preprocessing in Machine Learning Pipelines
• [MLSys'22] Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines
• [ISCA'22] Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training
• [SIGMOD'22] Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines
• [VLDB'21] Analyzing and Mitigating Data Stalls in DNN Training
• [VLDB'21] tf.data: A Machine Learning Data Processing Framework

Preprocessing stalls

• [FAST'26] Preparation Meets Opportunity: Enhancing Data Preprocessing for ML Training With Seneca
• [ATC'24] Pecan: Cost-Efficient ML Data Preprocessing with Automatic Transformation Ordering and Hybrid Placement
• [HotStorage'24] A Selective Preprocessing Offloading Framework for Reducing Data Traffic in DL Training
• [VLDB'24] FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation
• [CVPR'23] FFCV: Accelerating Training by Removing Data Bottlenecks
• [RecSys'23] InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models
• [SIGMOD'23] GoldMiner: Elastic Scaling of Training Data Pre-Processing Pipelines for Deep Learning
• [VLDB'23] FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline
• [SoCC'23] tf.data service: A Case for Disaggregating ML Input Data Processing
• [ATC'22] Cachew: Machine Learning Input Data Processing as a Service
• [OSDI'22] Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters
• [ICPP'19] DLBooster: Boosting End-to-End Deep Learning Workflows with Offloading Data Preprocessing Pipelines

Fetch stalls (I/O)

• [TACO'23] Fastensor: Optimise the Tensor I/O Path from SSD to GPU for Deep Learning Training
• [ICPP'22] Lobster: Load Balance-Aware I/O for Distributed DNN Training
• [SC'21] Clairvoyant Prefetching for Distributed Machine Learning I/O

Specific workloads (GNN, DLRM)

• [VLDB'25] Eliminating Data Processing Bottlenecks in GNN Training over Large Graphs via Two-level Feature Compression
• [ISCA'24] PreSto: An In-Storage Data Preprocessing System for Training Recommendation Models
• [MLSys'23] RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructure
• [ASPLOS'22] RecShard: statistical feature-based memory optimization for industry-scale neural recommendation
• [RecSys'23] InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models
• [SOSP'23] Bagpipe: Accelerating Deep Recommendation Model Training
• [SOSP'23] gSampler: General and Efficient GPU-based Graph Sampling for Graph Learning
• [NSDI'23] BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing
• [DAC'22] A Joint Management Middleware to Improve Training Performance of Deep Recommendation Systems with SSDs
• [VLDB'22] Accelerating Recommendation System Training by Leveraging Popular Choices

Caching and distributed storage for ML training

• [ATC'25] HyCache: Hybrid Caching for Accelerating DNN Input Preprocessing Pipelines
• [ICDE'25] MLKV: Efficiently Scaling up Large Embedding Model Training with Disk-based Key-Value Storage
• [TPDS'23] High-Level Data Abstraction and Elastic Data Caching for Data-Intensive AI Applications on Cloud-Native Platforms
• [SOSP'23] UGACHE: A Unified GPU Cache for Embedding-based Deep Learning
• [ATC'23] Tectonic-Shift: A Composite Storage Fabric for Large-Scale ML Training
• [EuroSys'23] SiloD: A Co-design of Caching and Scheduling for Deep Learning Clusters
• [FAST'23] SHADE: Enable Fundamental Cacheability for Distributed Deep Learning Training
• [HPCA'23] iCACHE: An Importance-Sampling-Informed Cache for Accelerating I/O-Bound DNN Model Training
• [NeurIPS'22] A Deep Learning Dataloader with Shared Data Preparation
• [CLUSTER'22] Hvac: Removing I/O Bottleneck for Large-Scale Deep Learning Applications
• [ICDE'22] Fluid: Dataset Abstraction and Elastic Acceleration for Cloud-native Deep Learning Training Jobs
• [ATC'21] Refurbish Your Training Data: Reusing Partially Augmented Samples for Faster Deep Neural Network Training
• [FAST'20] Quiver: An Informed Storage Cache for Deep Learning
• [ICPP'20] DIESEL: A Dataset-Based Distributed Storage and Caching System for Large-Scale Deep Learning Training
• [HotCloud'19] The Case for Unifying Data Loading in Machine Learning Clusters

LLM data plane

• [SIGMOD'26] Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
• [EMNLP'25] Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
• [ICDE'25] Training Data Distribution Estimation for Optimized Pre-Training Data Management

Data formats

• [ECCV'22] L3: Accelerator-Friendly Lossless Image Format for High-Resolution, High-Throughput DNN Training
• [VLDB'21] Progressive compressed records: Taking a byte out of deep learning data

Data pipeline fairness and correctness

• [CIDR'21] Lightweight Inspection of Data Preprocessing in Native Machine Learning Pipelines

Data labeling automation

• [VLDB'18] Snorkel: Rapid Training Data Creation with Weak Supervision

Training System

ML job analysis on GPU clusters

• [ICSE'24] An Empirical Study on Low GPU Utilization of Deep Learning Jobs
• [NSDI'24] Characterization of Large Language Model Development in the Datacenter
• [NSDI'22] MLaaS in the wild: workload analysis and scheduling in large-scale heterogeneous GPU clusters (PAI)
• [ATC'19] Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (Philly)

Resource scheduling

• [EuroSys'26] Suika: Efficient and High-quality Re-scheduling of 3D-parallelized LLM Training Jobs in Shared Clusters

• [EuroSys'26] Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design

• [EuroSys'26] Bridging the GPU Utilization Gap: Predictive Multi-Dimensional Resource Scheduling for AI Workloads

• [EuroSys'26] AdaGen: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Inference Serving

• [OSDI'25] Decouple and Decompose: Scaling Resource Allocation with DeDe

• [SoCC'25] Cuckoo: Deadline-Aware Job Packing on Heterogeneous GPUs for DL Model Training

• [EuroSys'25] Eva: Cost-Efficient Cloud-Based Cluster Scheduling

• [TACO'24] Taming Flexible Job Packing in Deep Learning Training Clusters

• [SoCC'24] Kale: Elastic GPU Scheduling for Online DL Model Training

• [SC'24] PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU Clusters

• [OSDI'24] MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale

• [ASPLOS'24] Heet: Accelerating Elastic Training in Heterogeneous Deep Learning Clusters

• [Middleware'24] Optimal Resource Efficiency with Fairness in Heterogeneous GPU Clusters

• [IPDPS'24] Hadar: Heterogeneity-Aware Optimization-Based Online Scheduling for Deep Learning Cluster

• [EuroSys'24] Blox: A Modular Toolkit for Deep Learning Schedulers

• [NSDI'24] Swing: Short-cutting Rings for Higher Bandwidth Allreduce

• [NSDI'24] Towards Domain-Specific Network Transport for Distributed DNN Training

• [NSDI'24] Vulcan: Automatic Query Planning for Live ML Analytics

• [NSDI'24] CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters

• [Survey :mag:] [ACM CSUR'23] Deep Learning Workload Scheduling in GPU Datacenters: A Survey

• [SC'23] EasyScale: Accuracy-consistent Elastic Training for Deep Learning

• [ICPP'23] CoTrain: Efficient Scheduling for Large-Model Training upon GPU and CPU in Parallel

• [ICPP'23] Embracing Uncertainty for Equity in Resource Allocation in ML Training

• [SOSP'23] Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling

• [NSDI'23] Shockwave: Proactive, Fair, and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning

• [EuroSys'23] SiloD: A Co-design of Caching and Scheduling for Deep Learning Clusters

• [EuroSys'23] Lyra: Elastic Scheduling for Deep Learning Clusters

• [EuroSys'23] ElasticFlow: An Elastic Serverless Training Platform for Distributed Deep Learning

• [ASPLOS'23] Lucid: A Non-intrusive, Scalable and Interpretable Scheduler for Deep Learning Training Jobs

• [SoCC'22] ESCHER: Expressive Scheduling with Ephemeral Resources

• [NSDI'22] MLaaS in the wild: workload analysis and scheduling in large-scale heterogeneous GPU clusters (PAI)

• [OSDI'22] Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters (Synergy)

• [SIGCOMM'22] Multi-resource interleaving for deep learning training (Muri)

• [MLSys'21] Wavelet: Efficient DNN Training with Tick-Tock Scheduling

• [SoCC'21] Chronus: A Novel Deadline-aware Scheduler for Deep Learning Training Jobs

• [SC'21] Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters (Helios)

• [OSDI'21] Privacy Budget Scheduling (DPF)

• [NSDI'21] Elastic Resource Sharing for Distributed Deep Learning (AFS)

• [OSDI'21] Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning

• [EuroSys'20] Balancing efficiency and fairness in heterogeneous GPU clusters for deep learning (GandivaFair)

• [NSDI'20] Themis: Fair and Efficient GPU Cluster Scheduling

• [OSDI'20] HiveD: Sharing a GPU Cluster for Deep Learning with Guarantees

• [OSDI'20] Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads (Gavel)

• [EuroSys'20] AlloX: Compute Allocation in Hybrid Clusters

• [MLSys'20] Resource Elasticity in Distributed Deep Learning

• [NSDI'19] Tiresias: A GPU Cluster Manager for Distributed Deep Learning

• [ATC'19] Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (Philly)

• [EuroSys'18] Optimus: an efficient dynamic resource scheduler for deep learning clusters

• [OSDI'18] Gandiva: Introspective Cluster Scheduling for Deep Learning

Distributed training

• [MLSys'26] FlexTrain: Scalable Hybrid-Parallel Training with Elastic Resource Utilization and Consistent Accuracy

• [MLSys'26] HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware

• [MLSys'26] DreamDDP: Accelerating Low-Bandwidth Geo-Distributed LLM Training with Layer-wise Partial Synchronization

• [ICML'26] One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

• [ICML'26] When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

• [ICML'26] AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training

• [EuroSys'26] HARP: Orchestrating Automated Parallel Training on Heterogeneous GPU Clusters

• [HPCA'26] Towards Compute-Aware In-Switch Computing for LLMs Tensor-Parallelism on Multi-GPU Systems

• [HPCA'26] AutoHAAP: Automated Heterogeneity-Aware Asymmetric Partitioning for LLM Training

• [HPCA'26] WATOS: Efficient LLM Training Strategies and Architecture Co-exploration for Wafer-scale Chip

• [ASPLOS'26] SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips

• [NeurIPS'25] Synergistic Tensor and Pipeline Parallelism

• [NeurIPS'25] First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training

• [SC'25] Hypertron: Efficiently Scaling Large Models by Exploring High-Dimensional Parallelization Space

• [CLUSTER'25] BMPipe: Bubble-Memory Co-Optimization Strategy Planner for Very-Large DNN Training

• [OSDI'25] WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training

• [ISCA'25] FRED: A Wafer-scale Fabric for 3D Parallel DNN Training

• [ISCA'25] MeshSlice: Efficient 2D Tensor Parallelism for Distributed DNN Training

• [ISCA'25] Scaling Llama 3 Training with Efficient Parallelism Strategies

• [ICML'25] HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training

• [MLSys'25] Radius: Range-based Gradient Sparsity for Large Foundation Model Pre-training

• [ICLR'25] TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training

• [INFOCOM'25] Espresso: Cost-Efficient Large Model Training by Exploiting GPU Heterogeneity in the Cloud

• [TPDS'25] HpT: Hybrid Acceleration of Spatio-Temporal Attention Model Training on Heterogeneous Manycore Architectures

• [ASPLOS'25] GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism

• [ASPLOS'25] FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism

• [ASPLOS'25] Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling

• [EuroSys'25] JABAS: Joint Adaptive Batching and Automatic Scaling for DNN Training on Heterogeneous GPUs

• [TPDS'24] UMPIPE: Unequal Microbatches-Based Pipeline Parallelism for Deep Neural Network Training

• [Survey :mag:] [ACM CSUR'24] Resource-efficient Algorithms and Systems of Foundation Models: A Survey

• [SOSP'24] Uncovering Nested Data Parallelism and Data Reuse in DNN Computation with FractalTensor

• [SOSP'24] Enabling Parallelism Hot Switching for Efficient Training of Large Language Models

• [TACO'24] ATP: Achieving Throughput Peak for DNN Training via Smart GPU Memory Management

• [NeurIPS'24] Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models

• [NeurIPS'24] SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation

• [SC'24] Accelerating Distributed DLRM Training with Optimized TT Decomposition and Micro-Batching

• [SC'24] Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers

• [SoCC'24] Distributed training of large language models on AWS Trainium

• [TPDS'24] AutoDDL: Automatic Distributed Deep Learning With Near-Optimal Bandwidth Cost

• [SOSP'24] Enabling Parallelism Hot Switching for Efficient Training of Large Language Models

• [SOSP'24] TENPLEX: Changing Resources of Deep Learning Jobs using Parallelizable Tensor Collections

• [ICPP'24] AutoPipe: Automatic Configuration of Pipeline Parallelism in Shared GPU Cluster

• [COLM'24] LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers

• [OSDI'24] nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training

• [ATC'24] Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Parallelism

• [ATC'24] Metis: Fast Automatic Distributed Training on Heterogeneous GPUs

• [ATC'24] FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences

• [ATC'24] OPER: Optimality-Guided Embedding Table Parallelization for Large-scale Recommendation Model

• [HPDC'24] DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Models

• [ICML'24] Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training

• [ICML'24] Integrated Hardware Architecture and Device Placement Search

• [MLSys'24] DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines

• [MLSys'24] Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping

• [MobiCom'24] Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices

• [EuroSys'24] DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines

• [EuroSys'24] ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling

• [EuroMLSys@EuroSys'24] ML Training with Cloud GPU Shortages: Is Cross-Region the Answer?

• [ASPLOS'24] AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning

• [ASPLOS'24] PrimePar: Efficient Spatial-temporal Tensor Partitioning for Large Transformer Model Training

• [EuroSys'24] Aceso: Efficient Parallel DNN Training through Iterative Bottleneck Alleviation

• [NSDI'24] MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

• [NSDI'24] DISTMM: Accelerating Distributed Multi-modal Model Training

• [NSDI'24] Accelerating Neural Recommendation Training with Embedding Scheduling

• [NSDI'24] Resiliency at Scale: Managing Google’s TPUv4 Machine Learning Supercomputer

• [NSDI'24] QuickUpdate: a Real-Time Personalization System for Large-Scale Recommendation Models

• [NSDI'24] Scaling Large Language Model Training to More Than 10,000 GPUs

• [TKDE'24] Improving Automatic Parallel Training via Balanced Memory Workload Optimization

  • extended version of Galvatron (VLDB'23)
  • arxiv version (2023): link

• [ICLR'24] Zero Bubble (Almost) Pipeline Parallelism

• [ICLR'24] CO2: Efficient Distributed Training with Full Communication-Computation Overlap

• [AAMAS'24] Holonic Learning: A Flexible Agent-based Distributed Machine Learning Framework

• [VLDB'24] Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads

• [HPCA'24] Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search

• [NSDI'24] Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances

• [EuroSys'24] HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

• [ICPP'23] Mercury: Fast and Optimal Device Placement for Large Deep Learning Models

• [IPDPS'23] MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

• [CLUSTER'23] Prophet: Fine-grained Load Balancing for Parallel Training of Large-scale MoE Models

• [NeurIPS'23] ASPEN: Breaking Operator Barriers for Efficient Parallelization of Deep Neural Networks

• [NeurIPS'23] DeepPCR: Parallelizing Sequential Operations in Neural Networks

• [DAC'23] MixPipe: Efficient Bidirectional Pipeline Parallelism for Training Large-Scale Models

• [SC'23] Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency

• [SOSP'23] PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation

• [SOSP'23] Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates

• [TPDS'23] Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models

• [MICRO'23] Grape: Practical and Efficient Graphed Execution for Dynamic Deep Neural Networks on GPUs

• [HPCA'23] Phloem: Automatic Acceleration of Irregular Applications with Fine-Grain Pipeline Parallelism

• [ACL'23] Sequence Parallelism: Long Sequence Training from System Perspective

• [CCGrid'23] A Deep Learning Pipeline Parallel Optimization Method

• [OSDI'23] MGG: Accelerating Graph Neural Networks with Fine-Grained Intra-Kernel Communication-Computation Pipelining on Multi-GPU Platforms

• [ATC'23] Accelerating Distributed MoE Training and Inference with Lina

• [ATC'23] SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization

• [ATC'23] MSRL: Distributed Reinforcement Learning with Dataflow Fragments

• [Survey :mag:] [TPDS'23] A Survey on Auto-Parallelism of Large-Scale Deep Learning Training

• [ICML'23] SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient

• [ICML'23] BPipe: Memory-Balanced Pipeline Parallelism for Training Large Language Models

• [ICS'23] A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training

• [NSDI'23] TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

• [NSDI'23] Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs

• [NSDI'23] ARK: GPU-driven Code Execution for Distributed Deep Learning

• [MLSys'23] On Optimizing the Communication of Model Parallelism

• [MLSys'23] MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

• [MLSys'23] Tutel: Adaptive Mixture-of-Experts at Scale

• [TPDS'23] Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

• [PPoPP'23] Elastic Averaging for Efficient Pipelined DNN Training

• [PPoPP'23] Efficient All-Reduce for Distributed DNN Training in Optical Interconnect Systems

• [VLDB'23] MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud

• [VLDB'23] Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

• [ASPLOS'23] Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers

• [ASPLOS'23] Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression

• [ICPP'22] Tesseract: Parallelize the Tensor Parallelism Efficiently

• [MLSys'22] Synthesizing optimal parallelism placement and reduction strategies on hierarchical systems for deep learning

• [NeurIPS'22] Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees

• [SoCC'22] Accelerating Large-Scale Distributed Neural Network Training with SPMD Parallelism

• [MLSys'22] Pathways: Asynchronous distributed dataflow for ML

• [MLSys'22] SRIFTY: Swift and Thrifty Distributed Neural Network Training on the Cloud

• [MLSys'22] Efficient Strong Scaling Through Burst Parallel Training

• [EuroSys'22] Varuna: scalable, low-cost training of massive deep learning models

• [ATC'22] Whale: Efficient Giant Model Training over Heterogeneous GPUs

• [NeurIPS'22] AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness

• [PPoPP'22] FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models

• [ICML'22] DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

• [ICML'22] GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

• [HPDC'22] Hare: Exploiting Inter-job and Intra-job Parallelism of Distributed Machine Learning on Heterogeneous GPUs

• [OSDI'22] Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

• [NSDI'22] Accelerating Collective Communication in Data Parallel Training across Deep Learning Frameworks

• [JMLR'21] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

• [TPDS'21] TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

• [ATC'21] Fine-tuning giant neural networks on commodity hardware with automatic pipeline model parallelism

• [SIGMOD'21] Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce(#210-communication-optimization)]

• [MLSys'21] PipeMare: Asynchronous Pipeline Parallel DNN Training

• [ICLR'21] GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

• [NeurIPS'21] Piper: Multidimensional Planner for DNN Parallelization

• [ICML'21] Memory-Efficient Pipeline-Parallel DNN Training

• [ICML'21] TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models

• [ICML'21] PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models

• [SC'21] Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines

• [SC'21] Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (PTD-P or Megatron-LM v2)

• [FAST'21] Behemoth: A Flash-centric Training Accelerator for Extreme-scale DNNs

• [PPoPP'21] DAPPLE: a pipelined data parallel approach for training large models

• [VLDB'21] Distributed Deep Learning on Data Systems: A Comparative Analysis of Approaches

• [HPCA'20] AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators

• [NeurIPS'20] Efficient Algorithms for Device Placement of DNN Graph Operators

• [KDD'20 Tutorial] DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

• [VLDB'20] PyTorch Distributed: Experiences on Accelerating Data Parallel Training

• [OSDI'20] A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (BytePS)

• [SOSP'19] PipeDream: Generalized Pipeline Parallelism for DNN Training

• [NeurIPS'20] Language Models are Few-Shot Learners

• [HPCA'19] HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array

• [IEEE MICRO'19] Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training

• [MLSys'19] Beyond data and model parallelism for deep neural networks (FlexFlow)

• [MLSys'19] TicTac: Accelerating Distributed Deep Learning with Communication Scheduling

• [EuroSys'19] Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks

• [EuroSys'19] Supporting Very Large Models using Automatic Dataflow Graph Partitioning (Tofu)

• [SOSP'19] A Generic Communication Scheduler for Distributed DNN Training Acceleration

• [NeurIPS'19] Mesh-TensorFlow: Deep Learning for Supercomputers

• [NeurIPS'19] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

• [ICML'18] Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks

• [Survey :mag:] [IJCAI'22] Survey on Effcient Training of Large Neural Networks

• [Survey :mag:] [ACM CSUR'19] Demystifying Parallel and Distributed Deep Learning

• [Survey :mag:] [ACM CSUR'19] Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools

RL post-training

• [COLM'26] QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization
• [ICML'26] TVCache: A Stateful Tool-Value Cache for Post-Training LLM Agents
• [ICML'26] Knapsack RL: Compute-Efficient Reinforcement Learning via Heterogeneous Rollout Allocation
• [ICML'26] DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
• [ASPLOS'26] History Doesn't Repeat Itself but Rollouts Rhyme: Accelerating Reinforcement Learning with RhymeRL
• [ICLR'26] Revisiting Parameter Server in LLM Post-Training
• [NeurIPS'25] Greedy Sampling Is Provably Efficient for RLHF
• [NeurIPS'25] AReaL: Asynchronous Reinforcement Learning for Efficient and Scalable Language Reasoning
• [COLM'25] Sample Efficient Preference Alignment in LLMs via Active Exploration
• [COLM'25] Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use
• [COLM'25] Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
• [IPDPS'25] FlexRLHF: A Flexible Placement and Parallelism Framework for Efficient RLHF Training
• [ACL'25] RLKGF: Reinforcement Learning from Knowledge Graph Feedback Without Human Annotations
• [MLSys'25] ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
• [NSDI'25] Optimizing RLHF Training for Large Language Models with Stage Fusion

AutoML

• [OSDI'23] Hydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters
• [NSDI'23] ModelKeeper: Accelerating DNN Training via Automated Training Warmup
• [OSDI'20] Retiarii: A Deep Learning Exploratory-Training Framework

GNN training

For comprehensive list of GNN systems papers, refer to https://github.com/chwan1016/awesome-gnn-systems.

• [SIGMOD'26] FastGNAS: Accelerating and Scaling Graph Neural Architecture Search on Multi-GPUs via Ring-Based Model Migration
• [MLSys'26] GriNNder: Breaking the Memory Capacity Wall in Full-Graph GNN Training with Storage Offloading
• [ICS'26] Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents
• [PPoPP'26] TAC: Cache-Based System for Accelerating Billion-Scale GNN Training on Multi-GPU Platform
• [PPoPP'26] ElasGNN: An Elastic Training Framework for Distributed GNN Training
• [SC'25] Plexus: Taming Billion-edge Graphs with 3D Parallel Full-graph GNN Training
• [SIGMOD'25] NeutronHeter: Optimizing Distributed Graph Neural Network Training for Heterogeneous Clusters
• [ICDE'25] CaliEX: A Disk-Based Large-Scale GNN Training System with Joint Design of Caching and Execution
• [HPCA'25] Mithril: A Scalable System for Deep GNN Training
• [VLDB'25] NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism
• [ICPP'24] GNNDrive: Reducing Memory Contention and I/O Congestion for Disk-based GNN Training
• [VLDB'24] NeutronStream: A Dynamic GNN Training Framework with Sliding Window for Graph Streams
• [MLSys'23] Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training
• [SIGMOD'23] DUCATI: A Dual-Cache Training System for Graph Neural Networks on Giant Graphs with the GPU
• [OSDI'23] MGG: Accelerating Graph Neural Networks with Fine-Grained Intra-Kernel Communication-Computation Pipelining on Multi-GPU Platforms
• [EuroSys'23] MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks
• [KDD'22] Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Heterogeneous Graphs
• [VLDB'22] TGL: a general framework for temporal GNN training on billion-scale graphs
• [OSDI'21] P3: Distributed Deep Graph Learning at Scale

Inference System

• [KDD'26] OrionInfer: Low-Overhead Parallelism Switching and Live Migration for Efficient LLM Serving

• [ISCA'26] CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM

• [ISCA'26] DynoPipe: Heterogeneous Edge-Cloud LLM Serving with Dynamically Orchestrated Pipeline Boundaries

• [ISCA'26] Tetris: Efficient Long-context LLM Serving with Chunkwise Dynamic Sequence Parallelism

• [ISCA'26] ConServe: Contiguity-Preserving Memory Management for Multi-Turn LLM Serving

• [ISCA'26] HybridSpec: Exploiting Hybrid-Bonding Memory to Accelerate LLM Serving Through Heterogeneous Architecture and Speculative Decoding

• [MICRO'26] NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

• [SIGOPS OSR'26] Rethinking LLM Deployment for Intent-Based Serving

• [SIGOPS OSR'26] Elastic Memory Remapping for Multi-tenant LLM Serving

• [ICML'26] Beyond Prediction: Tail-Aware Scheduling for LLM Inference

• [MLSys'26] SHIP: SRAM-Based Huge Inference Pipelines for Fast LLM Serving

• [MLSys'26] Dataflow Is All You Need

• [MLSys'26] Optimizing Deployment Configurations for LLM Inference

• [MobiSys'26] TimelyLLM: Time-sensitive LLM Serving System for Physical-I/O Limited Agents

• [SIGMOD'26] AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System

• [ISCA'26] Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles

• [SIGCOMM'26] KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

• [ICML'26] PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding

• [EuroSys'26] Efficient Data Passing for Serverless Inference Workflows: A GPU-Centric Approach

• [EuroSys'26] High Throughput and Low Latency LLM Serving via Adaptive KV Caching

• [EuroSys'26] Automated End-to-End Model Serving with Cooperative Compilation and Scheduling

• [SIGMOD'26] Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking

• [ASPLOS'26] BlendServe: Optimizing Offline Inference with Resource-Aware Batching

• [ASPLOS'26] DFVG: A Heterogeneous Architecture for Speculative Decoding with Draft-on-FPGA and Verify-on-GPU

• [ASPLOS'26] QoServe: Breaking the Silos of LLM Inference Serving

• [ASPLOS'26] Towards High-Goodput LLM Serving with Prefill-decode Multiplexing

• [ASPLOS'26] SwiftSpec: Disaggregated Speculative Decoding and Fused Kernels for Low-Latency LLM Inference

• [ASPLOS'26] ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression

• [HPCA'26] ELORA: Efficient LoRA and KV Cache Management for Multi-LoRA LLM Serving

• [FAST'26] Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness

• [FAST'26] CacheSlide: Unlocking Cross Position-Aware KV Cache Reuse for Accelerating LLM Serving

• [FAST'26] SolidAttention: Low-Latency SSD-based Serving on Memory-Constrained PCs

• [HPCA'26] PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models

• [MLSys'26] BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization

• [MLSys'26] Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT

• [PPoPP'26] Accelerating Sparse Transformer Inference on GPU

• [VLDB'26] ORBITFLOW: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration

• [IEEE Computer'26] Challenges and Research Directions for Large Language Model Inference Hardware

• [NSDI'26] FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees

• [NSDI'26] FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference

• [NSDI'26] HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds

• [FPGA'26] CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving

• [ASPLOS'26] XY-Serve: End-to-End Versatile Production Serving for Dynamic LLM Workloads

• [AAAI'26] Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving

• [EuroSys'26] FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters

• [EuroSys'26] KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving

• [EuroSys'26] TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling

• [SoCC'25] Multiplexed Heterogeneous LLM Serving via Stage-Aligned Parallelism

• [Middleware'25] Argus: Quality-Aware High-Throughput Text-to-Image Inference Serving System

• [NeurIPS'25] SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications

• [EMNLP'25] Distributed LLM Serving on Consumer-Grade GPUs by Reconciling Computation and Communication

• [MICRO'25] MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving

• [MICRO'25] Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing

• [CLUSTER'25] Scalable and Fast Inference Serving via Hybrid Communication Scheduling on Heterogeneous Networks

• [CLUSTER'25] Rock: Serving Multimodal Models in Cloud with Heterogeneous-Aware Resource Orchestration for Thousands of LoRA Adapters

• [Survey :mag:] [ACM CSUR'25] Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems

• [SOSP'25] Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market

• [SOSP'25] IC-Cache: Efficient Large Language Model Serving via In-context Caching

• [SOSP'25] DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction

• [COLM'25] OverFill: Two-Stage Models for Efficient Language Model Decoding

• [ACM MM'25] TinyServe: Query-Aware Cache Selection for Efficient LLM Serving

• [SC'25] Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism

• [SIGCOMM'25] SCX: Stateless KV-Cache Encoding for Cloud-Scale Confidential Transformer Serving

• [OSDI'25] BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching

• [OSDI'25] WaferLLM: Large Language Model Inference at Wafer Scale

• [OSDI'25] NanoFlow: Towards Optimal Large Language Model Serving Throughput

• [ICML'25] Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Serving

• [ACL'25] MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference

• [ACL'25] StitchLLM: Serving LLMs, One Block at a Time

• [ACL'25] SPECTRA: Faster Large Language Model Inference with Optimized Internal and External Speculation

• [CODEML @ ICML'25] TorchAO: PyTorch-Native Training-to-Serving Model Optimization

• [ICML'25] EPIC: Efficient Position-Independent Caching for Serving Large Language Models

• [ATC'25] DEEPSERVE: Serverless Large Language Model Serving at Scale

• [ISCA'25] WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling

• [ISCA'25] Hybe: GPU-NPU Hybrid System for Efficient LLM Inference with Million-Token Context Window

• [ICLR'25] TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention

• [OSDI'25] Clover: Exploiting Intra-device Parallelism for High Throughput Large Language Model Serving

• [MLSys'25] SOLA: Optimizing SLO Attainment for Large Language Model Serving with State-Aware Scheduling

• [MLSys'25] Marconi: Prefix Caching for the Era of Hybrid LLMs

• [Mobicom'25] D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving

• [ISPASS'25] Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures

• [SIGMOD'25] Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving

• [EuroMLSys'25] Performance Aware LLM Load Balancer for Mixed Workloads

• [MLSys'25] Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving

• [HPCA'25] PAISE: PIM-Accelerated Inference Scheduling Engine for Transformer-based LLM

• [HPCA'25] throttLL'eM: Predictive GPU Throttling for Energy Efficient LLM Inference Serving

• [ASPLOS'25] Aqua: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains

• [ASPLOS'25] Past-Future Scheduler for LLM Serving under SLA Guarantees

• [ASPLOS'25] Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management

• [EuroSys'25] SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs

• [EuroSys'25] Multiplexing Dynamic Deep Learning Workloads with SLO-awareness in GPU Clusters

• [EuroSys'25] NeuStream: Bridging Deep Learning Serving and Stream Processing

• [SoCC'25] ModServe: Scalable and Resource-Efficient Large Multimodal Model Serving

• [ISCA'25] Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization

• [FAST'25] Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot

• [NSDI'25] SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads

• [WWW'25] External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation

• [MLSys'25] ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments

• [ICLR'25] HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment

• [EuroSys'25] SkyServe: Serving AI Models across Regions and Clouds with Spot Instances

• [ASPLOS'25] Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow

• [ASPLOS'25] Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticity

• [MLSys'25] FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

• [EuroSys'25] A House United Within Itself: SLO-Awareness for On-Premises Containerized ML Inference Clusters via Faro

• [Survey :mag:] [ACM CSUR'24] Resource-efficient Algorithms and Systems of Foundation Models: A Survey

• [ICML'25] SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization [Code]

• [ICLR'25] SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration [Code]

• [ICML'25] SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference [Code]

• [ACL'24] LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

• [ACL'24] SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget

• [IPDPS'24] Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference

• [NeurIPS'24] Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting

• [NeurIPS'24] Toward Efficient Inference for Mixture of Experts

• [NeurIPS'24] Sequoia: Scalable and Robust Speculative Decoding

• [SC'24] PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation

• [SC'24] SMIless: Serving DAG-based Inference with Dynamic Invocations under Serverless Computing

• [SenSys'24] LiteMoE: Customizing On-device LLM Serving via Proxy Submodel Tuning

• [MICRO'24] Pushing the Performance Envelope of DNN-based Recommendation Systems Inference on GPUs

• [PML4LRS @ ICLR2024] Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

• [EuroSys'25] Fast State Restoration in LLM Serving with HCache

• [HPCA'24] KRISP: Enabling Kernel-wise RIght-sizing for Spatial Partitioned GPU Inference Servers

• [NeurIPS'24] Efficient LLM Scheduling by Learning to Rank

• [SOSP'24] PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

• [SOSP'24] LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

• [SOSP'24] Improving DNN Inference Throughput Using Practical, Per-Input Compute Adaptation

• [SOSP'24] Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving

• [ICPP'24] GMM: An Efficient GPU Memory Management-based Model Serving System for Multiple DNN Inference Models

• [SIGCOMM'24] CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

• [ES-FoMO @ ICML'24] CO2: Precise Attention Score Observation for improving KV Cache Replacement in Large Language Models

• [OSDI'24] dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving

• [OSDI'24] Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

• [OSDI'24] USHER: Holistic Interference Avoidance for Resource Optimized ML Inference

• [OSDI'24] Fairness in Serving Large Language Models

• [OSDI'24] MonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference Tasks on Modern GPU-Centric Architectures

• [OSDI'24] Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

• [OSDI'24] ServerlessLLM: Low-Latency Serverless Inference for Large Language Models

• [OSDI'24] InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

• [OSDI'24] Llumnix: Dynamic Scheduling for Large Language Model Serving

• [OSDI'24] DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

• [ATC'24] Power-aware Deep Learning Model Serving with μ-Serve

• [ATC'24] Fast Inference for Probabilistic Graphical Models

• [ATC'24] Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

• [ATC'24] PUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context Switch

• [ATC'24] Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Algorithm-System Co-Design on Modern GPUs

• [TPDS'24] ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG

• [OSDI'24] Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

• [ISCA'24] ElasticRec: A Microservice-based Model Serving Architecture Enabling Elastic Resource Scaling for Recommendation Models

• [ISCA'24] Splitwise: Efficient generative LLM inference using phase splitting

• [ICML'24] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

• [ICML'24] Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

• [ICML'24] HexGen: Generative Inference of Large Language Model over Heterogeneous Environment

• [ICML'24] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

• [ICML'24] MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving

• [HPCA'24] An LPDDR-based CXL-PNM Platform for TCO-efficient Inference of Transformer-based Large Language Models

• [MobiSys'24] ARISE: High-Capacity AR Offloading Inference Serving via Proactive Scheduling

• [MobiSys'24] Pantheon: Preemptible Multi-DNN Inference on Mobile Edge GPUs

• [MLSys'24] HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices

• [MLSys'24] S-LoRA: Serving Thousands of Concurrent LoRA Adapters

• [MLSys'24] Vidur: A Large-Scale Simulation Framework For LLM Inference

• [WWW'24] λGrapher: A Resource-Efficient Serverless System for GNN Serving through Graph Sharing

• [ICML'24] CLLMs: Consistency Large Language Models

• [EuroSys'24] Model Selection for Latency-Critical Inference Serving

• [ISCA'24] Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference

• [ASPLOS'24] ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference

• [ASPLOS'24] NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing

• [ICML'24] DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving

• [ICLR'24] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs

  • [PPoPP'24 poster] POSTER: LLM-PQ:Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization

• [NSDI'24] Approximate Caching for Efficiently Serving Diffusion Models

• [VLDB'24] Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity

• [ASPLOS'24] SpotServe: Serving Generative Large Language Models on Preemptible Instances

• [ASPLOS'24] SpecInfer: Accelerating Generative Large Language Model Serving with Speculative Inference and Token Tree Verification

• [EMNLP'23] Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding

• [NeurIPS'23] SpecTr: Fast Speculative Decoding via Optimal Transport

• [HPDC'23] Kairos: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources

• [SOSP'23] Paella: Low-latency Model Serving with Virtualized GPU Scheduling

• [SOSP'23] Efficient Memory Management for Large Language Model Serving with PagedAttention

• [MLSys'23] Efficiently Scaling Transformer Inference

• [EuroSys'23] Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access

• [EuroSys'23] Tabi: An Efficient Multi-Level Inference System for Large Language Models

• [EuroSys'23] Pocket: ML Serving from the Edge

• [OSDI'23] AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

• [NSDI'23] SHEPHERD: Serving DNNs in the Wild

• [VLDB'23] Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructures

• [ICML'23] Fast Inference from Transformers via Speculative Decoding

• [SIGMOD'22] Serverless Data Science - Are We There Yet? A Case Study of Model Serving

• [OSDI'22] Orca: A Distributed Serving System for Transformer-Based Generative Models

• [OSDI'22] Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences

• [ATC'22] SOTER: Guarding Black-box Inference for General Neural Networks at the Edge

• [ATC'22] Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing

• [ATC'22] Tetris: Memory-efficient Serverless Inference through Tensor Sharing

• [ATC'22] PetS: A Unified Framework for Parameter-Efficient Transformers Serving

• [ATC'21] INFaaS: Automated Model-less Inference Serving

• [SoCC'21] Morphling: Fast, Near-Optimal Auto-Configuration for Cloud-Native Model Serving

• [MobiCom'20] SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud

Attention Optimization

• [PPOPP'26] MetaAttention: A Unified and Performant Attention Framework across Hardware Backends
• [PPoPP'26] FlashAttention-T: Towards Fully Tensorized Attention by Exploiting Tensor-Vector Parallelism
• [SC'25] UltraAttn: Efficiently Parallelizing Attention through Hierarchical Context-Tiling
• [SC'25] RingX: Scalable Parallel Attention for Long-Context Learning on HPC
• [NeurIPS'25] Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning
• [NeurIPS'25] SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training [Code]
• [MLSys'25] FastTree: Optimizing Attention Kernel and Runtime for Tree-Structured LLM Inference
• [MLSys'25] FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
• [NeurIPS'24] FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
• [ICLR'24] FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• [NeurIPS'22] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Mixture of Experts (MoE)

• [ICPP'26] Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

• [CAL'26] HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training

• [SIGCOMM'26] Balancing and Beyond: Communication-Centric Optimizations in Expert Parallelism

• [SIGCOMM'26] UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods

• [OSDI'26] Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

• [MLSys'26] Demystifying the Mixture of Experts Serving Tax

• [MLSys'26] From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill

• [OCML'26] EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference

• [ICML'26] ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference

• [ISCA'26] Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs

• [ISCA'26] MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems

• [ISCA'26] Orders in Chaos: Enhancing Large-Scale MoE LLM Serving with Data Movement Forecasting

• [ASPLOS'26] EARTH: An Efficient MoE Accelerator with Entropy-Aware Speculative Prefetch and Result Reuse

• [ASPLOS'26] MoE-APEX: An Efficient MoE Inference System with Adaptive Precision Expert Offloading

• [DAC'26] TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading

• [TPDS'26] Reducing Cross-Pod Communication Overhead for MoE Model Training with Hybrid Parallelism in Multi-Tenant Clusters

• [NSDI'26] SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling

• [ASPLOS'26] LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training

• [EuroSys'26] Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading

• [EuroSys'26] MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production

• [PACT'25] ScaleMoE: A Fast and Scalable Distributed Training Framework for Large-Scale Mixture-of-Experts Models

• [SC'25] Diff-MoE: Efficient Batched MoE Inference with Priority-Driven Differential Expert Caching

• [SC workshop'25] Compression Error Sensitivity Analysis for Different Experts in MoE Model Inference

• [SC workshop'25] Batch Tiling on Attention: Efficient Mixture of Experts Training on Wafer-Scale Processors

• [MICRO'25] Optimizing All-to-All Collective Communication with Fault Tolerance on Torus Networks

• [MICRO'25] Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Serving

• [SOSP'25] KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models

• [ICML'25] Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

• [NeurIPS'25] BrainMoE: Cognition Joint Embedding via Mixture-of-Expert Towards Robust Brain Foundation Model

• [NeurIPS'25] S’MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning

• [NeurIPS'25] The Omni-Expert: A Computationally Efficient Approach to Achieve a Mixture of Experts in a Single Expert Model

• [NeurIPS'25] MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE

• [NeurIPS'25] FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts

• [NeurIPS'25] FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training

• [NeurIPS'25] FlashMoE: Fast Distributed MoE in a Single Kernel [Code]

• [SC'25] MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?

• [SIGCOMM'25] MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism

• [ICLR'25] Ada-K Routing: Boosting the Efficiency of MoE-based LLMs

• [ICML'25] Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference

• [ICML'25] I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts

• [SC'25] X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms

• [SIGCOMM'25] MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training

• [ATC'25] PopFetcher: Towards Accelerated Mixture-of-Experts Training Via Popularity Based Expert-Wise Prefetch

• [ACL'25] EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models

• [ACL'25] FOLDMOE: Efficient Long Sequence MoE Training via Attention-MoE Pipelining

• [ATC'25] PopFetcher: Towards Accelerated Mixture-of-Experts Training Via Popularity Based Expert-Wise Prefetch

• [ICML'25] FloE: On-the-Fly MoE Inference on Memory-constrained GPU

• [NAACL'25] MoLA: MoE LoRA with Layer-wise Expert Allocation

• [NAACL'25] Marrying LLMs with Dynamic Forecasting: A Graph Mixture-of-expert Perspective

• [NAACL'25] Sparser Mixture-of-Adapters with Cross-Layer Generalization

• [NAACL'25] SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse

• [Mobicom'25] D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving

• [DAC'25] HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference

• [TKDE'25] A Survey on Mixture of Experts

• [ICLR'25] NetMoE: Accelerating MoE Training through Dynamic Sample Placement

• [EuroSys'25] Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores

• [EuroMLSys'25] Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference

• [EuroMLSys'25] Accelerating MoE Model Inference with Expert Sharding

• [KDD'25] ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration

• [MLSys'25] Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts

• [CVPR'25] DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models

• [ASPLOS'25] CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory

• [TPDS'25] EfficientMoE: Optimizing Mixture-of-Experts Model Training with Adaptive Load Balance

• [NAACL'25] MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

• [ASPLOS'25] FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models

• [MICRO'24] SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts

• [TPDS'24] MPMoE: Memory Efficient MoE for Pre-Trained Models With Adaptive Pipeline Parallelism

  • Journal version of [IPDPS'23] MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

• [MLArchSys'24 @ ISCA'24] MoE-ERAS: Expert Residency Aware Selection

• [COLM'24] Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training

• [ME-FoMo @ ICLR'24] Scaling Laws for Fine-Grained Mixture of Experts

• [ML for Sys workshop @ NeurIPS'24] IFMoE: An Inference Framework Design for Fine-grained MoE

• [ML for Sys workshop @ NeurIPS'24] TurboMoE: Enhancing MoE Model Training with Smart Kernel-Fusion and Data Transformation

• [EMNLP'24] MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

• [EMNLP'24] Mixture of Diverse Size Experts

• [EMNLP'24] AdaMOE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models

• [ACL'24] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

• [ACL'24] SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget

• [SoCC'24] MoEsaic: Shared Mixture of Experts

• [KDD'24] Efficient Mixture of Experts based on Large Language Models for Low-Resource Data Preprocessing

• [IPDPS'24] Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference

• [NeurIPS'24] Toward Efficient Inference for Mixture of Experts

• [MLSys'24] SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models

• [SC'24] APTMoE: Affinity-Aware Pipeline Tuning for MoE Models on Bandwidth-Constrained GPU Nodes

• [NeurIPS'24] GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Experts

• [NeurIPS'24] LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing

• [NeurIPS'24] Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design

• [PML4LRS @ ICLR'24] Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

• [NeurIPS'24 (Splotlight)] Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts

• [SRW @ ACL'24] MoExtend: Tuning New Experts for Modality and Task Extension

• [ICML'24] Scaling Laws for Fine-Grained Mixture of Experts

• [ICML'24] Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training

• [MLSys'24] QMoE: Sub-1-Bit Compression of Trillion-Parameter Models

• [MLSys'24] Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping

• [SIGIR'24] M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Framework

• [EuroSys'24] ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling

• [ICLR'24] Mixture of LoRA Experts

• [IJCAI'24] LocMoE: A Low-overhead MoE for Large Language Model Training

• [ISCA'24] Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference

• [IPDPS'23] MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

• [EMNLP'23] Adaptive Gating in Mixture-of-Experts based Language Models

• [ACL'23] AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation

• [ICLR'23] Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

• [ICML'23] Brainformers: Trading Simplicity for Efficiency

• [ATC'23] Accelerating Distributed MoE Training and Inference with Lina

• [ATC'23] SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization

• [OSDI'23] Optimizing Dynamic Neural Networks with Brainstorm

• [SIGMOD'23] FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement

• [ICS'23] A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training

• [MLSys'23] MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

• [MLSys'23] Tutel: Adaptive Mixture-of-Experts at Scale

• [PPoPP'22] FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models

• [SustaiNLP @ EMNLP'22] Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production

• [NeurIPS'22] Mixture-of-Experts with Expert Choice Routing

• [ICML'22] DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

• [ICML'22] GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

• [JMLR'22] Switch transformers: scaling to trillion parameter models with simple and efficient sparsity

• [EMNLP'21] Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference

• [ICLR'17] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Communication Optimization & Network Infrastructure for Distributed ML

• [OSDI'26] Syncopate: Efficient Multi-GPU AI Kernels via Automatic Chunk-Centric Compute-Communication Overlap
• [SC'26] HCCL: Collective Communication for Meta Training and Inference Accelerators
• [SIGCOMM'26] UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
• [SIGCOMM'26] EPIC: Abstraction and Polymorphism of In-Network Collectives on Ethernet
• [ISCA'26] Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
• [HPCA'26] Towards Compute-Aware In-Switch Computing for LLMs Tensor-Parallelism on Multi-GPU Systems
• [EuroSys'26] Multipath Collective Communication Beyond Scale-up Networks in GPU Clouds
• [ASPLOS'26] Fine-grained and Non-intrusive LLM Training Monitoring via Microsecond-level Traffic Measurement
• [ASPLOS'26] Reconfigurable Torus Fabrics for Multi-tenant ML
• [NSDI'26] HeteCCL: Synthesizing Near-Optimal Collective Communication Schedules for Heterogeneous GPU Clusters
• [HPCA'26] SCALE: Tackling Communication Bottlenecks in Confidential Distributed Machine Learning
• [ToN'26] Virtual Slicing: Achieving Control Plane Availability and Traffic Engineering Efficiency in Data Centers
• [TPDS'26] Reducing Cross-Pod Communication Overhead for MoE Model Training with Hybrid Parallelism in Multi-Tenant Clusters
• [PPoPP'26] COCCL: A Collective Communication Library Supporting Easy Integration and Configuration of Customized Compression for Scalable LLM Training
• [EuroSys'26] Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering
• [SC workshop'25] Redesigning GROMACS Halo Exchange: Improving Strong Scaling with GPU-initiated NVSHMEM
• [SC'25] Understanding Communication Bottlenecks in Multi-node LLM Inference
• [SC'25] CPU- and GPU-initiated Communication Strategies for Conjugate Gradient Methods on Large GPU Clusters
• [SC'25] SDR-RDMA: Software-Defined Reliability Architecture for Planetary Scale RDMA Communication
• [HotNets'25] Photonic Rails in ML Datacenters
• [SOSP'25] Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training
• [MICRO'25] SuperMesh: Energy-Efficient Collective Communications for Accelerators
• [MICRO'25] SkipReduce: (Interconnection) Network Sparsity to Accelerate Distributed Machine Learning
• [MICRO'25] Optimizing All-to-All Collective Communication with Fault Tolerance on Torus Networks
• [APNET'25] Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelization
• [NAIC @ SIGCOMM'25] Chronos: Prescheduled circuit switching for LLM training
• [SIGCOMM'25] Falcon: A Reliable, Low Latency Hardware Transport
• [SIGCOMM'25] ByteScale: Communication-Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs
• [SIGCOMM'25] From ATOP to ZCube: Automated Topology Optimization Pipeline and A Highly Cost-Effective Network Topology for Large Model Training
• [SIGCOMM'25] Astral: A Datacenter Infrastructure for Large Language Model Training at Scale
• [SIGCOMM'25] ResCCL: Resource-Efficient Scheduling for Collective Communication
• [OSDI'25] ZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization
• [OSDI'25] Enabling Efficient GPU Communication over Multiple NICs with FuseLink
• [APNET'25] Congestion Control for AI Workloads with Message-Level Signaling
• [ASPLOS'25] Concerto: Automatic Communication Optimization and Scheduling for Large-Scale Deep Learning
• [ISCA'25] Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models
• [HPCA'25] Enhancing Large-Scale AI Training Efficiency: The C4 Solution for Real-Time Anomaly Detection and Communication Optimization
• [EuroMLSys'25] TAGC: Optimizing Gradient Communication in Distributed Transformer Training
• [MLSys'25] TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives
• [NSDI'25] AutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and Parallel DNN Training
• [NSDI'25] Efficient Direct-Connect Topologies for Collective Communications
• [IEEE MICRO'25] Understanding and Characterizing Communication Characteristics for Distributed Transformer Models
• [NSDI'25] OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the Cloud
• [APNET'24] Understanding Communication Characteristics of Distributed Training
• [HotInfra'24] Immediate Communication for Distributed AI Tasks
• [NeurIPS'24] SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
• [SC'24] Optimizing Distributed ML Communication with Fused Computation-Collective Operations
• [SC'24] Network-Offloaded Bandwidth-Optimal Broadcast and Allgather for Distributed AI
• [NeurIPS'24] LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
• [TPDS'24] AutoDDL: Automatic Distributed Deep Learning With Near-Optimal Bandwidth Cost
• [HOTI'24] Unified Collective Communication (UCC): An Unified Library for CPU, GPU, and DPU Collectives
• [HOTI'24] Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters
• [SC'24] Switch-Less Dragonfly on Wafers: A Scalable Interconnection Architecture based on Wafer-Scale Integration
• [HPDC'24] Near-Optimal Wafer-Scale Reduce
• [HPDC'24] Efficient all-to-all Collective Communication Schedules for Direct-connect Topologies
• [ICS'24] gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusters
• [ICS'24] Snoopie: A Multi-GPU Communication Profiler and Visualizer
• [ICPP'24] Sparse Gradient Communication with AlltoAll for Accelerating Distributed Deep Learning
• [NAIC @ SIGCOMM'24] Proof-of-Concept of a Flexible and High-Fidelity Approach to Distributed DNN Training Emulation
• [NAIC @ SIGCOMM'24] Eloquent: A More Robust Transmission Scheme for LLM Token Streaming
• [NAIC @ SIGCOMM'24] OmNICCL: Zero-cost Sparse AllReduce with Direct Cache Access and SmartNICs
• [HotNets'24] I've Got 99 Problems But FLOPS Ain't One
• [HotNets'24] MLTCP: A Distributed Technique to Approximate Centralized Flow Scheduling For Machine Learning
• [HotNets'22] Congestion Control in Machine Learning Clusters
• [SIGCOMM'24] Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem
• [SIGCOMM'24] RDMA over Ethernet for Distributed Training at Meta Scale
• [SIGCOMM'24] Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs
• [SIGCOMM'24] MCCS: A Service-based Approach to Collective Communication for Multi-Tenant Cloud
• [SIGCOMM'24] Crux: GPU-Efficient Communication Scheduling for Deep Learning Training
  • [HotNets'24] MLTCP: A Distributed Technique to Approximate Centralized Flow Scheduling For Machine Learning
• [APNet'24] Understanding Communication Characteristics of Distributed Training
• [ICLR'24] ZeRO++: Extremely Efficient Collective Communication for Large Model Training
• [ICLR'24] CO2: Efficient Distributed Training with Full Communication-Computation Overlap
• [MLSys'24] L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning
• [MLSys'24] Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping
• [ASPLOS'24] T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives
• [ASPLOS'24] TCCL: Discovering Better Communication Paths for PCIe GPU Clusters
• [ASPLOS'24] Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning
• [ASPLOS'24] Two-Face: Combining Collective and One-Sided Communication for Efficient Distributed SpMM
• [NSDI'24] THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression
• [INFOCOM'23] Libra: Contention-Aware GPU Thread Allocation for Data Parallel Training in High Speed Networks
• [ICDCS'23] bbTopk: Bandwidth-Aware Sparse Allreduce with Blocked Sparsification for Efficient Distributed Training
• [ICML'23] CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks
  • Related to DT-FM (NeurIPS'22)
• [IPDPS'23] MCR-DL: Mix-and-Match Communication Runtime for Deep Learning
• [ASPLOS'23] MSCCLang: Microsoft Collective Communication Language
• [ASPLOS'23] Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models
• [EuroSys'23] A2TP: Aggregator-aware In-network Aggregation for Multi-tenant Learning
• [MLSys'23] Cupcake: A Compression Optimizer for Scalable Communication-Efficient Distributed Training
• [MLSys'23] On Optimizing the Communication of Model Parallelism
• [NSDI'23] TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs
• [NSDI'23] Better Together: Jointly Optimizing ML Collective Scheduling and Execution Planning using SYNDICATE
• [NSDI'23] TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
• [NSDI'23] ARK: GPU-driven Code Execution for Distributed Deep Learning
• [EuroSys'22] Out-of-order backprop: an effective scheduling technique for deep learning
• [ISCA'22] Themis: a network bandwidth-aware collective scheduling policy for distributed training of DL models
• [ISCA'22] Software-hardware co-design for fast and scalable training of deep learning recommendation models
• [SC'22] HammingMesh: A Network Topology for Large-Scale Deep Learning
• [PPoPP'22] Near-optimal sparse allreduce for distributed deep learning
• [MLSys'22] Synthesizing optimal parallelism placement and reduction strategies on hierarchical systems for deep learning (P^2)
• [ASPLOS'22] Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads (CoCoNET)
• [EuroSys'21] DGCL: an efficient communication library for distributed GNN training
• [ICLR'21] Multi-Level Local SGD for Heterogeneous Hierarchical Networks
• [SIGMOD'21] Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce(#25-parallelism--distributed-training)]
• [SC'21] Flare: flexible in-network allreduce
• [NSDI'21] Scaling Distributed Machine Learning with In-Network Aggregation
• [ISCA'21] Enabling compute-communication overlap in distributed deep learning training platforms
• [PPoPP'21] Synthesizing optimal collective algorithms (SCCL)
• [SIGCOMM'21] SiP-ML: High-Bandwidth Optical Network Interconnects for Machine Learning Training
• [ISCA'20] An in-network architecture for accelerating shared-memory multiprocessor collectives
• [NeurIPS'20] Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning
• [PPoPP'20] Taming unbalanced training workloads in deep learning with partial collective operations
• [MLSys'20] Blink: Fast and Generic Collectives for Distributed ML
• [MLSys'20] PLink: Discovering and Exploiting Datacenter Network Locality for Efficient Cloud-based Distributed Training
• [OSDI'20] A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (BytePS)
• [MLSys'19] Priority-based Parameter Propagation for Distributed DNN Training (P3)
• [MLSys'19] TicTac: Accelerating Distributed Deep Learning with Communication Scheduling
• [SOSP'19] A generic communication scheduler for distributed DNN training acceleration (ByteScheduler)
• [ATC'17] Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters

Fault tolerance & Straggler mitigation

• [KDD'26] Don't Predict, Prioritize: Rethinking GPU Reliability Assessment
• [MLSys'26] RaidServe: High-performance Resilient Serving
• [CoNEXT'26] PRC: Process-centric Rate Control for Transient Stragglers in Hybrid-Parallel Training
• [ICML'26] Ekka: Automated Diagnosis of Silent Errors in LLM Inference
• [FSE'26] TSGuard: Automated User-Centric Incident Diagnosis for AI Workloads in the Cloud
• [ICS'26] Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference
• [MLSys'26] Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training
• [OSDI'26] RobustRL: Role-based Fault Tolerance System for RL Post-Training
• [OSDI'26] Continuum: An Interruption-Resilient Runtime for ML Training
• [FAST'26] AdaCheck: An Adaptive Checkpointing System for Efficient LLM Training with Redundancy Utilization
• [FAST'26] GPU Checkpoint/Restore Made Fast and Lightweight
• [MLSys'26] GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving
• [HPCA'26] SpotCC: Facilitating Coded Computation for Prediction Serving Systems on Spot Instances
• [PPoPP'26] CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training
• [PPoPP'26] Elastor: Elastic and Efficient Model Partitioning and Checkpointing for Fault-Tolerant Distributed Training
• [NSDI'26] Attack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
• [NSDI'26] Sparse Checkpointing for Fast and Reliable MoE Training
• [NSDI'26] Checkmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication
• [MICRO'25] Optimizing All-to-All Collective Communication with Fault Tolerance on Torus Networks
• [APSys'25] Indispensable CPU-centric Checkpointing for GPUs
• [CLUSTER'25] Capricorn: Efficient In-Memory Checkpointing for MoE Model Training with Dynamicity Awareness
• [SOSP'25] PhoenixOS: Concurrent OS-level GPU Checkpoint and Restore with Validated Speculation
• [SOSP'25] Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training
• [SOSP'25] Robust LLM Training Infrastructure at ByteDance
• [SC'25] LowDiff: Efficient Frequent Checkpointing via Low-Cost Differential for High-Performance Distributed Training Systems
• [OSDI'25] Understanding Stragglers in Large Model Training Using What-if Analysis
• [SIGMOD'25] Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
• [ATC'25] SAVE: Software-Implemented Fault Tolerance for Model Inference against GPU Memory Bit Flips
• [ATC'25] Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelism
• [NSDI'25] Evolution of Aegis: Fault Diagnosis for AI Model Training Service in Production
• [NSDI'25] BCP: A Unified Checkpointing System for Large Foundation Model Development
• [NSDI'25] Minder: Faulty Machine Detection for Large-scale Distributed Model Training
• [EuroSys'25] SkyServe: Serving AI Models across Regions and Clouds with Spot Instances
• [ASPLOS'25] MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
• [ASPLOS'25] PCcheck: Persistent Concurrent Checkpointing for ML
• [ICML'24] DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving
• [SOSP'24] ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
• [HPDC'24] DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Models
• [EuroSys'24] Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures
• [NSDI'24] MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• [NSDI'24] Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances
• [ASPLOS'24] SpotServe: Serving Generative Large Language Models on Preemptible Instances
• [VLDB'23] Eficient Fault Tolerance for Recommendation Model Training via Erasure Coding
• [SOSP'23] GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints
• [SOSP'23] Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates
• [NSDI'23] Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
• [EuroSys'22] Varuna: scalable, low-cost training of massive deep learning models
• [ATC'22] Sibylla: To Retry or Not To Retry on Deep Learning Job Failure
• [MLSys'21] Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
• [FAST'21] CheckFreq: Frequent, Fine-Grained DNN Checkpointing
• [ICSE'20] An Empirical Study on Program Failures of Deep Learning Jobs

GPU Memory Management & Optimization

• [ICML'26] Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
• [ASPLOS'26] T-Control: An Efficient Dynamic Tensor Rematerialization System for DNN Training
• [ASPLOS'26] ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
• [SC'25] HELM: Characterizing Unified Memory Accesses to Improve GPU Performance under Memory Oversubscription
• [SC'25] MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall
• [ISCA'25] Forest: Access-aware GPU UVM Management
• [EuroSys'25] MEPipe: Democratizing LLM Training with Memory-Efficient Slice-Level Pipeline Scheduling on Cost-Effective Accelerators
• [EuroSys'25] Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
• [FAST'25 WiP] Baton: Orchestrating GPU Memory for LLM Training on Heterogeneous Cluster
• [CGO'25] IntelliGen: Instruction-Level Auto-tuning for Tensor Program with Monotonic Memory Optimization
• [IJCAI'24] LLMem: Estimating GPU Memory Usage for Fine-Tuning Pre-Trained LLMs
• [MICRO'24] SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts
• [TACO'24] ATP: Achieving Throughput Peak for DNN Training via Smart GPU Memory Management
• [ICML'24] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• [ASPLOS'24] GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching
• [SoCC'23] Towards GPU Memory Efficiency for Distributed Training at Scale
• [VLDB'23] PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• [SOSP'23] Efficient Memory Management for Large Language Model Serving with PagedAttention
• [HPCA'23] MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism
• [HPCA'23] Tensor Movement Orchestration in Multi-GPU Training Systems
• [IJCAI'23] OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
• [ICLR'22] LoRA: Low-Rank Adaptation of Large Language Models
  • algorithmic method for memory efficiency
• [VLDB'22] Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers
• [ATC'21] ZeRO-Offload: Democratizing Billion-Scale Model Training
• [ICLR'21] ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
• [ICLR'21] Dynamic Tensor Rematerialization
• [SC'21] ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning
• [HPCA'21] Sentinel: Efficient Tensor Migration and Allocation on Heterogeneous Memory Systems for Deep Learning
• [MLSys'20] Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
• [ASPLOS'20] Capuchin: Tensor-based GPU Memory Management for Deep Learning
• [ASPLOS'20] SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
• [ESEC/FSE'20] Estimating GPU memory consumption of deep learning models
• [SC'20] ZeRO: memory optimizations toward training trillion parameter models
• [ISCA'18] Gist: Efficient Data Encoding for Deep Neural Network Training
• [PPoPP'18] Superneurons: dynamic GPU memory management for training deep neural networks
• [MICRO'16] vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design

GPU Sharing

• [TACO'26] Performance Prediction of Concurrent DNN Training Tasks in GPU Spatial Sharing Environments
• [HPCA'26] µShare: Non-Intrusive Kernel Co-Locating on NVIDIA GPUs
• [HPCA'26] eGPU: Production-Scale Elastic Sharing Over 10,000 GPUs
• [SC workshop'25] WAGES: Workload-Aware GPU Sharing System for Energy-Efficient Serverless LLM Serving
• [SOSP'25] LithOS: An Operating System for Efficient Machine Learning on GPUs
• [OSDI'25] XSched: Preemptive Scheduling for Diverse XPUs
• [EuroSys'25] Improving GPU Sharing Performance through Adaptive Bubbleless Spatial-Temporal Sharing
• [PPOPP'25] SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs
• [SC'24] ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments
• [ICPP'24] MIGER: Integrating Multi-Instance GPU and Multi-Process Service for Deep Learning Clusters
• [ASPLOS'24] RAP: Resource-aware Automated GPU Sharing for Multi-GPU Recommendation Model Training and Input Preprocessing
• [EuroSys'24] Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications
• [ATC'23] Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent
• [NSDI'23] Transparent GPU Sharing in Container Clouds for Deep Learning Workloads
• [ICPP'23] FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference
• [SoCC'22] MISO: exploiting multi-instance GPU capability on multi-tenant GPU clusters
• [PACT'22] GPUPool: A Holistic Approach to Fine-Grained GPU Sharing in the Cloud
• [ATC'21] Zico: Efficient GPU Memory Sharing for Concurrent DNN Training
• [MLSys'20] Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications
• [OSDI'20] AntMan: Dynamic Scaling on GPU Clusters for Deep Learning
• [OSDI'20] PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications
• [RTAS'19] Fractional GPUs: Software-Based Compute and Memory Bandwidth Reservation for GPUs

Compiler

• [OSDI'26] Syncopate: Efficient Multi-GPU AI Kernels via Automatic Chunk-Centric Compute-Communication Overlap
• [PLDI'26] Compiling Strassen-like Matrix Multiplication Algorithms to Fast CUDA Kernels
• [MLSys'26] Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel
• [CGO'26] Fast Autoscheduling for Sparse ML Frameworks
• [NeurIPS'25] REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving
• [SOSP'25] Mercury: Unlocking Multi-GPU Operator Optimization for LLMs via Remote Memory Scheduling
• [MICRO'25] StreamTensor: Make Tensors Stream in Dataflow Accelerators for LLMs
• [OSDI'25] PipeThreader: Software-Defined Pipelining for Efficient DNN Execution
• [OSDI'