Abstract

High-performance applications necessitate rapid and dependable transfer of massive datasets across geographically dispersed locations. Traditional file transfer tools often suffer from resource underutilization and instability due to fixed configurations or monolithic optimization methods. We propose AutoMDT, a novel Modular Data Transfer Architecture, to address these issues by employing a deep reinforcement learning agent to simultaneously optimize concurrency levels for read, network, and write operations. This solution incorporates a lightweight network-system simulator, enabling offline training of a Proximal Policy Optimization (PPO) agent in approximately 45 minutes on average, thereby overcoming the impracticality of lengthy online training in production networks. AutoMDT's modular design decouples I/O and network tasks, allowing the agent to capture complex buffer dynamics precisely and adapt to changing system and network conditions quickly. Evaluations on production-grade testbeds show that AutoMDT achieves up to 8x faster convergence and a 68% reduction in transfer completion times compared to state-of-the-art solutions.

Department(s)

Computer Science

Comments

National Science Foundation, Grant 2451376

Keywords and Phrases

concurrency control; data transfer optimization; high-performance networks; HPC; reinforcement learning

Document Type

Article - Conference proceedings

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2025 Association for Computing Machinery, All rights reserved.

Publication Date

15 Nov 2025

Download

Full Text Link

Share

 
COinS