Group File Operations for Scalable Tools and Middleware

Abstract

Group file operations are a new, intuitive idiom for tools and middleware - including parallel debuggers and runtimes, performance measurement and steering, and distributed resource management - that require scalable operations over large groups of distributed files. The group file operation idiom provides new semantics for using file groups as operands in standard file operations, thus eliminating costly iteration. A file-based idiom promotes conciseness and portability, and eases adoption. With explicit semantics for aggregation of group data results, the idiom addresses a key scalability barrier. We have designed TBON-FS, a new distributed file system that provides scalable group file operations by leveraging tree-based overlay networks (TBONs) for scalable distribution of group file operation requests and aggregation of group status and data results. Using a prototype TBON-FS system, we have integrated group file operations into several tools. Our experience verifies the group file operation idiom is intuitive and easily adopted, and improves performance at scale.