Distributed and Scalable Directory Service in a Parallel File System

Lixin WANG
Yutong LU

IEICE TRANSACTIONS on Information and Systems   Vol.E99-D    No.2    pp.313-323
Publication Date: 2016/02/01
Publicized: 2015/10/26
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2015EDP7009
Type of Manuscript: PAPER
Category: Fundamentals of Information Systems
parallel file system,  distributed and scalable directory service,  concurrent and unsynchronized splitting,  recursive split-ordering,  write-optimized index structure,  

Full Text: PDF>>
Buy this Article

One of the patterns that the design of parallel file systems has to solve stems from the difficulty of handling the metadata-intensive I/O generated by parallel applications accessing a single large directory. We demonstrate a middleware design called SFS to support existing parallel file systems for distributed and scalable directory service. SFS distributes directory entries over data servers instead of metadata servers to offer increased scalability and performance. Firstly, SFS exploits an adaptive directory partitioning based on extendible hashing to support concurrent and unsynchronized partition splitting. Secondly, SFS describes an optimization based on recursive split-ordering that emphasizes speeding up the splitting process. Thirdly, SFS applies a write-optimized index structure to convert slow, small, random metadata updates into fast, large, sequential writes. Finally, SFS gracefully tolerates stale mapping at the clients while maintaining the correctness and consistency of the system. Our performance results on a cluster of 32-servers show our implementation can deliver more than 250,000 file creations per second on average.