User Documentation

1. What we are doing

A short description for what we are trying to do is - demonstrate that technology changes (shingling of magnetic disk tracks) can be exposed by new magnetic disks to improve their storage capacity and at the same time force minimal change to the current production software infrastructure.

The new technique is called Shingled Magnetic Recording (SMR). Using SMR techniques it is possible to exploit the current magnetic read and write technologies to increase disk capacity by at least 1.25x and perhaps up to 2.5x.
Starting before real hardware for shingled disks was available (as of Sept 2013, Seagate reported that it had already shipped 1M shingled disks), we decided to develop an SMR Disk emulation environment for research purposes. The emulator allows us to study the behavior of SMR disks in absence of real hardware.
The emulator can be independent of the underlying filesystem, implemented either on raw (non-shingled disk) partitions or as a file in a traditional filesystem. This enables the use of the emulator on a variety of convenient test machines. The emulator is flexible, so it can be run with different parameters of the SMR disk that will be emulated. This helps in studying the impact of - band sizes, data placement, cleaning policies, and internal metadata structures.

We are releasing both this simple SMR emulator and an SMR-aware filesystem - SMRfs. SMRfs is a VFS compliant filesystem, with an intermediate buffer cache, implemented on UNIX systems with the FUSE infrastructure for user-level file systems. SMRfs is also intended for research and development, not production use, and the thoroughness of our testing (and lack of warranty) reflects this.

2. Related work

The fundamentals of the SMR design and working principles are described in PDL's report on the SMR disks [1]. The usability of SMR disks as Log-based System Model and Synergy with Solid-State Disk is covered in [2] serve as pre-cursor to ShingledFS design. The initial design and implementation of the ShingledFS is based on the CloudFS [3] developed as a part of Advanced Storage Systems (15/18-746) course. A custom-built SMR Device Emulator and ShingledFS, a FUSE-based SMR-aware file system that operates in tandem with the SMR Device Emulator is covered in [4]. Also, the valuation studies SMR for Big Data applications and examine the overheads introduced by the emulation. The TableFS [5], a stacked file system, which uses another local file system as an object store, is used for enhancing the metadata efficiency in ShingledFS. The Caveat Scriptor [6] along with other methodologies with exposed disk parameters provide an additional perspective in ShingledFS design. Cassuto et al. [8] have proposed two indirection systems (essentially data structures and algorithms) that can help mask the above metioned shortcoming of shingled disks while still maintaining performance for regular disk accesses. Hall et al. [9] have proposed a data handling algorithm that aids random-write situations for short-blocks. Amer et al. [10] discussed the requirements and issues of a system designed for shingled disks. Tan et al. [11] have constructed an SMR simulator using the disk geometry profile and performance profile. Pitchumani et al. [12][14] have implemented an SMR emulator on a traditional perpendicular magnetic recording disk. Lin et al. [13] proposed a window-based hot data identification scheme for SMR. Wan et al. [15] presents a new hybrid storage architecture that combines a shingled-recording magnetic disk and a fast SSD cache to achieve a high-capacity storage system without any compromise to performance. Shingled Disk products [16] are real!

List:
[1] G. Gibson and G. Ganger, "Principles of Operation for Shingled Disk Devices", Technical Report CMU-PDL-11-107, Parallel Data Lab, Carnegie Mellon University, Pittsburgh, PA, 2011. (http://www.pdl.cmu.edu/PDL-FTP/Storage/CMU-PDL-11-107.pdf)

[2] G. Gibson and M. Polte, "Directions for Shingled-Write and Two-Dimensional Magnetic Recording System Architectures: Synergies with Solid-State Disks," Technical Report CMU-PDL-09-104, Parallel Data Lab, Carnegie Mellon University, Pittsburgh PA, 2009. (http://www.pdl.cmu.edu/PDL-FTP/PDSI/CMU-PDL-09-104.pdf)

[3] "Programming Project 2: Storage Management in a Hybrid SSD/HDD File System," 14 Mar 2011. [Online]. Available: http://www.ece.cmu.edu/~ganger/746.spring11/projects/proj2_fuse/18746-s11-proj2.pdf [Accessed 25 May 2012]

[4] Anand Suresh, Garth Gibson, Greg Ganger: Shingled Magnetic Recording for Big Data Applications, Carnegie Mellon University Parallel Data Lab Technical Report CMU-PDL-12-105. May 2012. (http://www.pdl.cmu.edu/PDL-FTP/FS/CMU-PDL-12-105_abs.shtml)

[5] Kai Ren and Garth Gibson. 2013. TABLEFS: enhancing metadata efficiency in the local file system. In Proceedings of the 2013 USENIX conference on Annual Technical Conference (USENIX ATC'13). USENIX Association, Berkeley, CA, USA, 145-156. (http://www.cs.cmu.edu/~kair/papers/tablefs.pdf)

[6] Shingled Magnetic Recording: Areal Density Increase Requires New Data Management. Tim Feldman, Garth Gibson. USENIX v 38, n 3, June 2013. (http://www.pdl.cmu.edu/PDL-FTP/HECStorage/05_feldman_022-030.pdf)

[7] Garth Gibson: Directions for Shingled-Write and TDMR System Architectures: Synergies with Solid-State Disks (http://www.pdl.cmu.edu/PDL-FTP/PDSI/FA-06%20Gibson-Intermag-05-07-09.pdf)

[8] Cassuto Yuval Cassuto, Marco A. A. Sanvido, Cyril Guyot, David R. Hall, and Zvonimir Z. Bandic. 2010. Indirection systems for shingled-recording disk drives. In Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) (MSST '10). IEEE Computer Society, Washington, DC, USA, 1-14. DOI=10.1109/MSST.2010.5496971 (http://algo.epfl.ch/~yuval/papers/MSST10_shingled.pdf)

[9] Hall, D.; Marcos, J.H.; Coker, J.D., "Data Handling Algorithms For Autonomous Shingled Magnetic Recording HDDs," Magnetics, IEEE Transactions on , vol.48, no.5, pp.1777,1781, May 2012 URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6187785&isnumber=6187761

[10] Amer, A.; Holliday, J.; Long, D.D.E.; Miller, E.L.; Paris, J.; Schwarz, T., "Data Management and Layout for Shingled Magnetic Recording," Magnetics, IEEE Transactions on , vol.47, no.10, pp.3691,3697, Oct. 2011 (http://www.storageconference.org/2010/Papers/MSST/Amer.pdf)

[11] Tan, S.; Weiya Xi; Zhi Yong Ching; Chao Jin; Chun Teck Lim, "Simulation for a shingled magnetic recording disk," APMRC, 2012 Digest , vol., no., pp.1,7, Oct. 31 2012-Nov. 2 2012 URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6407526&isnumber=6401675

[12] Rekha Pitchumani, Andy Hospodor, Ahmed Amer, Yangwook Kang, Ethan L. Miller, and Darrell D. E. Long. 2012. Emulating a Shingled Write Disk. In Proceedings of the 2012 IEEE 20th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS '12). IEEE Computer Society, Washington, DC, USA, 339-346. DOI=10.1109/MASCOTS.2012.46 (http://www.ssrc.ucsc.edu/Papers/pitchumani-mascots12.pdf)

[13] Chung-I Lin ; Dept. of Comput. Sci. & Eng., Univ. of Minnesota-Twin Cities, Minneapolis, MN, USA ; Dongchul Park ; Weiping He ; Du, D.H.C. H-SWD: Incorporating Hot Data Identification into Shingled Write Disks, Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS), 2012 IEEE 20th International Symposium (http://www-users.cselabs.umn.edu/classes/Fall-2013/csci8980-adv/files/SWD/Layout_SWD.pdf)

[14] Rekha Pitchumani: Shingled Write Disk Emulator (http://www.ssrc.ucsc.edu/Papers/ssrctr-12-05.pdf)

[15] Jiguang Wan, Nannan Zhao, Yifeng Zhu, Jibin Wang, Yu Mao, Peng Chen, and Changsheng Xie. 2012. High Performance and High Capacity Hybrid Shingled-Recording Disk System. In Proceedings of the 2012 IEEE International Conference on Cluster Computing (CLUSTER '12). IEEE Computer Society, Washington, DC, USA, 173-181. DOI=10.1109/CLUSTER.2012.21 (http://web.eece.maine.edu/~zhu/papers/cluster2012_zhu.pdf)

[16] Breaking Capacity Barriers With Seagate Shingled Magnetic Recording (http://www.seagate.com/tech-insights/breaking-areal-density-barriers-with-seagate-smr-master-ti/)

3. Running the SMRfs code

Detailed instructions for running the SMRfs code can be found at - Build and Run SMRfs.

4. List of Benchmarks

SMRfs has been benchmarked against quite a few benchmarks. A complete list of all the benchmarks can be found at - SMRfs Benchmarks