Behind the Wizard’s Curtain (OneFS Revealed, Part III)

If you’re still reading, I can only presume you are enjoying this long walk behind the wizard’s curtain into the depths of OneFS®’ magic, or that you’re just hoping for another chili analogy. I’ll finish this series with a look at how OneFS scales to meet any and all performance demands and is, in a word, awesome 😉

A storage system can only be generally useful if it provides the necessary performance for a large variety of workflows; sequential, concurrent, and random are the typical categories that workflows fall into. Not only can different workflows exist within a storage system at the same time, and grow over time, but they also will tend to differ within a time period while still exhibiting principles of temporal or spatial locality. OneFS provides for all of these needs simultaneously – by distributing data across many spindles and processing nodes, the storage system is able to provide access to the raw storage hardware that a workflow needs and scale that hardware seamlessly over time.

This alone doesn’t exploit the spatial and temporal locality; in order to maximize workflow behavior, OneFS provides a globally accessible and coherent cache across all nodes. Each storage node contains standard DRAM (between 4 and 16 GB); this memory is primarily used to cache data that has been placed on that particular storage node and is actively being accessed. This cache grows as more nodes are added to a cluster, allowing an increasing working set to continually remain in cache – up to 2.3 TBs can be cached in a single system. In addition, OneFS allows the application architect to specify the type of workload on a per-file or per-directory basis – indicating whether the access pattern to a particular file/directory is random, concurrent, or sequential. This unique capability of OneFS, called SmartFiles, allows OneFS to tailor on-disk layout decisions, cache-retention policies, and data pre-fetch policies in order to maximize performance of the individual workflows.

In all the key aspects of scalability – simplicity, efficiency, reliability, flexibility, and performance – OneFS has been uniquely designed from the onset to provide not only a scalable platform today, but also a platform which can be evolved seamlessly over time. The trends that drove these design decisions – commoditization of hardware, acceleration of networking fabric, increasing user demands (both capacity and performance), and a organizational need to do more with less – are long-term, secular trends that will continue to be relevant in years to come.

Now, who’s up for chili?

About the Author: Nick Kirsch