Flex: Flexible and Energy Efficient Scheduling for Big Data Storage

Traditional distributed schedulers only consider the scheduling of jobs and treat the storage system as static and already deployed. However, individual application often needs its own configuration of storage system. Therefore, traditional distributed schedulers are not able to serve multiple tenants. To improve the resource utilization, some mechanisms are needed to consolidate multiple applications running on top of the same computing resources. Flex is such a system for helping the distributed scheduler to deploy a customized storage system before a job is scheduled without too much overhead incurred. The lightweight operating system level virtualization technology, aka containers in Linux, is used to wrap the computing environment and isolate different applications. Flex can achieve two goals that traditional distributed scheduler cannot provide. One is to deploy a specific storage system for the requirement of an individual application, the other is to consolidate applications for a better use of the underlying computing resources. Thus, by employing storage scheduling, Flex improves the resource utilization of distributed systems.