BlueDBM: A multi-access, distributed flash store for Big Data analytics

Complex analytics of the vast amount of data collected via social media, cell phones, ubiquitous smart sensors, and satellites is likely to be the biggest economic driver for the IT industry over the next decade. For many “Big Data” applications, the limiting factor in performance is often the transportation of large amount of data from hard disks to where it can be processed, i.e. DRAM. We will present BlueDBM, an architecture for a scalable distributed flash store which is designed to overcome this limitation in two ways. First, the architecture provides a high-performance, high-capacity, scalable random-access storage. It achieves high-throughput by sharing large numbers of flash chips across a low-latency, chip-to-chip backplane network managed by the flash controllers. Second, it permits some computation near the data via a FPGA-based programmable flash controller. We will present the preliminary results on accelerating complex queries using BlueDBM consisting of 20 nodes and up to 20TB of flash.