Developer toolchains for large-scale analytics: Two case studies

While big data analytics continue to grow in popularity among companies and organizations, their large-scale analytic implementations are often completed by software developers with little or no formal training in machine learning or data analysis. These developers are skilled at writing code but they do not have the understanding of the data analytics process to be efficient or necessarily accurate at it. These developers use processes and tools that are often ad hoc and incomplete as they learn by doing. We followed a development team through two analytics development cycles and analyzed their interactions with their data and tools. In this paper, we first describe the tools the developers used and then present concrete opportunities for the big data community to create tools that empower these developers to build more accurate analytics more efficiently.