Data Infrastructure at LinkedIn

Author

Auradkar, Aditya ; Botev, Chavdar ; Das, Shirshanka ; De Maagd, D. ; Feinberg, Alex ; Ganti, Phanindra ; Gao, Lei ; Ghosh, Bhaskar ; Gopalakrishna, Kishore ; Harris, Brendan ; Koshy, Joel ; Krawez, Kevin ; Kreps, Jay ; Lu, Shi ; Nagaraj, Sunil ; Narkhede,

fYear

2012

fDate

1-5 April 2012

Firstpage

1370

Lastpage

1381

Abstract

Linked In is among the largest social networking sites in the world. As the company has grown, our core data sets and request processing requirements have grown as well. In this paper, we describe a few selected data infrastructure projects at Linked In that have helped us accommodate this increasing scale. Most of those projects build on existing open source projects and are themselves available as open source. The projects covered in this paper include: (1) Voldemort: a scalable and fault tolerant key-value store, (2) Data bus: a framework for delivering database changes to downstream applications, (3) Espresso: a distributed data store that supports flexible schemas and secondary indexing, (4) Kafka: a scalable and efficient messaging system for collecting various user activity events and log data.

Keywords

database indexing; distributed databases; public domain software; social networking (online); software fault tolerance; storage management; Data bus; Espresso; Kafka; LinkedIn; Voldemort; activity events; core data sets; data infrastructure projects; database change delivery; distributed data store; fault tolerant key-value store; flexible schemas; log data; messaging system; open source projects; request processing requirements; secondary indexing; social networking sites; Companies; Indexes; LinkedIn; Pipelines; Routing; Servers;

fLanguage

English

Publisher

ieee

Conference_Titel

Data Engineering (ICDE), 2012 IEEE 28th International Conference on

Conference_Location

Washington, DC

ISSN

1063-6382

Print_ISBN

978-1-4673-0042-1

Type

conf

DOI

10.1109/ICDE.2012.147

Filename

6228206