Parallel Processing of Data, Metadata, and Aggregates within an Archival Storage System User Interface (Toward Archiving a Million Files and a Million Megabytes per Minute)

Author

Roschke, Mark A. ; Parliman, Bart J. ; Cook, Danny P. ; Sherrill, C. David

Author_Institution

High Performance Comput. Div., Los Alamos Nat. Lab., Los Alamos, NM

fYear

2008

fDate

22-22 Sept. 2008

Firstpage

36

Lastpage

43

Abstract

Archiving large datasets requires parallel processing of both data and metadata for timely execution. This paper describes the work in progress to use various processing techniques, including multi-threading of data and metadata operations, distributed processing, aggregation, and conditional processing to achieve increased archival performance for large datasets.

Keywords

data handling; information retrieval systems; meta data; multi-threading; archival storage system; conditional processing; data processing; large datasets; metadata processing; multithreading; parallel processing; user interface; Aggregates; Bandwidth; Conferences; File systems; High performance computing; Information retrieval; Laboratories; Multithreading; Parallel processing; User interfaces; aggregation; archive; data; interface; metadata; parallel;

fLanguage

English

Publisher

ieee

Conference_Titel

Storage Network Architecture and Parallel I/Os, 2008. SNAPI '08. Fifth IEEE International Workshop on

Conference_Location

Baltimore, MD

Print_ISBN

978-0-7695-3408-4

Type

conf

DOI

10.1109/SNAPI.2008.10

Filename

4685845