Calendar Details

For more information about item submission and attendance, see About the Technical Calendar.

Tuesday, August 06

Processing and Analysis of Very Large Data Sets

Fernanda Foertter and Bobby Whitten,
Computing and Computational Sciences Driectorate, ORNL
Computing and Computational Sciences Directorate,
Oak Ridge Leadership Computing Facility, Workshop
9:00 AM — 5:00 PM, Hilton Hotel, 501 West Church Avenue, Knoxville
Contact: Fernanda Foertter (, 865.576.9391


This workshop focuses on the whole lifetime of large datasets. From job prep, to jobs, to analysis, this workshop will help you better deal with large data from acquisition to publication. Planned topics include

  • How do I know if I have BIG data?
  • What you should use for large data prep and analysis
  • Why shuffling data during a job kills performance and how you can improve it
  • Libraries: better ways to do parallel I/O
  • Are all file formats the same?
  • How do I begin to visualize enormous datasets?
  • In situ analysis: a how-to
  • Sharing your massive data with friends and strangers
  • Future outlook on a growing data problem
  • Hands on tutorials are also planned for various scripting languages, parallel I/O libraries, and viz and analysis tools.

    For more information, go to .