Introduction

Purpose - December 2015, a university wide survey was sent out to inquire about research computing needs. 84% of survey participants indicated they have a direct need to be able to compute on “Big Data”. However, only 23% also indicated they have a need for high performance computing systems. How can this be possible?

How do you compute on data sets that by definition push memory and compute time limits, and not solve this through performance computing? This is when we realized that there is a large disconnect between what big Data is and how high performance computing can alleviate these limitations. This manual was created with the sole purpose of bridging big Data needs to HPC infrastructures within the scientific computing field. In this manual, we outline a process to simplify your computing needs to one of three choices. Given that choice, we outline the chosen solution. We are not going to re-invent the wheel here, so besides very large overviews of the given techniques, we direct users to the appropriate documentation elsewhere. Our goal is for you to be able to read this manual in a single 30 minute time window, and understand exactly where to find resources to implement needed solutions.

Audience - Any researcher that computes on data sets that impose burdens that slow down the research. This can either be due to the long compute time needed to complete the tasks, or due to memory limitations of existing hardware.

Pre-requisites - There is no skillset needed to read through this manual. However, it will make much more sense, and be exponentially more useful if you have some familiarity with linux command-line and some programming experience. Specifically, we wrote this manual for researchers that are already performing research computing, and need to know how to overcome a specific limitation as they scale their workflows to large problem sizes.