Supercomputing: Concepts

  • “Supercomputers are far more powerful than desktop computers”
  • “Most supercomputers today are HPC clusters”
  • “Supercomputers are intended to run large calculations or simulations that takes days or weeks to complete”
  • “HPC clusters are aggregates of machines that operate as a single entity to provide computational power to many users”

The Life Cycle of a Job

  • “To execute jobs on an HPC cluster you need to move across the filesystem, create folders, edit files and submit jobs to the cluster”
  • “All these elements will be learned in the following episodes”

Command Line Interface: The Shell

  • Get sufficient familiarity with basic comands such as ls, cp, cd, pwd and rm
  • If you want to know extra arguments for command that you know use the command man
  • Most Linux/UNIX commands are just programs that have existed for decades and whose use has been for most part standarized overtime. The commands that you learn will serve you in the future so it is a fruitful investment.

Terminal-based Text Editors: nano, emacs and vim

  • “There are three major editor in Linux/Unix: nano, emacs and vim”
  • “Once decided for on text editor it is important to the basic key combinations and operation of your editor of choice”

Data Transfer: Globus

  • “Supercomputers are far more powerful than desktop computers”
  • “Most supercomputers today are HPC clusters”
  • “Supercomputers are intended to run large calculations or simulations that takes days or weeks to complete”
  • “HPC clusters are aggregates of machines that operate as a single entity to provide computational power to many users”

Workload Manager: Slurm

  • “The scheduler handles how compute resources are shared between users.”
  • “Everything you do should be run through the scheduler.”
  • “A non-interactive job is expressed as a shell script that is submitted to the cluster.”
  • “Try to adjust the wall time to around 10-20% more than the expected time the job should need.”
  • “It is a good idea to keep aliases to common torque commands for easy execution.”

Software on HPC Clusters: modules, conda and containers

  • Modules. Use module avail to know all the modules on the cluster.
  • Modules. Use module load <module_name> to load a module.
  • Conda. Use conda env list to list the available environments.
  • Conda. Use conda activate to activate a conda environment.
  • Singularity. Use singularity shell <container> to get a shell inside the container.
  • Singularity. Use singularity exec <container> <command> to execute a command or script inside the container.

Terminal Multiplexing: tmux

  • “A terminal multiplexer protects programs such as text editors that are running on the login node from connection drops”
  • “Allow programs running on a remote server to be accessed from multiple different local computers.”
  • “Work with multiple programs and shells together in one terminal, a bit like a window manager.”