Summary and Setup
Lecturer: Guillermo Avendano Franco
Summary
This workshop is an introduction to High-Performance Computing (HPC) and Research Computing in general for users of our clusters. You will learn the basics of the Linux command line interface, terminal-based text editors, job submission via Slurm, using software and a few other topics that allow you to make efficient use of a High Performance Computing cluster.
This lesson pretends to go straight to the point. You will learn some useful commands, tools and techniques to start using WVU’s HPC clusters for your research. As on any introductory course, no completeness should be assumed. For each episode, there is a lot more to explore that could be interesting for more advanced users. Some references are left at the end of each section.
This lesson assumes no previous knowledge of Linux and the command line interface (CLI). If you are only familiar with working from a terminal expect a bit of a rough start but please do not lose your motivation. Learning Linux and command lines is like exploring a new language, expect mistakes and frustrations, but also rewards as you move forward and start grasping the tools that will make your interaction with the cluster more efficient and your research more dynamic.
For more documentation see our WVU HPC Docs.
There are two ways of today accessing WVU’s HPC Clusters: A secure remote shell (SSH) and via a Web Browser using the web application Open On-Demand. A secure remote shell is the traditional way of accessing a High-Performance Computing (HPC) cluster. A remote shell allows you to execute commands on another machine as you do sitting in front of it. A remote shell is convenient because it also allows other people to do the same with very little consumption of resources, so you get access to a resource being utilized by several users simultaneously.
Access via Secure Shell (SSH)
A remote shell is the most common way of accessing a High-Performance Computing (HPC) cluster. A remote shell allows you to execute commands on another machine as you do sitting in front of it. A remote shell is convenient because it also allows many other machine users to do the same, so you get access to a resource being utilized by tens or hundreds of users simultaneously.
All you need on your computer is a terminal emulator and an SSH client. A terminal emulator is a program that mimics the behavior of old dumb terminals from decades ago. An SSH client is a program on your computer that allows you to connect to another computer using an encrypted channel. In the old times (the 80s), people used to create a remote shell using an unencrypted channel known as Telnet. Today, SSH provides a secure channel over an unsecured network such as the Internet. SSH offers similar capabilities to Telnet but adds encryption, so all data sent and received between your computer and the remote host is encoded in such a way that only your computer and the remote computer can see the data. If you want to learn more about Secure Shell, you can find it on Wikipedia. Currently (2024), WVU offers access to three HPC clusters: Thorny Flat, Dolly Sods, and the CTSI HPC Cluster. You can access them using SSH. The instructions here are valid for our main two clusters, Thorny Flat and Dolly Sods.
Regardless of which operating system you use, there is always a way to install the terminal emulator and SSH client that you need to connect to an HPC cluster. Both Linux and macOS include an SSH client by default. For these operating systems, all that you have to do is to open a terminal. On macOS, the terminal is located in Applications under the Utilities folder. On Linux, the terminal is so central that most Linux distributions create an icon directly from the desktop. Otherwise, you can search for it in your applications bar.
On Windows machines, you have several options. You can install an external application or use Windows utilities such as Powershell or Visual Studio Code, which are included in modern versions of Windows. For the external application, one option is a free application called PuTTY. PuTTY offers a simple SSH client that is enough for this lesson. Another option is MobaXTerm, which offers a full-featured SSH client plus the ability to open X11 windows from the remote machine. There are many alternatives to these two. As soon as the application supports the SSH protocol, you can use it to connect to the clusters.
More recently, there have also been options for accessing a terminal and SSH clients using Microsoft products. One option is to install Visual Studio Code. There are instructions on Remote Development using SSH. Visual Studio Code will give you remote access and a file manager that you can use to move files between the cluster and your computer.
Another alternative is using the Windows Subsystem for Linux (WSL). WSL will allow you to run a full Linux OS inside Windows with all the commands that you will encounter on a typical Linux machine. With the latest version of WSL 2 and Windows 10 or 11, you can even run GUI applications from inside Windows alongside other Windows applications. You can see the files on your Windows machine. Running and completing Linux OS inside Windows will give you access not only to SSH client and terminal but also to the commands that you will learn during this lesson, as you will do from a Native Linux machine. Read also Windows Subsystem for Linux Documentation
Connecting to Thorny Flat and Dolly Sods
Most likely, you will connect to Thorny Flat, our flagship HPC cluster. The instructions below work for our two main HPC clusters Thorny Flat which is a general-purpose HPC cluster, and Dolly Sods, which is a GPU-based HPC cluster. Accounts on those clusters are independent, so you can access one without necessarily having an account on the other.
To access Thorny Flat from your computer, execute this command from your terminal (Linux, macOS):
~$ ssh <username>@ssh.wvu.eduThis command will allow you access to WVU’s SSH gateway. An intermediate machine that controls the access to machines inside the campus. The machine will ask for your password and DUO authentication. Once you are accepted into the system, you will see a command prompt and indication that you can start typing commands that will be executed on the SSH gateway.
The commands that will allow you to enter into the HPC clusters are: For Thorny Flat:
~$ ssh ts.hpc.wvu.eduFor Dolly Sods:
~$ ssh ds.hpc.wvu.eduIf you are using a GUI client such as PuTTY enter
ssh.wvu.edu as the server’s name and your username in the
corresponding field before connecting to the remote machine.
After entering your credentials with DUO, you get a prompt on the SSH gateway, and from there, you go to Thorny Flat.
Logging Out
Logging out of a cluster can be done with the exit command:
~$ exitExecuting this command once will return you to the SSH gateway.
Execute exit again, and you will leave the SSH gateway and
return to a prompt on your machine.
FIXME: Setup instructions live in this document. Please specify the tools and the data sets the Learner needs to have installed.
Alternative SSH clients
Details
There are more ways of accessing terminals and SSH clients in particular in Microsoft Windows that usually obscure the terminal access to casual users. The blocks below present a more detailed explanation of how to access a terminal in various OS from which you can execute the commands above to access our HPC clusters.
There is no bash terminal default on Windows. Below are the steps to acquire one.
Enable WSL
Press the Windows key and type “Windows Powershell”.
Right-click and “Run as administrator”.
-
Paste the following into the shell and press
enter.dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart Close the Powershell and restart the computer.
Download the Ubuntu Subspace
- Press the Windows key.
- Type “Microsoft Store” and press enter.
- Search for “Ubuntu” (the search bar is in the top right of the store) and click the top-most result.
- Install Ubuntu.
This installation gives your Windows a little bit of Linux inside of
it. Launching the Ubuntu app will give you a terminal and environment
quite similar to those of a Linux machine like our HPCs. Your normal “C
drive”, “D drive”, and so on are found under /mnt in the
subspace. More on accessing those later.
- The terminal is usually a button on your desktop.
- The icon is commonly depicted as a monitor with a black screen.
Access via Open On-Demand
Open On-Demand is an easy way to access our HPC clusters from a Web Browser. The only software that you need is a fairly modern web Browser on your side. Open On Demand works with Firefox, Safari, Microsoft Edge and many other web browsers.
You can connect to Thorny Flat using ood thorny. To connect to Dolly Sods use ood dolly.
Downloading the materials
The materials of examples and exercises can be downloaded by executing this command on the terminal.
~$ git clone https://github.com/WVUHPC/workshops_hands-on.gitThis command will download all the files used in our workshops into a
folder called workshops_hands-on. You can move into the
particular folder for this workshop using the command.
~$ cd workshops_hands-on/Introduction_HPCIf you are unfamiliar with these commands, do not worry; you are in the right class and will learn all these commands and how to use them. These and many other commands will be the topic of the material that follows.