Parallel Programming: Glossary

Parallel computing start with identifying the concurrent opportunities of a given problem and decide the best paradigm to exploit the concurrency with the minimal programming effort.

GNU parallel is just a tool to exploit embarrassing parallel problems. You can equally use multiprocessing in python or R parallel libraries for the same effect.

Both GCC and Intel compilers offer OpenMP support. Loops can be processed in parallel if there is no data dependency between each iteration of the loop.

MPI is the fact standard for large scale numerical computing, where a single node no matter how big, does not have the cores or memory able to run the simulation.

GPU computing is nowadays the landmark of HPC clusters, the most powerful computers in the world rely on them for part of most of its multi PetaFLOP capabilities.