How do I do parallel programming?

Overview of Parallel Programming

There are two basic flavors of parallel processing (leaving aside GPUs): distributed memory and shared memory. With shared memory, multiple processors (which I'll call cores) share the same memory. With distributed memory, you have multiple nodes, each with their own memory. Each node generally has multiple cores. You can think of each node as a separate computer connected by a fast network to the other nodes.

November 2016 workshop

A two-part tutorial with extensive information on: campus and department computing resources, submitting jobs to the department and campus cluster, and various parallel programming approaches/tools/strategies, including demo code, are available in this Git repository on Github (in particular see the Download ZIP button on the lower right of that link to get all the materials as a zip file).  The HTML of the first session on computing resources and job submission is also available here, while the HTML of the second session on parallel programming approaches/tools/strategies is available here. You can clone the repository with a git clone, which can be done from the Linux/Mac command line as:

git clone https://github.com/berkeley-scf/parallel-scf-2016

Videos of the first session and second session are available. 

October 2015 workshop

A tutorial with extensive information on both flavors of parallel processing and on how to submit jobs to the SCF Linux cluster, as well as demo code, are available in this Git repository on Github (in particular see the Download ZIP button on the lower right of that link to get all the materials as a zip file).  The HTML of the tutorial is also available here. You can clone the repository with a git clone, which can be done from the Linux/Mac command line as:

git clone https://github.com/berkeley-scf/parallel-scf-2015

October 2014 workshop

A tutorial with extensive information on both flavors of parallel processing and on how to submit jobs to the SCF Linux cluster, as well as demo code, are available in this Git repository on Github (in particular see the Download ZIP button on the lower right of that link to get all the materials as a zip file).  The PDF of the tutorial is also available here. You can clone the repository with a git clone, which can be done from the Linux/Mac command line as:

git clone https://github.com/berkeley-scf/parallel-workshop-2014

2012 and 2013 workshops

Before October 2014, we had older versions of this material provided separately for shared memory and distributed memory parallel processing:

  1. For information on shared memory parallel programming, please see these notes on the basics of shared memory parallel programming in R, Matlab, Python and C, and the accompanying template code files (most of the code is also embedded in the notes).
  2. For information on distributed memory parallel programming using message passing, such as with MPI, see these notes on the basics of distributed memory parallel programming in R and C, and the accompanying template code files (most of the code is also embedded in the notes).