Conda and Mamba

Conda and Mamba

Conda is a package management system which can be used to create Python and R development environments on many different platforms. You can install it on your own device, and it is often available at many High Performance Computing centers.

It's common that installing packages using Conda is slow or fails because Conda is unable to resolve dependencies. To get around this, we suggest the use of Mamba.

Mamba is a drop-in replacement for Conda that is generally faster and better at resolving dependencies.

In the examples below, we show the syntax for Conda, but you can simply replace "conda" with "mamba" on the SCF (or if you've installed Mamba on your own machine), e.g., "mamba install pytorch" rather than "conda install pytorch".

Installing software using Conda/Mamba

We have information on using Conda (and Mamba) to install software on the SCF.

Using Conda/Mamba

Once conda is installed, you can use it to search for and install other things.

(base) mylaptop:~$ conda search pytorch
Loading channels: done
# Name                       Version           Build  Channel
pytorch                        1.3.1 cpu_py27h0c87eb2_0  pkgs/main
pytorch                        1.3.1 cpu_py36h0c87eb2_0  pkgs/main
pytorch                        1.3.1 cpu_py37h0c87eb2_0  pkgs/main
pytorch                        1.4.0 cpu_py36hf9bb1df_0  pkgs/main
pytorch                        1.4.0 cpu_py37hf9bb1df_0  pkgs/main
pytorch                        1.4.0 cpu_py38hf9bb1df_0  pkgs/main
pytorch                        1.6.0 cpu_py37hd70000b_0  pkgs/main
pytorch                        1.6.0 cpu_py38hd70000b_0  pkgs/main

Here we have searched for `pytorch` and conda has found several versions of it. There are also multiple build per version. For example above you can see that the latest version is 1.6.0 and there are builds for python3.7 and python3.8. If we ask conda to install version 1.6.0, it will check to make sure that it can proceed, and will tell you what it will do:

(base) mylaptop:~$ conda install pytorch=1.6.0
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/me/miniconda3

  added / updated specs:
    - pytorch=1.6.0


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _pytorch_select-0.1        |            cpu_0         169 KB
    blas-1.0                   |              mkl           6 KB
    ca-certificates-2020.6.24  |                0         125 KB
    certifi-2020.6.20          |           py38_0         156 KB
    conda-4.8.4                |           py38_0         2.9 MB
    intel-openmp-2019.4        |              233         887 KB
    libmklml-2019.0.5          |                0        16.2 MB
    llvm-openmp-10.0.0         |       h28b9765_0         236 KB
    mkl-2019.4                 |              233       101.9 MB
    mkl-service-2.3.0          |   py38hfbe908c_0          42 KB
    mkl_fft-1.1.0              |   py38hc64f4ea_0         138 KB
    mkl_random-1.1.1           |   py38h959d312_0         290 KB
    ninja-1.10.0               |   py38h879752b_0         102 KB
    numpy-1.19.1               |   py38h3b9f5b6_0          21 KB
    numpy-base-1.19.1          |   py38hcfb5961_0         4.1 MB
    pytorch-1.6.0              |cpu_py38hd70000b_0        34.0 MB
    ------------------------------------------------------------
                                           Total:       161.2 MB

The following NEW packages will be INSTALLED:

  _pytorch_select    pkgs/main/osx-64::_pytorch_select-0.1-cpu_0
  blas               pkgs/main/osx-64::blas-1.0-mkl
  intel-openmp       pkgs/main/osx-64::intel-openmp-2019.4-233
  libmklml           pkgs/main/osx-64::libmklml-2019.0.5-0
  llvm-openmp        pkgs/main/osx-64::llvm-openmp-10.0.0-h28b9765_0
  mkl                pkgs/main/osx-64::mkl-2019.4-233
  mkl-service        pkgs/main/osx-64::mkl-service-2.3.0-py38hfbe908c_0
  mkl_fft            pkgs/main/osx-64::mkl_fft-1.1.0-py38hc64f4ea_0
  mkl_random         pkgs/main/osx-64::mkl_random-1.1.1-py38h959d312_0
  ninja              pkgs/main/osx-64::ninja-1.10.0-py38h879752b_0
  numpy              pkgs/main/osx-64::numpy-1.19.1-py38h3b9f5b6_0
  numpy-base         pkgs/main/osx-64::numpy-base-1.19.1-py38hcfb5961_0
  pytorch            pkgs/main/osx-64::pytorch-1.6.0-cpu_py38hd70000b_0

The following packages will be UPDATED:

  ca-certificates                                2020.1.1-0 --> 2020.6.24-0
  certifi                                 2020.4.5.1-py38_0 --> 2020.6.20-py38_0
  conda                                        4.8.3-py38_0 --> 4.8.4-py38_0


Proceed ([y]/n)?

Typing `y`, or accepting the default choice (surrounded by brackets []) of `y` by pressing return, will let conda proceed.

Downloading and Extracting Packages
certifi-2020.6.20    | 156 KB    | ##################################### | 100%
blas-1.0             | 6 KB      | ##################################### | 100%
...
[snip]
...
pytorch-1.6.0        | 34.0 MB   | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

You can then start using the installed library.

(base) mylaptop:~$ python
Python 3.8.3 (default, May 19 2020, 13:54:14)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>

Environments

Conda will install all of your packages into your default environment. However there are various situations in which you may want to create a separate environment, These include (1) if you are working on multiple projects simultaneously, (2) if you want to be able to eventually share your environment with a collaborator, and (3) if some package versions conflict with others in your default environment.

Creating

To create a new environment:

(base) mylaptop:~$ conda create --name myproject2

You can also create new environments with an entirely different version of python:

(base) mylaptop:~$ conda create --name myproject2 python=3.8

Note that under some situations, your environment might still make use of the default environment for the Python executable or certain packages already available in the default environment. See here for details and tips on creating a completely isolated environment.

To enter into your newly created environment:

(base) mylaptop:~$ source activate myproject2

The shell prompt will be updated to reflect the active environment name.

You can also activate an environment with `conda activate myproject2`, but see below for some cautionary notes before doing that.

(myproject2) mylaptop:~$ 

Also, note that your previously installed libraries are not in your new environment.

(myproject2) mylaptop:~$ python -m torch
/Users/me/miniconda3/envs/myproject2/bin/python: No module named torch

We will install a completely different version of pytorch into this environment.

(myproject2) mylaptop:~$ conda install pytorch=1.4.0
Collecting package metadata (current_repodata.json): done
...
[snip]
...
Downloading and Extracting Packages
pycparser-2.20       | 94 KB     | ##################################### | 100%
six-1.15.0           | 13 KB     | ##################################### | 100%
cffi-1.14.1          | 219 KB    | ##################################### | 100%
pytorch-1.4.0        | 26.3 MB   | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(myproject2) mylaptop:~$ python
Python 3.8.5 (default, Aug  5 2020, 03:39:04)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.4.0'

You can invoke the `activate` and `deactivate` conda commands to switch between the environments. The shell prompt is updated to reflect the activate environment.

(myproject2) mylaptop:~$ conda deactivate
(base) mylaptop:~$ 

Conda activate versus source activate

When you first try to use `conda activate`, Conda will prompt you to run `conda init` to initialize conda for the shell you are using. You can do this, but note that it will modify your `.bashrc` so that Conda commands are available whenever you log in. As part of this, Conda will put you in the base Conda environment automatically when a new shell starts. Note that on the SCF, this will prevent you from accessing the Python version and related packages that the SCF provides. To avoid this, we recommend running `conda config --set auto_activate_base False` after running `conda init` on the SCF.

Using `source activate` (and `source deactivate`) instead of `conda activate` (and `conda deactivate`) are deprecated but still work, and you may wish to use them in some situations (such as on the SCF) to avoid the behavior discussed above.

Reproducing an environment

You can create a snapshot of your current environment which can then be reproduced by collaborators, or even just yourself if you want to setup the same environment on another device. To export an environment:

(base) mylaptop:~$ source activate myproject2
(myproject2) mylaptop:~$ conda env export --from-history > environment.yml

Transmit the `environment.yml` file to the other person or device. One can then create a new environment from that file:

(base) mylaptop:~$ conda env create -f environment.yml

Installing Mamba/Conda on your own computer

On the SCF, we use Mamba as a drop-in replacement for Conda. For your own computer, we recommend using the Mambaforge distribution as your Python installation rather than Anaconda/Miniconda. Mambaforge by default uses the conda-forge channel, a popular channel providing a wide variety of up-to-date packages.

Using Mambaforge (recommended)

You can follow these instructions to install Mambaforge. 

Miniforge: a minimal installation

To install a minimal installation with a small set of packages, we recommend Miniforge, which uses the conda-forge channel, which provides a broader set of updated packages than the default anaconda channel. You can optionally have it use Mamba rather than Conda. 

Mamba for Mac ARM64-based machines (Apple Silicon M1 and M2 Macs)

With the M1 and M2 Macs, Apple is now using its own chips, referred to as Apple Silicon. These have a different architecture, ARM64, than standard x86-64 chips, such as those produced by Intel and AMD.

To take full advantage of the new chips, you can use either Mambaforge (and Miniforge); both provide versions for ARM64. 

Non-ARM64 programs, including non-ARM64 Mambaforge installations, will work on the new Apple Silicon-based machines by automatically making use of Apple's Rosetta2 system to translate machine code from ARM64 to x86_64, but you can expect some decrease in performance. 

Using Anaconda/Miniconda

While we recommend installing Mambaforge, you can also install Anaconda or a minimal installation called Miniconda. 

Miniconda

  1. Download Miniconda. There are installers for Linux, macOS, and Windows. Miniconda is a much smaller version of Anaconda which includes more packages out of the box as well as a graphical user interfaces to the installer. Miniconda is better if you don't want to install too many unnecessary packages right at the start.
  2. Run the installer. This will prompt you a couple of times, and it is usually sensible to accept the defaults. Conda alters your shell initialization, and updates your executable path to find the base installation. For example, Miniconda will be installed into ~/miniconda3/ by default, so ~/miniconda3/bin/ will be added to your PATH variable. Conda may update your shell prompt as well with status information.
  3. Open a new terminal window. This will start up a new shell which will be aware of the new Conda tools. For example, on a Mac:
    (base) mylaptop:~$ type -a python
    python is /Users/me/miniconda3/bin/python
    python is /usr/bin/python
    

Anaconda for Mac ARM64-based machines (Apple Silicon M1 and M2 Macs)

With the M1 and M2 Macs, Apple is now using its own chips, referred to as Apple Silicon. These have a different architecture, ARM64, than standard x86-64 chips, such as those produced by Intel and AMD.

To take full advantage of the new chips, you can use Anaconda with ARM64 support. You can either install Anaconda using the 64 -Bit (M1) installer, or update your existing conda installation:

conda install anaconda=2022.05

Non-ARM64 programs, including standard Miniconda and Anaconda installations, will work on the new Apple Silicon-based machines by automatically making use of Apple's Rosetta2 system to translate machine code from ARM64 to x86_64, but you can expect some decrease in performance.