Using Conda On Your Device

Using Conda On Your Device

Conda is a package management system which can be used to create Python and R development environments on many different platforms. You can install it on your own device, and it is often available at many High Performance Computing centers. To install it on your own device:

Installation

Miniconda

  1. Download Miniconda. There are installers for Linux, macOS, and Windows. Miniconda is a much smaller version of Anaconda which includes more packages out of the box as well as a graphical user interfaces to the installer. Miniconda is better if you don't want to install too many unnecessary packages right at the start.
  2. Run the installer. This will prompt you a couple of times, and it is usually sensible to accept the defaults. Conda alters your shell initialization, and updates your executable path to find the base installation. For example, Miniconda will be installed into ~/miniconda3/ by default, so ~/miniconda3/bin/ will be added to your PATH variable. Conda may update your shell prompt as well with status information.
  3. Open a new terminal window. This will start up a new shell which will be aware of the new Conda tools. For example, on a Mac:
    (base) mylaptop:~$ type -a python
    python is /Users/me/miniconda3/bin/python
    python is /usr/bin/python
    

Anaconda for Mac ARM64-based machines (Apple Silicon M1 and M2 Macs)

With the M1 and M2 Macs, Apple is now using its own chips, referred to as Apple Silicon. These have a different architecture, ARM64, than standard x86-64 chips, such as those produced by Intel and AMD.

To take full advantage of the new chips, you can use Anaconda with ARM64 support. You can either install Anaconda using the 64 -Bit (M1) installer, or update your existing conda installation:

conda install anaconda=2022.05

Non-ARM64 programs, including standard Miniconda and Anaconda installations, will work on the new Apple Silicon-based machines by automatically making use of Apple's Rosetta2 system to translate machine code from ARM64 to x86_64, but you can expect some decrease in performance. 

Using Conda

Once conda is installed, you can use it to search for and install other things.

(base) mylaptop:~$ conda search pytorch
Loading channels: done
# Name                       Version           Build  Channel
pytorch                        1.3.1 cpu_py27h0c87eb2_0  pkgs/main
pytorch                        1.3.1 cpu_py36h0c87eb2_0  pkgs/main
pytorch                        1.3.1 cpu_py37h0c87eb2_0  pkgs/main
pytorch                        1.4.0 cpu_py36hf9bb1df_0  pkgs/main
pytorch                        1.4.0 cpu_py37hf9bb1df_0  pkgs/main
pytorch                        1.4.0 cpu_py38hf9bb1df_0  pkgs/main
pytorch                        1.6.0 cpu_py37hd70000b_0  pkgs/main
pytorch                        1.6.0 cpu_py38hd70000b_0  pkgs/main

Here we have searched for `pytorch` and conda has found several versions of it. There are also multiple build per version. For example above you can see that the latest version is 1.6.0 and there are builds for python3.7 and python3.8. If we ask conda to install version 1.6.0, it will check to make sure that it can proceed, and will tell you what it will do:

(base) mylaptop:~$ conda install pytorch=1.6.0
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/rylo/miniconda3

  added / updated specs:
    - pytorch=1.6.0


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _pytorch_select-0.1        |            cpu_0         169 KB
    blas-1.0                   |              mkl           6 KB
    ca-certificates-2020.6.24  |                0         125 KB
    certifi-2020.6.20          |           py38_0         156 KB
    conda-4.8.4                |           py38_0         2.9 MB
    intel-openmp-2019.4        |              233         887 KB
    libmklml-2019.0.5          |                0        16.2 MB
    llvm-openmp-10.0.0         |       h28b9765_0         236 KB
    mkl-2019.4                 |              233       101.9 MB
    mkl-service-2.3.0          |   py38hfbe908c_0          42 KB
    mkl_fft-1.1.0              |   py38hc64f4ea_0         138 KB
    mkl_random-1.1.1           |   py38h959d312_0         290 KB
    ninja-1.10.0               |   py38h879752b_0         102 KB
    numpy-1.19.1               |   py38h3b9f5b6_0          21 KB
    numpy-base-1.19.1          |   py38hcfb5961_0         4.1 MB
    pytorch-1.6.0              |cpu_py38hd70000b_0        34.0 MB
    ------------------------------------------------------------
                                           Total:       161.2 MB

The following NEW packages will be INSTALLED:

  _pytorch_select    pkgs/main/osx-64::_pytorch_select-0.1-cpu_0
  blas               pkgs/main/osx-64::blas-1.0-mkl
  intel-openmp       pkgs/main/osx-64::intel-openmp-2019.4-233
  libmklml           pkgs/main/osx-64::libmklml-2019.0.5-0
  llvm-openmp        pkgs/main/osx-64::llvm-openmp-10.0.0-h28b9765_0
  mkl                pkgs/main/osx-64::mkl-2019.4-233
  mkl-service        pkgs/main/osx-64::mkl-service-2.3.0-py38hfbe908c_0
  mkl_fft            pkgs/main/osx-64::mkl_fft-1.1.0-py38hc64f4ea_0
  mkl_random         pkgs/main/osx-64::mkl_random-1.1.1-py38h959d312_0
  ninja              pkgs/main/osx-64::ninja-1.10.0-py38h879752b_0
  numpy              pkgs/main/osx-64::numpy-1.19.1-py38h3b9f5b6_0
  numpy-base         pkgs/main/osx-64::numpy-base-1.19.1-py38hcfb5961_0
  pytorch            pkgs/main/osx-64::pytorch-1.6.0-cpu_py38hd70000b_0

The following packages will be UPDATED:

  ca-certificates                                2020.1.1-0 --> 2020.6.24-0
  certifi                                 2020.4.5.1-py38_0 --> 2020.6.20-py38_0
  conda                                        4.8.3-py38_0 --> 4.8.4-py38_0


Proceed ([y]/n)?

Typing `y`, or accepting the default choice (surrounded by brackets []) of `y` by pressing return, will let conda proceed.

Downloading and Extracting Packages
certifi-2020.6.20    | 156 KB    | ##################################### | 100%
blas-1.0             | 6 KB      | ##################################### | 100%
...
[snip]
...
pytorch-1.6.0        | 34.0 MB   | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

You can then start using the installed library.

(base) mylaptop:~$ python
Python 3.8.3 (default, May 19 2020, 13:54:14)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>

Environments

Conda will install all of your packages into your default environment. You may want to create a separate environment however if you are working on multiple projects simultaneously, if you want to be able to eventually share your environment with a collaborator, or if some package versions conflict with others.

Creating

To create a new environment:

(base) mylaptop:~$ conda create --name myproject2

You can also create new environments with an entirely different version of python:

(base) mylaptop:~$ conda create --name myproject2 python=3.8

To enter into your newly created environment:

(base) mylaptop:~$ conda activate myproject2

The shell prompt will be updated to reflect the active environment name.

(myproject2) mylaptop:~$ 

Also, note that your previously installed libraries are not in your new environment.

(myproject2) mylaptop:~$ python -m torch
/Users/me/miniconda3/envs/myproject2/bin/python: No module named torch

We will install a completely different version of pytorch into this environment.

(myproject2) mylaptop:~$ conda install pytorch=1.4.0
Collecting package metadata (current_repodata.json): done
...
[snip]
...
Downloading and Extracting Packages
pycparser-2.20       | 94 KB     | ##################################### | 100%
six-1.15.0           | 13 KB     | ##################################### | 100%
cffi-1.14.1          | 219 KB    | ##################################### | 100%
pytorch-1.4.0        | 26.3 MB   | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(myproject2) mylaptop:~$ python
Python 3.8.5 (default, Aug  5 2020, 03:39:04)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.4.0'

You can invoke the `activate` and `deactivate` conda commands to switch between the environments. The shell prompt is updated to reflect the activate environment.

(myproject2) mylaptop:~$ conda deactivate
(base) mylaptop:~$ 

Reproducing

You can create a snapshot of your current environment which can then be reproduced by collaborators, or even just yourself if you want to setup the same environment on another device. To export an environment:

(base) mylaptop:~$ conda activate myproject2
(myproject2) mylaptop:~$ conda env export --from-history > environment.yml

Transmit the `environment.yml` file to the other person or device. One can then create a new environment from that file:

(base) mylaptop:~$ conda env create -f environment.yml