Using Pyenv-virtualenv+direnv

Some times ago, I made a post entitled ‘Sandboxed Python with Pyenv, virtualenv and autoenv’. I’m a Ruby fan, and a rather poor python developer, however at the occasion of SC’17, I was playing with jupyter notebooks and tensorflow within a Machine Learning tutorial, two components that made me really enjoy (for the first time) Python. It was also the occasion to refresh my notes on virtualenv and to discover the excellent direnv tool, a better replacement to autoenv (from the own terms of autoenv author).

Here are my updated notes to setup a clean and sandboxed environment for python projects (similar to what can be done in Ruby with RVM and project gemsets).

Reference:

Installation

Under Mac OS X, prefer (as always) an installation through Homebrew:

1
2
3

$> brew update
$> brew install pyenv pyenv-virtualenv
$> brew install direnv

Under Linux, see pyenv-installer:

1
2
3
4

# Under Linux
$> curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash
$> git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv
$> { apt-get | yum } install direnv

You probably want also to activate the pyenv bash/zsh completion

Configuration

Adapt your environment i.e. ~/.{profile | bash* | zsh* etc.} to support pyenv shims, virtualenv and direnv:

1
2
3
4
5
6
7
8
9

# Bash
$> echo 'eval "$(pyenv init -)"'            >> ~/.bash_profile
$> echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile
$> echo 'eval "$(direnv hook bash)"'        >> ~/.bashrc
#
# ZSH
$> echo 'eval "$(pyenv init -)"'            >> ~/.zshenv
$> echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.zshenv
$> echo 'eval "$(direnv hook zsh)"'         >> ~/.zshrc

If you’re using oh-my-zsh, you probably want to enable the pyenv plugin

Now you can configure direnv:

1

$> mkdir -p ~/.config/direnv

You can now edit ~/.config/direnv/direnvrc to define the functions common and available to all your .envrc files. Add the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# ~/.config/direnv/direnvrc: Global configuration for direnv to make it compliant
# with pyenv -- see https://direnv.net/
#
# Adapted from
#  - https://github.com/direnv/direnv/wiki/Python#-pyenv and
#  - https://github.com/direnv/direnv/wiki/Python#-virtualenvwrapper
#
# use a certain pyenv version
use_python() {
    if [ -n "$(which pyenv)" ]; then
        local pyversion=$1
        pyenv local ${pyversion}
    fi
}
layout_virtualenv() {
    local pyversion=$1
    local pvenv=$2
    if [ -n "$(which pyenv-virtualenv)" ]; then
        pyenv virtualenv --force --quiet ${pyversion} ${pvenv}-${pyversion}
    fi
}
layout_activate() {
    if [ -n "$(which pyenv)" ]; then
        local pyenvprefix=$(pyenv prefix)
        local pyversion=$(pyenv version-name)
        local pvenv="$1"
        source ${pyenvprefix}/envs/${pvenv}-${pyversion}/bin/activate
    fi
}

Time for the awesome magic

Install a few different pythons

1
2
3
4
5
6
7

$> pyenv install --list
$> pyenv install 2.6.9
$> pyenv install 2.7.14
$> pyenv install 3.1.5
$> pyenv install 3.2.6
$> pyenv install 3.4.7
$> pyenv install 3.5.4

Assuming you wish to configure you project hosted within the /path/to/myproject directory:

Define the expected python version under .python-version:

1
2

$> cd /path/to/myproject
$> echo '2.7.14' > .python-version

Define the expected name for the virtualenv under .python-virtualenv:

1

$> echo 'myenv' > .python-virtualenv

Define the direnv configuration file .envrc for your project (you can actually copy/paste the below content for all your files):

1
2
3
4
5
6
7
8
9
10
11

# -*- mode: sh; -*-
# (rootdir)/.envrc : direnv configuration file
# see https://direnv.net/
pyversion=$(head .python-version)
pvenv=$(head     .python-virtualenv)
#
use python ${pyversion}
# Create the virtualenv if not yet done
layout virtualenv ${pyversion} ${pvenv}
# activate it
layout activate ${pvenv}-${pyversion}

1

$> curl -o .envrc https://raw.githubusercontent.com/Falkor/dotfiles/master/direnv/envrc

Allow the .envrc file

1
2
3
4

$> direnv allow .
direnv: loading .envrc
direnv: using python 2.7.14
direnv: export +VIRTUAL_ENV ~PATH

Enjoy your sand-boxed environment

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$> pip list
pip (9.0.1)
setuptools (37.0.0)
wheel (0.30.0)
#
# install your packages
$> pip install numpy scipy matplotlib
$> pip install jupyter ipykernel
$> python -m ipykernel install --user --name=$(head .python-virtualenv)
$> jupyter notebook
#
# freeze your environment to pass it around
$> pip freeze -l            # List all the pip packages used in the virtual environment
$> pip freeze -l > requirements.txt  # Dump it to a requirements file in the project folder

Leave your directory to unload the virtualenv and thus ensure the isolation:

1
2
3
4
5
6
7
8

$> pyenv version
2.7.14 (set by /path/to/myproject/.python-version)
$> python -V
Python 2.7.14
$> cd
direnv: unloading
$> python -V
Python 2.7.10

Reminder: You can later on install all pip packages back from the requirements.txt file (generated by pip freeze -l > requirements.txt) via:

1

$> pip install -r requirements.txt