Using Pyenv-virtualenv+direnv

Some times ago, I made a post entitled ‘Sandboxed Python with Pyenv, virtualenv and autoenv’. This new post, considering direnv tool as a better replacement to autoenv (from the own terms of autoenv author) was initially created on Nov 22, 2017 and reviewed/updated in Sept 10, 2019.

So here are my updated notes to setup a clean and sandboxed environment for python projects (similar to what can be done in Ruby with RVM and project gemsets).

Reference:

• pyenv
• pyenv-virtualenv
  • see Note on virtualenv and venv: pyenv-virtualenv uses python -m venv if it is available (CPython 3.3 and newer) and the virtualenv command is not available.
• Blog posts:
  • The definitive guide to setup my Python workspace

Summary of the pyenv/pyenv-virtualenv most important commands:

Installation

Under Mac OS X, prefer (as always) an installation through Homebrew:

1
2

$> brew update
$> brew install pyenv pyenv-virtualenv direnv

Under Linux, see pyenv-installer:

1
2
3
4

# Under Linux
$> curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash
$> git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv
$> { apt-get | yum } install direnv    # direnv install

You probably want also to activate the pyenv bash/zsh completion

Configuration for pyenv[-virtualenv]

Adapt your environment i.e. ~/.{profile | bash* | zsh* etc.} to support pyenv shims, virtualenv and direnv. See my own pyenv.sh and direnv.sh common shell hook scripts for up-to-date content.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# To add in your favorite shell configuration (.bashrc, .zshrc etc.)
#
# - pyenv: https://github.com/pyenv/pyenv
# - pyenv-virtualenv: https://github.com/pyenv/pyenv-virtualenv
if [ -n "$(which pyenv)" ]; then
  eval "$(pyenv init -)"
  eval "$(pyenv virtualenv-init -)"
  export PYENV_VIRTUALENV_DISABLE_PROMPT=1
fi
#
# - direnv: https://direnv.net/
if [ -n "$(which direnv)" ]; then
  eval "$(direnv hook $(basename $SHELL))"
  # export DIRENV_WARN_TIMEOUT=100s
fi

If you’re using oh-my-zsh, you probably want to enable the pyenv plugin

Configuration for direnv

• Reference documentation:
• Direnv has a standard library of functions / layout
  • See direnv-stdlib - man page
• You can override the direnv-stdlib with your own set of function though ~/.config/direnv/direnvrc.
  • This file is loaded and it’s content made available to any .envrc file.
  • below notes permits to install my own version of this file used to make direnv interacts nicely and easily with pyenv and pyenv-virtualenv.
  • For review: my custom direnvrc configuration file installed below

1
2
3
4
5
6

# Prepare [custom] configuration directory for direnv
mkdir -p ~/.config/direnv
# Grab a copy of my own custom version of ~/.config/direnv/direnvrc
curl -o ~/.config/direnv/direnvrc https://raw.githubusercontent.com/Falkor/dotfiles/master/direnv/direnvrc
# Grab a copy of the generic 'envrc' file to be used as template '.envrc' for all your projects
curl -o ~/.config/direnv/envrc https://raw.githubusercontent.com/Falkor/dotfiles/master/direnv/envrc

From this global direnvrc configuration, a typical .envrc at the root of your project directory would look like.

For instance:

1
2
3
4
5
6
7
8
9
10
11

# Just to load the expected version from '.python-version' file if it exists
if [ -f ".python-version" ]; then
    pyversion=$(head .python-version)
else
    pyversion=3.7.4
fi
pvenv=$(basename $PWD)
#
use python ${pyversion}
layout virtualenv ${pyversion} ${pvenv}
layout activate ${pvenv}

You can take a look at my default template for the .envrc file

Note: DEEPRECATION: since Python 2.7 will reach the end of its life on January 1st, 2020, the above configuration enforces to use the latest Python 3.x version (at the time of writing).

Time for the awesome magic

First of all, you should have installed a few different recent pythons

1
2
3
4
5
6
7

pyenv install --list
pyenv install 2.7.16
pyenv install 3.7.4
pyenv install pypy2.7-7.1.1
pyenv install pypy3.6-7.1.1
# check available versions
pyenv versions

Assuming you wish to configure you project hosted within the /path/to/myproject directory:

Define the expected python version under .python-version and copy the default template for the local direnv configuration file .envrc:

1
2
3
4
5

cd /path/to/myproject
echo '3.7.4' > .python-version
cp ~/.config/direnv/envrc .envrc   # Assuming you copied my template
# OR
curl -o .envrc https://raw.githubusercontent.com/Falkor/dotfiles/master/direnv/envrc

Eventually, if you don’t want to use the virtualenv named myproject (i.e. the basename of the directory), you can define the expected name for the virtualenv under .python-virtualenv as follows:

1

$> echo 'myenv' > .python-virtualenv

That’s all ! Just allow (one time) the .envrc file and enjoy from now on the automatic loading/unloading of your virtualenv (created if needed) every time you visit/leave your project [sub]directories!

1
2
3
4
5

$> direnv allow .
direnv: loading .envrc
direnv: using python 3.7.4
[...]
direnv: export +PYENV_ACTIVATE_SHELL +PYENV_VERSION +PYENV_VIRTUAL_ENV +VIRTUAL_ENV ~PATH ~PYENV_SHELL

In particular, the following action were performed (and this will happen every time you enter this directory):

• the expected python version is loaded (default or from .python-version)
  • An error message is raised to ask you to install it if that version cannot be found
• a virtualenv myproject is created (or whatever name you specified in .python-virtualenv)
• this virtualenv is activated

Enjoy your sand-boxed environment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

$> pip list
Package    Version
---------- -------
pip        19.0.3
setuptools 40.8.0
#
#
# install your packages
$> pip install numpy scipy matplotlib
$> pip install jupyter ipykernel
#
# OR install all pip packages listed in 'requirements.txt' (see below)
#
$> pip install -r requirements.txt
#
# Play with it
$> python -m ipykernel install --user --name=$(head .python-virtualenv)
$> jupyter notebook
#
# freeze your environment to pass it around
$> pip freeze                        # List all the pip packages used in the virtual environment
$> pip freeze -l > requirements.txt  # Dump it to a requirements file in the project folder
#
#

Leave your directory to unload the virtualenv and thus ensure the isolation:

1
2
3
4
5
6
7
8
9
10

$> pyenv version
myproject (set by PYENV_VERSION environment variable)
$> python -V
Python 3.7.4
#
# Leave the directory
$> cd
direnv: unloading
$> python -V
Python 2.7.16

Reminder: You can later on install all pip packages back from the requirements.txt file (generated by pip freeze -l > requirements.txt) via:

1

$> pip install -r requirements.txt