Create a Sandboxed Python/ruby Environment

This post describes the configuration we use to have sandboxed environments for projects developped in Ruby or python. By sandboxed, we mean that for each project (typically hosted in a specific git repository), the set of packages (i.e. ruby gems or python modules) required for the projects are all separated and self-contained - from the system, and from each other.

Table of Content

Sandboxed Ruby with RVM

RVM is a command-line tool which allows you to easily install, manage, and work with multiple ruby environments from interpreters to sets of gems.

Installation

Under Mac:

  • Install XCode command lines – Maverick guide

  • Check that gcc is installed and available by command line

      $>  gcc --version

  • Install Homebrew

      $> ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

  • Install Git

      $> brew update
  $> brew install git

    At least, you shall configure git with your name and mail (adapt accordingly) and configure colors:

    $> git config –global user.name “Your Name Comes Here” $> git config –global user.email you@yourdomain.example.com # configure colors $> git config –global color.diff auto $> git config –global color.status auto $> git config –global color.branch auto

    You might be interested here in my personal git configuration

  • Install RVM

      $> \curl -sSL https://get.rvm.io | bash -s stable

Now you can install different versions of the official ruby interpreters:

$> rvm list known
$> rvm install 1.9.3 --default
$> ruby -v
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-darwin12.5.0]

At least, you shall now install by default the bundler gem

$> rvm @global do gem install bundler

Sandboxed ruby project

RVM gives you compartmentalized independent ruby setups. This means that ruby, gems and irb are all separate and self-contained - from the system, and from each other.

Sandboxing with RVM is straight-forward via the notion of gemsets. So for a given ruby project hosted in a given directory on your machine:

  • create a .ruby-version file at the root directory containing a single line: the ruby version to use.

      $> cat .ruby-version
  1.9.3

  • create a .ruby-gemset file containing the name of the gemset to create. Ex:

      $> cat .ruby-gemset
  myprojectname

  • create a Gemfile which list all the ruby packages used in the gemset. Ex:

      $> cat Gemfile
  source "https://rubygems.org"

  gem 'bundler', ">= 1.5.2"
  gem 'capistrano', "= 2.15.5"
  gem 'colored', "= 1.2"
  gem 'git_remote_branch', "= 0.3.7"
  gem 'puppet', "= 3.4.2"
  gem 'facter', "= 1.7.5"
  gem 'hiera', "= 1.3.2"
  gem 'hiera-json', "= 0.4.0"
  gem 'highline', "= 1.6.21"
  gem 'ipaddress', "= 0.8.0"
  gem 'json', "= 1.8.1"
  gem 'json_pure', "= 1.8.1"
  gem 'net-scp', "= 1.1.2"
  gem 'net-sftp', "= 2.1.2"
  gem 'net-ssh', "= 2.8.0"
  gem 'net-ssh-gateway', "= 1.2.0"
  gem 'rainbow', "= 2.0.0"
  gem 'rest-client', "= 1.6.7"
  gem 'ruby-ldap', "= 0.9.16"

  case RUBY_PLATFORM
  when /darwin/
        gem 'ruby-shadow', :git => 'https://github.com/apalmblad/ruby-shadow.git', :ref => 'osx'
  else
        gem 'ruby-shadow', "= 2.2.0"
  end
  gem 'term-ansicolor', "= 1.3.0"
  gem 'thor', "= 0.18.1"
  gem 'unicode_utils', "= 1.4.0"

    You can then install all of the required gems from that specified sources:

      $> bundle install
  $> git add .ruby-* Gemfile Gemfile.lock

    The second command adds in particular the Gemfile and Gemfile.lock to your repository. This ensures that the other developers on your project, as well as your deployment environment, will all use the same third-party code that you are using now.

Sandboxed Python with Pyenv, virtualenv and autoenv.

While I confess I’m not a python lover, I still had to work sometimes with projects written in python (such as Easybuild). In the attempt to setup a clean and sandboxed ennviroment (similar to the above one mentioned under ruby with RVM and project gemsets), I arrived to the following setup (typically assuming you installed Homebrew if you’re running Mac)

Installation

  • Install pyenv
    • Mac OS: brew install pyenv
    • Linux: curl https://raw.github.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash

    You probably want also to activate the pyenv bash/zsh completion

  • Install latest version pyenv-virtualenv (to support the activate/deactivate options)
  • Install autoenv
    • Mac OS: brew install autoenv
    • Linux: pip install autoenv

  • Adapt your environment i.e. ~/.{profile | bashrc | zshrc etc.} to support pyenv shims, virtualenv and autoenv

      #~/.profile
  #
  # [...]
  #
  if which pyenv > /dev/null; then;
      PATH="$PYENV_ROOT/bin:$PATH"
      eval "$(pyenv init -)"
  fi

  PATH=$PATH:$HOME/.rvm/bin # Add RVM to PATH for scripting

  AUTOENV_INIT=/usr/local/opt/autoenv/activate.sh

  if [ -f "${AUTOENV_INIT}" ]; then
      . ${AUTOENV_INIT}
  fi

Now you can typically:

  • install a set of python versions:

      $> pyenv install 2.7.6
  $> pyenv install 3.3.3
  $> pyenv install pypy-2.2.1

  • create a new virtual environment to isolate the packages you will install for this project / repository

      $> pyenv local 2.7.6                               # select a version of python
  $> pyenv virtualenv 2.7.6 ulhpc-env-modules-2.7.6  # create the virtual environment
  $> pyenv activate ulhpc-env-modules-2.7.6          # activate it

  # note: use `pyenv uninstall ulhpc-env-modules-2.7.6` to delete it

  • install python packages for this project

      $> pyenv versions
      system
      2.7.6
      3.3.3
      pypy-2.2.1
    * ulhpc-env-modules-2.7.6 (set by PYENV_VERSION environment variable)

  $> pip list
  pip (1.5.4)
  setuptools (2.2)
  wsgiref (0.1.2)

  $> pip install easybuild
  $> pip install GitPython

  $> pip list
  easybuild (1.11.1)
  easybuild-easyblocks (1.11.1)
  easybuild-easyconfigs (1.11.1.0)
  easybuild-framework (1.11.1)
  GitPython (0.1.7)
  pip (1.5.4)
  setuptools (2.2)
  wsgiref (0.1.2)

  • freeze your environment to pass it around

      $> pip freeze -l            # List all the pip packages used in the virtual environment
  GitPython==0.1.7
  easybuild==1.11.1
  easybuild-easyblocks==1.11.1
  easybuild-easyconfigs==1.11.1.0
  easybuild-framework==1.11.1

  $> pip freeze -l > requirements.txt  # Dump it to a requirements file in the project folder

  • ensure the isolation:

      $> pyenv version
  ulhpc-env-modules-2.7.6 (set by PYENV_VERSION environment variable)
  $> pyenv deactivate
  $> pyenv local 3.3.3
  $> pip list
  pip (1.4.1)
  setuptools (2.0)

Sandboxed Python project

pyenv gives you compartmentalized independent python setups. pyenv-virtualenv permits also to isolate python modules for your project.

So for a given ruby project hosted in a given directory on your machine:

  • create a .python-version file containing the python version to use (for pyenv)

      $> cat .python-version
  2.7.6

  • create a .python-virtualenv file containing the name of the virtual python environment to use for this repository, independent of the python version.

      $> cat .python-virtualenv
  ulhpc-env-modules
  • create a .env file, i.e. the autoenv setup for this repository which:
    • use (and create if non-existing) a new virtual environment ulhpc-env-modules-2.7.6

    • activate it

      $> cat .env
pyversion=`head .python-version`
pvenv=`head .python-virtualenv`

pyenv virtualenv --force --quiet ${pyversion} ${pvenv}-${pyversion}
# activate it
pyenv activate ${pvenv}-${pyversion}

  • as mentioned above, the equivalent of Gemfile for ruby can be generated using the pip freeze -l command, and a classical convention is to name this file requirements.txt. It lists all the pip packages used in the virtual environment.

    You can then install all pip packages back from the requirements.txt file via:

    $> pip install -r requirements.txt
$> git add .python-* .env requirements.txt

    The second command adds all the required files to your repository. This ensures that the other developers on your project, as well as your deployment environment, will all use the same third-party code that you are using now.