Below are some short notes explaining these elements, and how I like to install Easubuild
Table of Content
- Environment modules
Environment Modules are a standard and well-established technology across HPC sites, to permit developing and using complex software builds with dependencies, allowing multiple versions of software stacks and combinations thereof.
Modules – Software Environment Management, offers a standard and well-established approach common to most HPC facilities to provide complex [tweaked] software and libraries with dependencies in multiple versions to their users.
The tool in itself is used to manage environment variables such as
MANPATH, enabling the easy loading and unloading of application/library profiles and their dependencies.
||Lists all the modules which are available to be loaded|
||Load a module|
||Unload a module|
||List loaded modules|
||Unload all modules (purge)|
||Display what a module does|
||Prepend the directory to the MODULEPATH environment variable|
||Remove the directory from the MODULEPATH environment variable|
You can also see the following pages on the UL HPC website:
At the heart of environment modules interaction resides the following components:
MODULEPATHenvironment variable, which defined the list of searched directories for modulefiles
modulefile(see an example) associated to each available software.
Lmod is a new implementation of Environment Modules that easily handles the MODULEPATH Hierarchical problem. It is drop-in replacement for TCL/C modules and reads TCL modulefiles directly. In particular, Lmod add many interesting features on top of the traditional implementation focusing on an easier interaction (search, load etc.) for the users. Thus that’s the tool I would advise to deploy.
Installation Notes (Mac OS X)
The best is to use HomeBrew
1 2 3 4
Note: You can also use
--local option (or the
--tree <path>) to have the LUA packages installed in
<path>). If you use
--tree <path>, you need to update the environmental variables
LUA_CPATH as follows:
1 2 3
Now it should be fine to install LMod:
After this installation:
lmodcommand is located in
$(brew --prefix lmod)/libexecand you probably wants to make an alias for it
alias lmod='$(brew --prefix lmod)/libexec/lmod'
You may want to load LMOD variables from your favorite shell init script:
source $(brew --prefix lmod)/init/$(basename $SHELL)
Installation Notes (Linux/CentOS 7)
Prerequisites: You need to have the EPEL testing repositories in the sources list. (Do not enable it by default). Then install the Lmod package using this repo.
$> yum install epel-release -y $> yum install --enable-repo=epel-testing Lmod
You can now use Lmod in a version compatible with EasyBuild.
- Reference documentation
- Reference Github repositories:
Easybuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. It is motivated by the need for a tool that combines the following features:
- a flexible framework for building/installing (scientific) software
- fully automates software builds
- full support and setup of Environment Modules or Lmod, eventually with a hierarchical layout
- divert from the standard
configure / make / make installwith custom procedures
- allows for easily reproducing previous builds
- keep the software build recipes/specifications simple and human-readable
- supports co-existence of versions/builds via dedicated installation prefix and module files
- enables sharing with the HPC community
- automagic dependency resolution
- retain logs for traceability of the build processes
For all these reasons, Easybuild has been selected as the reference middleware to handle the building and the installation of the software provided via the modules environment.
You can install Easybuild following the official instructions. Here is how I personally like to proceed:
Add the following entries to your
1 2 3 4 5 6 7
Then source this file to expose the environment variables:
1 2 3
Now let’s install Easybuild following the boostrapping procedure
1 2 3 4 5 6 7 8 9 10 11 12 13
From now on, you can search for existing recipes / easyconfigs and try to build it.
1 2 3 4 5 6 7 8 9 10 11
In the above example, Spark easyconfigs can be found from two source directories
Now you can try to build and install once of these recipes, generally the most recent one:
1 2 3