Software

Introduction

Users have access to several third-party and vendor-provided software packages on the ACF. Rather than install and configure these packages for each user manually, the ACF uses modulefiles that contain the necessary instructions a software package needs to execute within a user’s environment. Additionally, the ACF supports several compilers from both GNU and Intel. In this document, you will learn how to view and manipulate software on the ACF. To see a list of available software packages on the ACF, please refer to the relevant section of this document.

Using Modules

In the context of the ACF, modules are software packages that can be added and removed from your user environment as necessary. Modulefiles handle the changes made to your user environment by these modules so that they function as intended. The module command has several subcommands that allow you to manage the modules in your environment. Table 1.1 lists and describes these subcommands.

When you execute module avail, it outputs information to stderr (standard error) rather than stdout (standard output). Simply put, this means you cannot use the grep command or any other search utility on the output of module avail in standard fashion. Instead, you must use the statement in Figure 1.1 to filter the output of module avail. Alternatively, if you know the name of the module you require, use module whatis to determine if it exists on the ACF.

In the output of the module help command, several init subcommands appear. In some contexts, they automatically load modules from a file so that they are available when you log in. Ignore these subcommands. They are not supported on the ACF. Instead, add the modules you wish to load upon log in to your ~/.bashrc file.¹ For example, to load Conda 3 into your environment and activate it upon log in, use the commands shown in Figure 1.2. Execute these commands in the order they appear. The module must always be loaded before you can execute anything specific to that module. In the second echo command, you will need the path to the configuration script for the given module. The /sw/acf/ directory contains the subdirectories to each module, which will lead to the configuration script for the modules.

echo “module load anaconda3/5.1.0” >> ~/.bashrc
echo “. /sw/acf/anaconda3/5.1.0/centos7.3_gnu6.3.0/anaconda3-5.1.0/etc/profile.d/conda.sh” >> ~/.bashrc
echo “conda activate” >> ~/.bashrc

To verify these changes, log out of the ACF and log back in. You should see (base) to the left of your prompt. If you do not, open your .bashrc file and ensure that the text following the echo commands in Figure 1.2 is present. Next, use the module list command to verify that the module(s) you placed in the .bashrc file are loaded into your environment. If they are not, attempt to manually load them with the module load command. If the process still fails, use the module avail command to verify that the module exists and that you have spelled it correctly.

By default, several modules are already loaded into your environment. To view these modules, execute module list when you initially log in to the ACF. In particular, the DefApps module is loaded into your environment by default to support third-party applications and programming environments. Do not unload the modules that are loaded into your environment by default. Doing so may produce undesirable results.

Available Software

The ACF has an extensive library of software available for your use. To learn more about a specific package, follow the links below. To request a software package be installed on the ACF, please submit a software request form.

• ATLAS
• RevBayes
• abinit
• afni
• anaconda2
• bbcp
• bedtools
• blast
• boost
• bowtie2
• caffe
• cmake
• diamond
• dmtcp
• dplasma
• espresso
• fastqc
• fftw
• geant4
• git
• gmp
• gnuplot
• graphviz
• gromacs
• gsl
• hadoop
• hdf5
• hmmer
• hpc-blast
• hypre
• imagemagick
• ioapi
• keras
• lammps
• lapack
• llvm
• maven
• metis
• molcas
• molden
• mothur
• mpc
• mpfr
• mpich
• mxml
• nco
• neper
• netcdf
• nlopt
• nwchem
• openBLAS
• openDIEL
• openfoam
• openmpi
• papi
• pcre
• perl
• planetocosmics
• plasma
• postgis
• postgresql
• povray
• python
• q-espresso
• r
• raxml
• root
• samtools
• scalapack
• siesta
• silo
• skewer
• spades
• sprng
• star
• subversion
• swig
• szip
• tensorflow
• theano
• tophat
• transdecoder
• trimmomatic
• turbomole
• usearch
• vasp
• vmd
• xalt
• xml2
• xz
• zlib

Compiling Software

At the time of this writing, the ACF hosts C, C++, and Fortran compilers from both GNU and Intel. By default, the Intel programming environment is loaded. In the output of module list, it appears as PE-intel. To switch to the GNU compiler, use module swap PE-intel PE-gnu. To switch back to the Intel compiler, execute the same command but change the positions of PE-intel and PE-gnu. For usage information on the C compilers, type man mpiicc for the Intel compilers and man gcc for the GNU compilers. For usage information on the GNU Fortran compiler, use man gfortran. The man page for the Intel compilers has information on Intel’s Fortran compilers.

If you wish to target the ACF’s KNL (Knight’s Landing) nodes, it is best to use the Intel compilers. It is recommended that code designed for the KNL nodes be recompiled on the ACF. Table 3.1 shows the commands to use with the Intel compilers so that KNL features are targeted.

If you must use the GNU compilers on the KNL nodes, use the -mavx option. Be aware that compiling on these nodes with GNU compilers will not provide the same optimization or performance as Intel compilers.

To link Intel’s MKL (Math Kernel Library) with your application, refer to Intel’s documentation on the process.

Conda Environments

Conda is a package, dependency, and environment manager installed on the ACF. It can create an isolated area in which commands can be executed, software can be installed, and many other tasks can be performed. With Conda, you can use specific software packages on ACF without requiring the packages to be installed as separate modules. For more information on Conda, consult the official Conda documentation.

To create a Conda environment, identify the necessary version to use. Documentation for the software package you intend to use should indicate the appropriate Conda version. To determine which versions of Conda exist on the ACF, type module avail. The various versions of Conda will be listed as anaconda. To avoid searching through the output of module avail, execute the command in Figure 4.1 to quickly see the various versions of Conda on the ACF. Alternatively, execute module avail anaconda.

Next, load the appropriate Conda version. Use the module load <anaconda-version> command. Replace <anaconda-version> with the Conda version you require. After the module loads, type conda create --name <env-name>. Replace <env-name> with the desired name of the Conda environment. The name will not affect its operation, but it will identify it. If you wish to create a Conda environment for a specific package, specify the package name you desire after the environment name. For example, to create a Python 3.3 Conda environment, you would use conda create --name <env-name> python=3.3. If you needed a Ruby Conda environment, you would use conda create --name <env-name> ruby.

Upon creation, the Conda environment will appear in your home directory. Conda will prompt you to allow the creation. Before you confirm the creation, verify that the directory Conda uses is in your home directory. If it attempts to install the environment in another location, do not allow the creation. Otherwise, type y and press Enter (Return) when it asks. You will find your Conda environments in the ~/.conda/envs/ directory.

If you need to create the environment in another location, use the --prefix long option of conda create. For example, to create a Conda environment in ~/testing_envs/, execute conda create --prefix ~/testing_envs/<env-dir-name>. Replace <env-dir-name> with the name of the directory that will contain the Conda environment. You may specify the packages to install after the pathname.

Quota limitations on your home directory may require you to create your Conda environments in Lustre space. Before creating an environment in this space, be aware that it is purged every 30 days. Please refer to the File Systems document for more information.

To create a Conda environment in Lustre space, execute conda create --prefix $SCRATCHDIR/<env-dir-name>. Replace <env-dir-name> with the name of the directory in which the environment will reside. Conda will automatically create the directory. To verify the environment’s creation, execute ls -l $SCRATCHDIR. The environment’s directory should appear in the listing.

To activate a Conda environment, use the source activate <env-name> command.² When the environment becomes active, your shell prompt will change. It will show the name of the Conda environment in parentheses before your username and the hostname of the node on which you are currently working. After the prompt changes, you can freely use the environment. If you created your environment in Lustre space, use source activate $SCRATCHDIR/<env-dir-name> where <env-dir-name> is the name of the directory in which the environment resides.² Once active, the prompt will show the absolute path to the environment.

To deactivate a Conda environment, use the conda deactivate command. You will return to the shell prompt and can load another environment or use standard shell commands. Alternatively, you can use source activate from within an active environment to return to the base environment.² To delete a Conda environment, use the conda remove --name <env-name> --all command. You may also use conda env remove --name <env-name> command. When prompted to confirm the removal, type y and press Enter (Return). To verify the removal, use the conda info --envs command. For environments in Lustre space, use the conda env remove --prefix $SCRATCHDIR/<env-dir-name> command where <env-dir-name> is the name of the directory in which the environment resides.

To install a specific package into an existing Conda environment, load Conda and activate the appropriate environment. Search for your desired package with the conda search <package-name> command. Replace <package-name> with the software package you wish to find. If it is present in the Anaconda repository, it will appear in the search listing. To install the package, use the conda install <package-name> command. When prompted, type y and press Enter (Return) to confirm the installation. To verify that the package was successfully installed, type conda list within the environment.

If you regularly use a specific Conda environment, you can automate the loading process of that environment. To automate this process, execute the command in Figure 4.2. Replace the <env-name> argument with the environment you wish to load each time you log in to the ACF. Ensure that you have configured your .bashrc file to load Conda when you log in. Refer to the Using Modules section for more information on automating the module loading process.

To verify that your desired Conda environment automatically loads when you log in, log out of the ACF and log back in. Your prompt should show the name of your environment in parentheses to the left of your username and the hostname of the node on which you are working. If it does not, open your .bashrc file and verify that the command in Figure 4.2 was added to the file correctly. Next, verify that the environment you specified exists in ~/.conda/envs. Attempt to load it manually; if this fails, use module list to verify that Conda is loaded and functional. If it is not, load Conda manually and then attempt to load the environment.

For environments created with the --prefix long option, specify the pathname to the environment rather than its name. Figure 4.3 provides the command to execute to automate the loading process of an environment created in Lustre space.

To change or remove the automatic loading of a Conda environment, edit your .bashrc file with your desired text editor. Delete the entire line that loads the Conda environment or change the environment’s name. Save the file, then verify the changes took effect by logging off of the ACF and logging back in.

Using pip and virtualenv

Though Conda’s feature set is more robust than pip and virtualenv, these tools are still useful when you require certain packages that can only be retrieved from the PyPI (Python Package Index). Conda can interoperate with pip, but it is not advised. Instead, use pip with virtualenv when you require PyPI packages. For more information on pip, please refer to the official pip documentation. Please consult the official virtualenv documentation for additional information on how to use virtualenv environments.

Before you can install packages, you must create the environment that will contain those packages. The virtualenv command handles environment creation. To create a virtualenv environment, execute virtualenv ~/<env-dir-name>. Replace <env-dir-name> with the name you will assign to the environment. virtualenv will create the environment in your home directory. If you wish to create the environment in Lustre space, execute virtualenv $SCRATCHDIR/<env-dir-name>. Be aware that Lustre scratch space is purged every 30 days. Please review the File Systems document for more information before creating a virtualenv environment in Lustre space.

You may require Python 3 for your environment. By default, virtualenv creates environments with Python 2. To create a virtualenv environment with Python 3 installed, execute virtualenv ~/<env-dir-name> --python=python3.

After the environment is successfully created, activate it with the source command. You must specify the location of the environment’s activation script. The path to the activation script is always located in the <env-dir-name>/bin/ directory. The script itself is simply named activate. For instance, to activate an environment named test_venv_py3 in your home directory, execute source ~/test_env_py3/bin/activate. If you had an environment in Lustre space, you would activate it with the source $SCRATCHDIR/<env-dir-name>/bin/activate command. Your prompt will change to display the name of the environment in parentheses to the left of your username.

To deactivate an environment, execute the deactivate command. The prompt will return to normal to reflect the deactivation. If you wish to delete an environment, remove the directory in which the environment resides. For example, to delete an environment named matlab_venv in your home directory, you would execute the rm -r ~/matlab_venv command. In Lustre space, execute rm -r $SCRATCHDIR/<env-dir-name>. Before you execute the rm command, be absolutely sure you have specified the correct directory.

Packages within your environment are managed by pip. The usage of pip is the same in and out of your virtualenv environment. To install a package, execute pip install <package-name>. If you do not know the full name of the package you intend to install, search for it with the pip search <query> command. If your query is broad, be prepared for an extensive list of results. Attempt to narrow down your searches when possible. After you have installed all the packages you require, execute the pip list command to review the packages you have installed and verify they use the correct versions. To see information on a specific package, execute pip show <package-name>. If a package requires an update, execute pip install --upgrade <package-name>.

It is possible to automatically load a virtualenv environment when you log in to the ACF. Before you attempt this, verify that the environment operates as you expect and produces no errors. If it does and you wish to automate its loading process, execute the command shown in Figure 5.1.¹ Replace <env-dir-name> with the name of the environment’s directory.

Test this change by logging out of the ACF and logging back in. If your prompt reflects the name of the environment you specified in your shell rc file, the change was successful; if not, verify that the line in Figure 5.1 was added to the shell’s rc file. Next, check the spelling of the environment’s name. Ensure that the path to the activation script is accurate. If these steps fail, attempt to load the environment manually with the source command. After that, remove and recreate the environment, if possible.

If you wish to stop automatically loading a virtualenv environment upon login, edit your shell rc file with your preferred text editor. Remove or place a hashtag (#) before the line that has the source command on it. Save the file, then verify the changes by logging out of the ACF and logging back in. The prompt should then only display your username and the host name of the node into which you are logged, which indicates the environment is no longer automatically loading.

Footnotes

1. If you are using another shell, add the module load commands and the associated scripts to the rc file for that shell. Shell rc files all appear as hidden files in your home directory.
2. If you use a Conda version greater than version 4.6, use conda activate <env-name>.

Last Updated: 01 / 07 / 2020