User Tools

Site Tools


biac:cluster:modules

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
biac:cluster:modules [2021/05/12 12:42]
cmp12
biac:cluster:modules [2023/04/06 18:35] (current)
cmp12
Line 1: Line 1:
 ====== Accessing Available Software ====== ====== Accessing Available Software ======
  
-The cluster uses [[https://modules.readthedocs.io/en/latest/module.html|environment modules v3.2]] to control what software is loaded in your environment.  Certain software will be loaded by default, but there are other versions available to keep analyses consistent across a study or additional software that you may wish to load.  Modules work by adding specific environmental variables needed for each package and/or adding them to your default search path. +The cluster uses [[https://modules.readthedocs.io/en/latest/module.html|environment modules]] to control what software is loaded in your environment.  Certain software will be loaded by default, but there are other versions available to keep analyses consistent across a study or additional software that you may wish to load.  Modules work by adding specific environmental variables needed for each package and/or adding them to your default search path. 
  
 ==== What's Loaded? ==== ==== What's Loaded? ====
Line 28: Line 28:
 dramms/1.5.1                fscan                       hcp_workbench/1.5.0         medinria/2.2.3(default)     rar/5.70 dramms/1.5.1                fscan                       hcp_workbench/1.5.0         medinria/2.2.3(default)     rar/5.70
 dtiprep/1.2.4               fsl/5.0.1                   laynii/1.5.6                medinria/4.0.0              simnibs/2.0.1 dtiprep/1.2.4               fsl/5.0.1                   laynii/1.5.6                medinria/4.0.0              simnibs/2.0.1
 +
 +</code>
 +
 +==== Get more info about a module ====
 +To get more info about a specific module use the "module display" or "module whatis" command.
 +
 +<code>
 +[cmp12@blade01 ~]$ module display fsl
 +-------------------------------------------------------------------
 +/etc/modulefiles/fsl/6.0.3:
 +
 +module-whatis loads the fsl v6.0.3 environment 
 +setenv FSLDIR /usr/local/packages/fsl-6.0.3 
 +setenv FSLSUBMIT 0 
 +setenv FSL_DIR /usr/local/packages/fsl-6.0.3 
 +setenv FSL_BIN /usr/local/packages/fsl-6.0.3/bin 
 +setenv FSLTCLSH /usr/local/packages/fsl-6.0.3/bin/fsltclsh 
 +setenv FSLWISH /usr/local/packages/fsl-6.0.3/bin/fslwish 
 +setenv FSLOUTPUTTYPE NIFTI_GZ 
 +prepend-path PATH /usr/local/packages/fsl-6.0.3/bin 
 +-------------------------------------------------------------------
 +
 +
 +[cmp12@blade01 ~]$ module whatis fsl
 +fsl                  : loads the fsl v6.0.3 environment
  
 </code> </code>
Line 109: Line 134:
 ~/bin/cmake3 ~/bin/cmake3
 </code> </code>
 +
 +
 +====== Setting Up Experiment Specific Conda Environments ======
 +
 +[[https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html|Conda]] is available via modules to setup and maintain your own specific environments if you want to use specific versions of python ( or R ) modules that are not tied to the default cluster installation.
 +
 +First, you want to unload the default python3 and load miniconda via modules to begin:
 +<code>
 +[cmp12@blade29 ~]$ module unload python3
 +[cmp12@blade29 ~]$ module load miniconda
 +
 +[cmp12@blade29 ~]$ conda --version
 +conda 22.11.1
 +
 +#set the default location to download packages to TMPDIR so you do not fill up your home directory
 +[cmp12@blade29 ~]$ export CONDA_PKGS_DIRS=${TMPDIR}/pkgs
 +
 +</code>
 +
 +
 +Now, create your environment in a mounted location.  For example and Lab directory for general access from your other lab members, or in an experiment so it can be used throughout all the analysis of an entire experiment.
 +
 +<code>
 +conda create -p /mnt/munin/Song/Lab/Chris/myenv python
 +
 +#activate it
 +conda activate /mnt/munin/Song/Lab/Chris/myenv
 +
 +#install some of the python modules you may need
 +(/mnt/munin/Song/Lab/Chris/myenv) [cmp12@blade29 ~]$ which pip3
 +/mnt/munin/Song/Lab/Chris/myenv/bin/pip3
 +
 +(/mnt/munin/Song/Lab/Chris/myenv) [cmp12@blade29 ~]$ pip3 install numpy scipy nibabel ipython
 +
 +#verify
 +(/mnt/munin/Song/Lab/Chris/myenv) [cmp12@blade29 ~]$ which python
 +/mnt/munin/Song/Lab/Chris/myenv/bin/python
 +
 +
 +(/mnt/munin/Song/Lab/Chris/myenv) [cmp12@blade29 ~]$ pip3 freeze
 +ipython==8.11.0
 +jedi==0.18.2
 +matplotlib-inline==0.1.6
 +nibabel==5.0.1
 +numpy==1.24.2
 +.......
 +</code>
 +
 +Now you have a consistent environment that you can use/maintain throughout the experiment.
 +
 +To deactivate just run "conda deactivate" which will remove the environment from your path.
 +
 +To make it consistent, you can modify your ~/.bash_profile ( or ~/.bashrc ) to automatically add miniconda and activate the environment:
 +
 +<code>
 +module remove python3
 +module add miniconda
 +
 +conda activate /mnt/munin/Song/Lab/Chris/myenv
 +</code>
 +
 +
 +Alternatively, you can activate within submission scripts and/or make sure you are explicitly calling your own version of python within scripts:
 +
 +<code>
 + [cmp12@blade29 ~]$ //mnt/munin/Song/Lab/Chris/myenv/bin/python
 +Python 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] on linux
 +Type "help", "copyright", "credits" or "license" for more information.
 +>>> import numpy
 +>>> numpy.__version__
 +'1.24.2'
 +>>> numpy.__file__
 +'//mnt/munin/Song/Lab/Chris/myenv/lib/python3.10/site-packages/numpy/__init__.py'
 +</code> 
 +
 +Add something else to your environment ( like R ):
 +<code>
 +(/mnt/munin/Song/Lab/Chris/myenv) [cmp12@blade29 ~]$ conda search r-base
 +
 +#if you don't see the version you want you can add additional channels
 +conda config --add channels conda-forge
 +conda config --set channel_priority strict
 +
 +#then search again
 +conda search r-base
 +
 +
 +#install the version you would like to use
 +(/mnt/munin/Song/Lab/Chris/myenv) [cmp12@blade29 ~]$ conda install r-base=4.2.2
 +
 +(/mnt/munin/Song/Lab/Chris/myenv) [cmp12@blade29 ~]$ which R
 +/mnt/munin/Song/Lab/Chris/myenv/bin/R
 +
 +(/mnt/munin/Song/Lab/Chris/myenv) [cmp12@blade29 ~]$ R --version
 +R version 4.2.2 (2022-10-31) -- "Innocent and Trusting"
 +Copyright (C) 2022 The R Foundation for Statistical Computing
 +Platform: x86_64-conda-linux-gnu (64-bit)
 +
 +
 +</code>
 +
 +
 +====== Other Software ======
 +
 +Matlab Libraries : /usr/local/packages/MATLAB
 +
 +Premade Singularity Images : /usr/local/packages/singularity/images/
 +
 +
  
  
  
biac/cluster/modules.1620823344.txt.gz · Last modified: 2021/05/12 12:42 by cmp12