This is an old revision of the document!
Table of Contents
Interactive Jobs
The cluster allows for interactive jobs where you can login to machines and interact with your scripts/etc directly. There are currently 51 available interactive slots, which will each be allocated 4GB of memory. QSUB submissions can be run from interact slots in the same manner as described under Job submission
Starting an interactive job
Interactive jobs are started with qinteract. To start an interactive job,
> qinteract
Running an interactive job for the first time may launch a prompt confirming the connection
The authenticity of host 'node4.biac.duke.edu (152.3.98.178)' can't be established. RSA key fingerprint is d8:ae:54:b5:99:3b:34:c2:40:da:d0:70:b2:e5:9b:b0. Are you sure you want to continue connecting (yes/no)?
A “yes” response followed by “Enter” key will establish the connection. A password prompt requesting the cluster password ( used while logging into einstein) will appear.
deshmukh@node4.biac.duke.edu's password:
Once the connection has been established, the prompt should indicate the node where the interactive job has been started.
[deshmukh@node4 ~]$
Job restrictions
Each node has a finite amount of memory installed and due to the disk-less nature of the nodes there are restrictions set on the amount of ram used. Currently, the default is to assign 8G of ram per job that is submitted. If your job requires more than 8GB, then you may request a higher limit with the “-l h_vmem” directive … otherwise you don't have to do anything. This is done to prevent memory over subscription and to better distribute the load across the available machines.
> qrsh -q interact.q -V -verbose -N interact -l h_vmem=10G bash
The above example will request/reserve 10G of available memory. Your job will not go to a node unless it has the required amount available. Also, if you exceed the requested amount the grid engine will terminate the job and you will receive notice. In most cases you will not have to do anything, since 8G is a significant amount. The amount of ram used in your jobs is listed as “Max vmem” in the emails set from the cluster. The restriction is put in place to prevent memory being over allocated and jobs crashing an entire node, which would therefore kill other users' jobs.
The maximum available is 187GB on any node, so if you request more than that, the job will just sit in the queue waiting indefinitely.
Please do not request additional resources unless you absolutely need them. If additional resources are requested, they are deducted from the amount available to everyone else. If unneeded resources are requested, this reduces the capacity on a given node for other potential usage.
There is a 5GB cumulative quota on all $HOME directories ( shared with your window's home directory ) and a 32GB cumulative quota for space in /tmp ( or $TMP ) across all nodes combined.
Accessing experiments
Experiments can be reached multiple ways:
There is an automounter that is running on each node that can mount experiments when they are accessed through it's proxy filesystem “/mnt/BIAC” A call to a valid path will be intercepted by the proxy, and mounted. Paths are /mnt/BIAC/server.dhe.duke.edu/share/Experiemnt.01
If you are unsure, you can call the helper function findexp in various ways to return a valid path:
cd /mnt/munin/BIAC/Dummy.01 cd `findexp Dummy.01` ls `findexp Dummy.01` EXP1=`findexp Dummy.01`; cd $EXP1;
All of those instances would mount the experiment Dummy.01 within the proxy filesystem. The experiment paths within the proxy filesystem are consistent across all of the nodes, therefore you can access the data with the same path on any batch or interactive job.
using MATLAB
To load the entire graphical matlab desktop:
matlab
Please not that this requires a lot of memory to run the desktop, so if the environment is not needed consider running:
matlab -nodesktop
This turns the linux command line into the matlab command line. You'll still have access to all the functions, displays, etc … however this requires less overhead. If the full desktop is required, you will likely have to request slightly more ram than the default 4G.
If you do not need to interactively run matlab you can run a .m script without actually getting into the matlab environment. This is also the type of thing that should be submitted for batch processing:
matlab -nodesktop < myscript.m
You can run matlab without the desktop and without the java-virtual-machine if you continue to have “out of memory” errors: - some functions that require java may no longer be accessible
matlab -nodesktop -nojvm -nosplash
Also, if you continue having JAVA memory errors you can create a java.opts file to increase the JAVA memory that matlab uses. In a directory where you launch matlab, create a file names “java.opts” containing the following lines:
-Xms128m -Xmx1g
This will increase the initial java-virtual-machine to 128 megabytes from the default of 64, it will also allow it to grow to 1gigabyte from the previous default of 128mb. This is only relevant if NOT using the “-nojvm” flag.
You can also set the heap space preference through the graphical desktop in matlab: http://blogs.mathworks.com/community/2010/04/26/controlling-the-java-heap-size/
Just a word of cause, setting it to the max available space in the gui caused matlab to not open ( for me )
Ending a job
To end a job type exit or ctr+D It may take a minute or two before the terminal is closed. Killing the terminal at this point will result in a failed job and affect the unmount process. Please wait till exit is complete. On completion you should see the prompt for the head node.
[deshmukh@head ~]$
Using FSL
To launch the fsl GUI for creating fsf design files used by feat,
> fsl
OpenGL
For programs that require OpenGL rendering, like fsleyes, make sure your local X11 client supports openGL and is enabled.
on mac make sure xquartz is installed:
Via Terminal: defaults read org.xquartz.X11 { "NSWindow Frame x11_apps" = "330 482 454 299 0 0 1792 1095 "; "NSWindow Frame x11_prefs" = "291 401 484 336 0 0 1792 1095 "; SUHasLaunchedBefore = 1; SULastCheckTime = "2023-03-02 13:55:55 +0000"; "app_to_run" = "/opt/X11/bin/xterm"; "cache_fonts" = 1; "done_xinit_check" = 1; "enable_iglx" = 1; "login_shell" = "/bin/sh"; "no_auth" = 0; "nolisten_tcp" = 1; "startx_script" = "/opt/X11/bin/startx -- /opt/X11/bin/Xquartz"; }
If iglx is not 1, enable openGL. Afterwards reboot the computer.
defaults write org.xquartz.X11 enable_iglx -bool true
Versions of Xquartz before 2.8.0 used configuration paths of : org.macosforge.xquartz.X11