Brain Imaging & Analysis Center

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Interactive Jobs ======
The cluster allows for interactive jobs where you can login to machines and interact with your scripts/etc directly.  There are currently 51 available interactive slots, which will each be allocated 4GB of memory.  QSUB submissions can be run from interact slots in the same manner as described under [[biac:cluster:submit|Job submission]]

====== Starting an interactive job ======
Interactive jobs are started with **qinteract**. To start an interactive job,

  > qinteract

Running an interactive job for the first time may launch a prompt confirming the connection

  The authenticity of host 'node4.biac.duke.edu (152.3.98.178)' can't be established.
  RSA key fingerprint is d8:ae:54:b5:99:3b:34:c2:40:da:d0:70:b2:e5:9b:b0.
  Are you sure you want to continue connecting (yes/no)?

A "yes" response followed by "Enter" key will establish the connection. A password prompt requesting the cluster password ( used while logging into einstein) will appear.

  deshmukh@node4.biac.duke.edu's password:

Once the connection has been established, the prompt should indicate the node where the interactive job has been started.
  
  [deshmukh@node4 ~]$


====== Job restrictions ======
Each node has a finite amount of memory installed and due to the disk-less nature of the nodes there are restrictions set on the amount of ram used.  Currently, the default is to assign 8G of ram per job that is submitted.  If your job requires more than 8GB, then you may request a higher limit with the **"-l h_vmem"** directive ... otherwise you don't have to do anything.  This is done to prevent memory over subscription and to better distribute the load across the available machines.

  > qrsh -q interact.q -V -verbose -N interact -l h_vmem=10G bash

The above example will request/reserve 10G of available memory.  Your job will not go to a node unless it has the required amount available.  Also, if you exceed the requested amount the grid engine will terminate the job and you will receive notice.  In most cases you will not have to do anything, since 8G is a significant amount.  The amount of ram used in your jobs is listed as **"Max vmem"** in the emails set from the cluster.  The restriction is put in place to prevent memory being over allocated and jobs crashing an entire node, which would therefore kill other users' jobs.  

The maximum available is 187GB on any node, so if you request more than that, the job will just sit in the queue waiting indefinitely. 

//Please do not request additional resources unless you absolutely need them.  If additional resources are requested, they are deducted from the amount available to everyone else.  If unneeded resources are requested, this reduces the capacity on a given node for other potential usage.// 

There is a 5GB cumulative quota on all $HOME directories ( shared with your window's home directory ) and a 32GB cumulative quota for space in /tmp ( or $TMP ) across all nodes combined.
====== Accessing experiments ======
Experiments can be reached multiple ways:

There is an automounter that is running on each node that can mount experiments when they are accessed through it's proxy filesystem **"/mnt/BIAC"**
A call to a valid path will be intercepted by the proxy, and mounted.  Paths are /mnt/server/share/Experiemnt.01

If you are unsure, you can call the helper function **findexp** in various ways to return a valid path:
<code bash>
cd /mnt/munin/BIAC/Dummy.01
cd `findexp Dummy.01`
ls `findexp Dummy.01`
EXP1=`findexp Dummy.01`; cd $EXP1;
</code>

All of those instances would mount the experiment Dummy.01 within the proxy filesystem.  The experiment paths within the proxy filesystem are consistent across all of the nodes, therefore you can access the data with the same path on any batch or interactive job.


====== using MATLAB ======

To load the entire graphical matlab desktop:
<code bash>
matlab
</code>

Please not that this requires a lot of memory to run the desktop, so if the environment is not needed consider running:
<code bash>
matlab -nodesktop
</code>

This turns the linux command line into the matlab command line.  You'll still have access to all the functions, displays, etc ... however this requires less overhead.  If the full desktop is required, you will likely have to request slightly more ram than the default 4G.

If you do not need to interactively run matlab you can run a .m script without actually getting into the matlab environment.  This is also the type of thing that should be submitted for batch processing:
<code bash>
matlab -nodesktop < myscript.m 
</code>

You can run matlab without the desktop and without the java-virtual-machine if you continue to have "out of memory" errors:
- some functions that require java may no longer be accessible
<code bash>
matlab -nodesktop -nojvm -nosplash
</code>

Also, if you continue having JAVA memory errors you can create a java.opts file to increase the JAVA memory that matlab uses.
In a directory where you launch matlab, create a file names "java.opts" containing the following lines:
<code bash>
-Xms128m
-Xmx1g
</code>

This will increase the initial java-virtual-machine to 128 megabytes from the default of 64, it will also allow it to grow to 1gigabyte from the previous default of 128mb.  This is only relevant if NOT using the "-nojvm" flag.

You can also set the heap space preference through the graphical desktop in matlab:
[[http://blogs.mathworks.com/community/2010/04/26/controlling-the-java-heap-size/]]

Just a word of cause, setting it to the max available space in the gui caused matlab to not open ( for me )


====== Ending a job ======

To end a job type **exit** or **ctr+D** It may take a minute or two before the terminal is closed. Killing the terminal at this point will result in a failed job and affect the unmount process. Please wait till **exit** is complete. On completion you should see the prompt for the head node.
  
  [deshmukh@head ~]$

====== Using FSL ======

To launch the fsl GUI for creating fsf design files used by feat,

  > fsl


====== OpenGL ======

For programs that require OpenGL rendering, like fsleyes, make sure your local X11 client supports openGL and is enabled.

on mac make sure xquartz is installed:
<code>
Via Terminal:

defaults read org.xquartz.X11

{
    "NSWindow Frame x11_apps" = "330 482 454 299 0 0 1792 1095 ";
    "NSWindow Frame x11_prefs" = "291 401 484 336 0 0 1792 1095 ";
    SUHasLaunchedBefore = 1;
    SULastCheckTime = "2023-03-02 13:55:55 +0000";
    "app_to_run" = "/opt/X11/bin/xterm";
    "cache_fonts" = 1;
    "done_xinit_check" = 1;
    "enable_iglx" = 1; 
    "login_shell" = "/bin/sh";
    "no_auth" = 0;
    "nolisten_tcp" = 1;
    "startx_script" = "/opt/X11/bin/startx -- /opt/X11/bin/Xquartz";
}
</code>

If iglx is not 1, enable openGL.  Afterwards reboot the computer.
<code>
defaults write org.xquartz.X11 enable_iglx -bool true
</code>

Versions of Xquartz before 2.8.0 used configuration paths of : org.macosforge.xquartz.X11
Brain Imaging & Analysis Center

User Tools

Site Tools

Page Tools
