User Tools

Site Tools


biac:cluster:submit

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
biac:cluster:submit [2011/09/07 13:44]
petty [Template Script]
biac:cluster:submit [2023/02/23 18:43] (current)
Line 20: Line 20:
  
  
 +====== Job restrictions ====
 +==
 +Each node has a finite amount of memory installed and due to the disk-less nature of the nodes there are restrictions set on the amount of ram used.  Currently, the default is to assign 10G of ram per job that is submitted.  If your job requires more than 10GB, then you may request a higher limit with the **"-l h_vmem"** directive ... otherwise you don't have to do anything.  This is done to prevent memory over subscription and to better distribute the load across the available machines.
  
 +  > qsub -N run_script.1 -l h_vmem=12G,vf=12G script.sh run1
 +
 +The above example will request/reserve 12G of available memory.  **"vf"** will ensure your job will not go to a node unless it has the required amount available.  Also, if you exceed the requested amount of **"h_vmem"** the grid engine will terminate the job and you will receive notice.  In most cases you will not have to do anything, since 10G is a significant amount.  The amount of ram used in your jobs is listed as **"Max vmem"** in the emails set from the cluster.  The restriction is put in place to prevent memory being over allocated and jobs crashing an entire node, which would therefore kill other users' jobs.
 +
 +You can also get it from previous jobs if you have the job number with qacct ( there will be a resulting entry for "maxvmem" ) :
 +  > qacct -j JOBNUM
 +  maxvmem   4.315G
 +  
 +Also, you can request the info from currently running jobs with qstat ( look for the "usage" information ) : 
 +  > qstat -j JOBNUM
 +  usage    1:                 cpu=3:07:56:02, mem=36938.10261 GBs, io=15.57702, vmem=769.992M, maxvmem=1.451G
 +  
 +The maximum available is ~750GB on any node, so if you request more than that, the job will just sit in the queue waiting indefinitely.
 +
 +//Please do not request additional resources unless you absolutely need them.  If additional resources are requested, they are deducted from the amount available to everyone else.  If unneeded resources are requested, this reduces the capacity on a given node for other potential usage.// 
 +
 +There is a global limit on any single user of 60 slots and/or 1920G of ram.
 +
 +There is a 6GB cumulative quota on all HOME directories
 ====== Job status ====== ====== Job status ======
  
-The current status of a job can be checked with **qstat**. This will return the current list of jobs owned by the user.+The current statu 
 +of a job can be checked with **qstat**. This will return the current list of jobs owned by the user.
   > qstat    > qstat 
   job-ID  prior    name       user      state  submit/start at      queue                        slots ja-task-ID   job-ID  prior    name       user      state  submit/start at      queue                        slots ja-task-ID
Line 60: Line 83:
  
  
 +All jobs for a particular user can be deleted with the following command.
  
- +  > qdel -U username
 ====== Template Script ====== ====== Template Script ======
-Jobs are usually written in bash. They are similar to local bash scripts in syntax and usage. In  addition, they contain cluster related directives identified by lines starting with " #$ ". These are used to send job related setup information to the cluster. Scripts also contain requests for access to experiment data. The BIAC template script is a good starting point for testing job submission and as a base script for all jobs. Begin, by making a copy of the template script.  +Jobs are usually written in bash. They are similar to local bash scripts in syntax and usage. In  addition, they contain cluster related directives identified by lines starting with " #$ ". These are used to send job related setup information to the cluster. Scripts also contain requests for access to experiment data. The BIAC template script is a good starting point for testing job submission and as a base script for all jobs. Begin, by making a copy of the template script below
- +
-  cp /usr/local/packages/qsub_templates/basic.sh myscript.sh+
    
 The template script requests access to an experiment folder and lists its contents. It needs a valid BIAC Experiment Name (case-sensitive) that is accessible by the user. Submit myscript.sh using qsub . The template script requests access to an experiment folder and lists its contents. It needs a valid BIAC Experiment Name (case-sensitive) that is accessible by the user. Submit myscript.sh using qsub .
Line 101: Line 122:
   * All terminal output is routed to the " Analysis " folder under the Experiment directory i.e. <color navy>$EXPERIMENT</color>/Analysis. To change this path, set the <color navy>OUTDIR</color> variable at the beginning of this section to another location under your experiment folder.    * All terminal output is routed to the " Analysis " folder under the Experiment directory i.e. <color navy>$EXPERIMENT</color>/Analysis. To change this path, set the <color navy>OUTDIR</color> variable at the beginning of this section to another location under your experiment folder. 
 <code bash> <code bash>
-  OUTDIR=$EXPERIMENT/Analysis/ClusterOut+  OUTDIR=$EXPERIMENT/Analysis/ClusterLogs
 </code> </code>
   * On successful completion the job will return 0. If you need to set another return code, set the <color navy>RETURNCODE</color> variable in this section. To avoid conflict with system return codes, set a <color navy>RETURNCODE</color> higher than 100.    * On successful completion the job will return 0. If you need to set another return code, set the <color navy>RETURNCODE</color> variable in this section. To avoid conflict with system return codes, set a <color navy>RETURNCODE</color> higher than 100. 
Line 109: Line 130:
   * Arguments to the USER SCRIPT are accessible in the usual fashion eg:  <color navy>$1 $2 $3</color>   * Arguments to the USER SCRIPT are accessible in the usual fashion eg:  <color navy>$1 $2 $3</color>
  
-<code bash>+<code bash basic.sh>
 #!/bin/sh #!/bin/sh
  
Line 161: Line 182:
 # -- END POST USER-- # -- END POST USER--
 </code> </code>
 +
 +==== Notes ====
 +  * if you ever edit your scripts on a non-unix machine, please run dos2unix on them before submitting
 +  * sometimes there are hidden window's characters that will prevent the script from running
biac/cluster/submit.1315403078.txt.gz · Last modified: 2014/08/04 16:03 (external edit)