Getting Started Guide

Sources of information

• The Data storage and analysis server usage guidelines section of the Cluster Users' wiki area contains information about running data processing applications and data transfer.
• ClusterVision Administrator Manual. This is only available on paper at ESSC.
• Sun Grid Engine (SGE) documentation can be found on the SGE Web site. The Current version of SGE installed at ESSC is 6.0u7. A good place to start is the SGE User Guide.
• The man pages for the SGE "q" commands (see below) and configuration files can be very helpful.
• A brief summary of the ESSC cluster is given in the ReSC wiki. A longer version of that document, attached here, gives more background information and more detailed explanations of the scheduling, priority and backfilling strategies used at ESSC.

Changing passwords

User accounts on the cluster are administered by gorgon using NIS. To change your cluster user account password, log onto gorgon and use the command yppasswd (not passwd). The password change will take up to ten minutes to be propagated to the slave nodes and storage/analysis servers.

Using Sun Grid Engine (SGE)

• To use SGE commands the SGE module must be loaded with command " module add sge ". It is a good idea to put this command in .bashrc.
• The SGE Graphical User Interface is called Qmon, which can be launched with the command qmon. Qmon can be used by system adminstrators and ordinary users, and is a useful way to learn about SGE's capabilities. Ordinary users can view all the configuration parameters but can only change parameters that affect their own waiting and running jobs.
• The SGE commands can be found in directory /usr/local/Cluster-Apps/sge/bin/lx26-amd64, which is added to the user's path when the SGE module is loaded as described above.
• These are the most commonly used commands: qstat shows waiting and running jobs, qsub sumbits jobs and qdel deletes jobs.
• Job parameters can be specified on the qsub command line or in shell scripts. SGE parameters in shell scripts passed to qsub must be preceded by "#$".
• UMUI users don't need to submit jobs with qsub.
• Jobs can be given a higher priority by submitting with the high_priority attribute, which can be shortened to hp. For example, " qsub -l hp urgent.sh " submits urgent.sh with high priority. Campus Grid users can not use this facility.
• Use " qstat -r " to see all the resource attributes belonging to running and waiting jobs, including the run time limit parameter h_rt. This is useful for working out when running jobs are likely to finish.
• In SGE, queues are places where jobs run, not where they wait.
• SGE jobs sometimes leave behind rogue processes, which cause problems if they consume resources such as CPU time and Myrinet endpoints. Some of the additional command utilities described in the next section are useful for detecting and removing rogue processes.
• To submit a job that will only run on completely free nodes (i.e. nodes with four processors available), the boolean resource attribute " free " can be requested in the qsub command. For example, " qsub -l free=TRUE fussy_job.ksh " submits fussy_job.ksh to run only on completely free nodes. It might be necessary to do this to avoid small jobs running slowly due to being distributed over too many nodes.

Additional command utilities

A set of bash shell scripts intended to supplement SGE commands are located in /home/users/dab/Scripts. They all display a "Usage" string if run without any parameters.

which-nodes.sh This outputs a list of nodes involved in running a job. The user specifies the SGE job number and the script queries the scheduler log file to find out which nodes the job is, or was running on. It should be able to find jobs that ran during the past week. Use " qstat -s z " to find the job numbers of old jobs.

check-nodes.sh This outputs a ps listing of processes belonging to the specified user on the nodes that are, or were, running the specified job. It can be used to check the nodes after deleting a job, to make sure that nothing of yours is still running.

kill-my-procs.sh This kills (nearly) all the processes belonging to the specified user on the specified node using the specified signal. The idea is that it kills all the processes involved in a SGE job, leaving in place processes that are needed to run the script itself.

terminate_method.sh This deletes the specified SGE job belonging to the specified user. It can be run by the job owner from the command prompt in place of qdel, or by SGE in place of the default job deletion mechanism. More on this in the Technical Tips section in this wiki. WARNING: This script deletes ALL processes belonging to the specified user on ALL the nodes where the specified job is or was running. Therefore, care must be taken when deleting the remnants of old jobs, in case a new job that you want to keep has started running on one or more of the same nodes as the old one.

To avoid typing in your user name every time you use these scripts you can set up an alias. For example: alias domestos="/home/users/dab/Scripts/terminate_method.sh $USER"

With this in mind, the user name is always the first argument. The only script that doesn't take the user name is which-nodes.sh.

Usage Policies

• No job should run for more than three hours. Longer jobs should be split into a series of three-hour chunks, which ensures that no job has to wait more than three hours before being able to run.
• All jobs must be submitted with a run-time limit specified by the h_rt parameter, which allows other users to see when running jobs are going to finish and enables SGE to do backfilling (see below).
• Users must check their disk usage regularly. All unneccesary files must be moved or deleted as soon as possible.

---

---

Reservation and Backfilling

Jobs that require a large number of processors (20 or more, let's say) may have to wait a long time to run because it is less likely that enough free processors will become available all at the same time. To avoid this, processors can be reserved for a job by submitting with " -R y " in the qsub command. For example, the command " qsub -R y big_job.sh " submits big_job.sh with reservation. No other job can run on the reserved processors until the reserving job has finished. It is not possible to reserve resources for a specific date and time in SGE.

SGE uses a technique known as backfilling to allow short jobs to use reserved resources (which are usually processors) while the reserving job is waiting to run. Backfilling only works if all jobs have a maximum run time specified with the h_rt parameter in the qsub command. This represents the elapsed time, not the CPU time. The time limit can be specified as a number of seconds, or in the form hh:mm:ss. For example, the command " qsub -l h_rt=2:30:0 good_job.sh " submits good_job.sh with a time limit of 2 hours and 30 minutes. The run time limit should be set to a value that is comfortably longer than the expected duration of the job, so that the job does not actually need to be terminated by SGE (which can result in loss of data). For example, if the job is expected to last two hours, an appropriate time limit might be two and a half hours. A time limit that is much greater than the expected run time is not useful at all; it does not provide useful information for other users, and it does not enable backfilling to be coordinated effectively.

-- DanBretherton - 23 Jul 2009