Obsolete:Toolforge/Grid

Every non-trivial task performed in Toolforge should be dispatched by the Grid Engine, which ensures that the job is run in a suitable place with sufficient resources.

The basic principle of running jobs is fairly straightforward:

• You submit a job to a work queue from a submission server (for example login.toolforge.org)
• The grid engine master finds a suitable execution host to run the job on, and starts it there once resources are available
• As it runs, your job will send output and errors to files until the job completes or is aborted.

Jobs can be scheduled synchronously or asynchronously, continuously, or simply executed once. If a continuous job fails, the grid will automatically restart the job so that it keeps going.

The grid engine is a highly flexible system for assigning resources to jobs, including parallel processing. The Toolforge Grid Engine was implemented with Son of Grid Engine (an inactive open-source fork of Oracle Grid Engine, previously known as Sun Grid Engine). You can find more documentation on sourceforge.net and the archived Son of Grid Engine website.

Commonly used Grid Engine commands include:

• jsub: Toolforge specific wrapper for qsub that makes submitting a job much easier
• qsub: submit jobs to the grid
• qalter: modify job settings (while the job is waiting or running)
• qstat: get information about a queued or running job
• qacct: extracts arbitrary accounting information from the cluster logfile (also after job termination, useful for debugging)
• qdel: abort or cancel a job

You can find detailed information about these commands in the Grid Engine Manual

The Grid Engine commands are very flexible, but a little complex at first – you might prefer to use the helper scripts instead (jsub, jstart, jstop) described in more detail in the next sections.

Jobs can be submitted to the work queue with either Grid Engine’s 'qsub' command or the 'jsub' helper script, which is simpler to use and described in this section. (For information about qsub, please see the the Grid Engine Manual.)

To run a job on demand (on a schedule via cron, for instance, or from a web tool or the command line), simply use the 'jsub' command:

$ jsub [options…] program [args…]

By default, jsub will schedule the job to be run as soon as possible, and print the eventual output to files (‘jobname.out’ and ‘jobname.err’) in your home directory. Unless a job name is explicitly specified with jsub options, the job will have the same name as the program, minus extensions (e.g., if you have a program named foobot.pl and start it with jsub, the job's name will be foobot.)

Once your jobs has been submitted to the grid, you will receive an output similar to the one below, which includes the job id and job name.

Your job 120 ("foobot") has been submitted

$ ssh login.toolforge.org
$ become shtest
$ ls
logs  mybot.sh  public_html  replica.my.cnf
$ jsub mybot.sh
Your job 8326234 ("mybot") has been submitted
$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
8326234 0.00000 mybot      tools.shtest qw    09/11/2019 19:41:43                                    1
$ qstat
$ ls
logs  mybot.err  mybot.out  mybot.sh  public_html  replica.my.cnf
$ cat mybot.out
user_editcount
1363

In addition to a number of customized options, jsub supports many, but not all, qsub options:

Run jsub --help and man jsub to learn more.

The job name identifies the job and can also be used to control it (for example to suspend or stop it). By default, jobs are assigned the name of the program or script, minus its extension. For instance, if you started a program named 'foobot.pl' with jsub, the job's name would be 'foobot'.

It's important to note that you can have more than one job, running or queued, bearing the same name. Some of the utilities that accept a job name may not behave as expected in those cases.

Specify a different name for the job using the jsub’s -N option:

$ jsub -N NewName program [args…]

By default, jobs are allowed 512 MB of memory; you can request more (or less) with jsub’s '-mem' option (or qsub's '-l h_vmem=memory'). Keep in mind that a job that requests more resources may be penalized in its priority and may have to wait longer before being run until sufficient resources are available.

$ jsub -mem 500m program [args…]

For example, running a PHP script which requires 350MB of memory to work properly:

$ jsub -mem 350m php i_like_more_ram.php

By default, jobs are processed asynchronously in the background. If you need to wait until the job has completed (for instance, to do further processing on its output), you can add the '-sync y' (for sync y[es]!) option to the jsub command:

$ jsub -sync y program [args...]

If you need to make certain that the job isn't running multiple times in parallel (such as when you invoke it from a crontab), you can add the '-once' option. If the job is already running or queued the grid engine will simply mark the failed attempt in the error file and return immediately.

$ jsub -once program [args...]

Jsub and qsub always strip quotes in the arguments of a job. If the arguments include any special shell characters like spaces, "|" or "&" the job submission will likely fail, even when the arguments are given quoted to jsub (see phab:T50811). For instance with jsub myScript.php "Foo bar", myScript.php may only see $argv[1] as "Foo" and not the expected "Foo bar".

A workaround is to use two layers of quotes:

$ jsub myScript.php \''Foo bar'\'

Alternatively you can create a wrapper script, for example runMyScript.sh, that contains php myScript.php "Foo bar" and call that with jsub sh runMyScript.sh.

The only operating system release currently available on the Toolforge grid is Debian Buster. Stretch was deprecated in 2022. Trusty was deprecated on Monday 2019-03-25.

Prior to 14 March 2017, there were two different versions of Ubuntu in use on Toolforge: Ubuntu 12.04 ('precise') and Ubuntu 14.04 ('trusty'). The -l release=... option to jsub allowed a tool to choose which release to execute under. This option is currently not needed, but may be useful again in the future when multiple Linux distributions are available simultaneously.

Continuous jobs, such as bots, have a dedicated queue ('continuous') which is set up slightly differently from the standard queue:

• Jobs started on the continuous queue are automatically restarted if they, or the node they run on, crash
• In case of outage or lack of resources, continuous jobs will be stopped and restarted automatically on a working node
• Only tool accounts can start continuous jobs
• Continuous jobs are not restarted if they end normally (with the exit status 0)

For convenience, the jstart script (which accepts all the jsub options) facilitates the submission of continuous jobs:

$ jstart [options…] program [args…]

The jstart script will start the program in continuous mode (if it is not already running), and ensure that the program keeps running.

The jstart script is exactly equivalent to:

$ jsub -once -continuous [options…] program [args…]

jsub's '-once' option is important for ensuring that the job can be managed reliably with job and jstop utilities. The '-continuous' option ensures that the job will be restarted automatically until it exits normally with an exit value of zero, indicating completion.

Bigbrother was a job monitoring tool that has been decommissioned. It watched jobs specified in a .bigbrotherrc file and restarted them if they were not running.

If you're using Grid jobs that sometimes are killed even when running in the continuous queue (when started with jstart), you could consider having a simple shell script watching over them, like the following:

#!/bin/bash

set -o pipefail

if [ $# -ne 2 ]; then
  echo "Usage: $0 <jobname> <command>"
  exit 1
fi

JOBNAME="$1"
COMMAND="$2"

function log {
  echo "$(date -Iseconds) $1"
}

function restart_needed {
  if ! /usr/bin/qstat | awk '{ print $3 }' | grep "${JOBNAME:0:10}" >/dev/null 2>&1; then
    return 0
  else
    return 1
  fi
}

function submit_job {
  /usr/bin/jstart -N "$JOBNAME" "$COMMAND"
}

if restart_needed; then
  log "Restarting job '$JOBNAME' ('$COMMAND')"
  submit_job
fi

Save this to a bigbrother.sh file (or any other name you want) in the tool's home directory and add a crontab entry to trigger it:

*/5 * * * * /data/project/tool_name/bigbrother.sh my_job /data/project/tool_name/my_command.sh

Cron will trigger the bigbrother.sh script every 5 minutes. The script will look for a job named my_job and, if it's not running, it will run the command you specify.

Note that Bigbrother was never necessary for web services. The webservice system uses a built-in system called "manifest monitors" to provide similar functionality automatically.

Each job submitted to the grid has a unique job id as well as a job name (which will not be unique if you have more than one instance running). The name and id identify the job, and can also be used to retrieve information about its status.

If you don’t know the job id, you can find it with either the ‘job’ command or the ‘qstat’ command. Both of these commands can also be used to return additional status information, as described in the next sections.

If you know that your job has only one instance running (if you used the -once option when starting it, for example) you can use the ‘job’ command to get its job id:

tools.xbot@tools-login:~$ job xbot
717898

Use the job command’s -v (‘verbose’) option to return additional status information:

tools.xbot@tools-login:~$ job -v xbot
Job 'xbot' has been running since 2013-04-01T21:00:00 as id 717898

The verbose response is particularly useful from scripts or web services.

Once you know the job id, you can use the ‘qstat’ command to return additional information about it. See Returning the status of a particular job for more information.

The ‘qstat’ command returns detailed information about the status of queued jobs. If you know the job id of a particular job, you can use qstat’s ‘-j’ option to return information about that job. If you use the ‘qstat’ command without options, it will return the status of all your currently running and pending jobs. More information about running qstat without options and with the -j option is included in the following sections. For more information about qstat in general, please see the Grid Engine Manual.

To see the status of all of your running and pending jobs (including the job number), use the ‘qstat’ command without options. ‘qstat’ will then return the job id, priority, name, owner, state (e.g., r(unning) or s(uspended)), the date and time the job was submitted or started, and the name of the assigned job queue (e.g., continuous) for each job.

For example:

tools.xbot@tools-login:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------    
120    0.50000   xbot   tools.xbot         r     04/01/2013 21:00:00 continuous@tools-exec-01.pmtpa     1        

Common job states include:

• r (running)
• qw (queued/waiting)
• d (deleted)
• E (error)
• s (suspended)

See the Grid Engine Manual for a complete list of states and abbreviations.

If you know the job Id of a job, you can find out more information about it using the 'qstat command's ‘-j’ option. qstat will only show information about currently running jobs. For historical jobs, use qacct (which may take minutes to return information). For example, the following command returns detailed information about job id 990.

tools.toolname@tools-login:~$ qstat -j 990
==============================================================
job_number:                 990
exec_file:                 job_scripts/990
submission_time:            Wed Apr 13 08:32:39 2013
owner:                      tools.toolname
uid:                        40005
group:                      tools.toolname
gid:                        40005
sge_o_home:                 /data/project/toolname/ sge_o_log_
name:                           tools.toolname
sge_o_path:                 /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin
sge_o_shell:                /bin/bash
sge_o_workdir:              /data/project/toolname
sge_o_host:                 tools-login
account:                    sge
stderr_path_list:           NONE:NONE:/data/project/toolname//taskname.err
hard resource_list:         h_vmem=256m
mail_list:                  tools.toolname@tools-login.pmtpa.wmflabs
notify:                     FALSE
job_name:                   epm
stdout_path_list:           NONE:NONE:/data/project/toolname//taskname.out
jobshare:                   0
hard_queue_list:            task
env_list:
script_file:                /data/project/toolname/taskname.py
usage    1:                 cpu=00:21:08, mem=158.09600 GBs, io=0.00373, vmem=127.719M, maxvmem=127.723M

Common shell exit code numbers^[1] returned e.g. by qacct include (there are no standard exit codes, aside from 0 meaning success - non-zero doesn't necessarily mean failure):

See the signal(.h) man pages for a more comprehensive list of the values ("n") of the possible fatal error signals (SIG...) issued by the kernel.

If you started a job with the 'jstart' command, or if you know there is only one job with the same name, then you can also use the 'jstop' utility command with the job name to stop it:

jstop job_name

You can also use the underlying ‘qdel’ command with a job’s number or name:

qdel job_number/job_name

This will also delete matching jobs that have only been queued, but not started yet. Do note that if you specify a 'job_name', all queued or running jobs with that name are deleted.

If you do not know the job number, you can find it using the ‘qstat’ command.

In some cases, jobs can get stuck on a host. This happens, for example, if the job somehow does not respond to SIGSEGV and continues running. In these cases, try the following steps:

• Find the host the job is running on: run qstat -xml, and find the relevant queue_name. The part after the @ is the host the job is running on.
• Ssh to that host, e.g. ssh tools-webgrid-generic-1404.tools.eqiad1.wikimedia.cloud
• Find all your running jobs using ps ux
• Kill them: kill <pid>, where <pid> is the number in the second column of ps ux.
• Check if the jobs have been killed with ps ux. If not, try again, but using kill -9 <pid>.

• Maximum of 16 active jobs simultaneously allowed per tool user
  • The scheduler will hold additional job submissions in the qw (queued/waiting) until an active slot is available.
• Maximum of 50 active and queued jobs simultaneously allowed per tool user
  • The scheduler will reject additional job submissions by exiting with a status code of 25 and writing "Unable to run job: job rejected: only 50 jobs are allowed per user (current job count: 50)" to stderr

Implementing these limits has allowed us enable job submission from the continuous and and task job queues.

To schedule jobs to be run at specific days or time of days, you can use cron to submit the jobs to the grid.

Scheduling a command more often than every five minutes (e.g. * * * * * command) is highly discouraged, even if the command is "only" jsub. In these cases, you very probably want to use 'jstart' instead. The grid engine ensures that jobs submitted with 'jstart' are automatically restarted if they exit.

Crontabs are set (as on any Unix system) using crontab -e or crontab FILE.

Please be aware that any submitted crontab is automatically going to be edited to send any jobs to the grid directly (by prepending a default jsub invocation unless the cron entry already had one).

If your cron entry only includes a brief script that, itself, sends any real work to the grid then you may skip that automatic invocation by prepending jlocal explicitly marking it as a local job. Any script or job invoked with jlocal should not be running more than a few seconds and use minimal resources; misuse of that feature may have severe impact on general reliability for all users and is not allowed.

Implementation Detail: Because the $PATH environment variable is set differently for interactive shells and cron jobs, please be aware that the crontab command is a symbolic link to a special executable (/usr/bin/oge-crontab) which will create your crontab in a special way so it's correctly recognized by Toolforge. If you run /usr/bin/crontab directly, that is the local crontab command which will NOT create a crontab in the grid (it will create only a local crontab in the server you're on at the moment and, since cron does not run on all servers, nothing will run based on your crontab). In other words, just use crontab directly or, if you want to specify the full path, use /usr/local/bin/crontab.

The ‘tools’ project, like other hosting environments, uses the time zone UTC (to view UTC time just write date). If you need to schedule a job for another time zone, you can specify so in the crontab. For example, to schedule a job for midnight in Germany, you can use the crontab line:

0 22,23 * * * [ "$(TZ=Europe/Berlin date +\%H)" = "00" ] && jsub ...

The above crontab line instructs the system to check on 22:00 UTC (23:00 CET and 0:00 CEST) and 23:00 UTC (0:00 CET and 1:00 CEST) whether it is midnight in Berlin, and if so, calls jsub. Note that you can't just replace "Berlin" with "Hamburg"; the values for TZ are limited to those found at /usr/share/zoneinfo. If you're unsure what the offset of your time zone to UTC is, you can run the check hourly by replacing 22,23 with *.

Note that the deployed version of crontab currently does not support CRON_TZ (phab:T208561).

The program you want to execute must either be a binary executable or a script. In the latter case, it must contain a shebang line with the name of the interpreter (/usr/bin/perl, /usr/bin/python, etc.). For shell scripts that means in most cases the first line needs to be #!/bin/bash.

When you run a script may be a problem with non-ascii characters.

• Make sure that the script is saved in utf-8 encoding.
• The error may occur if the bash scripts saved with CRLF newlines format of Windows, but it is necessary saved in Unix LF format.
• You must set the LANG environment variable, it you can add it to a bash script. Or: add -v LC_ALL=en_US.UTF-8 parameter to jsub.

For Python:

• Set the LC_ALL environment variable, add -v LC_ALL=en_US.UTF-8 parameter to jsub (as with all jsub parameters, make sure to place it before the python command or else it will be passed to script and not parsed by jsub).
• Run script on python 3 via the command: $ python3 myscript.py. Or, set as the first line of script: #!/usr/bin/env python3, and run it like $ ./myscript.py^[2].
• Make sure that in the begin of script is the line # coding: utf8.
• Set the PYTHONIOENCODING environment variable. Set it through a bash script or by sending -v PYTHONIOENCODING=UTF-8 jsub argument.

The following tools have been built by the Toolforge admin team to help others see grid engine job status:

• sge-jobs.toolforge.org — web-based tool for statistics on current and past Grid engine jobs
• sge-status.toolforge.org — status board of Grid engine nodes and jobs they are currently running (also available as an API)

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

1. ↑ https://stackoverflow.com/questions/1101957/are-there-any-standard-exit-status-codes-in-linux
2. ↑ google about it

• Source code for sge-jobs.toolforge.org
• Wikimedia Techblog: Toolforge and Grid Engine Arturo Borrero González, Site Reliability Engineer, Wikimedia Cloud Services Team, March 14, 2022.