SQS - the Simple Queueing System.

Hosted on SourceForge

Latest version 3.1 (Released privately in May, 2011, and on sourceforge in October, 2011.)

There are many queueing and scheduling systems around, but they were all rather over-kill for what we needed. This was originally a single queue on a single machine allowing one or two jobs to run at once. This version allows several queues, primarily because it is useful to have a main working queue, and a queue for runs that just test the input files. It also has the ability to submit jobs to machines on a cluster and to do some inspection and manipulation of queues on a truly remote machine (see the section at the end of the README file on remote use).

SQS has two uses for us, but it may have others:-

  1. For the general running of computational chemistry codes, which use a lot of scratch disk space, a lot of time and often a lot of memory.

  2. For memory, disk intensive jobs submitted from web form pages.

Here are some examples of using SQS with cluster or remote machines:-
  1. We use a cluster of Linux machines. They are not connected directly to the internet and communicate with ssh and friends. Almost the whole filesystem of each cluster is in common, mounted using NFS. We want to work on any one machine, and run jobs on all machines. We call this cluster use. There is one set of work directories, one set of SQS queue files and one set of executables.

  2. We have used two machines that are identical, but we did not have root access and nothing was mounted in common. The administrator did not want us to do this. We wanted to run jobs on both machines, with each running its own SQS. We wanted to inspect and modify queues on each machine from the other, depending on which one we are connected to. We used slogin, ssh and friends on this pair of machines. We call this remote use. Connecting to the first cluster, which is at a university, from home is another example of remote use.

  3. At home, I link a laptop running Linux to a PC running Linux, with a direct connection. We can treat this pair as cluster use or remote use. It was used to develop earlier versions of this code.

The system runs under linux and is written in perl. It should run on any system with perl. Earlier versions were also tested on a Tru64 Compaq Alpha. I have used it regularly on a laptop running Cygwin. However, I am not a brilliant programmer. There may still be bugs. There must certainly be more elegant and efficient ways of doing things. Please join this project to make it better.

We had previously used Generic NQS, but this contains many features we did not want and now seems to have little support for new platforms. We want just a few queues. Initially we wanted this to run on a stand-alone single machine, but we now want to submit jobs to a cluster and have implemented this. We want to be able to inspect the queue, delete jobs in the queue, including killing a running job. We want to be able to put jobs on hold, and release them later. We now have a priority system with different jobs having different priorities. We want to move queued jobs from one queue to another. the capabilities are discussed in more details in the README linked below.

I am happy to receive contributions that extend the way SQS works, if there seems a demand for them. However, let us try to still keep it the Simple Queueing System.

It consists of 12 scripts, including 'install.pl' for making the installation and deinstallation more automatic. There are man pages that are installed during the installation process. The original files are templates named 'file.t', which are used by 'install.pl' to create the following scripts:-

There are also include files called sqs.conf and sqs.inc

See the README and INSTALL files for more information.

See the ChangeLog for the development of this project.

This page is maintained by Brian Salter-Duke.

Last Modified: 28th October 2011 by : b_duke@bigpond.net.au