Check Status of Batch Jobs

The cqstat command is available on all user-accessible HPC systems at the participating centers. It is able to produce one of two reports: A Default Report, and a Priority Report. In both reports, information is displayed in two sections: Active Jobs, and Waiting Jobs. The two sections are separated by a summary that shows the number of cores and nodes in use for each type of node configured on the system. The Priority Report is identical to the Default Report with the addition of a priority column in the Waiting Jobs section.

Sample Output

$ cqstat --prio
---------------------------------------------------------ACTIVE JOBS-------------------------------------------------------
JOBID                  JOBNAME  USERNAME        SUBPROJECT  ST   CORES   NODES        QUEUE  REMAINING       START TIME
---------------------------------------------------------------------------------------------------------------------------
1082416               Job_1234  useraaaa     PROJECT012345   R   18432     512   background    0:03:59  Thu Jun  8 07:52:19
1082445            TestingAAAA  userbbbb     PROJECTAAAAAA   R      64       2        debug    0:13:22  Thu Jun  8 08:05:35
1081154              prodrun03  usercccc     PROJECT987654   R     252       9     standard   11:58:22  Wed Jun  7 20:35:50
1082185               my_projX  userdddd     PROJECT000000   R     560      20         high   14:03:05  Thu Jun  8 00:40:06

          4 Active Jobs
                               18432 of 124416 Standard Compute Cores Active ( 14.81%)
                                 512 of   3456 Standard Compute Nodes Active ( 14.81%)

                                 252 of    896 GPU Co-Processors Active ( 28.13%)
                                   9 of     32 GPU Nodes Active ( 28.13%)

                                  64 of    128 Large Memory Cores Active ( 50.00%)
                                   2 of      4 Large Memory Nodes Active ( 50.00%)

                                 560 of   4984 Phi-Attached Standard Cores Active ( 11.24%)
                                  20 of    178 Phi Nodes Active ( 11.24%)


--------------------------------------------------------WAITING JOBS---------------------------------------------------------------------------------------------------------------
JOBID                  JOBNAME  USERNAME        SUBPROJECT  ST   CORES   NODES        QUEUE  WALLCLOCK       QUEUE TIME         WAIT TIME       EST START TIME             PRIORITY
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1076944                    run  usereeee     PROJECTZYXWVU   Q    4608     128     standard   24:00:00  Thu Jun  1 10:08:55    6:22:28:35  Thu Jun  8 09:29:14 2017         86.8373
1080013               my_projY  userdddd     PROJECT000000   Q     560      20         high   72:00:00  Tue Jun  6 12:11:46    1:20:25:44  Sat Jun 10 20:35:51 2017         61.6933
1073628         Data_Coll_Main  userffff     PROJECT135790   Q    1080      30     standard   96:00:00  Tue May 30 18:54:02    8:13:43:28  Tue Jun 20 04:34:49 2017         46.5995
1082404               Job_5678  useraaaa     PROJECT012345   Q      28       1   background   00:40:00  Thu Jun  8 07:48:24       0:49:06  Thu Jun  8 10:28:53 2017       -999.8600

Total Jobs:     8   Active jobs:     4   Waiting Jobs:     4

Description of Report Columns

Columns included in all reports:

  • JOBID - ID number given to the job by the batch queuing system
  • JOBNAME - Name given to the job by either the user or the batch queuing system
  • USERNAME - Username of the person who submitted the job
  • SUBPROJECT - Subproject whose allocation will be charged for time spent running this job
  • ST - Job status: Q(ueued), H(eld), W(aiting), R(unning), etc.
  • CORES - Number of computational cores of all types requested by the job
  • NODES - Number of computational nodes of all types requested by the job
  • QUEUE - Queue in which the job has been placed

Additional columns included in the Active Jobs list:

  • REMAINING - Time remaining for the job to complete execution (HHH:MM:SS)
  • START TIME - Start time of job execution (DAY MON DD HH:MM:SS)

Additional columns included in the Waiting Jobs list:

  • WALLCLOCK - Requested maximum wall time for job execution (HHH:MM:SS)
  • QUEUE TIME - Time when the job entered the batch queue (DAY MON DD HH:MM:SS)
  • WAIT TIME - Time that the job has been waiting to run (DDD:HH:MM:SS)
  • EST START TIME - Estimated start time for the job, based on current queue conditions (DAY MON DD HH:MM:SS)

Additional columns included in the Priority Report:

  • PRIORITY - Calculated relative priority for each job, based on current queue conditions

*** NOTE: The values for Estimated Start Time and Priority change quickly.

Back to BC Policy FY10-06.