Test jobs are generated periodically to each queue visible from the IC RB for a typical CMS user (generated with a dg-job-list-match command).
Each test job will require output to the queue's local SE, if one is available (whether or not this is true is shown in the "CloseSE" column with a grey ball showing none found).
The right hand six columns shows information on the current test job being performed. The "Last reported status" column shows the status derived from a dg-job-status command. In addition this column has a hyperlink to drill through to the full logging information obtained from a dg-job-get-logging-info command. Note that the status and logging information are not necessarily synched up (the logging trails the status by about three minutes).
The column labelled "Last Status" shows the last confirmed status of the queue (i.e. a test job ran and succeeded/failed with a known status) and a time stamp showing the time at which the test that confirmed this status ran.
At the start of each test, a check of the LDAP server is done (as part of a search for a local closeSE for the particular queue). A failure here will be shown in the LDAP server status column. In this particular case, though the test will fail as it cannot be performed, the status of the queue will remain unchanged.
Note the colour scheme is RED=bad, GREEN=good, GREY=unknown, YELLOW=waiting, flashing GREEN=running.
The dg-job-list-match is performed every 60 minutes. At this time, a check is made to ensure a test process is running on each queue that is listed - if not, a new test is launched. Note that tests for each queue are run independently and in parallel.
For the test of a particular queue, once the job has been successfully submitted to the queue, the status of the job is checked every two minutes.
Finally, the web page with the status table is updated every minute with the latest information from all test processes.