Job Submission

Job submission

Overview

The Satisfaction Survey revealed that most researchers and students do not face major difficulties in processing jobs. However, a critical analysis of the emails sent by users indicated significant opportunities for improvement in the documentation on job submission.

Note

The documentation improvements have been implemented. The documentation has been reviewed continuously.

Factors Affecting Wait Time

Queue wait time

A critical point identified in the survey is the wait time for a submitted job to start running on the processing nodes. This time depends on several factors:

  1. Resources available at the moment

    • Demand for the cluster varies constantly

    • Peak periods may have longer queues

    • Resources are finite and shared

  2. Resources requested by the job

    • Jobs that request many resources (CPUs, nodes, memory) tend to wait longer

    • Jobs with very long run times have reduced priority

    • Realistic specifications help with faster allocation

  3. Fair Share policy

    • Based on the recent usage of the user and the group

    • Users with frequent usage have reduced priority

    • Groups with many active users have lower priority per member

    • Full details in Priority Policy

Job Queue Dynamics

Cluster usage by researchers varies constantly with respect to:

  • Number of jobs submitted

  • Resources requested by each job

  • Processing time required

Because resources are finite, during periods of high demand:

  1. Some jobs remain in the PENDING state (waiting)

  2. Jobs start running (RUNNING) as resources are freed up

  3. Priority is recalculated continuously

Queue Ordering

The submission system gives preference to jobs that:

  • Require a smaller amount of resources

  • Have a more favorable Fair Share index

  • Request a shorter run time

Important

Because of these factors, some users may have to wait hours or even days for their jobs to start running, especially during periods of high demand.

Practical Example

Consider two jobs submitted simultaneously:

Job comparison

Characteristic

Job A

Job B

CPUs requested

28

4

Time requested

7 days

6 hours

User Fair Share

0.2 (intense recent usage)

0.8 (moderate usage)

Resulting priority

Low

High

Position in queue

End

Start

In this scenario, Job B will likely start running well before Job A, even though it was submitted later.

Note

At the time the Satisfaction Survey was conducted, most users had not yet started using the new resources acquired in the upgrade. Currently, the cluster has 56 new processing nodes with 56 threads via CPU, greater storage capacity and more available memory, in addition to a server with 4 GPUs.

Recommendations to Reduce Wait Time

✅ Do:

  • Request only the resources you really need

  • Estimate the run time with a realistic margin (but not an excessive one)

  • Test jobs with small samples before submitting the full set

  • Use job arrays to split large processing into smaller jobs

  • Check your Fair Share regularly with sshare -U $USER

❌ Avoid:

  • Requesting all the cluster’s resources for a single job

  • Estimating times far longer than necessary “just in case”

  • Submitting hundreds of identical jobs at once

  • Ignoring the Fair Share policy

See also