Skip to main content
Version: 2023.3

Statuses and results

Below are the possible statuses and descriptions for suite, job and test executions.

Note that every API method responds with the standard HTTP error codes (200, 300, 400…) but also includes a response body for errors. Within the error body, it is possible to find the list of errors in a human-readable manner together with an internal application error code that can help Quali when debugging an issue. For example, trying to stop an already finished suite will result in a 400 HTTP error code with the following JSON response body:

{
"errors":[
{
"message":"Suite execution already completed",
"name":"SuiteExecutionCompleted",
"code":1006
}
]
}

In the above response, 1006 is the internal application error code for the specific use case and the message is the human-readable error description

Suite statuses

API StatusStatusDescription
0Not startedTriggered. None of the suite’s jobs have started running, waiting for either slot or sandbox.
1RunningAt least one job started running.
2SucceededAll tests in all jobs have executed with test result either COMPLETED or PASSED.
3FailedAll tests in all jobs have executed with at least 1 test with test result FAILED.
4TerminatedAt least one test has test result TERMINATED.
5Automatically CancelledOption 1 - sandbox error (active with error, didn't find a sandbox in the retries). Option 2 - no test slot found
6Manually CancelledSuite execution was cancelled manually.

Job statuses

API StatusStatusDescription
0Not StartedTriggered. Job has not started running, waiting for either slot or sandbox.
1RunningTest execution is in progress (test has not failed/terminated).
2SucceededAll tests have been executed with COMPLETED or PASSED result.
3FailedAll tests have been executed. At least 1 test with result FAILED.
4TerminatedAt least 1 test with result TERMINATED.
5Automatically Cancelled

Job was automatically cancelled.

Possible reasons are:

  • Validation of job test data failed

  • Max retry count limit exceeded when trying to find an available slot

  • Max retry time limit exceeded when trying to find an available slot

  • Teardown started while tests are running

  • Job details:

    • Job's blueprint was deleted or modified in the online repository
    • CloudShell detected that the job's test(s) was deleted or modified
    • Job duration was reached while some of the job’s tests did not execute
  • Sandbox errors:

    • Sandbox creation took too long (configurable value ~35 minutes)
    • Sandbox cannot be used (has errors or failed to be created)
  • Connection errors:

    • Test Execution Service was unable to load blueprint details due to Quali Server connection failure
    • Job Scheduling Service was unable to send tests to Test Execution Service for execution
    • Job Scheduling Service was unable to publish the sandbox request to RabbitMQ
6Manually CancelledThe job was cancelled manually while in progress (Not Started, Running).
10Waiting for Job SlotJob is waiting for a suitable test execution server slot to become available.
11Waiting for SandboxJob is waiting for sandbox to become Active.

Test results

API ResultResultDescription
0CompletedTest finished execution.
1PassedTest completed successfully.
2Failed*An error during the test’s execution caused it to fail.
3Terminated*

Possible reasons are:

  • Test execution was stopped by the user or job status is "Automatically Cancelled”
  • The Test execution service failed to start the test:
  • API request failed
  • Test execution service failed to clone/pull repository (failed to create workspace)
  • Test execution service failed to create virtual environment
  • Requirements file is missing
  • Test execution failed/aborted on the Test execution service
  • Test execution service, which had a running job, was deleted from the Job Scheduling portal
4FailedToStart*

Job triggered. Test failed to execute.

Possible reasons:

  • Test execution service failed to clone/pull the repository (= failed to create workspace)
  • Test execution service ES failed to created virtual environment
  • Test's requirements file is missing

* Test execution failure does not stop the job from running its other tests.