Statuses and results

Below are the possible statuses and descriptions for suite, job and test executions.

Note that every API method responds with the standard HTTP error codes (200, 300, 400…) but also includes a response body for errors. Within the error body, it is possible to find the list of errors in a human-readable manner together with an internal application error code that can help Quali when debugging an issue. For example, trying to stop an already finished suite will result in a 400 HTTP error code with the following JSON response body:

{
   "errors":[
      {
         "message":"Suite execution already completed",
         "name":"SuiteExecutionCompleted",
         "code":1006
      }
   ]
}

In the above response, 1006 is the internal application error code for the specific use case and the message is the human-readable error description

Suite statuses

API Status	Status	Description
0	Not started	Triggered. None of the suite’s jobs have started running, waiting for either slot or sandbox.
1	Running	At least one job started running.
2	Succeeded	All tests in all jobs have executed with test result either COMPLETED or PASSED.
3	Failed	All tests in all jobs have executed with at least 1 test with test result FAILED.
4	Terminated	At least one test has test result TERMINATED.
5	Automatically Cancelled	Option 1 - sandbox error (active with error, didn't find a sandbox in the retries). Option 2 - no test slot found
6	Manually Cancelled	Suite execution was cancelled manually.

Job statuses

API Status	Status	Description
0	Not Started	Triggered. Job has not started running, waiting for either slot or sandbox.
1	Running	Test execution is in progress (test has not failed/terminated).
2	Succeeded	All tests have been executed with COMPLETED or PASSED result.
3	Failed	All tests have been executed. At least 1 test with result FAILED.
4	Terminated	At least 1 test with result TERMINATED.
5	Automatically Cancelled	Job was automatically cancelled. Possible reasons are: Validation of job test data failed Max retry count limit exceeded when trying to find an available slot Max retry time limit exceeded when trying to find an available slot Teardown started while tests are running Job details: Job's blueprint was deleted or modified in the online repository CloudShell detected that the job's test(s) was deleted or modified Job duration was reached while some of the job’s tests did not execute Sandbox errors: Sandbox creation took too long (configurable value ~35 minutes) Sandbox cannot be used (has errors or failed to be created) Connection errors: Test Execution Service was unable to load blueprint details due to Quali Server connection failure Job Scheduling Service was unable to send tests to Test Execution Service for execution Job Scheduling Service was unable to publish the sandbox request to RabbitMQ
6	Manually Cancelled	The job was cancelled manually while in progress (Not Started, Running).
10	Waiting for Job Slot	Job is waiting for a suitable test execution server slot to become available.
11	Waiting for Sandbox	Job is waiting for sandbox to become Active.

Test results

API Result	Result	Description
0	Completed	Test finished execution.
1	Passed	Test completed successfully.
2	Failed*	An error during the test’s execution caused it to fail.
3	Terminated*	Possible reasons are: Test execution was stopped by the user or job status is "Automatically Cancelled” The Test execution service failed to start the test: API request failed Test execution service failed to clone/pull repository (failed to create workspace) Test execution service failed to create virtual environment Requirements file is missing Test execution failed/aborted on the Test execution service Test execution service, which had a running job, was deleted from the Job Scheduling portal
4	FailedToStart*	Job triggered. Test failed to execute. Possible reasons: Test execution service failed to clone/pull the repository (= failed to create workspace) Test execution service ES failed to created virtual environment Test's requirements file is missing

* Test execution failure does not stop the job from running its other tests.