Online Help
You are here: CloudShell Administration > CloudShell Execution Server Configurations > Setting Up Execution Servers to Run Commmands

Setting Up Execution Servers to Run Commands

This article explains how to configure execution servers to run commands in CloudShell. Note that associating execution servers to automation suites is done on the job level, as explained in Add jobs to an automation suite.

In this article: 

Managing execution servers

The Execution Servers management pages in the web portal allow you to both manage and troubleshoot your execution servers, providing critical, real time data about the status of your execution servers, command and job executions, and troubleshooting information and options. For additional information, see Managing Execution Servers.

Optimizing execution capacity

Available capacity for a job's selected execution servers, license usage, resource availability, and the number of other jobs in the queue can all influence how long it takes to schedule and execute an enqueued job.

In theory, the number of TestShell clients in use determines how many execution servers you can install and your overall execution capacity. In practice, the number of installed execution servers is only part of the equation that determines your maximum test execution capacity.

Actual execution capacity is also determined according to the maximum capacity of each execution server (how many tests it can run in parallel), by the number of available execution licenses, and by the offline | online and included | excluded status of each server at the time of execution.

A shortage of execution servers – or execution server licenses – can create a backlog in the execution of scheduled jobs.

You can maximize available execution capacity by installing execution servers on every available TestShell client, by adding additional concurrent execution licenses, and by configuring the execution servers for optimal performance.

  • To optimize run-time performance for individual tests, Quali recommends setting server capacity to 1. This setting ensures that maximum resources are always available to the running instance.
  • If the time it takes to execute each test is less important than the time it takes to run a series of tests, you can set execution server capacity according to the number of cores or the amount of RAM on each machine.

Note: Setting the maximum execution server capacity above the recommended settings may cause performance failures.

Distributed command execution

Multiple execution servers can be deployed in order to scale out the provisioning and resource commands tasks. By default, blueprint and resource commands will be distributed between the execution servers according to their capacity. It is possible, however, to specify more explicit rules to control the execution server selection for commands. In this section we'll review some of those options and understand how distributed provisioning is implemented.

Resource commands

When a resource command is executed, the system first checks whether a driver for that resource is already active on one of the execution servers. If it is, the command is automatically sent to that driver to be queued and handled. In case no driver is currently active, the resource driver is started on an available execution server.

Controlling execution server selection for resource commands

Attributes can be used to match resource commands to the right execution server based on geographical location of the server and resource, execution server capabilities or other concerns. In a multi-site deployment, for example, there is an advantage in ensuring that overseas lab resources are only controlled by an on-premise execution server. This will help reduce the network latency and improve performance.

Associating an execution server to a resource is a simple process that involves assigning one or more linking attributes to both the resource and the execution server and ensuring that both have the same values.

Note: Resources based on 2nd Generation Shells include the Execution Server Selector attribute.

To associate an execution server to a resource:

  1. Add the selector attribute to the relevant execution server. 

    1. In CloudShell Portal, navigate to the Manage dashboard and select Execution Servers from the side pane.
    1. Below Execution Servers in the left pane, select Servers.

      A list of all currently registered execution servers is displayed.

    1. Click the name of an execution server.

      The Attributes dialog box is displayed.

    1. Select the relevant attribute and specify the resource's domain. For example, "London":

  2. If your shell does not have the Execution Server Selector attribute, do one of the following:

    • For 1st Gen shells:

      1. In Resource Manager Client, open the Resource Families explorer.
      2. Add the attribute to the relevant models/families.
    • For 2nd Gen shells, see Adding custom attributes to a Shell.
  3. In the resource's Execution Server Selector attribute, specify a value. For example, the resource's domain - "London".
  4. Rediscover the resource.

    When running a command on the resource, CloudShell will use this execution server. If additional execution servers have the same attribute value, CloudShell will select the first available one.

In some scenarios, you may want to specify additional criteria for execution server selection. For example, an execution server in India that is running on a Windows OS. To achieve this, make sure the resource has these two selector attributes (for example, 'Lab Location' and 'OS'), with 'India' and 'Windows' as their values and at least one execution server with these two attributes selected and with the same values.

To achieve this, you will need to create the attributes with the Execution Server Selector rule, as explained below, and then add the attributes to the shell.

To create Execution Server Selector attributes:

  1. In Resource Manager Client, open the Attributes pane.

  2. Create the new attribute. The attribute can be of any type except Password.
  3. Attach the Execution Server Selector rule to the attribute.
  4. Save your changes.
  5. Repeat steps 1-4 to add additional linking attributes

Controlling execution server selection for App deployments

To learn how to do this, see the appropriate article:

Controlling execution server selection for blueprint commands

Blueprint driver commands are also distributed between the execution servers based on availability and capacity. In order to restrict blueprint commands to a specific group of execution servers, a configuration key needs to be added specifying the relevant selector attribute value.

  1. Create a selector attribute and assign it to an execution server, as explained in Controlling execution server selection for resource commands.

  2. Specify an attribute value for blueprint commands. Add the following configuration key to the Quali server customer.config file: EnvironmentCommandsESRestrictions

The value should be the attribute name and value given the syntax Name=Value. For example:

Configuring the Execution Server to Run as a Process by Default

See this article in the CloudShell Suite Installation Guide.

Working with local tests

If you are using a source control tool and wish to configure CloudShell to work with your local tests, contact Quali support or your Customer Success representative.

Configuring Execution Servers to Deploy vCenter Apps

Number of execution slots for VM deployments

Take the following considerations into account when setting the number of Execution Server slots for the deployment of vCenter VMs.

App deploymentNumber of slots required
Apps deployed manually 3 slots per App
Apps deployed automatically by CloudShell setup

2n + 1 (n = total number of Apps to be deployed at the same time)

For example, deploying 5 Apps requires at least 11 execution slots.

Deployments from OVF image files

The following configurations should be performed on each execution server machine that will be used by the vCenter resource to deploy VMs from OVF image files during App deployments.

  • Configure access to vCenter OVF image file path
  • Install the OVF tool

Enabling custom execution servers

You can enable or disable the use of your own custom-built execution server. This feature is enabled by default.

To enable custom execution servers:

  1. Go to C:\Program Files (x86)\QualiSystems\CloudShell\Portal\customer.config file, and add the following key:
  2. <add key="EnableCustomServerTypes" value="True"/>

  3. Restart the CloudShell Portal IIS service.

Setting the logging level for python drivers and scripts running on an Execution Server

It is possible to set the logging level for automation processes running on a specific Execution Server or python driver (for drivers, see CloudShell Dev Guide's Tips and Tricks article). The logging levels are: DEBUG, INFO, WARN and ERROR.Only messages that are greater than the specified log level are saved to the logs.

Logs are organized in different files according to resource and sandbox. The folder location will be relative to the driver in the virtual environment location at: %venv%\[drivername]\Lib\site-packages\cloudshell\Logs (e.g. C:\ProgramData\QualiSystems\venv\Deployment_Orchestrator_5_2\Lib\site-packages\cloudshell\Logs). Under windows, [venv] will be located at %programdata%\qualisystems\venv.

To set the logging level for python drivers and scripts:

  1. Go to C:\Program Files (x86)\QualiSystems\TestShell\ExecutionServer\customer.config file, and add the following key:
  2. <add key="DefaultPythonEnvrionmentVariables" value="LOG_LEVEL=<log-level>"/>

  3. Replace <log-level> with the desired level.

    For example:

    <add key="DefaultPythonEnvrionmentVariables" value="LOG_LEVEL=DEBUG"/>

  4. Restart the TestShell Execution Server service.

Setting environment variables to be used by python driver instances

Using the DefaultPythonEnvrionmentVariables key, you can define environment variables to be used by the driver. The environment variable is defined on the Execution Server and will be used by the appropriate driver instances that are running on that Execution Server.

To set environment variables for python driver instances:

  1. Go to C:\Program Files (x86)\QualiSystems\TestShell\ExecutionServer\customer.config file, and add the following key:
  2. <add key="DefaultPythonEnvrionmentVariables" value="<variable-pairs>"/>

  3. Replace each <variable-pairs> with a semicolon (;) separated list of the appropriate variable name-value pairs.

    For example:

    <add key="DefaultPythonEnvrionmentVariables" value="name1=value1;name2=value2,"/>

  4. Restart the TestShell Execution Server service.

Troubleshooting

  • What happens if the Execution Server is offline while I run the update process?

    The execution server will still change to a state of Waiting for update and once it gets back online, it will start the update process first, and only then it will get jobs to run.

  • What happens if I (or someone else) start another update process before the previous one is done?

    You cannot start another process until the update on the QualiServer is done.

    Once its done, and the system is updating or waiting to update the execution servers, you can start another update process and even provide different blueprint parameters.

    Idle stations that were already updated by the previous process and stations that are currently updating will start the update again. Each station that is in a Waiting for update state will remain in that state and will execute the batch file with the new parameters when it becomes idle

  • If the batch files fails, how can I check what happened?

    On each machine, you define the batch file to launch and a folder in which we save the outputs from these batch files. If your batch files outputs any information about the process, then you'll be able to see it in these files and check what might have gone wrong. If the process stopped on TestShell Studio, you can check the logs of the portal for other relevant details. If it failed on the execution server, check the logs of it to see if there is more information.