This topic will guide you on how to troubleshoot CloudShell execution errors. CloudShell saves details of execution activity in different logs, according to automation piece and execution server type (Windows or Linux).
If you are a user or an Admin, you can use these logs to troubleshoot your issue. In some cases, if you have approached Quali Customer Support for assistance, they may ask for these logs to analyze the circumstances that led up to the error.
CloudShell logs include the following:
- QualiX logs
- Linux execution server logs
- Windows execution server logs
- Shell/script execution logs
- L1 shell execution logs
- New Job Scheduling Logs
- QAC logs
We also recommend that you use the system messages displayed in CloudShell Portal, for example, the Activity Details Window, which presents detailed information pertaining to a specific error or message in the activity feed, including the event's logging information and (for command execution errors) the execution server on which the command is supposed to run. For more information on all CloudShell system messages, see the CloudShell help topic Viewing System Messages.
CloudShell Monitor is a tool that collects CloudShell logs and data from the machine it is running on. It delivers a snapshot of the situation immediately prior to when the logs were extracted.
We suggest that you watch this short video demonstrating how to collect and save system log files using CloudShell Monitor:
To collect the system logs, perform the following steps:
You will need the logs from the Quali Server machine and the relevant client machine (Studio, Execution Server, CloudShell Portal, etc.).
- On the appropriate machine, open CloudShell Monitor.
Click Save Logs and from the drop-down list, select Save Logs and Windows Events.
A zip file containing the log files is downloaded to your computer.
Save the zip file.
Open a support ticket at https://support.quali.com/ and attach the zip file to your ticket.
If the zip file is too large to attach to your support ticket (greater than 20 MB), you will receive a Payload is Too Large message in the top right corner of your screen. If this is the case, do the following:
- Log on to the QualiSystems FTP site.
If you do not have the required credentials, contact customer support.
- Save the zip file on QualiSystems FTP.
The file is saved in your account's FTP folder.
- In the ticket, state that you uploaded the log files to QualiSystems FTP.
A Quali support representative will review the logs and contact you.
The zip file contains the following files, depending on the CloudShell component:
Log name Type EventLogs File folder RemoteTeamServer File folder SetupLogs File folder Monitor File folder CloudShell Authoring File folder CloudShell Configuration File folder CloudShell License Server File folder CloudShell Remote Runner File folder CloudShell Resource Management Client File folder CloudShell Runtime File folder CloudShell Sandbox API File folder CloudShell Sandbox API Gateway File folder CloudShell Server Proxy File folder Quali Server File folder TestShell Execution Server File folder TestShell QSMQ Server File folder TestShell Studio File folder RunningProcesses.txt Text Document SytemInfo.exe.txt Text Document SystemInformation.txt Text Document QualisystemsRegistry.txt Text Document InstallInfo.txt Text Document cs_usage_data.zip ZIP package
- Log on to the QualiSystems FTP site.
Collecting QualiX logs
- Open Quali's QualiX Download Page, search the page for "utilities", and click the Utilities link of the appropriate QualiX version.
- Extract the files to your local directory.
- Edit the .\ GetQualiXInfo.ps1 file and set the Username and Password to the QualiX machine.
Save the file and run the .\ GetQualiXInfo.ps1 file with Windows PowerShell.
The script connects to QualiX using SSH and extracts the logs from the QualiX machine.
When prompted, enter the IP address of the QualiX machine.
The command returns the QualiX logs.
- Transfer the files from the Linux machine to your local machine.
- Open a support ticket and attach the files to your ticket.
To get the connection request logs for all services (QualiX over vSphere/Docker):
SSH to the QualiX Server/Docker hostand run:
For details about the Guacamole services, see QualiX High-level Setup Process.
To get the following logs, SSH to QualiX Server and run the appropriate command.
To get the Qualix-tomcat service application logs:
journalctl -u qualix-tomcat
To get the Qualix-guacd service application logs:
journalctl -u qualix-guacd
To set the log level for the Docker logs:
SSH to the Docker host and run:
To get the docker logs for a specific container:
- SSH to the Docker host and run:
sudo docker logs -f <container_id>
To get the container ids, run
sudo docker ps -a.
To get the logs for the guacamole/guacd services running in Docker:
- SSH to the Docker host.
If containers are running, destroy them:
sudo docker stop guacamole guacd && docker rm guacamole guacd
Edit /opt/qualix/docker-compose.yml and set:
Start the containers:
cd /opt/qualix && ./start.sh
View the logs:
sudo docker logs guacamole
sudo docker logs guacd
Note: If you are using a Windows machine to extract logs from the Linux execution server, we recommend using an SCP client to connect via SSH to the Linux execution server to browse though the logs. You can also transfer the files to a Windows machine in order to send them, if needed.
To extract logs if you are using Linux execution server on a Virtual Appliance (VA):
- General execution server logs: ~/ExecutionServer/Logs
- Shell execution logs: /var/log/qualiSystems/
- Orchestration script execution logs: ./usr/share/QualiSystems/logs
To extract logs if you are using Linux ES on CloudFormation (AWS) or CloudShell Template (Azure):
When a Windows-based execution server runs a Python command, either shell-based or an orchestration script, the execution server's logs will be saved here: C:\ProgramData\QualiSystems\logs.
Shell execution logs (resource discovery and inventory commands): C:\ProgramData\QualiSystems\logs\inventory
Note that older logging packages are sometimes logged to C:\ProgramData\QualiSystems\logs\Autoload
- Latest shell/cloud provider execution logs: C:\ProgramData\QualiSystems\logs\<reservation-id>\<driver name>
- Latest orchestration script logs: C:\ProgramData\QualiSystems\logs\<reservationid>
Shell execution logs
Shell execution logs reside on the Execution Server running the shell at %ProgramData%\Qualisystems\logs\<<reservation_id>>\<<shell name>>\*.logs folder in a dedicated sub-folder with the reservation ID as the name.
To customize the shell’s logging policy, enable debug logging:
- See Logging in shells.
To change the log level:
- Navigate to C:\ProgramData\QualiSystems\venv\<shell-name>\Lib\site-packages\cloudshell\core\logger\qs_config.ini.
LOG_LEVEL='INFO'as appropriate. For example,
Orchestration script execution logs
Orchestration script logs reside on the Execution Server running the script at %programdata%\QualiSystems\logs<reservationid> folder in a dedicated sub-folder with the reservation ID as the name.
Setting up and retrieving script logs is detailed in Logging in orchestration scripts.
Virtual environment logs
The virtual environment creation process is as follows:
- Shell/script package installations logged at InstallRequirements.txt file: ~\QualiSystems\TestShell\ExecutionServer\Logs\QsPythonDriverHost\
- If the virtual environment fails to build, CloudShell will create a log record in the ExecutionServer\Logs file (~\QualiSystems\TestShell\ExecutionServer\Logs\QsRuntimeService.txt). But in some cases you might want to enable DEBUG logs to get more details.
Look here for the L1 shell execution logs on the Quali Server machine:
~QualiSystems\CloudShell\Server\Logs\<L1 shell name>.
CloudShells logs events related to the New Job Scheduling's Kubernetes services. For details, see New Job Scheduling Service Logs.
QAC does the following:
- Scheduler of jobs in the original Job Scheduling: Determines whether a job can be executed within a given time range and when a blueprint can be resolved (for jobs that have an attached blueprint)
- Determines the next available time slot (available time slots are proposed in the Resolve Conflicts window that is displayed when reserving a blueprint with resource conflicts)
Note: QAC logs are disabled by default. Once enabled, future QAC logs will be included in the CloudShell system logs ZIP file.
To enable QAC logs:
- Edit the file C:\Program Files (x86)\QualiSystems\CloudShell\Server\log4net.config.
Remove <-- and --> at the beginning and end of the qacLogs section:
So it look as follows:
- Restart the Quali Server service.