Use the following steps to help troubleshoot the installed Automation Worker framework. A flowchart is provided as a reference.
1.1 Checking the connection
The first step is to check the connection between the Clarity LIMS server and the Automation Worker node.
Use the -n option of the ai-monitor.jar tool script to see if the Clarity LIMSserver is currently able to communicate with the AI node.
To check the status of ai-monitor.jar:
As the glsjboss user open a SSH session to the Clarity LIMS server.
Run the following command:
Copy
If the Clarity LIMS server cannot connect to any of the AI nodes the response will be as follows:
Copy
In this scenario, proceed to Step 2. Verify Windows Service or Linux Daemon.
If the Clarity LIMS server can connect to the Automation Worker nodes, the response will resemble the following:
Copy
Determine if the Windows service or Linux daemon for the Automation Worker is running.
2.1 Starting and stopping the Windows service / Linux daemon
To start, stop, or restart the Windows service:
From the Start menu, select Run.
In the Open text field, type ‘services.msc’ and select OK.
In the Services dialog, locate the Automation Worker service.
Right-click the service and select Start, Stop, or Restart. If the service is stopped, start the service.
If the service is running, stop and start it again.
Wait for a minimum of three minutes, and then check if the AI node is communicating with the Clarity LIMS server by running the ai-manager.sh script with the status argument, as described in Step 1.
To start or stop the Automation Worker Linux daemon:
To verify current status:
Copy
To restart a running daemon:
Copy
To stop a running daemon:
Copy
To start a stopped daemon:
Copy
Once Started/Restarted:
Wait for a minimum of three minutes, and then check if the AI node is communicating with the Clarity LIMS server by running the ai-manager.sh script with the status argument, as described in Step 1.
If the daemon is not recognized, list out the contents of the /etc/init.d directory and determine the exact name of the Automation Worker daemon.
The name typically contains 'automation_worker', but may vary—particularly if there is more than one daemon on the same Linux server, or if the Automation Worker is installed on a server other than the Clarity LIMS application server.
3. Automation Worker Log Files
Automation Worker creates history and log files and stores them on laboratory computers in the logs folder of the Automation Worker installation directory.
3.1 Reviewing Automation Worker log files
After performing the steps described above, reviewing these log files may help to determine the cause of the issue.
For details on the Automation Worker log files, and instructions on how to view them, refer to Clarity LIMS Log Files.
3.2 Turning on debug logging
After reviewing the log files, if the cause of the issue is not evident, the next stage is to turn on debug logging. This outputs DEBUG messages to the log files.
Contact the Clarity LIMSsupport team for instructions on turning on DEBUG mode.
Review the log files to determine if the DEBUG messages help to find resolution.
After turning on debug logging, ensure that you restart the Windows service or Linux daemon.
All Clarity LIMS installations include the installation of a separate component known as an Automation Worker (AW) node (formerly known as Automated Informatics (AI) node).
When writing automation-based triggers, the code invoked by an automation runs on the AW node.
While the AW node is a critical component, it typically does not require much attention. However, there are many other options that must be considered. These options include how many AW nodes a Clarity LIMS system uses, and where they are placed. This section discusses some of these options.
The AW node is installed adjacent to the Clarity LIMS application. Its original purpose was to enable remote computing.
To illustrate these features, assume that a need requires Clarity LIMS to produce a file in a specific location. The file is then processed and an action occurs.
A good example is label printing via the BarTender application. The BarTender application picks up the new file, associates it with a printer and a template, and causes a label to be printed.
The infrastructure for this file storage and processing likely occurs on a separate server from the one that contains the Clarity LIMS application and the AW node.
The Clarity LIMS application invokes the command to create the file on the AW node. After the file is in the file store, it is processed by the file processing application.
How does the AW node get the file to the file store, even if it is on a different server and possibly on a different network? The solution is to install an AW node on the external server.
The addition of the second AW node, which is local to the external server but remote as far as Clarity LIMS is concerned, provides a solution to the problem. This demonstrates how AW nodes can support remote computing..
Clarity LIMS invokes the production of the file via the remote AW node.
The remote AW node copies the file to the local file store.
The file processing application processes the file.
An AW node is a Java-based application and can run on most PCs/servers.
The AW node and the Clarity LIMS application must be able to communicate through networking firewalls.
When the Clarity LIMS application has the choice of sending the task to multiple AW nodes, the channel name property is used to determine which one receives the job. For example, the AW node installed on the Clarity LIMS server has the default channel name of limsserver. This is why you must specify the limsserver value when defining an automation command.
When defining the automation, the following two items are defined:
Which command should be run by the AW node.
Which AW node should receive the job via the channel name property.
Typically, for the AW node that runs on the Clarity LIMS server, the convention is to place scripts in the /opt/gls/clarity/customextensions folder. The log file is stored in /opt/gls/clarity/automation_worker/node/logs/.
For the remote AW node, store the scripts in any folder, and choose where its log file gets stored (it is running on your hardware).
For the external AW node to run Clarity LIMS toolkits, such as the Lab Logic or Lab Instrument toolkit, make copies of the JAR files that contain these toolkits. Place them on the external server, so the AW node can access them.
There is no real limit to how many AW nodes you can have. Place them wherever they are needed.
Consider the AW node that is installed on the Clarity LIMS server for a cloud-based implementation.
Although cloud-based Clarity LIMS servers contain an AW node, the best practice is not to run your code on it.
Why not? It is a question of security policies for both Illumina and our cloud-hosting provider. If we provide customers with command-line access to the AW node, we are allowing them command-line access to the Clarity LIMS server and, as a consequence, the Clarity LIMS application itself. If using Clarity LIMS in a clinical environment, this makes it more difficult to pass security and access audits.
If the Clarity LIMS instance is cloud-hosted, and you need to run custom code via an AW node, there is a solution.
You could install an AW node on your local architecture and have it interact seamlessly with Clarity LIMS, as illustrated in Figure 1. You can control access to the remote AW node, the infrastructure of the Clarity LIMS server is safely hidden behind firewalls, and security policies remain intact.
However, for some customers, part of the attraction of a cloud-based system is not having to maintain mission-critical hardware. To address this, we can offer an external AW node that does not live on local hardware but is also in the cloud.
Thus, we can provide an external AW node that lives on a separate machine, known as the Automation Worker host. This Software-as-a-Service (SaaS) model gives access to only those parts of the system that require it. You can access the Automation Worker host, and its AW node can interact with Clarity LIMS. However, you cannot access the Clarity LIMS server.
Because the AW node is running on a separate machine to the Clarity LIMS server, it needs its own copy of the toolkit JAR files. For some, this feature has an additional bonus in that the Automation Worker host hardware supports Python 3 for scripting.
For customers who are cloud-based and need a true local AW node, this is not a problem. They can have an AW node in the Automation Worker host and as many local AW nodes as needed. Place them wherever they are needed.
In Clarity LIMS v5.4 and later, you can install the Automation Worker Node onto the Windows server. Before you begin the installation, make sure that you have met the following requirements:
The clarity-aiinstaller-x-deployment-bundle.zip file must be retrieved from the server where Clarity LIMS is installed. You can find this file at /opt/gls/clarity/config/.templates/automation_worker/.
For Windows 10 users, the command prompt must be specified in the Automation command line (eg, cmd.exe / c echo 'Hello World').
If VISTA-SETUP.bat does not display in the installation window after Run as Administrator is selected, start the installation window as follows.
Launch the command prompt as an Administrator.
Change the directory where VISTA-SETUP.bat is located with cd C:<DIRECTORY>.
Execute the java -jar .\GLSAutomatedInformatics-Installer.jar command.
Install the Automated Worker Node as follows.
Copy the SecretUtil deployment bundle ZIP file to the remote Automation Worker node.
Extract the contents of the ZIP file to a folder named secretutil. You can add this folder to C:\opt\gls\clarity\tools or another location you choose.
Edit vault.properties of the file in the conf folder to update application.mode to file.
Make sure that the following System Environment Variables are set:
CLARITYSECRET_HOME (eg, C:\opt\gls\clarity\tools\secretutil)
CLARITYSECRET_ENCRYPTION_KEY (minimum 24 characters)
Using secretutil.jar, set the required secrets. For a basic installation of AutomationWorker, you must set the passwords for apiuser and glsftp using the following commands:
# For glsftp
java -jar C:\opt\gls\clarity\tools\secretutil\secretutil.jar -u=<secret> app.ftp.password
# For apiuser
java -jar C:\opt\gls\clarity\tools\secretutil\secretutil.jar -u=<secret> -n=integration apiusers\<username of the API user, e.g. apiuser>
After setting the secrets, attempt to retrieve them with the following command:
java -jar C:\opt\gls\clarity\tools\secretutil\secretutil.jar app.ftp.password
Restart the Automation Worker service.
