Attach Files Located Outside the Default File Storage Repository

In Next Generation Sequencing (NGS), a large amount of data is generated in a single run. Because the volume of data is so large, it makes sense to link to data files as they exist on the network, rather than copying files to the Clarity LIMS file server.

This example shows how you can use the files resource to associate a process output result file with a file on the network using the HTTP protocol.

Prerequisites

Before you follow the example, make sure you have the following items:

• Samples that have been added to the system.

• A NGS process that takes analyte (derived sample) inputs and creates a single result file output for each input.

• A flow cell container with 8 rows and 1 column. Both the rows and columns are numbers and start at 1.

• Samples that have been added to a flow cell and have run the flow cell through your NGS process.

• An HTTP file store that has been set up in the Clarity LIMS configuration file.

• The appropriate directory structure and files to be linked to your HTTP file store. For this example, you need a directory with the same name as the flow cell on which you run your process.

  • You require a subdirectory for each lane named by lane number (eg, 1 containing the file you want to link).

  • The name of the file must be easily programmatically determined (eg, s_1_export.txt for lane number 1).

  • A compatible version of API v2.

Setup

This example assumes you are using the standard non-production scripting sandbox server, which uses Apache to serve files with HTTP. For more information on this server, see Useful Tools.

If you plan to use an alternative file storage configuration, contact the IT administrator of the Clarity LIMS server.

The administrator uses the omxprops-ConfigTool.jar configuration tool. For more information on this configuration tool, refer to Non-Production Scripting Sandbox Server - IT Admin Guide.

The omxprops-ConfigTool.jar tool is at /opt/gls/clarity/tools/propertytool/. For more information, see Remote HTTP Filestore Setup.

Code example

After running a NGS process on a full flow cell, in the Operations Interface, the Input/Output Explorer for the process run shows the relationship between the inputs in the flow cell and the result file output placeholders.

Attaching_files_not_on_default_file_repository_before.png

Associating a file that resides on a server that is accessible via the HTTP protocol requires the following steps:

1. Matching up the file with the correct file placeholder.

2. Constructing the XML to POST to the files resource.

3. A POST to the files resource, which associates the file and creates the link between the artifact and the file.

Step 1. Match the File with the Correct File Placeholder

The following code snippet shows how, by starting with only the name of the flow cell, we can obtain the API resource for the flow cell container and the NGS process run.

Because the location of all the files we want to link is known, the combination of the flow cell and NGS process contains all the additional information required to link each file to the correct file placeholder, so that the results are associated with the correct sample.

1. Get a containers list filtering by container name, where the name is equal to the name of the directory. Your container name must be unique in the system as only one result is expected by this script.

   def containers = GLSRestApiUtils.httpGET("${baseURI}containers?name=${containerName}", username, password)
   println(containers)

2. Get the full XML for the container. The container list only contains the URI and LIMS ID of the container and we require all the placements as well.

```
def container = GLSRestApiUtils.httpGET(containers.'container'[0].@uri, username, password)
```

3. Using the LIMS ID of one of the placements in the container, use the API to find the processes that have used it as an input. For this example, the expectation is that only one process has been run on the flow cell. If that is not the case, the first one returned is used.

```
def processes = GLSRestApiUtils.httpGET("${baseURI}processes?inputartifactlimsid=${container.'placement'[0].@limsid}", username, password)
```

4. Complete the XML with the input-output-map and make another call to the API to retrieve the complete process. As for the container case, the list resource just gives us the URI and LIMS ID of the process.

```
def process = GLSRestApiUtils.httpGET(processes.'process'[0].@uri, username, password)
```

The code in this example includes some helper closures and a persistent single HTTP connection. The persistent connection helps improve the performance of the script.

The location of the file to be linked is constructed for each POST, based on the directory structure. For each input-output-map element in the process it is stored in the FileURI variable. The value provided here is used in both the <content-location> and <original-location> elements in the file node that will be posted.

String placement = container.placement.find{ it['@limsid'] == inputlimsid }.value[0].text()
    String laneNum = placement.split(':')[0]
    String fileURI = "${illuminaRunDirectoryURI}/${laneNum}/s_${laneNum}_export.txt"

The location of the result file artifact is stored in the variable outputURI. It is obtained from the <output> element in the input-output-map:

String outputURI = it.output[0][e-mail]("\\?")[0]

Step 2. Construct the XML to POST to the Files Resource

The first requirement is to create an XML resource for the file you want to submit to the system.

As in the example from Attach a File to a File Placeholder with REST, you create the resource using StreamingMarkupBuilder, which you use in a POST to the files resource.

When you construct the individual file XML, you must specify the content-location, attach-to, and original-location child nodes.

In this example, you are posting a file link to a file in an HTTP file storage location, so the content-location and original-location are the same.

def fileBuilder = new StreamingMarkupBuilder()
    fileBuilder.encoding = "UTF-8"
    def fileXML = fileBuilder.bind {
        mkp.xmlDeclaration()
        mkp.declareNamespace(file: 'http://genologics.com/ri/file')
        'file:file'{
            'content-location'(fileURI)
            'attached-to'(outputURI)
            'original-location'(fileURI)
        }
    }

Step 3. POST to the Files Resource

The POST requires an XML node as input. Therefore, you must convert fileDoc from a writable closure to an XML node using GLSGeneusRestApiUtils.xmlStringToNode.

You can then submit the new file resource to the API via a POST.

// Convert the fileXML to a node and link the file by posting the node to the files resource in the API
    Node fileNode = new XmlParser().parseText(fileXML.toString())
    GLSRestApiUtils.httpPOST(fileNode, "${baseURI}files", username, password)

The following example shows the XML for FileNode, which is used in the POST:

<?xml version='1.0' encoding='UTF-8'?>
<file:file xmlns:file='http://genologics.com/ri/file'>
    <content-location>http://Root of your http file store/illumina/27-18-1/1/s_1_export.txt</content-location>
    <attached-to>http://yourIPaddress/api/v2/artifacts/ADM1A1NE3</attached-to>
    <original-location>http://Root of your http file store/illumina/27-18-1/1/s_1_export.txt</original-location>
</file:file>

The POST command does the following:

• POSTs the XML constructed by StreamingMarkupBuilder to the files resource.

• Adds a link from the files list resource to the new file.

• After the POST executes, a link to the new file resource is added to the ResultFile artifact resource – because it was specified in the attached-to field of the new file resource.

The following code is the XML for the ResultFile artifact after the POST, which contains a link to the new file resource on the last line:

<art:artifact uri="http://yourIPaddress/api/v2/artifacts/ADM1A1NE3?state=48" limsid="ADM1A1NE3">
   <name>results.txt</name>
   <type>ResultFile</type>
   <output-type>ResultFile</output-type>
   <parent-process uri="http://yourIPaddress/api/v2/processes/NEX-SA1-101126-24-1" limsid="NEX-SA1-101126-24-1"/>
   <qc-flag>UNKNOWN</qc-flag>
   <sample uri="http://yourIPaddress/api/v2/samples/ADM1A1" limsid="ADM1A1"/>
   <file:file limsid="ADM1A1NE3-40-112" uri="http://yourIPaddress/api/v2/files/ADM1A1NE3-40-112"/>
</art:artifact>

The file resource available from the API is slightly altered from the XML submitted to the POST. The file is assigned a LIMS ID and a URI.

The following code is the XML resource for the file that is available from the API:

<file:file uri="http://yourIPaddress/api/v2/files/ADM1A1NE3-40-112" limsid="ADM1A1NE3-40-112">
   <attached-to>http://yourIPaddress/api/v2/artifacts/ADM1A1NE3</attached-to>
   <content-location>http://Root of your http file store/illumina/27-18-1/1/s_1_export.txt</content-location>
   <original-location>http://Root of your http file store/illumina/27-18-1/1/s_1_export.txt</original-location>
   <is-published>false</is-published>
</file:file>

The POST to the files resource associates a physical file to the ResultFile artifact. In the user interface, this POST changes the file icon.

Before POSTing to the files resource, you must make sure that the file exists in the location referenced by the content-location element. If the file does not exist in this location, the POST fails.

Expected Output and Results

The value returned from the POST is stored in the returnNode variable.

If the command was successful, the above XML is returned. If the command was unsuccessful, an XML document explaining the error is returned. For example, if a file was already attached to the result file at artifactURI, the following document is stored in returnNode, as returned from the POST. In this case, the file that is already attached to the artifact must first be removed via the file resource DELETE method.

<exc:exception xmlns:exc="http://genologics.com/ri/exception">
    <message>Removing file in this manner not permitted</message>
</exc:exception>

After a successful POST HTTP command, in the Operations Interface, the process summary view's Input/Output Explorer shows all the attached files.

Attachments

AttachingFileNotOnClarity.groovy:

4KB

AttachingFileNotOnClarity.groovy