# Transfer Files using Rsync Between Hosted Clarity Instance and Customer File Server For File-Based Integrations

In Clarity file-based integrations such as NextSeq 500/550, there may be a need for file transfer between hosted Clarity LIMS instance and Customer File Server.

Examples of file transfer scenarios:

* Transfer of sample sheet from Clarity LIMS instance to Customer File Server
* Transfer of sequencing/analysis run data from Customer File Server to Clarity LIMS instance

## Rsync-based File Transfer

Rsync can be used as an alternative to VPN for transferring files between Clarity LIMS instance and Customer File Server. Rsync is a command-line tool to copy and synchronize files and directories between two locations and leverages SSH network protocol through port 22.

<figure><img src="https://2084401275-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfjuebS41N49G1Eh55hP7%2Fuploads%2Fgit-blob-f122fdb4c681959cb78329433adb25efb900b824%2Frsync-architecture.png?alt=media" alt=""><figcaption></figcaption></figure>

### Prerequisites

1. Same version of Rsync should be installed on both Clarity LIMS instance and Customer File Server.
   * Install Rsync by running either commands as root user:

     <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">apt install rsync
     yum install rsync
     </code></pre>
   * Check the rsync version and protocol version:

     <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">rsync --version
     </code></pre>
2. SSH (port 22) on the Clarity LIMS instance must be accessible from the Customer's instance where the rsync script will be run.
3. Requires appropriate permissions:

   * Read permissions to the source directory and files in the Clarity LIMS instance
   * Write permissions to the destination directory in the Clarity LIMS instance

   {% hint style="danger" %} It is recommended to use */opt/gls/clarity/customextensions* directory. {% endhint %}

### Command

* Transfer file from customer file server to Clarity LIMS instance.

  {% hint style="danger" %} The destination directory to transfer the file must exist in the Clarity LIMS instance and the user must have write permissions to this directory. {% endhint %}

  <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">rsync {File To Transfer} {user}@{Clarity LIMS Instance Name}.claritylims.com:{Target Path}
  </code></pre>
* Transfer file from Clarity LIMS instance to customer file server.

  {% hint style="danger" %} The file to transfer must exist in the Clarity LIMS instance and the executing user must have read permissions to the source directory containing the file and the file itself. {% endhint %}

  <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">rsync {user}@{Clarity LIMS Instance Name}.claritylims.com:{File To Transfer} {Target Path}
  </code></pre>

Refer to <https://download.samba.org/pub/rsync/rsync.1> for detailed information on *rsync*

### Example

<details>

<summary>Transfer sample sheet from Clarity LIMS instance to customer file server</summary>

* transfer\_samplesheet.sh:

  <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">#!/bin/bash
  rsync -av --include='*/' --include='*.csv' "bob@mylimsdomain.com:/path/to/samplesheet/" "/path/to/destination"
  </code></pre>

</details>

<details>

<summary>Transfer run data customer file server to Clarity LIMS instance</summary>

* transfer\_run\_data.sh:

  <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">#!/bin/bash
  pushd "/path/to/instrument/run/root/dir"
  find . -type f -mtime -1 -print0 | rsync -aRv --include='*/' --include='*.bin' --include='*.xml' --exclude='*' --files-from=- --from0 . "bob@mylimsdomain.com:/path/to/destination/"
  popd
  </code></pre>

</details>

transfer\_samplesheet.sh:

{% file src="<https://2084401275-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfjuebS41N49G1Eh55hP7%2Fuploads%2Fgit-blob-23f94aecdcdbcb3a3e6a75aaec4628c4e36cacd8%2Ftransfer_samplesheet.sh?alt=media&token=9ea8cc93-8a5e-4ca3-830c-40325e095984>" %}

transfer\_run\_data.sh:

{% file src="<https://2084401275-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfjuebS41N49G1Eh55hP7%2Fuploads%2Fgit-blob-1258e537bb750181d7c67c29fb5a089ad5cdfa25%2Ftransfer_run_data.sh?alt=media&token=06eb5112-8b7b-4731-8058-6e9581128602>" %}

## Data Clean Up

{% hint style="danger" %}
The Clarity LIMS instance is able to perform at the same efficiency up till storage is full. However, it is recommended to clean up files before storage capacity is full. Upon reaching maximum capacity, Clarity LIMS UI will not be functional and will throw an error, prompting user to approach their admin.
{% endhint %}

<details>

<summary>Clean up old files in the Clarity LIMS instance</summary>

* cleanup\_rundata.sh:

  <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">#!/bin/bash
  find /path/to/destination -type f -mtime +7 -exec rm -f {} \;
  </code></pre>

</details>

<details>

<summary>Delete unwanted files in target folder during rsync file transfer</summary>

* The flag *--delete* can be added to the rsync command to delete any files in the target directory that is not in the source directory.

  <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">Rsync --delete ...
  </code></pre>
* When unwanted files are cleaned up on the customer file server, Rsync will also clean up those files on the Clarity LIMS instance.

</details>

cleanup\_rundata.sh:

{% file src="<https://2084401275-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfjuebS41N49G1Eh55hP7%2Fuploads%2Fgit-blob-e9f475200fd1032cdd1fef0543f9a28197289399%2Fcleanup_rundata.sh?alt=media&token=6621f648-1928-411d-afe3-3b107d0e1c47>" %}

## Automated Bash Script

Crontab can be used to automatically schedule bash scripts to run at a given set interval.

**Example**

1. Run the following command to open and edit the current cron jobs for the current user.

   <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">crontab -e
   </code></pre>
2. Add the following lines to run the transfer scripts every 15 minutes.

   <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">*/15 * * * * /bin/bash /usr/local/bin/transfer_run_data.sh
   */15 * * * * /bin/bash /usr/local/bin/transfer_samplesheet.sh
   </code></pre>
3. Add the following lines to run the clean up script every Sunday at midnight.

   <pre class="language-markup" data-overflow="wrap"><code class="lang-markup">0 0 * * 0 /bin/bash /usr/local/bin/cleanup_rundata.sh
   </code></pre>

{% hint style="danger" %}
Ensure that bash and script locations are correctly specified.
{% endhint %}

{% hint style="info" %}
For more information on crontab, refer to the official documentations: <https://man7.org/linux/man-pages/man5/crontab.5.html>
{% endhint %}