FOURPROC

Giving Docker a Try

In the past few weeks there has been lots of buzz about Docker and linux containers, including big announcements for Docker support and management tools by

In my futile effort to stay relevant in this ever changing world, I gave Docker a try. This weekend I spent some time experimenting and eventually building a Docker image that will run LabKey Server.

You can download the image from DockerHub. It is named bconn/labkey-standalone.

The Dockerfile, scripts and instructions for building your own image are checked into LabKey’s public samples repository.

Give it a try.

.....

Using Python to Manage Data in a LabKey Server

At LabKey, I regularly use Python to create tools for managing the servers we use. I often use the LabKey Server Python API to interact with the LabKey Servers we are running. However, in a few instances I have been forced to write code to interact with a LabKey Server in ways not yet supported by the Python API.

Eventually this code might be included in the Python API, but until that happens I decided to share a few examples of what can be done.

upload_file.py:

  • This script shows you how to upload a file from your workstation to the FileRoot of a Folder on your LabKey Server
  • How do I use this functionality?
    • LabKey uses AWS Spot Instances when running the servers used to test LabKey Server. These Spot Instances are started using a Python script which tracks each request in a log file. To make cost accounting easier, each start/stop of a Spot Instance is tracked in a List. When the script finishes the log file is uploaded to the FileRoot of the Folder containing the List and the URL of the uploaded file is included in the List record.

download_study_archive.py:

  • This script shows you how to create a Study Archive and download the resulting Archive (in the form of a ZIP file) to your workstation.
  • How do I use this functionality?
    • I use a similar script to periodically move Study data from one LabKey Server to a second LabKey Server.

These scripts are available in LabKey’s Samples repository at Github. This repository is used by LabKey to share tools/scripts with the LabKey Server community. It currently contains scripts for the install and upgrade of a LabKey Server and the python scripts described above. Keep an eye on it for additional scripts that are coming soon.

.....

Creating a Repository to Hold Example Code and Tools for LabKey Server

This weekend I created a new GitHub repository called samples. This repository was created to hold scripts, tools and/or sample code that LabKey wants to make available to our customers and other LabKey Server users.

Repository Information:

URL: https://github.com/LabKey/samples

What is in this repository as of Today?

Currently the repository contains two folders

ops:

  • Install and upgrade scripts to be used on Windows server (where LabKey is manually installed)
  • These scripts used to be maintained in a repository created by my personal account (https://github.com/bdconnolly/labkey-ops).

python:

  • sample python scripts for uploading files and downloading study archives to/from a LabKey Server

Over the next few months, I will be adding some additional python tools for import/export of Study Archives and for managing AWS instances.

.....

How LabKey uses Vagrant to make development easier

As you may or may not know, LabKey Server has the capability to offload certain types of processing to separate computers such as

  • Rserv: offload the execution of R reports/views to an Rserv running on a separate computer
  • Remote Pipeline: offload proteomics and sequence analysis computation to a separate computer
  • Galaxy: offload sequence analysis and genotyping to a separate Galaxy server.

Last month, one of LabKey’s developers, Kevin Krouse, wrote up instructions for using Vagrant to make development and testing of features using Rserv easier. I took his work and expanded it

  • to support Remote Pipeline servers
  • uses chef-solo to handle configuration during provisioning.
  • [update 9/11] Ben Bimber added support for his sequence analysis pipeline.

Instructions

Once these VMs have been installed, all you need to do is run vagrant up and the VM will be started on your dev machine.

.....

Create a DEBUG log file for your LabKey Server

There are times, when I am investigating a problem, it is helpful to turn on DEBUG logging for the LabKey Server. As you would expect, enabling DEBUG logging, produces LOTS of log entries, so it is not something that you want to keep enabled forever, but it can be helpful to enable while debugging.

There are a number of ways to enable DEBUG logging. The easiest will simply flood the labkey.loglog file with the DEBUG messages, which is not ideal. I like to write the DEBUG logs to a separate, new log file.

The LabKey Server uses log4j for logging. The configuration file is located at LABKEY_HOME/labkeywebapp/WEB-INF/classes/log4j.xml, where LABKEY_HOME is the installation directory for your LabKey Server.

With the changes documented below, the new logging behavior will be

  1. The log file, labkey-errors.log will continue to receive all LabKey-specific messages with priority of ERROR or greater
  2. The log file, labkey.log will continue to receive all LabKey-specific messages with priority of INFO or greater
  3. The new log file, labkey-debug.log will receive all LabKey-specific messages with priority of DEBUG or greater

Please be warned, the labkey-debug.log file can grow very fast. The configuration below sets the max log file size to 10MB. When the file size reaches 10MB, it will be automatically rollover to a new file. Four previous log files will be retained.

Enable DEBUG level logging

You can follow these instructions to enable DEBUG level logging for your LabKey Server

1) Stop your LabKey Server

2) Make a backup of the existing log4j configuration file

cd LABKEY_HOME/labkeywebapp/WEB-INF/classes/log4j.xml
cp log4j.xml log4j.xml.original

3) Edit log4j.xml file

4) Start your LabKey Server

Now if you go directory which holds the log files, CATALINA_HOME/logs you should see a file named labkey-debug.log.

Once you are done debugging your problem, you can disable the DEBUG logging by

1) Stop your LabKey Server

2) Restore the backup of the log4j configuration file

cd LABKEY_HOME/labkeywebapp/WEB-INF/classes/log4j.xml
cp log4j.xml.original log4j.xml

3) Start your LabKey Server

Using the DEBUG log file.

labkey-debug.log will contain lots and lots of DEBUG messages. The best way to use the new log file, is to use a tool like grep to only output the messages you are interested in. For example, if you are interested in tracking the progress of the deletion of a Project, you can use

tail -f labkey-debug.log | grep ContainerManager

or you see all messages about the deletion by running

less labkey-debug* | grep ContainerManager

.....