One of the latest features of the QuickReduce program is a small tool that listens to the SAMP Message bus that is widely used across many VO tools and, if signaled to do so, goes off and reduces a pODI frame. This reduction can either be done locally on the same machine, or remotely by ssh-ing into a different machine and running QR there.

Configuring the Listener:

Here I describe how to setup the SAMP Listener tool to reduce files remotely. You’ll need the latest version of QR, so first run “svn update” on both the local machine running the Listener AND the remote machine running the actual reduction. With that in place, the next step is the configuration:

All relevant options are found in the podi_SAMPsetup.py file:

message_queue = "odi.image.load"

output_format = "/scratch/%OBSID.fits"

use_ssh = True
ssh_user = "rkotulla"
ssh_host = "localhost"
ssh_executable = "/work/podi_devel/podi_collectcells.py"

def translate_filename_local2remote(filename):
    # This is a template to re-write the filenames from local paths to the
    # path used on the remote system
    if (filename.startswith("/Volumes")):
        # This is on a Mac
        new_filename = filename.replace("/Volumes/", "/mnt/")
    else:
        new_filename = filename
    return new_filename

The first line

message_queue = "odi.image.load"

configures the name of the message you want the SAMPListener to listen to. Feel free to chose your own name, but the above is what’s used at WIYN.

The second block tells the Listener to run the reduction remotely,

use_ssh = True

and defines the account information to use for the ssh command.

ssh_user = "rkotulla"
ssh_host = "localhost"

Make sure you can log into the remote machine without needing to type a password, otherwise the Listener won’t work. Nothing needs to be done if you decide to run the reduction locally.

The last line defines the full path to the executable script on the remote machine.

ssh_executable = "/work/podi_devel/podi_collectcells.py"

Before you start the Listener, log into the remote machine by hand and make sure the installation is setup correctly and you can reduce frames with error messages.

Checking the SAMP connection:

Before you start the Listener for real you should check if the SAMP connection works. To do so, run

podi_SAMPListener.py -testconnect

This will try to establish the connection and either state that the connection was successful or that there were some problems.

The most likely reason for a failed connection are:

You don’t have the necessary python package installed (The error message says something like “unable to import sampy”. In this case you can get the SAMPy package here or install it using “sudo pip install sampy”.
You don’t have a SAMP Hub running. The SAMP hub is the central message broker that handles all communication between different programs. Many VO compatible tools start their own Hub (e.g. Topcat), but you can also start a dedicated SAMP Hub using the JSAMP tool. Download the small Java tool from here, and then start a hub by running “java -jar jsamp-1.3.2.jar Hub” in a terminal.

Retry the connection check to make sure everythings works.

Starting the Listener:

To start the listener, run

podi_SAMPListener.py (options)

(options) here are all options that are understood by collectcells, as theyare simply forwarded to the remote machine during execution or directly read by collectcells if you reduce the files locally.

Important notes for remote reduction:

All filenames you specify have to be valid filenames on the remote machine. This is particularly important for calibration files, WCS models, etc.
The remote reduction is started from home directory of the remote-user (or whatever directory is the starting directory when you ssh into that machine). It thus is advisable to use full path names rather than relative path names (e.g. /home/user/calib rather than just calib/). Also note that the ~ sign as placeholder for the home-directory is resolved on the local machine and might not work on the remote machine.

With this in place, you can start reducing your frames.

Running a test-frame to make sure things work

In the test/ subdirectory within the QR source directory is a small test-tool named “sampy_send.py”. This can be used to send a filename over the message bus to test and potentially troubleshoot the SAMPListener. Ifyou choose a custom message name you might have to edit this file and specify the new name at the top of the file.

To send a SAMP command, run

sampy_send.py qr /some/dir/podi_dir/ (e.g. /my/files/o12131211T123456.0/)

Running the SAMPListener at WIYN

The most likely use-case for the SAMPListener is as automatic reduction tool while observing at WIYN. For this case I have bundled the command and all its options into a little shell script that you can simply run in a terminal:

sh ~/bin/start_listener.sh

Once started you should receive an output as the following.

 *******************************************************************
 * SAMPListener for automatic image reduction (locally/remote)     *
 * Part of the QuickReduce package for the WIYN One Degree Imager  *
 * Author: Ralf Kotulla, kotulla@uwm.edu                           *
 *******************************************************************

Starting receiver ...
Starting execution process...
Setup complete, waiting for messages...
Worker process started, ready for action...
Current system-time: 2014-01-05 14:20:23.582003 (press Crtl-C to quit)

Once this is started, the following things will happen:

Every time you double-click a object frame in the ODI File Browser (the upper window on the right ODI screen), the file will be displayed in the OTA Browser (the lower window on the right screen as usual.
At the same time, the frame will be send off for reduction on a different machine (it’s configured to use the mostly idle odidb machine). The reduction includes overscan-, bias-, and dark-correction, flat-fielding, correction for nonlinearity, and astrometric and photometric calibration, all taking less than typically 30 seconds.
Once the frame is reduced, the central OTA (ID 3,3 – the central OTA of the 3×3 science field) is loaded into ds9.

If you prefer to view the entire focal-plane, you can also load the frame manually – in ds9, go to File -> Open As -> Mosaic IRAF (open files as Mosaic WCS is possible, but not recommended because of the way ds9 handles WCS distortions in multi-extension FITS files); the reduced files are located under

/Volumes/odifile/qr

and named after the OBSID, i.e. the timestamp shown as filename in the ODI File Browser. One example would be 20140104T034405.2.fits.

Note:

The start_listener.sh script on wiyn-5 supports passing additional parameters on to collectcells.

This means that if you, for example, want to apply fringing as correction, just add the fringing option for podi_collectcells to the start_listener.sh script, and run

~/bin/start_listener.sh -fringe

You can even get your non-sidereal motion taken care that way, although this case is a little more complicated. What you need in this case is the reference MJD as number, e.g. by copying the value from a fits header of your reference frame. Then kill the currently active start_listener, and add your updated nonsidereal option when you restart the command. One example could be the following call.

~/bin/start_listener.sh -nonsidereal=-12.0,+14.2,53765.836583

Here, the -12.0 is the non-sidereal motion in Ra, given in arcsec/hour, the +14.2 is the velocity in declination, and the 53765.836583 is the MJD of the reference frame. Keep in mind, however, that overplotting reference catalogs in ds9 won’t work properly, as the reduced frame has its WCS modified to account for the non-sidereal motion. Additionally, already existing reduced frames are not re-reduced even if you double-click them in the File Browser, so you might be stuck with the corrected frame if you choose to go down this route.

Warning:

The automatic reduction uses some pre-installed reduction files that could be outdated or incomplete – I installed basic calibrations (bias, darks) and flat-fields for the broad-band filters based on data from 2013/01/03 when writing this. This means that the files should be good enough to judge data quality etc, they do not yield the best possible reduction (also there’s no fringe or pupilghost correction). For optimal science-grade reductions use the ODI PPA and process the data either with the official AuCaP pipeline or my QuickReduce pipeline.