pODI Tutorial: Master calibration products

This tutorial will give you a primer on how to create calibration products, i.e. combined bias, dark, and flat-field frames from your raw data. If you are interested in how to create other calibration products have a look at the other tutorials for fringe-maps, pupilghost, and non-linearity curves.

Download and prepare all raw files

First of all, make sure you downloaded all the raw files you need. This includes a bunch of bias-frames (there are typically at least 10, according to ODI’s standard calibration plan), dark-frames and flat-fields for all filters you might need, plus of course the actual science frames. I personally like to sort my files into directories, so for the sake of this tutorial I put all raw-files in a sub-directory named raw. This might look like this:

(galev) ...wiyn/ison > ls raw/
b20131105T145150.0  d20131105T211237.0  f20131107T161236.5  o20131107T043525.2
b20131105T145150.1  d20131105T223921.0  f20131107T161713.1  o20131107T043525.3
b20131105T145150.2  d20131106T035826.0  f20131107T161713.2  o20131107T043525.4
b20131105T145150.3  d20131106T163430.0  f20131107T161713.3  o20131107T044951.1
b20131105T145150.4  d20131106T163430.1  f20131107T161713.4  o20131107T044951.2
b20131105T145150.5  d20131106T163430.2  f20131107T161713.5  o20131107T044951.3
b20131105T145150.6  d20131106T174330.0  f20131107T162229.1  o20131107T044951.4
b20131105T145150.7  d20131106T174330.1  f20131107T162229.2  o20131107T044951.5
b20131105T145150.8  d20131106T174330.2  f20131107T162229.3  o20131107T044951.6

Preparing the calibration file catalog

Next I create a file containing the names or directories of all calibration files. You could do this by simply running

ls -1 $PWD/raw/[bdf]20*/*33.fits* > calib.list

the $PWD makes sure the files contain the full path name including sub-directories, otherwise the next step might cause trouble. The output is then pipe’d into the calib.list file that we need for the next step. The [bdf]20* part only selects files labeled as bias, darks, or flats, while the *33.fits* only selects the central OTA (no worries, all OTAs will be included in the reduction, but we only need a single frame for the file. If you select all OTAs into the list you effectively reduce every frame 13 times), and the final .fits* makes sure we also include the potential .fits.fz files.

Alternatively, I actually prefer to use the podi_getinfo.py tool to create a file-list with some extra information:

/work/podi_devel/podi_getinfo.py $PWD/raw/[bdf]*/*33.fits* >calib.list

This gives me a small log entry for each file:

/nas/wiyn/ison/raw/b20131105T145150.2/b20131105T145150.2.33.fits.fz    BIAS    1   odi_r   0.0   scp bias                +17:58:52.291  +83:43:31.925
/nas/wiyn/ison/raw/b20131105T145150.3/b20131105T145150.3.33.fits.fz    BIAS    1   odi_r   0.0   scp bias                +17:59:19.641  +83:43:31.335
/nas/wiyn/ison/raw/f20131105T163059.4/f20131105T163059.4.33.fits.fz    DFLAT   1   odi_g   5.0   scp dflat ODI SDSS g'"  +19:33:51.399  +82:58:17.74
/nas/wiyn/ison/raw/f20131105T163059.5/f20131105T163059.5.33.fits.fz    DFLAT   1   odi_g   5.0   scp dflat ODI SDSS g'"  +19:34:30.279  +82:58:17.02
/nas/wiyn/ison/raw/f20131105T163621.1/f20131105T163621.1.33.fits.fz    DFLAT   1   odi_z   5.0   scp dflat ODI SDSS z'"  +19:37:07.178  +82:58:16.698

For each frame it lists in this order:

Filename
Type of exposure (bias, dark, flat, object=science)
Binning (either 1 or 2)
Filter
exposure time in seconds
object name (this is what you filled in in the ODI Exposure GUI)
Pointing information (Ra/Dec)

Based on this file I can delete files I don’t want (e.g. test biases) or select only certain frames (e.g. only sky-flats or only dome-flats)

Creating the calibration products – all at once

To keep my files in order, I put all my calibration files into a sub-directory parallel to the “raw”-directory. All remaining reduction steps will be executed from within this directory.

mkdir calib
cd calib

The actual work of creating calibration products is entire handled by the podi_makecalibrations script. To create all files, this is how you run it:

podi_makecalibrations.py ../raw/calib.list . (options)

The first parameter is the file containing the filenames of all calibration raw-files that we created above. The second parameter specifies the directory to hold all final calibration products. Options specify additional reduction options. In principle all options understood by collectcells are supported, but you will likely only want the following two:

-nonlinearity
-pupilghost=/some/dir

That’s all. Go and run the script and you will see something like the following output:

work/podi_devel/podi_makecalibrations.py ../raw/calib_short.list .
Yippie, found 8 CPUs to use in parallel!

    **********************************************************************
    * This is podi_makecalibrations                                      *
    * (c) 2012-2013: Ralf Kotulla, kotulla@uwm.edu                       *
    *                University of Wisconsin (Milwaukee & Madison)       *
    *                WIYN Observatory, Inc                               *
    *                                                                    *
    * Please acknowledge the author when using any products generated    *
    * with this tool. For comments, questions or ideas for improvement   *
    * please send an email to kotulla@uwm.edu. Thank you!                *
    **********************************************************************

####################
#
# Sighting input data
#
####################
   /nas/wiyn/ison/raw/b20131105T145150.0 --> BIAS BIN=1
   /nas/wiyn/ison/raw/b20131105T145150.1 --> BIAS BIN=1
   /nas/wiyn/ison/raw/d20131105T152034.0 --> DARK BIN=1
   /nas/wiyn/ison/raw/d20131105T165311.0 --> DARK BIN=1
   /nas/wiyn/ison/raw/f20131105T162421.1 --> DFLAT BIN=1
   /nas/wiyn/ison/raw/f20131105T163059.1 --> DFLAT BIN=1
####################
#
# Creating bias-frame (binning 1)
#
####################
writing output file (./tmp/bias.b1.b20131105T145150.0.33.fits)... done!
writing output file (./tmp/bias.b1.b20131105T145150.1.33.fits)... done!
Combining frames for OTA OTA00.SCI (# 13/ 14) ... writing results to file ./bias_bin1.fits ... done!
done!
####################
#
# Creating dark-frame (binning 1)
#
####################
writing output file (./tmp/dark.b1.d20131105T152034.0.33.fits)... done!
writing output file (./tmp/dark.b1.d20131105T165311.0.33.fits)... done!
Combining frames for OTA OTA00.SCI (# 13/ 14) ... writing results to file ./dark_yes_bin1.fits ... done!
done!
filter set set(['odi_g', 'odi_r'])
Workiing on odi_g 1
pupilghost dir= None
####################
#
# Reducing flat-field odi_g (binning=1)
#
####################
Adding OTA 16 to flat-field ... computing median ( 28751.9) ... normalizing ... writing results ... done!
Stacking 1 frames into ./flat_odi_g_bin1.fits ...  done!
Workiing on odi_r 1
pupilghost dir= None
####################
#
# Reducing flat-field odi_r (binning=1)
#
####################
Adding OTA 16 to flat-field ... computing median ( 29718.3) ... normalizing ... writing results ... done!
Stacking 1 frames into ./flat_odi_r_bin1.fits ...  done!

All done, yippie :-)

The routine automatically creates all calibration products in order, starting from bias, darks, to flats, using the earlier calibration products to reduce the later ones (i.e. using the new bias-frame to reduce the dark-frames, etc.). It also knows not to mix and match data with different binnings, so if your input file contains binned and unbinned bias-frames you’ll end up with a binned and an unbinned master bias frame.

While this is very convenient, you might want to inspect the master-bias frame before you use it to create the dark-frame. For this you can either use different input files, one only containing biases, the next only containing darks, and so on. Or you simply use the step-by-step method outlined next.

Creating the calibration products – step by step

make_calibrations understands two command line options that are specific to this tool:

-only=(bias|dark|flat)
This tells make_calibration to only reduce and produce the specified master calibration product. For example, with the -only=bias you will only create the bias frame, although the input catalog might also contain darks and flats.
-keeptemps
All reduced but not yet combined frames are stored in a “tmp” sub-directory and deleted when they are no longer needed. If you suspect you might want to tweak the input catalog to only include the best files for your needs, adding this flag speeds up execution time as the individual files do not need to be re-reduced. However, if you created, e.g, flat-fields before but now decide to change the files that go into your bias frame, make sure to delete the respective files in the tmp directory.

With this information at hand, I assume you can guess the next steps:

/work/podi_devel/podi_makecalibrations.py ../raw/calib_short.list . -only=bias

Check bias_binX.fits to make sure you are happy, otherwise change the input catalog and re-run.

/work/podi_devel/podi_makecalibrations.py ../raw/calib_short.list . -only=dark

Check dark_yes_binX.fits and fiddle with input catalog until happy.

/work/podi_devel/podi_makecalibrations.py ../raw/calib_short.list . -only=flat

Again, check flats and fiddle input catalog until happy.

Once done, if you used the -keeptemps option at any point make sure to delete all files in the tmp directory to free up the disk-space.