Getting started with the ULTRACAM pipeline software

This page attempts to give you an idea of how to get going with the software, without going into extensive details. Follow the links for further information; see towards the end for a bit on spectra.

Let's suppose that you are observing, or that you have come back with data from a run. Your data will consist of pairs of files of the form 'run012.xml' and 'run012.dat'. The XML file is an ASCII file which describes the format of the data in a fairly obvious html-ish way. The '.dat' file contains raw binary data plus times stored during a complete ULTRACAM run. These store all the data from, say, 45 minutes of 1 second exposures. These files can be large. Here is how you might go about dealing with this data.

1 Firing up the software First of all you need to to define the aliases for the ULTRACAM programs. This requires sourcing a script file called Ultracam that should be located in the top-level of the installed distribution. It is well worth defining an alias e.g. 'ultracam' to carry this out for you as in
alias ultracam 'source $TRM_SOFTWARE/bin/ultracam/Ultracam'
which you can stick in your '.cshrc' or equivalent. In addition, if you are accessing the data at the telescope, you need to start the ATC FileServer which dishes out the raw data to the pipeline across a network via http requests; this is independent of the pipeline software. If you have the raw data on a local disk, you will not need it.

2 What do my data look like? Typically you first want to plot the data. The program you want is rtplot which will allow you to plot data even as it accumulates into the .dat file. It accesses the large data files and plots images sequentially. It also contains facilities for setting up your windows, avoiding CCD defects and for fitting profiles to stars which is useful when focussing. Useful tip: a backslash is response to any input means assume default values for all following arguments, see the user input guide.

3 Setting up windows (observing only) rtplot has an option to load an ASCII file defining the windows which it plots over the data. As each new frame comes in it re-reads the file so you can adjust it by editing the file. The idea is to run in a heavily binned (4 by 4) fast acquisition mode and ask the TO to move the telescope and/or rotator until your field is where you want it. You must always save the window setup file to disk for it to be re-loaded by rtplot. Beware CCD defects! You need to look at all 3 CCDs to ensure that you are not affected by any. rtplot now includes the option of accessing windows contained by a Java-based program called Udriver which can act as a server.

4 Grabbing frames rtplot is very much a first look. To do more you may want to grab individual exposures and save them to the pipeline's native format, ".ucm" files. The program you want is grab. This spits out one file per exposure, which could amount to hundreds or thousands for a single run so very often you may want to ctrl-C it. For example, this is useful if you want to apply bias subtraction in rtplot. First grab a bias run, then combine the resulting files with combine (see also makebias). Incidentally, if you do ctrl-C during grab, you may want to delete the last file which may be corrupted (typically it is corrupted if grab did not report that it was written to disk and yet it exists). Once you have data in native form, a suite of standard programs can be run on them. Alternatively you may prefer to operate with a reduction code of your choice, in which case you can save to FITS files using grab2fits. This is certainly useful if you want access to good image display programs such as 'ds9' and 'gaia' because the pipeline does not attempt to replicate the features of such programs, and has just a couple of simple display programs plot and cplot.

5 Getting a light curve Next you may want to see a light curve of some sort. The program reduce is what you want. This can run off the large raw files, or sequences of files as produced by grab. I normally run off the raw files, but you will in any case need to grab some of the data initially in order to set up apertures with setaper. reduce is mainly controlled by an input ASCII file which you can (and should) edit with normal text editors. reduce saves its results to an ASCII file which is the key product of the software. Once you have set up the apertures and edited the control file, you can run reduce. You will be able to determine whether you want to plot images and/or the light curve at the time you invoke reduce.

6 Going further with your data The light-curve files are ASCII files with one line per CCD and a set of instructions at the top detailing their contents. Any standard package should be able to read these files and make suitable plots. However note that you may want to re-do all the reduction at the end of a run with the two-pass mode of reduce which has certain advantages. The ASCII files are somewhat unwieldy and if you are happy with the idea of reading in FITS rather than ASCII files, I suggest converting them to FITS using ulog2fits for which you must have Python along with the 3rd-party modules numpy and pyfits. 'fv' is a convenient routine to examine the FITS file output by this script.

7 Extra There are a fair number of helper scripts around including ones for debiassing, for making biases and many others.

Spectra

If you have spectra to reduce then there are some other routines that you may want to use.

1 Profiles With spectra one often wants to looks at cuts through the data. The commands to do this are collapse and lplot.

2 Defining apertures To define extraction and sky regions, use the command setreg.

3 Reducing The command for reducing spectra is sreduce. It runs off an ASCII file in the same way that reduce does. See sreduce.sre for an example.

4 Looking at the results The spectra are dumped to a molly file, so it is useful to have molly installed.

5 Measuring avalanche gain L3CCDs with low count levels have either 0 or 1 electron per pixel. The histogram of these gives a spike for all those with 0 electrons plus an exponential distribution for the single electron pixels. This can be fitted to give an estimate of the gain. I have made a crude script to do this called l3gain.

6 Spectrograph setup There are a few routines helpful in setting up spectrographs with Hartmann shutter exposures of arcs. These are collapse, lplot, ppos and possibly expand.

Tom Marsh, Warwick

1	Firing up the software	First of all you need to to define the aliases for the ULTRACAM programs. This requires sourcing a script file called Ultracam that should be located in the top-level of the installed distribution. It is well worth defining an alias e.g. 'ultracam' to carry this out for you as in alias ultracam 'source $TRM_SOFTWARE/bin/ultracam/Ultracam' which you can stick in your '.cshrc' or equivalent. In addition, if you are accessing the data at the telescope, you need to start the ATC FileServer which dishes out the raw data to the pipeline across a network via http requests; this is independent of the pipeline software. If you have the raw data on a local disk, you will not need it.
2	What do my data look like?	Typically you first want to plot the data. The program you want is rtplot which will allow you to plot data even as it accumulates into the .dat file. It accesses the large data files and plots images sequentially. It also contains facilities for setting up your windows, avoiding CCD defects and for fitting profiles to stars which is useful when focussing. Useful tip: a backslash is response to any input means assume default values for all following arguments, see the user input guide.
3	Setting up windows (observing only)	rtplot has an option to load an ASCII file defining the windows which it plots over the data. As each new frame comes in it re-reads the file so you can adjust it by editing the file. The idea is to run in a heavily binned (4 by 4) fast acquisition mode and ask the TO to move the telescope and/or rotator until your field is where you want it. You must always save the window setup file to disk for it to be re-loaded by rtplot. Beware CCD defects! You need to look at all 3 CCDs to ensure that you are not affected by any. rtplot now includes the option of accessing windows contained by a Java-based program called Udriver which can act as a server.
4	Grabbing frames	rtplot is very much a first look. To do more you may want to grab individual exposures and save them to the pipeline's native format, ".ucm" files. The program you want is grab. This spits out one file per exposure, which could amount to hundreds or thousands for a single run so very often you may want to ctrl-C it. For example, this is useful if you want to apply bias subtraction in rtplot. First grab a bias run, then combine the resulting files with combine (see also makebias). Incidentally, if you do ctrl-C during grab, you may want to delete the last file which may be corrupted (typically it is corrupted if grab did not report that it was written to disk and yet it exists). Once you have data in native form, a suite of standard programs can be run on them. Alternatively you may prefer to operate with a reduction code of your choice, in which case you can save to FITS files using grab2fits. This is certainly useful if you want access to good image display programs such as 'ds9' and 'gaia' because the pipeline does not attempt to replicate the features of such programs, and has just a couple of simple display programs plot and cplot.
5	Getting a light curve	Next you may want to see a light curve of some sort. The program reduce is what you want. This can run off the large raw files, or sequences of files as produced by grab. I normally run off the raw files, but you will in any case need to grab some of the data initially in order to set up apertures with setaper. reduce is mainly controlled by an input ASCII file which you can (and should) edit with normal text editors. reduce saves its results to an ASCII file which is the key product of the software. Once you have set up the apertures and edited the control file, you can run reduce. You will be able to determine whether you want to plot images and/or the light curve at the time you invoke reduce.
6	Going further with your data	The light-curve files are ASCII files with one line per CCD and a set of instructions at the top detailing their contents. Any standard package should be able to read these files and make suitable plots. However note that you may want to re-do all the reduction at the end of a run with the two-pass mode of reduce which has certain advantages. The ASCII files are somewhat unwieldy and if you are happy with the idea of reading in FITS rather than ASCII files, I suggest converting them to FITS using ulog2fits for which you must have Python along with the 3rd-party modules numpy and pyfits. 'fv' is a convenient routine to examine the FITS file output by this script.
7	Extra	There are a fair number of helper scripts around including ones for debiassing, for making biases and many others.

1	Profiles	With spectra one often wants to looks at cuts through the data. The commands to do this are collapse and lplot.
2	Defining apertures	To define extraction and sky regions, use the command setreg.
3	Reducing	The command for reducing spectra is sreduce. It runs off an ASCII file in the same way that reduce does. See sreduce.sre for an example.
4	Looking at the results	The spectra are dumped to a molly file, so it is useful to have molly installed.
5	Measuring avalanche gain	L3CCDs with low count levels have either 0 or 1 electron per pixel. The histogram of these gives a spike for all those with 0 electrons plus an exponential distribution for the single electron pixels. This can be fitted to give an estimate of the gain. I have made a crude script to do this called l3gain.
6	Spectrograph setup	There are a few routines helpful in setting up spectrographs with Hartmann shutter exposures of arcs. These are collapse, lplot, ppos and possibly expand.