At the telescope

Here it is assumed that you are observing with HiPERCAM, ULTRACAM or ULTRASPEC (hereafter ULTRA(CAM|SPEC) for short) for the first time and want to look at your data. It is written in tutorial style for someone starting afresh who just wants to get going, so it is as short as possible, with details listed elsewhere. The first thing to realise is that the pipeline is run entirely from command lines entered in a terminal, so first of all open a terminal on the “drpc” (data reduction PC).

Note

Many of the precise arguments & prompts you will see on these pages have changed with software updates, so don’t worry if you spot differences.

Plotting the data coming in

Assuming that you are observing and data are being acquired to the rack PC at the telescope, the first command to try is rtplot 1 or, preferably, nrtplot, a developmental upgrade to rtplot based on matploltib. Try typing in a terminal as follows:

rtplot

You should see something like 2

rtplot
run - run name [run0068]:

which is prompting you to supply a name for the run. If this is your first time, the suggested value ‘run0068’, which remains from the last time the command was run, is likely to be of little use 3 . So you need to type the name of your run, which might be ‘run0002’. If you don’t know it, either look at the observing GUI to see the run number, or ctrl-C out of rtplot (a safe option to quit any pipeline command) and type instead:

hls

for HiPERCAM, or:

uls

for ULTRA(CAM|SPEC) (which work with a different server). This returns a list of runs; you probably want the last one of the list, which will be the most recent. Note that all HiPERCAM runs appear as ‘run####’, while ULTRA(CAM|SPEC) have just three digits. Returning to rtplot we can enter more inputs:

rtplot
run - run name [run0068]: run0002
first - first frame to plot [100]: 11
ccd - CCD(s) to plot [0 for all] [3]: 2 3 4 5
nx - number of panels in X [2]:
bias - bias frame ['none' to ignore] [none]:
defect - defect file ['none' to ignore] [defect]:
setup - display current hdriver window settings [True]:
msub - subtract median from each window? [True]:
iset - set intensity a(utomatically), d(irectly) or with p(ercentiles)? [p]:
plo - lower intensity limit percentile [10.0]:
phi - upper intensity limit percentile [99.0]:
xlo - left-hand X value [1360.0]: min
xhi - right-hand X value [1400.0]: max
ylo - lower Y value [280.0]: min
yhi - upper Y value [310.0]: max

Each line consists of an input parameter name (e.g. bias, xlo), followed by a dash, a prompt, a default value in square brackets then a colon. In some lines the default has been accepted (e.g. plo) by hitting <enter>, whereas in others, something new has been entered, e.g. ccd where the default value ‘3’ has been replaced with ‘2 3 4 5’. (for ULTRACAM you would want some combination of ‘1’, ‘2’ and ‘3’; ULTRASPEC has a single CCD and you will not be prompted).

If we wanted to change one parameter, say first, the first frame to plot, we can type:

rtplot
run - run name [run0002]:
first - first frame to plot [11]: 12
ccd - CCD(s) to plot [0 for all] [3]: \

The backslash ‘\\’ means adopt the default values for all remaining parameters. An alternative with the same effect is:

rtplot first=12 \\

where the ‘\\\\’ (double backslash) is needed because the shell will strip one off. A simple rtplot \\ at the terminal will repeat the command with no changes of any parameters. The ability to specify parameters on the command line is particularly useful for the later ones e.g.:

rtplot xlo=100 xhi=1100 \\

will reduce the X-range without the need to wind through the initial parameters. The one other thing to be aware of are “hidden” parameters. For instance:

rtplot source=hs

ensures that rtplot tries to get the data from the HiPERCAM server on the rack (which must itself be running of course), while you would want to substitute ‘us’ when using the ULTRA(CAM|SPEC) server, or ‘ul’ when accessing ULTRA(CAM|SPEC) data locally. One very useful feature of rtplot is for quick examination of the FWHM of images through a parameter called profit that will only be prompted if you display just a single CCD. This is useful to see if your target has reasonable count levels, and as a quick check on focus, although focussing can only be done properly by looking at the FWHMs of all CCDs during reduction. The section on Parameter specification has many further details on the parameter input system and should be read the first time you use the software. From now it will be assumed that you know the parameter input system.

Note

rtplot is one of the most useful commands during observing runs. Three useful features are (1) the ability to fit the profiles of selected targets when displaying a single CCD; (2) the option to plot a file of CCD defects. Use this to avoid bad regions of the CCDs; (3) the setup option which interrogates the hdriver GUI for the current set of windows. This is useful for optimising the CCD setup on a target.

nrtplot adds the ability to fit the profiles of more than one CCD which it plots as a FWHM versus frame number. This is useful when observing for focussing a run. It also allows you to zoom and pan the plots which is very helpful for checking bad pixels.

Commands used in this section: rtplot, nrtplot, hls

Looking in more detail

To look in more detail at particular images, you can save individual exposures to disk using the command grab as e.g.:

grab
run - run name [run0002]:
ndigit - number of digits in frame identifier [3]:
first - first frame to grab [1]:
last - last frame to grab [2]: 5
bias - bias frame ['none' to ignore] [none]:
Written frame 1 to run0002_001.hcm
Written frame 2 to run0002_002.hcm
Written frame 3 to run0002_003.hcm
Written frame 4 to run0002_004.hcm
Written frame 5 to run0002_005.hcm

In this case the first 5 exposures have been saved to “hcm” files. These are in fact FITS files but the extension is to indicate that they have a format specific to HiPERCAM. For more on the format of these files consult HiPERCAM’s hcm file format.

You can then look at these with the HiPERCAM command hplot which, if you use the matplotlib option (device=/mpl), can allow you to zoom in on the same part of all CCDs simultaneously and also enables some simple statistics. These frames can also be displayed with other packages such as ds9 or gaia.

Commands mentioned in this section: grab, hplot.

Plotting defects during target acquisition

IT IS STRONGLY RECOMMENDED THAT YOU PLOT CCD DEFECTS DURING ACQUISITION!. We have created defect files for each camera for this purpose from flat-field and bias frames, but it’s never a bad idea to look for changes. Flat-field defects in particular can often be hard to see, during acquistion, especially if the sky is dark. Plotting them can help you avoid the worst defects which you might only otherwise discover during reduction, by which time it is too late. Check defects near your target and the comparisons in all CCDs. nrtplot, which allows zooming and panning (in contrast to rtplot) is particularly useful for such checking.

Getting reduction going

Typically one of the first things you want to do during a run is to start a reduction which allows you to monitor the conditions, above all the transparency and seeing (and therefore the telescope focus, although see the not concerning nrtplot above). For multi-CCD images it is important to measure the FWHM in all CCDs because if the cameras are not par-focal, you may have good images in one but not another, but even with just one CCD as with ULTRASPEC, one of the first things you should do is check the focus.

The HiPERCAM reduction is essentially a three-step process:

  1. Define target apertures with the setaper command.

  2. Generate a text file (“reduce file”) defining the reduction with the genred command.

  3. Run the reduce command to see lightcurves.

Setting the apertures

Beginning with setaper, one selects the targets to be extracted using a cursor over an image. For the image, you can simply extract a single frame from the run using grab, but it is better practice to average a few frames which is easily done with averun:

averun run0076 1 10 \\

which averages (actually take the median in this case) the first 10 frames, and stores the result in run0076.hcm. Next run setaper on this image. This should display the image and allow you to add apertures one by one. Standard HiPERCAM practice is to label the main target ‘1’, the main comparison star ‘2’, and then number them from then on in order of increasing faintness. See the docs on setaper for the many features of the program. The end result of setaper is an “aperture file” with extension .ape, which it is recommended you call by the same name as the run itself, and thus in this case you would end up with a file called run0076.ape. This is a JSON-format text file, and can be edited directly, although it is much safer to let setaper do the job for you.

Commands mentioned in this section: averun, grab, setaper.

Note

Many of the pipeline commands generate default names for the output using the input as the template. Thus the input file for averun above was run0076 (a HiPERCAM raw .fits files) and the output defaulted to run0076.hcm. The different extensions avoid clashes between different files. setaper similarly produces run0076.hcm. The different extensions avoid clashes between different files but keep them all connected by the root name, run0076 in this case.

Generating a reduce file

Once you have defined the apertures, you are ready to create a “reduce file”, which is a text file with a series of directives specifying how the reduction is to be carried out. The easiest way to do this is through the command genred which generates the file given the aperture file you just created. In this case the command would be:

genred run0076 run0076 "A test" none none none false

which generates a reduce file called run0076.red with no bias, flat or dark frame (‘none none none’) and using a magnitude scale (linear=’false’) at the end. It is likely that the resultant reduce file will not be right for you, and you will want to adjust it, but the aim is to provide a start to get you going quickly. The reduce file is just a text file and is easy to edit.

Note

genred has a huge number of hidden parameters. The very first time you run it for a given observing run, specify “prompt” on the command line so that these hidden parameters are prompted. Once you have done this once, they can usually be left the same value from one run to the next. This way you only need modify a few obvious parameters for subsequent invocations of genred.

Commands mentioned in this section: genred.

Reducing your data

You are now ready to run the reduce. This should generate a plot in which you can see light curves, the object X and Y position, the transmission and the FWHM of the images. You will almost certainly need to adjust the plots, especially the light curve plots, by adding offsets to the plot lines in the [light] section of the reduce file. Apart from this, the first thing to do is to look at the FWHM plots (at the bottom of the light curve plot panel) and see whether the telescope focus can be adjusted to improve them. It is also worth displaying images at least once (parameter implot) in order to see that things are doing what you expected, but, once you are going, you may want to switch off this option for speed.

If you have got this far, well done. You will see some steps, such as bias subtraction have been skipped. This was in the spirit of getting going fast. Look at Reduction for a longer and slower look at how to get more out of the reduction.

Commands mentioned in this section: reduce.

plog

Once you have a reduction log file, then the command plog can be useful to make simple plots of counts, count ratios, FWHMs etc, while allowing to zoomand pan in an interactive matplotlib plot. It can be run while reduce itself is running.

Footnotes

1

All HiPERCAM commands are implemented as entry points to functions, all of which are part of the hipercam.scripts module. This means that documentation on any HiPERCAM pipeline command can be looked up from a terminal with pydoc, e.g. for rtplot you would type pydoc hipercam.scripts.rtplot. To invoke the same command, simply type rtplot in a terminal.

2

This may fail if the data source is set incorrectly. If so, re-type as``rtplot prompt`` to see all the options, many of which are hidden by default. The one you might want to change is source which should be hs for data from the HiPERCAM server at the telescope, or hl for data from a disk file local to your computer (or us and ul for ULTRA(CAM|SPEC)).

3

All raw HiPERCAM file names take the form ‘run####.fits’ but the extension ‘.fits’ can be omitted. ULTRA(CAM|SPEC) files take the form ‘run###.dat’ / ‘run###.xml’. Note that HiPERCAM files have 4 digits compared to ULTRA(CAM|SPEC)’s 3.