API quick start

Your best start for developing scripts which use the HiPERCAM API is probably to look at whichever one of the existing scripts seems closest to your task. Hunt out the location of hipercam.scripts. But to give a bit more background, here are some basic concepts to get you going.

HiPERCAM hcm files and hipercam.MCCD objects

The pipeline stores individual multi-CCD exposures as hipercam.MCCD objects in memory. These contain all the windows of all the CCDs, along with header information. e.g. for HiPERCAM itself, there will be 5 CCDs, each with 4 or more windows of data. They quickly allow access to the data. Thus given an hipercam.MCCD object called ‘mccd’, the following lines return first a single hipercam.CCD labelled ‘2’, then a hipercam.Window of that CCD labelled ‘E1’, then a 2D numpy array containing the data of that hipercam.Window:

ccd  = mccd['2']
wind = ccd['E1']
data = wind.data

At this point you have direct access to the data. e.g. The value of the pixel at (x,y) = (10,20) from the lower-left corner of the pixel can be printed and set equal to 150 with:

print(data[20,10])
data[20,10] = 150.

Note the C-style ordering, with Y coming first, and that the pixel indices are C-style 0-offset, i.e. (0,0) is the lower-left pixel itself; apologies to FORTRAN devotees. hipercam.MCCD objects are derived from collections.OrderedDict objects, and are keyed by string labels assigned to the CCDs they contain. In HiPERCAM data the CCDs are labelled ‘1’, ‘2’, ‘3’, ‘4’ and ‘5’ for the ugriz arms. A statement such as ccd = mccd['2'] returns CCD ‘2’ as a hipercam.CCD object which is also dictionary-like so that ccd['E1'] returns window ‘E1’ as a hipercam.Window object. It is worth perusing the methods of all of these classes.

How does one get an hipercam.MCCD in the first place? One way, probably the usual way, is to read it in from a file using the class method hipercam.MCCD.read() as follows:

import hipercam as hcam
mccd = hcam.MCCD.read('myfile.hcm')

assuming that there is some pre-written file myfile.hcm, as generated by e.g. grab. If we do something to ‘mccd’ and want to save it then:

mccd.write('mynewfile.hcm')

or:

mccd.write('mynewfile.hcm',True)

if we don’t mind over-writing pre-existing files in the latter case, will do the job. “hcm” is the standard extension the pipeline uses when saving hipercam.MCCD objects to disk, but these files are in fact FITS files and can be looked at with standard tools such as “fv”. The extension highlights that they are a particular style of FITS files and keeps them separate from the raw data that comes with the extension ‘.fits’ (see below). Here is a script to subtract two files and save the result:

import hipercam as hcam

# input the two frames
mccd1 = hcam.MCCD.read('input1.hcm')
mccd2 = hcam.MCCD.read('input2.hcm')

# subtract in place
mccd1 -= mccd2

# output the result
mccd1.write('output.hcm')

If the two hipercam.MCCD objects are compatible, this will subtract them CCD by CCD, and window by window.

Dealing with raw data files

Sometimes you may prefer to deal directly with the raw data files. These have names like run0034.fits. They are FITS files, but are not directly usable as the data is scrambled within a 3D FITS “cube” owing to the constraints of continually adding to FITS during data acquisition. To minimize risk to the data, the pipeline has no mechanism to write raw files, and does not use the extension ‘.fits’ for any of the various files it produces. If you want to modify data, you need to write out lists of hcm files.

Here is an example of subtracting the median values from all the windows of all CCDs of a run, writing out the results in a series of hcm files:

from hipercam.hcam import Rdata

input_name = 'run0034'

for n, mccd in enumerate(Rdata(input_name)):

    # subtract median from each window of each CCD
    for ccd in mccd.values():
        for wind in ccd.values():
            wind -= wind.median()

    output_name = '{:s}_{:03d}.hcm'.format(input_name,n+1)
    mccd.write(output_name)

Line 1 imports the pipeline software. Line 5 creates a hipercam.hcam.Rdata object which can deliver each frame one-by-one in the form of what is known as “iterator” in Python-speak. Any iterator can be called in a for-loop as in line 5. The successive frames are returned as hipercam.MCCD objects labelled mccd in line 5. After the median is subtracted from each :They are written to files which will have names like run0034_001.hcm, run0034_002.hcm, etc. The code above loads frames from a raw data file assumed to be in the directory the script is run in. If instead you want to load from the server, then line 5 would be replaced by

for n, mccd in enumerate(Rdata(input_name,True)):

A similar construction serves for ULTRACAM files, with the difference that Rdata is loaded from the hipercam.ucam sub-module:

from hipercam.ucam import Rdata
input_name = 'run0034'
for mccd in Rdata(input_name):
   .
   .
   .

The resource that produces the file can also be called as a “context manager” with the with statement

import hipercam as hcam

with hcam.HcamDiskSpool(input_name) as spool:

    for mccd in spool:
        # do something
        .
        .

The advantage of this is that the input file will be closed as soon at the with is exited. HcamDiskSpool is one of 5 similar iterable context managers to return data. Another useful one is HcamListSpool which can be attached to a file list to return each file as an MCCD. Others source data from servers for both HiPERCAM and ULTRA(CAM|SPEC). Several of the standard HiPERCAM scripts (rtplot, reduce, grab for example) use a special method data_source() to return any one of these 5 types. For instance these lines are extracted from ‘rtplot’:

import hipercam as hcam

.
.
.

with hcam.data_source(source, resource, first) as spool:

    for mccd in spool:

‘source’ here takes values one of 5 values: ‘hl’ for a local HiPERCAM file, ‘hs’ for the HiPERCAM server, ‘hf’ for an hcm file list, ‘ul’ for a local ULTRA[CAM|SPEC] file, or ‘us’ for the ULTRA[CAM|SPEC] server. ‘resource’ is the run or in the case of a file list, the file name, while ‘first’ is the number of the first frame to return. This provides uniform access code to the different types of data.

Command line inputs

The pipeline uses a style of command-line inputs derived from STARLINK routines that is distinct from the usual UNIX-style command-line options. See the Parameter specification section for details of this. This has great benefits in reducing the amount of typing. If you want to write scripts that follow the same convention then you should look at hipercam.cline.Cline. Here is an example taken from a stripped-down version of the stats script:

 def stats(args=None):

     command, args = hcam.script_args(args)

     # get input section
     with Cline('HIPERCAM_ENV', '.hipercam', command, args) as cl:

         # register parameters
         cl.register('input', Cline.LOCAL, Cline.PROMPT)
         cl.register('format', Cline.LOCAL, Cline.HIDE)

         # get inputs
         frame = cl.get_value('input', 'frame to lists stats of',
                              cline.Fname('hcam', hcam.HCAM))
         mccd = hcam.MCCD.read(frame)

         cl.set_default('format','9.3f')
         form = cl.get_value('format', 'output format for numbers', '9.3f')

A Cline object ‘cl’ is called as a context manager (with statement again, line 6). The command name and any arguments are passed to it along with parameters to determine the location of any default file. Two parameter names are registered, namely ‘input’ and ‘format’ (lines 9 and 10). The second of these is not prompted for by default (‘Cline.HIDE’). Following the ‘register’ lines, the inputs themselves are obtained (lines 13-14 and 18). Once the ‘with’ statement is exited, any values entered will be saved to disk in a directory whose location is specified by the first two arguments to ‘Cline’ and whose name is determined by the command name, e.g. ‘stats.def’. In this way, previously entered values can be retrieved. In a script like rtplot these input sections rival the “doing stuff” part of the script for length. Your best place to start is very probably one of the existing scripts where you can see how to deal with input of existing files, versus output of new one, getting floating point, integer or string input and other details.