.. Quick start guide to the API, started 24/10/2017 .. include:: ../globals.rst .. |CCD| replace:: :class:`hipercam.CCD` .. |MCCD| replace:: :class:`hipercam.MCCD` .. |Window| replace:: :class:`hipercam.Window` .. |Winhead| replace:: :class:`hipercam.Windhead` API quick start *************** Your best start for developing scripts which use the |hiper| API is probably to look at whichever one of the existing scripts seems closest to your task. Hunt out the location of :mod:`hipercam.scripts`. But to give a bit more background, here are some basic concepts to get you going. .. contents:: :local: |hiper| hcm files and |MCCD| objects ==================================== The pipeline stores individual multi-CCD exposures as |MCCD| objects in memory. These contain all the windows of all the CCDs, along with header information. e.g. for HiPERCAM itself, there will be 5 CCDs, each with 4 or more windows of data. They quickly allow access to the data. Thus given an |MCCD| object called 'mccd', the following lines return first a single |CCD| labelled '2', then a |Window| of that CCD labelled 'E1', then a 2D numpy array containing the data of that |Window|: .. code-block:: python ccd = mccd['2'] wind = ccd['E1'] data = wind.data At this point you have direct access to the data. e.g. The value of the pixel at (x,y) = (10,20) from the lower-left corner of the pixel can be printed and set equal to 150 with: .. code-block:: python print(data[20,10]) data[20,10] = 150. Note the C-style ordering, with Y coming first, and that the pixel indices are C-style 0-offset, i.e. (0,0) is the lower-left pixel itself; apologies to FORTRAN devotees. |MCCD| objects are derived from :class:`collections.OrderedDict` objects, and are keyed by string labels assigned to the CCDs they contain. In |hiper| data the CCDs are labelled '1', '2', '3', '4' and '5' for the ugriz arms. A statement such as ``ccd = mccd['2']`` returns CCD '2' as a |CCD| object which is also dictionary-like so that ``ccd['E1']`` returns window 'E1' as a |Window| object. It is worth perusing the methods of all of these classes. How does one get an |MCCD| in the first place? One way, probably the usual way, is to read it in from a file using the class method :func:`hipercam.MCCD.read` as follows: .. code-block:: python import hipercam as hcam mccd = hcam.MCCD.read('myfile.hcm') assuming that there is some pre-written file :file:`myfile.hcm`, as generated by e.g. |grab|. If we do something to 'mccd' and want to save it then: .. code-block:: python mccd.write('mynewfile.hcm') or: .. code-block:: python mccd.write('mynewfile.hcm',True) if we don't mind over-writing pre-existing files in the latter case, will do the job. "hcm" is the standard extension the pipeline uses when saving |MCCD| objects to disk, but these files are in fact FITS files and can be looked at with standard tools such as "fv". The extension highlights that they are a particular style of FITS files and keeps them separate from the raw data that comes with the extension '.fits' (see below). Here is a script to subtract two files and save the result: .. code-block:: python import hipercam as hcam # input the two frames mccd1 = hcam.MCCD.read('input1.hcm') mccd2 = hcam.MCCD.read('input2.hcm') # subtract in place mccd1 -= mccd2 # output the result mccd1.write('output.hcm') If the two |MCCD| objects are compatible, this will subtract them CCD by CCD, and window by window. Dealing with raw data files =========================== Sometimes you may prefer to deal directly with the raw data files. These have names like :file:`run0034.fits`. They are FITS files, but are not directly usable as the data is scrambled within a 3D FITS "cube" owing to the constraints of continually adding to FITS during data acquisition. To minimize risk to the data, the pipeline has no mechanism to write raw files, and does not use the extension '.fits' for any of the various files it produces. If you want to modify data, you need to write out lists of hcm files. Here is an example of subtracting the median values from all the windows of all CCDs of a run, writing out the results in a series of hcm files: .. code-block:: python :linenos: from hipercam.hcam import Rdata input_name = 'run0034' for n, mccd in enumerate(Rdata(input_name)): # subtract median from each window of each CCD for ccd in mccd.values(): for wind in ccd.values(): wind -= wind.median() output_name = '{:s}_{:03d}.hcm'.format(input_name,n+1) mccd.write(output_name) Line 1 imports the pipeline software. Line 5 creates a :class:`hipercam.hcam.Rdata` object which can deliver each frame one-by-one in the form of what is known as "iterator" in Python-speak. Any iterator can be called in a for-loop as in line 5. The successive frames are returned as |MCCD| objects labelled `mccd` in line 5. After the median is subtracted from each :They are written to files which will have names like :file:`run0034_001.hcm`, :file:`run0034_002.hcm`, etc. The code above loads frames from a raw data file assumed to be in the directory the script is run in. If instead you want to load from the server, then line 5 would be replaced by .. code-block:: python for n, mccd in enumerate(Rdata(input_name,True)): A similar construction serves for ULTRACAM files, with the difference that :class:`Rdata` is loaded from the :mod:`hipercam.ucam` sub-module: .. code-block:: python from hipercam.ucam import Rdata input_name = 'run0034' for mccd in Rdata(input_name): . . . The resource that produces the file can also be called as a "context manager" with the `with` statement .. code-block:: python import hipercam as hcam with hcam.HcamDiskSpool(input_name) as spool: for mccd in spool: # do something . . The advantage of this is that the input file will be closed as soon at the `with` is exited. :class:`HcamDiskSpool` is one of 5 similar iterable context managers to return data. Another useful one is :class:`HcamListSpool` which can be attached to a file list to return each file as an :class:`MCCD`. Others source data from servers for both HiPERCAM and ULTRA(CAM|SPEC). Several of the standard HiPERCAM scripts (rtplot, reduce, grab for example) use a special method :func:`data_source` to return any one of these 5 types. For instance these lines are extracted from 'rtplot': .. code-block:: python import hipercam as hcam . . . with hcam.data_source(source, resource, first) as spool: for mccd in spool: 'source' here takes values one of 5 values: 'hl' for a local HiPERCAM file, 'hs' for the HiPERCAM server, 'hf' for an hcm file list, 'ul' for a local ULTRA[CAM|SPEC] file, or 'us' for the ULTRA[CAM|SPEC] server. 'resource' is the run or in the case of a file list, the file name, while 'first' is the number of the first frame to return. This provides uniform access code to the different types of data. Command line inputs =================== The pipeline uses a style of command-line inputs derived from STARLINK routines that is distinct from the usual UNIX-style command-line options. See the :ref:`command-calling` section for details of this. This has great benefits in reducing the amount of typing. If you want to write scripts that follow the same convention then you should look at :mod:`hipercam.cline.Cline`. Here is an example taken from a stripped-down version of the |stats| script: .. code-block:: python :linenos: def stats(args=None): command, args = hcam.script_args(args) # get input section with Cline('HIPERCAM_ENV', '.hipercam', command, args) as cl: # register parameters cl.register('input', Cline.LOCAL, Cline.PROMPT) cl.register('format', Cline.LOCAL, Cline.HIDE) # get inputs frame = cl.get_value('input', 'frame to lists stats of', cline.Fname('hcam', hcam.HCAM)) mccd = hcam.MCCD.read(frame) cl.set_default('format','9.3f') form = cl.get_value('format', 'output format for numbers', '9.3f') A :class:`Cline` object 'cl' is called as a context manager (`with` statement again, line 6). The command name and any arguments are passed to it along with parameters to determine the location of any default file. Two parameter names are registered, namely 'input' and 'format' (lines 9 and 10). The second of these is not prompted for by default ('Cline.HIDE'). Following the 'register' lines, the inputs themselves are obtained (lines 13-14 and 18). Once the 'with' statement is exited, any values entered will be saved to disk in a directory whose location is specified by the first two arguments to 'Cline' and whose name is determined by the command name, e.g. 'stats.def'. In this way, previously entered values can be retrieved. In a script like |rtplot| these input sections rival the "doing stuff" part of the script for length. Your best place to start is very probably one of the existing scripts where you can see how to deal with input of existing files, versus output of new one, getting floating point, integer or string input and other details.