HiPERCAM commands

This page documents the pipeline commands which are all part of hipercam.scripts. Help on any given command can be obtained at the terminal using e.g. pydoc hipercam.scripts or pydoc hipercam.scripts.rtplot. This is often the most useful way to get information quickly. If you are new to the pipeline, it would be worth reading the Parameter specification section on how to call the pipeline commands and specify parameters.

Main commands

This section lists the main HiPERCAM commands 1 ; see also the bottom of the page for all the documentation on the commands in one long list. Clicking on a command name in the table below will take you to the relevant section of this long list. The table also indicates contexts in which each command is particularly useful.

Command	Purpose	Observing	Reduction	Plots & analysis	Arithematic	Information
add	add two frames				Yes
averun	average a series of frames in a run	Yes	Yes
cadd	add a constant to a frame				Yes
cdiv	divide a frame by a constant				Yes
cmul	multiply a frame by a constant				Yes
combine	combine a list of frames	Yes	Yes		Yes
csub	subtract a constant from a frame				Yes
div	divide one frame by another				Yes
exploss	plot reduce log readout noise loss factor	Yes			Yes	Yes
fits2hcm	convert foreign FITs files to hcm format		Yes
flagcloud	flag cloudy and junk data in a reduce log		Yes	Yes
ftargets	automatically find targets using “sep”	Yes	Yes
genred	create a reduce file	Yes	Yes
grab	split frames out of a run	Yes	Yes
hfilter	filter a HiPERCAM image				Yes
hinfo	lists information on a HiPERCAM image					Yes
hist	plot a histogram of a frame			Yes		Yes
hlog2col	produce ASCII column data from a reduce log		Yes	Yes		Yes
hlog2fits	convert a reduction log file to FITS		Yes	Yes
hls	list the runs on the HiPERCAM server	Yes				Yes
hpackage	bundles up reduce data files	Yes	Yes			Yes
hplot	plot a frame	Yes	Yes	Yes
joinup	joins windows into a single image per CCD	Yes	Yes
ltimes	list times of a run					Yes
makebias	combine a run to make a bias frame	Yes	Yes
makedark	combine a run to make a dark frame		Yes
makeflat	combine a set of frames into a flat		Yes
makefringe	combine a set frames into a fringe map	Yes	Yes
makemovie	makes stills for movies from a run			Yes		Yes
mstats	list stats of multiple frames from a run					Yes
mul	multiply two frames				Yes
ncal	noise calibration	Yes	Yes	Yes		Yes
nrtplot	plot frames as they come in [matplotlib]	Yes	Yes	Yes
pbands	plot reduce log as a lightcurve per CCD	Yes		Yes
plog	flexible plot of a reduce log	Yes		Yes
redanal	analyse a reduction log file		Yes			Yes
reduce	carry out photometric reduction	Yes	Yes
rtplot	plot frames as they come in [pgplot]	Yes	Yes	Yes
rupdate	updates old reduce files	Yes	Yes
setaper	define the photometric apertures	Yes	Yes
setdefect	define a file of CCD defects	Yes
setfringe	define peak/trough pairs for fringe measurement	Yes	Yes
splice	splice two frames together				Yes
stats	report statistics of a frame					Yes
sub	subtract two frames				Yes
uls	list the runs on the ULTRACAM server	Yes

Parameter specification

The pipeline commands are distinct from standard unix commands in having a ‘memory’, which is implemented through storage of inputs in disk files for each command, and also in prompting you if you don’t specify a parameter on the command line. Parameters can be ‘global’ or ‘local’ depending upon whether they are reset across multiple commands or just the command of interest. The parameter memory, along with the use of backslashes ‘\\’ to accept default values can save a huge amount of typing making for efficient operation once you get up to speed.

Basic parameter input

The command rtplot has more parameters than most others and is a good one to start with (see also its replacement nrtplot). Suppose then that we have a raw HiPERCAM file, run0076.fits, that we want to plot. If we type ‘rtplot’ and follow the prompts, the first few lines might be:

rtplot
run - run name [run0064]: run0076
first - first frame to plot [10]: 1

This shows that the last time rtplot was invoked, it was used to look as run0064.fits starting with frame 10. Note that the extension ‘.fits’ is not needed: HiPERCAM makes significant use of extensions to differentiate between different forms of file, all of which could be associated with the same run and therefore start with the ‘run0064’ or whatever. Alternatively we could have typed:

rtplot run0076
first - first frame to plot [10]: 1

which would accomplish the same, setting the run name according to its position in the command-line arguments. For this to work, you need to be certain of the ordering. If you are not sure of the order, but you are sure that the parameter is called run, then:

rtplot run=run0076
first - first frame to plot [10]: 1

will do, and the first two parameters could be similarly specified:

rtplot first=1 run=run0076

Note that by naming the parameters, the order becomes immaterial. Now, assuming that the command was completed, if you run rtplot again you might get:

rtplot
run - run name [run0076]:
first - first frame to plot [1]:
ccd - CCD(s) to plot [0 for all] [3]: 3 4
nx - number of panels in X [2]:
bias - bias frame ['none' to ignore] [none]:
msub - subtract median from each window? [False]: True

You will see that the defaults for run and first have been updated by the previous invocations of the command. This the parameter “memory” referred to earlier. Now suppose that we wish to repeat the command without changing any parameter. Then a simple:

rtplot \\

will do. The double backslash ‘\\\\’ indicates “take the default value for all remaining parameters”. Two are needed because the shell will strip one of them. If you let at least one prompting line be output then a single backslash will accomplish the same thing:

rtplot
run - run name [run0076]: \

If you wish to change just one parameter, say msub, then much typing can be saved with:

rtplot msub=no \\

(NB: Boolean True/False parameters like msub can be reset False with either ‘no’, ‘false’ or even just ‘n’.) Combined with up and down arrow keys, and commands can be quickly repeated or modified.

Hidden parameters and special keywords

A look through the Command definitions will reveal that many commands have “hidden” parameters that are not usually prompted for. The idea behind this is to reduce the level of prompting, particularly for those parameters that hardly ever change. The values of all parameters can be revealed through the use of a special keyword list`:

rtplot list \\
rtplot
source = hl
device = 1/xs
width = 0.0
height = 0.0
run = run0076
first = 1
twait = 10.0
tmax = 20.0
ccd = 3 4
nx = 2
pause = 0.0
plotall = False
bias = none
msub = True
iset = p
plo = 10.0
phi = 99.7
xlo = 600.0
xhi = 950.0
ylo = 100.0
yhi = 400.0

This reveals parameters source, device, width and height that were never prompted for in the commands of the previous section. Their values can be changed by giving another special keyword prompt:

rtplot prompt source - data source [hs, hl, us, ul, hf] [hl]: device - plot device [1/xs]: width - plot width (inches) [0.0]: 10 height - plot height (inches) [0.0]: 8 run - run name [run0076]:

etc. Usually once changed in this way, the hidden parameters will keep the new values, although there are some which will return to a standard default on the basis that it is almost always what is wanted and it would be dangerous to default to a different value.

In addition to list and prompt, there is a third special keyword nodefs which means do not read or write any default parameter values from files. If nodefs is specified, then all parameters values need to be spelt out; it’s use is inside scripts in circumstances where multiple instances are being used, or when there is interactive work going on too, to avoid causing problems by over-writing the default files. Avoid using these keywords in any other context. e.g. you would be asking for trouble if you named your files list.hcm or prompt.hcm.

Strings with spaces

If you need to specify a string with a space in it, either let a command prompt you and type as normal, or quote it on the command line, e.g.:

rtplot run="silly file name" \\

Setting a string to be blank

If you want to set a string to a blank, then a carriage return won’t work when prompted for it since this will just keep whatever default is set. Instead type ‘’ (two single quotes) which will be interpreted as implying an empty string.

Parameter ranges

Many numerical parameters have mimimum and/or maximum values set. Typing ‘?’ at the prompt should tell you what the values are. Typing ‘min’ or ‘max’ will set the parameter to have the appropriate value.

Tab completion

File name input is helped by tab completion: start typing, hit <tab> and if the file exists, the name might be completed.

Global vs local parameters

All HiPERCAM pipeline parameters fall into one of two classes, either being ‘local’ to a command or ‘global’ to multiple commands. The run parameter of rtplot for instance also appears in grab and if you change it in rtplot, it will be changed in grab. This is very useful when running a series of commands on the same file as the commands almost ‘know’ what you want, saving much typing.

Trouble-shooting parameter input

It very rarely happens that the file I/O needed to read the default values can get confused. This can happen e.g. if the data type of a parameter has changed. If very odd things seems to happen when you try to start a command, then you might want to track down where the default files are located. Usually this will be in .hipercam in your home directory, unless the environment variable HIPERCAM_ENV has been defined to re-direct where the default files are stored. Once you have located them it is always safe to delete one or more or all of the default files (end with .def). The worst that happens is that the commands have lost the default values.

Errors reported by scripts

If you use the pipeline for any significant time, you will at some point get a rather forbidding-looking error traceback such as:

reduce run=run013 \\
Traceback (most recent call last):
  File "/storage/astro1/phsaap/software/python/bin/reduce", line 11, in <module>
    load_entry_point('hipercam==0.19.9.dev22+g758df9d.d20200303', 'console_scripts', 'reduce')()
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 405, in reduce
    with spooler.data_source(source, resource, first, full=False) as spool:
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/spooler.py", line 323, in data_source
    return UcamDiskSpool(resource, first)
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/spooler.py", line 108, in __init__
    self._iter = ucam.Rdata(run, first, False)
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/ucam.py", line 637, in __init__
    Rhead.__init__(self, run, server)
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/ucam.py", line 152, in __init__
    udom = xml.dom.minidom.parse(run + '.xml')
  File "/warwick/desktop/2018/software/MPI/GCC/7.3.0-2.30/OpenMPI/3.1.1/Python/3.6.6/lib/python3.6/xml/dom/minidom.py", line 1958, in parse
    return expatbuilder.parse(file)
  File "/warwick/desktop/2018/software/MPI/GCC/7.3.0-2.30/OpenMPI/3.1.1/Python/3.6.6/lib/python3.6/xml/dom/expatbuilder.py", line 910, in parse
    with open(file, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'run013.xml'

Yikes! Although it looks awful, it simply reflects the chain of function calls that led to the problem, an extremely useful diagnostic of problems in Python code. Such tracebacks look a bit ugly, but almost certainly, in most cases, including this one, they are caused by incorrect parameter inputs. The one to look at is probably the last line or two, which reveals in this case that an expected file ‘run013.xml’ was not found; a directory listing would confirm this.

Such errors follow from a standard Python approach of not trying to add endless checking code, but to let the code tell you what happened when errors are encountered. This has the merit of being very informative (and de-clutters code), but it can make it hard to distinguish between an essentially trivial issue, such as a missing file, and a genuine problem with the code. If however you find you can’t get round a problem and the error reported does not look innocent, then it might be a time to report the problem. We will do our best to provide fast solutions to critical issues.

Command definitions

This section contains documentation auto-generated from the code. This is the same as is returned from clicking command names in the lists at the top of this page or from using pydoc in a terminal (e.g. try pydoc hipercam.scripts.reduce). Each command appears as a function (an implementation detail), followed by a highlighted line showing the parameters one can use on the command-line. Inputs in square brackets such as [source] are hidden by default; those in round brackets e.g. (plot) may or may not be prompted depending upon earlier inputs. It is always safest when first running a command simply to type its name and hit enter and let the command itself prompt you for input. Many commands have hidden parameters that can only be revealed by typing e.g. rtplot prompt. These are usually parameters that rarely need changing, but you are sure sometimes to need to alter them. See the Parameter specification section for details on how to specify command parameters.

In the one-line descriptions below, run refers to a complete run, containing multiple images, stored in a .fits file. frame refers to a single image from a run as might be extracted using grab. These have file extension ‘.hcm’ to distinguish them, although they are also FITS-format files.

hipercam.scripts.add(args=None)

add input1 input2 [ccd win] output

Adds two hcm frames and outputs the result. Can be applied only to particular CCDs and windows if wanted.

Parameters:

input1string

first input hcm file

input2string

second input hcm file to add to the first.

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as either input1 or input2 in which case the input file will be over-written.

hipercam.scripts.averun(args=None)

averun [source] (run first last twait tmax | flist) trim ([ncol nrow]) bias dark flat fmap (fpair [nhalf rmin rmax]) [method sigma adjust clobber] output

Averages images from a run using median combination, skipping the junk frames that result from NSKIP / NBLUE options in HiPERCAM and ULTRACAM data. averun is meant to be a simple tool to create median frames suitable prior to aperture selection with setaper. See combine if you want more fine-grained control over frame averaging. (averun uses a combination of grab [if needed] and combine).

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststr [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame (‘0’ is not supported).

lastint [if source ends ‘s’ or ‘l’]

last exposure number must be >= first, or 0 for the lot

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool

True to trim columns and/or rows off the edges of windows nearest the readout. Useful for ULTRACAM particularly.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame to subtract, ‘none’ to ignore.

flatstr

Name of flat field to divide by, ‘none’ to ignore.

fmapstr

Name of fringe map (see e.g. makefringe), ‘none’ to ignore.

fpairstr [if fmap is not ‘none’]

Name of fringe pair file (see e.g. setfringe). Required if a fringe map has been specified.

nhalfint [if fmap is not ‘none’, hidden]

When calculating the differences for fringe measurement, a region extending +/-nhalf binned pixels will be used when measuring the amplitudes. Basically helps the stats.

rminfloat [if fmap is not ‘none’, hidden]

Minimum individual ratio to accept prior to calculating the overall median in order to reduce the effect of outliers. Although all ratios should be positive, you might want to set this a little below zero to allow for some statistical fluctuation.

rmaxfloat [if fmap is not ‘none’, hidden]

Maximum individual ratio to accept prior to calculating the overall median in order to reduce the effect of outliers. Probably typically < 1 if fringe map was created from longer exposure data.

methodstr [hidden, defaults to ‘m’]

‘m’ for median, ‘c’ for clipped mean. See below for pros and cons.

sigmafloat [hidden; if method == ‘c’]

With clipped mean combination, pixels that deviate by more than sigma RMS from the mean are kicked out. This is carried out in an iterative manner. sigma <= 0 implies no rejection, just a straight average. sigma=3 is typical.

adjuststr [hidden; defaults to ‘i’]

adjustments to make: ‘i’ = ignore; ‘n’ = normalise the mean of all frames to match the first; ‘b’ = add offsets so that the mean of all frames is the same as the first. Option ‘n’ is useful for twilight flats; ‘b’ for combining biases.

clobberbool [hidden]

clobber any pre-existing output files

outputstr

output file. Set by default to match the last part of “run” (but it will have a different extension so they won’t clash)

hipercam.scripts.cadd(args=None)

cadd input const [ccd win] output

Adds a constant to a HiPERCAM hcm frame. Can be applied only to particular CCDs and windows if wanted.

Parameters:

inputstring

input hcm file name

constfloat

constant to add

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as input in which case the file will be over-written.

hipercam.scripts.cdiv(args=None)

cdiv input const [ccd win] output

Divides a HiPERCAM hcm frame by a constant. Can be applied only to particular CCDs and windows if wanted.

Parameters:

inputstring

input hcm file name

constfloat

constant to divide by

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as input in which case the file will be over-written.

hipercam.scripts.cmul(args=None)

cmul input const [ccd win] output

Multiplies a HiPERCAM hcm frame by a constant. Can be applied only to particular CCDs and windows if wanted.

Parameters:

inputstring

input hcm file name

constfloat

constant to multiply by

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as input in which case the file will be over-written.

hipercam.scripts.combine(args=None)

combine list bias dark flat method (sigma) adjust (usemean) [plot clobber] output

Combines a series of images defined by a list using median or clipped mean combination. Only combines those CCDs for which is_data() is true (i.e. it skips blank frames caused by NSKIP / NBLUE options)

Parameters:

liststring

list of hcm files with images to combine. The formats of the images should all match

biasstring

Name of bias frame to subtract, ‘none’ to ignore.

darkstring

Name of dark frame to subtract, ‘none’ to ignore.

flatstring

Name of flat field frame to subtract, ‘none’ to ignore.

methodstring

‘m’ for median, ‘c’ for clipped mean. See below for pros and cons.

sigmafloat [if method == ‘c’]

With clipped mean combination, pixels that deviate by more than sigma RMS from the mean are kicked out. This is carried out in an iterative manner. sigma <= 0 implies no rejection, just a straight average. sigma=3 is typical.

adjuststring

adjustments to make: ‘i’ = ignore; ‘n’ = normalise the mean or median of all frames to match the first; ‘b’ = add offsets so that the mean or median of all frames is the same as the first. Option ‘n’ is useful for twilight flats and fringe frames; ‘b’ is good for combining biases.

usemeanbool [if adjust == ‘n’ or ‘b’]

True to use the mean rather than the median for normalisation or biass offsetting.

plotbool [hidden, if adjust == ‘n’ or ‘b’; defaults to False]

make a plot of the mean versus frame number. This can provide a quick check that the frames are not too different.

clobberbool [hidden]

clobber any pre-existing output files

outputstring

output file

Clipped mean can work well for large numbers of frames but gets worse for small numbers as the RMS can be heavily influenced by a single bad value. The median can be better in such cases, but has the downside of digitisation noise. For instance, the average of 100 bias frames could have a noise level significantly below 1 count, depending upon the readout noise, and the +/- 0.5 count uncertainty of median combination may be worse than this.

Note

This routine reads all inputs into memory, so can be a bit of a hog. However, it does so one CCD at a time to alleviate this. It will fail if it cannot find a valid frame for any CCD.

hipercam.scripts.csub(args=None)

csub input const [ccd win] output

Subtracts a constant from a HiPERCAM hcm frame. Can be applied only to particular CCDs and windows if wanted.

Parameters:

inputstring

input hcm file name

constfloat

constant to subtract

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as input in which case the file will be over-written.

hipercam.scripts.div(args=None)

div input1 input2 [ccd win] output

Divides two hcm frames and outputs the result. Can be applied only to particular CCDs and windows if wanted.

Parameters:

input1string

first input hcm file

input2string

second input hcm file to divide into the first.

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as either input1 or input2 in which case the input file will be over-written.

hipercam.scripts.exploss(args=None)

exploss log [device] aper readout gain

Computes the equivalent exposure time needed to match the signal-to-noise ratio of the current input log file as a fraction of the actual exposure time, assuming one had an identical detector except it had zero readout noise. The purpose of the routine is to help in judging how readout noise limited a given run is. The result is a loss factor between 0 and 1. If close to 1, readout noise is not significant. A value of 0.5 means that, in the absence of readout noise, you could have achieved the same signal-to-noise ratio in half the time you actually used. The point is one can tilt towards effectively zero readnoise case by exposing longer / reading out less often.

As well as the loss factor for the supplied run (plotted in blue), the case for double the exposure time is plotted (in red). If this is significantly larger than the blue values, it may be advisable to increase the exposure time or add to NBLUE / NSKIP. Basically you want all curves to be close to the upper level of 1, or you are effectively throwing away time by reading out too often. Of course you may have little choice, and so it is important to consider what exposure time you really need.

The key determinant as to whether readout noise matters is how the quantity n*gain*readout**2, where n = number of pixels in target aperture, compares with obj+n*sky, where obj is the total number of object counts in the aperture and sky in the sky background per pixel (both ADU). Thus readout noise tends to be of less significance during bright time, and seeing is important, particularly during dark time, since that is what determines “n”. Always remember, in very bad seeing, binning might be an option.

Arguments:

log : string
    name of |reduce| ASCII log file (text file with loads of columns)

device : string [hidden, defaults to 'term']
   'term' for interactive plot, file name such as 'plot.pdf'
   for a hardcopy.

aper : str
   aperture to consider

readout : float
   readout noise, RMS ADU.

gain : float
   Gain,  electrons/ADU.

Note

The results from this script are meaningless if the data were not bias-subtracted during reduce! You have been warned.

hipercam.scripts.fits2hcm(args=None)

fits2hcm flist origin

This routine converts “foreign” data into a format suitable for the pipeline. HiPERCAM’s hcm files are in fact FITS-format so this is mostly a case of re-organising the files.

Parameters:

fliststr

name of list of input FITS-format files. ‘flist’ should end ‘.lis’. The output file names will have the same rootname as the input files (but without any leading directories) but end in ‘.hcm’

originstr

origin of the data. Currently recognised:

HICKS :

University of Sheffield 16” Hicks Telescope SBIG ST10-XME CCD

INTWFC :

Wide field Camera on the INT. Just operates on a single CCD’s-worth of data.

LCOGT :

Las Cumbres Observatory, based on some FTN data taken in 2019.

LTIO :

Liverpool Telescope IO camera.

LTRISE :

Liverpool Telescope RISE camera. I don’t know how general my code is in this case. I assume XBIN=YBIN=2.

PT5M :

University of Sheffield/Durham pt5m telescope on La Palma QSI 532 CCD

ROSA :

University of Sheffield 10” Hicks Telescope ATIK ONE 6.0 CCD

WHTCAM :

Camera used by Richard Ashley in Feb 2021.

overwritebool

overwrite files on output

hipercam.scripts.flagcloud(args=None)

flagcloud hlog aper1 aper2 ccd delta output

Interactive flagging of cloud-affected or otherwise bad points in a HiPERCAM log file. You either mark a range of times as cloudy, or individual points as junk. If you mark a range, then all apertures of all CCDs will be flagged with the bitmask value CLOUDS. Individual points will be flagged as JUNK. Note that nothing is done to the data apart from changing the bitmask flags, so it is then up to you to test for these later on. This also means the flagging operation is easily reversible by clearing the CLOUD and JUNK flags, e.g. by AND-ing the flags with ~JUNK and ~CLOUDS. It is also possible to flag individual points as CLOUDS but these will not propagate across CCDs so it is probably not advisable to do this. Similarly range can be flagged as JUNK although it is unusual for all apertures of all CCDs to be junk, although bad seeing / awful cloud can cause this.

Junk points are marked red, cloudy points orange. OK aperture 1 points are plotted green, aperture 2 blue, and their ratio black. What is meant by ‘junk’ as opposed to ‘cloud’ is really down to the user. I tend to reserve junk for one-off points affected by bad cosmic rays and satellites, but in general it is probably sensible to think of “junk” as points you never want to see again versus “clouds” meaning data that you might want either to mask or use down the line, but would would like some way to know that it was affected by clouds. Some genuine “cloudy” data will be so bad that it will be better flagged as junk however. You can flag the same point as both “cloud” and “junk”, but “junk” is the stronger condition.

Bitmasks propagate when data are combined so a point flagged junk in aperture 2 but not aperture 1 will be flagged junk in the ratio of 1 divided by 2. You can also recover points in this routine; doing so will clear both their junk and/or cloud status. At the moment this is only possible on a point-by-point basis.

Interaction is via the cursor and hitting specific keys. Common options are lower case; less common ones upper case. Hitting the X on the plot will abort without saving the results. ‘q’ to quit saves the results. ‘h’ gives further help on the options.

Parameters:

hlogstr

ASCII log file, as produced by reduce.

aper1str

the name of first aperture to look at

aper2str

the name of second aperture to look at. The ratio aper1 / aper2 will be plotted along with the two separately, scaled by their maximum, all in the same panel.

ccdstr

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’, or even ‘1 2 3’ are possible inputs (without the quotes). Note the space separation when multiple CCDs are specified. If you want to plot more than one CCD, then you will get multiple panels in the Y direction, but their X-axes are kept in lock step when panning or zooming.

deltafloat

separation to use to space the plots in a given panel, each of which is normalised to 1. A value of 1 is recommended because then the second aperture should end with a typical level of 0, and any dips below this show the extent of the cloud. e.g. -0.9 would suggest an approximate 90% loss of flux due to cloud, ignoring extinction.

outputstr

name of modified version of the Hlog for output. Can overwrite the original if you dare.

hipercam.scripts.ftargets(args=None)

ftargets [source device width height] (run first [twait tmax] | flist) trim ([ncol nrow]) (ccd (nx)) [pause] thresh fwhm minpix output bias flat msub iset (ilo ihi | plo phi) xlo xhi ylo yhi

This script carries out the following steps for each of a series of images:

1. detects the sources,

2. identifies isolated targets suited to profile fits,

3. fits 2D Moffat profiles to these,

4. Saves results to disk.

The profile fits are carried out because sep does not return anything that can be used reliably for a FWHM.

Several parameters depends on the object detection threshold retuned by the source detection. This is referred to as threshold. The source detection is carried out using sep which runs according to the usual source extractor algorithm of Bertin.

The script plots the frames, with ellipses at 3*a,3*b indicated in red, green boxes indicating the range of pixels identified by sep, and blue boxes marking the targets selected for FWHM fitting (the boxes indicate the fit region).

Parameters:

sourcestring [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

devicestring [hidden]

Plot device. PGPLOT is used so this should be a PGPLOT-style name, e.g. ‘/xs’, ‘1/xs’ etc. At the moment only ones ending /xs are supported.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

runstring [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststring [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed). For data from the HiPERCAM server, a negative number tries to get a frame not quite at the end. i.e. -10 will try to get 10 from the last frame. This is mainly to sidestep a difficult bug with the acquisition system.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstring

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

nxint [if more than 1 CCD]

number of panels across to display.

pausefloat [hidden]

seconds to pause between frames (defaults to 0)

threshfloat

threshold (mutiple of RMS) to use for object detection. Typical values 2.5 to 4. The higher it is, the fewer objects will be located, but the fewer false detections will be made.

fwhmfloat

FWHM to use for smoothing during object detection. Should be comparable to the seeing.

minpixint

Minimum number of pixels above threshold before convolution to count as a detection. Useful in getting rid of cosmics and high dark count pixels.

rminfloat

Closest distance of any other detected object for an attempt to be made to fit the FWHM of an object [unbinned pixels].

pminfloat

Minimum peak height for an attempt to be made to fit the FWHM of an object. This should be a multiple of the object detection threshold (returned by sep for each object).

pmaxfloat

Maximum peak height for an attempt to be made to fit the FWHM of an object. Use to exclude saturated targets [counts]

emaxfloat

Maximum elongation (major/minor axis ratio = a/b), > 1. Use to reduce very non-stellar profiles.

nmaxint

Maximum number of FWHMs to measure. Will take the brightest first, judging by the flux.

biasstring

Name of bias frame to subtract, ‘none’ to ignore.

flatstring

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

output: string

Name of file for storage of results. Will be a fits file, with results saved to the HDU 1 as a table.

isetstring [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilofloat [if iset=’d’]

lower intensity level

ihifloat [if iset=’d’]

upper intensity level

plofloat [if iset=’p’]

lower percentile level

phifloat [if iset=’p’]

upper percentile level

xlofloat

left-hand X-limit for plot

xhifloat

right-hand X-limit for plot (can actually be < xlo)

ylofloat

lower Y-limit for plot

yhifloat

upper Y-limit for plot (can be < ylo)

hipercam.scripts.genred(args=None)

genred apfile rfile bias dark flat fmap fpair seeing (sworst binfac) template (inst (nccd (ccd) nonlin sat scale readout gain))

Generates a reduce file as needed by reduce or psf_reduce. You give it the name of an aperture file, calibration frames and a few other parameters and it will write out a reduce file which you can then refine by hand. The aim is to try to get a reduce file that is self-consistent and works to get observing started as soon as possible. The parameters are not necessarily the best.

It is assumed that the target is called ‘1’, the main comparison ‘2’.

To avoid endless prompts, genred does not try to prompt for all parameters even via hidden ones (a change from its previous behaviour). However, if you find yourself always making the samed edits to the file produced by genred, you may start wanting a way to alter some of the otherwise fixed parameters. You can do so by supplying a template file, which can just be from a previous run of genred where such parameters can be altered by editing with a standard text editor before running genred.

genred recognises some standard instrument telescope combinations which it uses to set parameters such as the pixel scale and readout noise, but if you choose ‘other’, you will be prompted for these details. The main parameters it does prompt for are calibration files, and it allows you to define a “seeing” which then controls the setting of various profile fit parameters. This can be ignored by setting = 0, in which case any template parameters will be passed unchanged.

Parameters:

apfilestr

the input aperture file created using setaper (default extension .ape). This will be read for the targets. The main target will be assumed to have been called ‘1’, the main comparison ‘2’. If there is a ‘3’ it will be plotted relative to ‘2’; all others will be ignored for plotting purposes. Target ‘2’ will be used to define the position and transmission plots for one CCD only [user definable]. Target ‘1’ will be used for the seeing plot unless it is linked when target ‘2’ will be used instead.

rfilestr

the output reduce file created by this script. The main target will be assumed to have been called ‘1’, the main comparison ‘2’. If there is a ‘3’ it will be plotted relative to ‘2’; all others will be ignored for plotting purposes.

biasstr

Name of bias frame; ‘none’ to ignore.

darkstr

Name of dark frame; ‘none’ to ignore.

flatstr

Name of flat field frame; ‘none’ to ignore.

fmapstr

Name of fringe frame; ‘none’ to ignore.

fpairstr [if fmap not ‘none’]

File defining pairs to measure fringe amplitudes.

seeingfloat

estimate of seeing which will be used to define several of the profile fitting parameters. Enter 0 to ignore and use defaults from the template (or genred if no template) instead.

sworstfloat [if seeing > 0]

worst seeing expected during run. Expands the search and fitting boxes which can cause difficulties if they are too small. It also sets the maximum target aperture radius which will set to ~ 1.8*sworst, converted to pixels, and the sky annulus radii. It has to be at least 2*seeing. Always remember that you can override the automated settings using a pre-edited template and seeing = 0.

binfacint [if seeing > 0]

binning factor. e.g. 4 if using 4x4. Needed to optimise some profile parameters.

templatestr

Reduce file to use as a template for any parameters not set by genred. This allows one to tweak settings by running genred without a template, then modifying the resultant file and using it as a template. ‘none’ to ignore, and then these values will be set by genred by default. Some will be modified anyway (e.g. the calibration file names), and if seeing > 0, then several of the profile fitting paramaters will be adapted to the value given along with the instrument parameters. genred will try to keep as many of the ‘extraction’, ‘position’, ‘transmission’, ‘light’ etc lines associated with particular CCDs in this case without modification, and will pass readout, saturation levels etc unchanged from the template.

inststr [if template == ‘none’]

the instrument. ‘hipercam-gtc’, ‘ultracam-ntt’, ‘ultraspec-tnt’, ‘ultracam-wht’, ‘ultracam-vlt’, or ‘other’. Sets pixel scale and standard readnoise params.

nccdint [if inst == ‘other’]

number of CCDs.

ccdint [if nccd > 1]

the CCD to use for position plots

nonlinlist(float) [if inst == ‘other’]

values for warning of non linearity

satlist(float) [if inst == ‘other’]

values for warning of saturation

scalefloat [if inst == ‘other’]

image scale in arcsec/pixel

readoutfloat | string [if inst == ‘other’]

readout noise, RMS ADU. Can either be a single value or an hcm file.

gainfloat | string [if inst == ‘other’]

gain, electrons per ADU. Can either be a single value or an hcm file.

hipercam.scripts.grab(args=None)

grab [source temp] (run first last (ndigit) [twait tmax] | flist (prefix)) trim ([ncol nrow]) bias dark flat fmap (fpair [nhalf rmin rmax verbose]) [dtype]

This downloads a sequence of images from a raw data file and writes them out to a series CCD / MCCD files.

Parameters:

sourcestr [hidden]

Data source, four options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’. ‘hf’ is useful if you want to apply pipeline calibrations (bias, flat, etc) to files imported from a ‘foreign’ format. In this case the output files will have the same name as the inputs but with a prefix added.

tempbool [hidden, defaults to False]

True to indicate that the frames should be written to temporary files with automatically-generated names in the expectation of deleting them later. This also writes out a file listing the names. The aim of this is to allow grab to be used as a feeder for other routines in larger scripts. If temp == True, grab will return with the name of the list of hcm files. Look at ‘makebias’ for an example that uses this feature.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

firstint [if source ends ‘s’ or ‘l’]

First frame to access

lastint [if source ends ‘s’ or ‘l’]

Last frame to access, 0 for the lot

ndigitint [if source ends ‘s’ or ‘l’ and not temp]

Files created will be written as ‘run005_0013.fits’ etc. ndigit is the number of digits used for the frame number (4 in this case). Any pre-existing files of the same name will be over-written.

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds.

fliststr [if source ends ‘f’]

name of file list. Output files will have the same names as the input files with a prefix added

prefixstr [if source ends ‘f’ and not temp]

prefix to add to create output file names

trimbool

True to trim columns and/or rows off the edges of windows nearest the readout. Particularly useful for ULTRACAM windowed data where the first few rows and columns can contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame, ‘none’ to ignore.

flatstr

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

fmapstr

Name of fringe map (see e.g. makefringe), ‘none’ to ignore.

fpairstr [if fmap is not ‘none’]

Name of fringe pair file (see e.g. setfringe). Required if a fringe map has been specified.

nhalfint [if fmap is not ‘none’, hidden]

When calculating the differences for fringe measurement, a region extending +/-nhalf binned pixels will be used when measuring the amplitudes. Basically helps the stats.

rminfloat [if fmap is not ‘none’, hidden]

Minimum individual ratio to accept prior to calculating the overall median in order to reduce the effect of outliers. Although all ratios should be positive, you might want to set this a little below zero to allow for some statistical fluctuation.

rmaxfloat [if fmap is not ‘none’, hidden]

Maximum individual ratio to accept prior to calculating the overall median in order to reduce the effect of outliers. Probably typically < 1 if fringe map was created from longer exposure data.

verbosebool

True to print lots of details of fringe ratios

dtypestring [hidden, defaults to ‘f32’]

Data type on output. Options:

‘f32’ : output as 32-bit floats (default)

‘f64’ : output as 64-bit floats.

‘u16’ : output as 16-bit unsigned integers. A warning will be issued if loss of precision occurs; an exception will be raised if the data are outside the range 0 to 65535.

Note

grab is used by several other scripts such as averun so take great care when changing anything to do with its input parameters.

If you use the “temp” option to write to temporary files, then those files will be deleted if you interrup with CTRL-C. This is to prevent the accumulation of such frames.

hipercam.scripts.hfilter(args=None)

hfilter input ccd nx filter [fwhm] output

Filters a multi-CCD image. e.g. smooths it. Can be useful in some analysis steps, e.g. for picking out defects, division by a smoother version of an image can be useful.

Parameters:

inputstring

name of MCCD input file

ccdstring

CCD(s) to filter, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes).

filterstring [single character]

type of filter. ‘g’ = gaussian. Uses an FFT-based approach which regards the boundaries as periodic, so you will get significant edge-effects if the values on opposite edges of a window are significantly different from each other.

clobberbool [hidden]

clobber any pre-existing output files

outputstring

name of MCCD output file

hipercam.scripts.hinfo(args=None)

hinfo input

Prints out an hcm file along with its name and the number of CCDs. You will get information CCD-by-CCD first followed by the top level header. You will also get at least partial printouts of the data arrays. If you want still more detail, I recommend the FITS viewer tool ‘fv’, or ‘ds9’ for examining images (also :hplot:).

Parameters:

inputstring

name of the MCCD file

hipercam.scripts.hist(args=None)

hist ccd window x1 x2 nbins gplot (nx) msub (nint) [device width height]

Plots histograms of MCCD objects. Histograms can be powerful diagnostics of CCD problems.

Parameters:

inputstr

name of the MCCD file

ccdstr

CCD or CCDs to plot in histogram form, one CCD per panel. Can be ‘0’ for all of them or a specific set ‘2 3 5’

windowstr

the window label to show. ‘0’ will merge all windows on each CCD.

x1float

lower end of histogram range

x2float

upper end of histoigram range. Set == x1 to get automatic min to max.

nbinsint

number of bins to use for the histogram

ymaxfloat

maximum y value for plot; 0 for automatic

gplotbool

plot a gaussian of the same mean and RMS as the data. Note this is not a fit.

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

msubbool

True/False to subtract median from each window before scaling

nintbool [if msub]

True/False to take the nearest integer of the median or not. Can help with histograms of digitised data.

devicestr [hidden]

Plot device name. Either ‘term’ for an interactive plot or a name like ‘plot.pdf’ for a hard copy.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

hipercam.scripts.hlog2col(args=None)

hlog2col log ap1 ap2

Converts a HiPERCAM ASCII log into one or more column format files of one aperture versus another for each CCD. This tacks on the date of the start of the night in order to distinguish run log names of the the same number.

Parameters:

logstr

name of the log file (should end .log). The output FITS file will have the same root name but end .fits. The routine will abort if there is a pre-existing file of the same name.

originstr

‘h’ or ‘u’ depending upon whether the log file was created with the hipercam or old ultracam pipeline.

offset :float [hidden]

offset in days to subtract from first time to get date at start of the night. 0.7 should mostly work. offset is subtracted and the MJD truncated to integer

ap1str

name of first aperture

ap2str

name of second aperture

typestr

output type ‘l’ for linear; ‘m’ for magnitudes. Linear has the advantage of handling negative values.

The output files will get names like 2021-01-12_run014_2_3_4.dat meaning CCD 2, aperture 3 divided by 4 of run014 from the night starting 2021-01-12.

hipercam.scripts.hlog2fits(args=None)

hlog2fits log [origin dir]

Converts a HiPERCAM ASCII log into a FITS file. As well as a modest reduction in file size (~40%, the ASCII logs are written relatively efficiently), the resulting file is faster to read than the ASCII log so this may be useful for very large log files [test of 78,000 frame file: 12.9 seconds to read the ASCII file, 1.9 to read the FITS version]. The FITS log is also much easier to understand than the ASCII files, but they don’t have all the header information, so are not a replacement. At the moment no significant header information is transferred beyond the CCD names. Each CCD appears as a single binary table, starting at the second HDU (or HDU 1 if you number them 0,1,2 ..). This can be read using hipercam.hlog.Hlog.from_fits().

Parameters:

logstr

name of the log file (should end .log). The output FITS file will have the same root name but end .fits. The routine will abort if there is a pre-existing file of the same name.

originstr [hidden]

‘h’ or ‘u’ depending upon whether the log file was created with the hipercam or old ultracam pipeline. Defaults to ‘h’.

dirstr [hidden]

directory for the output; defaults to the present working directory

NB Because of the danger of over-writing raw data (also ends .fits), this routine will not over-write pre-existing files. You should delete clashing files if you really want to proceed.

hipercam.scripts.hls(args=None)

Gives an ‘ls’-like listing of the runs available on the HiPERCAM file server. Just invoke as ‘hls’ with no arguments. This should work correctly whether it in fact its the HiPERCAM or ULTRACAM file server that is running.

hipercam.scripts.hplot(args=None)

hplot input [device] ccd nx msub ([cmap]) hsbox iset (ilo ihi | plo phi) xlo xhi ylo yhi [width height]

Plots a multi-CCD image. Can use PGPLOT or matplotlib. The matplotlib version is slightly clunky in its choice of the viewing area but has some features that could be useful, in particular, the interactive plot (device=’/mpl’) allows one to pan and zoom and to compare the same part of multiple CCDs easily.

Parameters:

inputstring

name of MCCD file

devicestring [hidden]

Plot device name. Uses characters after a final trailing ‘/’ to identify the type in PGPLOT style. Thus:

/xs : PGPLOT xserver interactive plot

1/xs : PGPLOT xserver interactive plot called ‘1’

plot.ps/cps : PGPLOT colour postscript called ‘plot.ps’

plot.ps/vps : PGPLOT B&W portrait oriented plot

/mpl : matplotlib interactive plot

plot.pdf/mpl : matplotlib PDF plot

ccdstring

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes). ‘3 4’ will plot CCD ‘3’ and CCD ‘4’. If you want to plot more than one CCD, then you will be prompted for the number of panels in the X direction. This parameter will not be prompted if there is only one CCD in the file.

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

msubbool

True/False to subtract median from each window before scaling

cmapstr [if matplotlib; hidden]

The colour map to use. “Greys” is the usual, but there are many others. Typing an incorrect one will give a list. “none” for matplotlib default.

hsboxint [if device = ‘/mpl’; hidden]

half-width in binned pixels of stats box as offset from central pixel hsbox = 1 gives a 3x3 box; hsbox = 2 gives 5x5 etc.

isetstring [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD bais.

ilofloat [if iset==’d’]

lower intensity level

ihifloat [if iset==’d’]

upper intensity level

plofloat [if iset==’p’]

lower percentile level

phifloat [if iset==’p’]

upper percentile level

xlofloat

left X-limit, for PGPLOT plots. Applies to matplotlib plots to restrict region used to compute percentile limits. This is useful in case where bias strips otherwise distort the plot limits (e.g. ultraspec full frame images)

xhifloat

right X-limit. See comments for xlo as well.

ylofloat

bottom Y-limit. See comments for xlo as well.

yhifloat

top Y-limit. See comments for xlo as well.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

hipercam.scripts.hpackage(args=None)

hpackage runs dname tar

hpackage looks for standard reduce data products and bundles them up into a single directory and optionally creates a tar file. The idea is to copy all the files needed to be able to re-run the reduction with the pipeline, while also adding a few helpful extras where possible. This is to make export of a reduction relatively painless.

Given ‘run123’ for instance, it looks for:

run123.hcm – typically the result from a run of averun. This must

exist for each run included.

run123.ape – file of photometric apertures. Also must exist.

run123*.red – reduce file as made by genred (* == any string)

run123*.log – result from reduce (* == any string)

Note the ‘*’ means that multiple .red / .log pairs are possible to allow for multiple alternative reductions. But they must start ‘run123’ and end ‘.red’ and ‘.log’ to be included. There has to be at least one such pair. It also looks for calibration files inside the reduce file and copies them. It requires them to be within the same directory and will fail if they are not.

It produces several extra files which are:

run123.fits – FITS version of the log file

run123_ccd1.fits – joined-up ds9-able version of run123.hcm

(and ccd2 etc) [but only if the windows are in sync.

run123_ccd1.reg – ds9-region file representing the apertures

from run123.ape [see above re synced windows]

README – a file of explanation.

The files are all copied to a temporary directory.

Arguments:

runstr

Series of run names of the ones to copy, separated by spaces.

dnamestr

Name for the directory to store all files forming the root of any output tar file created from it.

tarbool

Make a tar.gz file of the directory at the end; the directory and the files in it will be deleted. Otherwise, no tar file is made and the directory is left untouched. The directory will however be deleted if the program is aborted early.

hipercam.scripts.joinup(args=None)

joinup [source] (run first [twait tmax] | flist) trim ([ncol nrow]) (ccd) (aper) bias dark flat fmap (fpair nhalf rmin rmax) msub (ndigit) dtype dmax nmax [overwrite] compress odir

Converts a run or a list of hcm images into as near as possible “standard” FITS files with one image in the primary HDU per file, representing a single CCD with all windows merged. The aim above all is to have a file that plays nicely with ‘ds9’. A file such as ‘run0002.fits’ (HiPERCAM) will end up producing files with names like run0002_0001_ccd1.fits, run0002_001_ccd2.fits, etc and the same for any of the other CCDs for frame 1, then run0002_0002_ccd1.fits. ‘_ccd1’ etc will be tacked onto the names froma list of files. The generated files will be written to the present working directory. If the windows have gaps, then they will be filled with zeroes.

When binning has been used it is possible for sub-windows to be out of sync with each other so they cannot be simply added into to a single image. In this case the routine will instead try to place them within an unbinned image with each binned pixel repeated within this image xbin by ybin times. This and the zero filling can mean that the resulting images are much larger than the originals. There are safety parameters to guard against disaster in such cases.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstring [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame. For data from the HiPERCAM server, a negative number tries to get a frame not quite at the end. i.e. -10 will try to get 10 from the last frame. This is mainly to sidestep a difficult bug with the acquisition system.

lastint [if source ends ‘s’ or ‘l’]

Last frame to access, 0 for the lot

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

fliststr [if source ends ‘f’]

name of file list or just a single hcm file

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstring

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

aperstr [if source ends ‘s’ or ‘l’ or just a single file]

A photometric aperture file associated with the run or images. This will be translated into a file of “regions” (extension .reg) for each CCD they are defined for. The file can be loaded into ds9 showing the apertures over the image.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame to correct for dark counts. ‘none’ to ignore.

flatstr

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

fmapstr

Fringe map to remove fringes, ‘none’ to ignore.

fpairstr [if fmap is not == ‘none’]

File of peak/trough pairs for fringe amplitude measurement.

nhalfint [if fmap is not == ‘none’, hidden]

Half-size of regions around each point for measuring intensity differences from peak/trough pairs.

rminfloat [if fmap is not == ‘none’, hidden]

Minimum individual ratio for pruning peak/tough ratios prior to taking their median.

rminfloat [if fmap is not == ‘none’, hidden]

Maximum individual ratio for pruning peak/trough ratios prior to taking their median.

msubbool

subtract the median from each window. If set this happens after any bias subtraction etc.

ndigitint [if source ends ‘s’ or ‘l’]

number of digits to be used in the frame counter attached to the output file names. These are zero-padded so that the frames order alphabetically. Thus ‘run0002_ccd1_0001.fits’, ‘run0002_ccd1_0002.fits’, ‘run0002_ccd1_0003.fits’ … for instance.

dtypestr

output data type. ‘unit16’, ‘float32’, ‘float64’. The first of these (2-byte unsigned) is only probably a good idea if no bias, flat field or median subtraction has been applied because it involves rounding and it will fail if any data are out of the range 0 to 65535. 32-bit (4 byte) floats should be OK for most purposes, but require twice the space of uint16.

dmaxfloat

Maximum amount of data in GB to write out. A safety device to avoid disaster in case this script was applied to highly windowed data where you can end up expanding the total amount of data by a large factor.

nmaxint

Maximum number of frames. A similar safety device to dmax to avoid inadvertent application of this script to a million+ frame run. File systems tend not to behave well with vast numbers of files.

overwritebool [hidden]

overwrite any pre-existing files. Always defaults to ‘False’ for safety.

compressstr

allows data to be compressed with FITS’s internal lossless compression mechanisms. The file will still end as “.fits” but has a different internal format; ‘ds9’ copes seamlessly with all of them. The options are: ‘none’, ‘rice’, ‘gzip1’, ‘gzip2’. ‘rice’ gave about a factor of 2 compression in a short test I ran, and was as fast as gzip2, but it may depend upon the nature of the data. ‘none’ is fastest. See astropy.io.fits for further documentation.

odirstr

directory for output files. This routine can produce many files so it often makes sense to direct them to a specific directory. ‘.’ for the PWD. [Must exist prior to running the script.]

Note

Be careful of running this on highly windowed data since it could end up expanding the total amount of “data” hugely. It’s really aimed at full frame runs above all. The “dmax” and “nmax” parameters are aimed at heading off disaster.

The routine only creates a window big enough to contain all the windows. Thus it might end up representing a sub-array of the CCD as opposed to all of it. The location can be determined from the ‘LLX’ and ‘LLY’ parameters that are written to the header. These represent the location of the lowest and leftmost unbinned pixel that is contained within the data array. The bottom-left pixel of the CCD is considered to be (1,1), so full frame images have LLX=LLY=1.

A HipercamError will be raised if an attempt is made to write out data outside the range 0 to 65535 into unit16 format and nothing will be written

hipercam.scripts.ltimes(args=None)

ltimes [source] run first last [twait tmax tdigit edigit]

Lists timing information from a run in machine-readable space-separated column style. Integer status flags are used, 1=OK, 0=not OK.

For HiPERCAM, the following items are listed: (1) the frame number, (2) the raw GPS timestamp as an MJD, (3) the raw GPS timestamp as a UTC string, (4) a status flag, 0 or 1, then for each CCD: (i) the CCD number [1-5], (ii) the mid exposure time as an MJD, (iii) the mid-exposure time as a UTC string, (iv) the exposure length (seconds), (v) a status flag (0 or 1), determined by the NSKIP parameters for each CCD.

For ULTRACAM, the following items are listed: (1) the frame number, (2) the raw GPS timestamp as an MJD, (3) the raw GPS timestamp as a UTC string, (4) the red & green mid exposure time as an MJD, (5) the red & green mid-exposure time as a UTC string, (6) a status flag to indicate whether the mid-exposure time is considered to be good, (7) the exposure time in seconds, (8) the blue mid-exposure time as an MJD, (9) the blue mid-exposure time as a UTC string, (10) status flag of the blue data, which reflects the “nblue” option where only some of the blue frames are valid data.

For ULTRASPEC, the following items are listed: (1) the frame number, (2) the raw GPS timestamp as an MJD, (3) the raw GPS timestamp as a UTC string, (4) the mid-exposure time as an MJD, (5) the mid-exposure time as a UTC string, (6) a status flag to indicate whether the mid-exposure time is considered to be good, (7) the exposure time in seconds.

Parameters:

sourcestring [hidden]

Data source, two options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : local ULTRA(CAM|SPEC) file

‘ul’ : ULTRA(CAM|SPEC) server

runstring

run number to access, e.g. ‘run0034’

firstint

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it changes)

lastint

last exposure number, 0 for all.

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds. Set to 0 to avoid waiting and extra output messages.

tdigitint [hidden]

number of digits of precision for the seconds after the decimal point when reporting the times.

edigitint [hidden]

number of digits of precision after the decimal point when reporting the exposure times

Warning

This routine does not yet work with ULTRACAM data.

hipercam.scripts.ltrans(args=None)

ltrans posns

Reads a file of target positions dumped by ftargets and uses it to derive the transformations needed to co-align images.

Parameters:

posnsstring

file of positions generated by ftargets

cmaxfloat

maximum number of counts at peak allowed for a target. Targets with greater than this will be ignored when computing the transform.

emaxfloat

maximum ratio a/b, major/minor axis elongation. Targets more elongated than this will be ignored.

hipercam.scripts.makebias(args=None)

makebias [source] run first last twait tmax sigma plot output

Combines the frames of a single run (or list) into a single frame using clipped-mean averaging appropriate for biases. This uses grab to get the frames and combine to combine them. If you already have the frames separately, then just use combine directly.

Parameters:

sourcestr [hidden]

Data source, four options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr

run name to access

firstint

First frame to access

lastint

Last frame to access, 0 for the lot

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds.

sigmafloat

The value of ‘sigma’ to pass to the clipped mean combination in ‘combine’

plotbool

make a plot of the mean level versus frame number for each CCD. This can provide a quick check that the frames are not too different. You will need explicitly to close the plot generated at the end of the script

outputstr

name of final combined file. Set by default to match the last part of “run” (but it will have a different extension so they won’t clash)

Note

This routine writes the files returned by ‘grab’ to automatically generated files, typically in .hipercam/tmp, to avoid polluting the working directory. These are removed at the end, but may not be if you ctrl-C. You should check .hipercam/tmp for redundant files every so often

hipercam.scripts.makedark(args=None)

makedark [source] (run first last [twait tmax] | flist) bias sigma output

Combines the frames of a single run into a single frame using clipped-mean averaging. This uses grab to get the frames and combine to combine them. It subtracts a bias and corrects the exposure time by the exposure time of the bias.

Parameters:

sourcestring [hidden]

Data source, four options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run name to access

firstint [if source ends ‘s’ or ‘l’]

First frame to access

lastint [if source ends ‘s’ or ‘l’]

Last frame to access, 0 for the lot

twaitfloat [[if source ends ‘s’ or ‘l’, hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [[if source ends ‘s’ or ‘l’, hidden]

maximum time to wait between attempts to find a new exposure, seconds.

fliststr [if source ends ‘f’]

name of file list. Assumed that these are dias and dark corrected.

biasstr

Names of bias frame (made e.g. with makebias). This is so that the counts left in the dark frame are genuine dark counts which can be scaled by the ratio of exposure lengths during dark subtraction.

sigmafloat

The value of ‘sigma’ to pass to the clipped mean combination in ‘combine’

outputstr

name of final combined file

Note

This routine writes the files returned by ‘grab’ to automatically generated files, typically in .hipercam/tmp, to avoid polluting the working directory. These are removed at the end, but may not be if you ctrl-C. You should check .hipercam/tmp for redundant files every so often

hipercam.scripts.makeflat(args=None)

makeflat [source] (run first last [twait tmax] | flist) bias dark ngroup ccd lower upper [clobber] output

Averages a set of images to make a flat field.

Typically flat-fields for HiPERCAM and ULTRA(CAM|SPEC) are taken with a strongly time-variable twilight sky as the Sun sets or rises. A typical flat field run may start out bright, or even saturated, but by the end be only a few thousand counts above bias. Moreover, there are very often stars visible in the images, so we usually take them while offsetting the telescope in a spiral pattern. The challenge is to combine these images while rejecting the stars and saturated frames and giving due weight to the better exposed images. This moreover has to be done for each CCD which vary significantly in sensitivity.

‘makeflat’ does this as follows: given an input list of files (or optionally a single run), it reads them all in, debiases them (optionally), and calculates the mean count level in each CCD, normalises by the mean and writes out the results to temporary files. For each CCD it then sorts the files by their (original) mean level, and for those that lie between defined limits it takes the median of the mean-mormalised frames in groups of defined size. Thus, say one had 75 OK images, then these would be divided into 10 groups, the first 9 having 7 frames, the last having 16. The median average of each of these would be taken. In each case the mean levels would be adjusted to be the same before taking the average to overcome the problem of taking a median of a time-variable sky. The assumption is that while the level may vary, the pattern of the image does not. It is up to the user to check that this is correct. Each of the medians is adjusted to have a mean equal to the sum of the means of the input frames. Finally the normal average of all of these median frames is taken and the mean level of the final output normalised to 1. The first step, taking the median in groups is designed to remove the stars assuming that the telescope was spiralled. The size of the groups (‘ngroup’ below is a crucial parameter in whether this works). A good strategy is to run makeflat for a succession of ever larger ‘ngroup’ and then to divide the results into each other to see if stars are visible.

The final step, the average of the medians with adjusted mean levels, is to ensure that the flats are combined in a way that reflects the level of signal that they have, i.e. to avoid giving equal weights to the median of a series of flats with 20,000 counts per pixel and another series with 1,000 counts per pixel. This somewhat complex procedure is implemented through a series of temporary files which are written and read as the script runs, but deleted at its end. This allows very large numbers to be combined as long as there is enough memory to load ‘ngroup’ CCDs simultaneously, which should usually be fine.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame (‘0’ is not supported).

lastint [if source ends ‘s’ or ‘l’]

last exposure number must be >= first or 0 for the whole lot.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

fliststr [if source ends ‘f’]

name of file list. Assumed that these are dias and dark corrected.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame to subtract, ‘none’ to ignore. Note that it is assumed all CCDs have the same exposure time when making a dark correction.

ngroupint

the number of frames. Probably should be at least 5, preferably more. Experiment to see its effect.

ccdstr

CCD(s) to process, ‘0’ for all, ‘1 3’ for ‘1’ and ‘3’ only, etc. Would almost always expect this to be set = ‘0’.

lowerlist of floats

Lower limits to the mean count level for a flat to be included. The count level is determined after bias subtraction. Should be the same number as the selected CCDs, and will be assumed to be in the same order. Use this to elminate frames that are of so low a level that the accuracy of the bias subtraction could be a worry. Suggested hipercam values: 3000 for each CCD. Enter values separated by spaces.

upperlist of floats

Upper limits to the mean count level for a flat to be included. The count level is determined after bias subtraction. Should be the same number as the selected CCDs, and will be assumed to be in the same order. Use this to eliminate saturated, peppered or non-linear frames. Suggested hipercam values: 58000, 58000, 58000, 40000 and 40000 for CCDs 1, 2, 3, 4 and 5. Enter values separated by spaces. ULTRACAM values 49000, 29000, 27000 for CCDs 1, 2 and 3.

clobberbool [hidden]

clobber any pre-existing output files

outputstr

output file. will be set by default to match the input name.

Note

This routine writes the files returned by ‘grab’ to automatically generated files, typically in .hipercam/tmp, to avoid polluting the working directory. These are removed at the end, but may not be if you ctrl-C. You should check .hipercam/tmp for redundant files every so often

hipercam.scripts.makefringe(args=None)

makefringe [source] (run first last [twait tmax] | flist) bias dark flat fpair ([nhalf]) ccd fwhm [clobber] output

Averages a set of images to make a frame for defringing (referred to elsewhere as a “fringe map”).

At long wavelengths, CCDs suffer what is known as “fringing”, which in terms of structure looks something like the coloured patterns you see when there is a thin layer of oil on water. Both patterns are caused by interference. In the case of CCDs this is caused by sky emission lines and is variable in strength, and additive in nature, but seems mostly fairly fixed in position.

De-fringing in HiPERCAM is implemented according to the method suggested by Snodgrass & Carry (2013Msngr.152…14S). The idea is to create a set of pairs of points marking the peak and troughs of fringes. The differences in intensity between these point pairs in the data to be corrected is compared with the differences for the same pairs in a reference “fringe map” by taking their ratios. Many pairs are used so that the median can be taken to eliminate ratios affected by celestial targets or cosmic rays. The median ratio is used to scale the reference data and thereby subtract the fringes. makefringe is used to make the reference fringe map. It works as follows: given a single run or a list of files, it reads them all in, optionally debiases, dark-subtracts and flat-fields them, calculates a median count level of each one which is subtracted from each CCD individually. The pixel-by-pixel median of all frames is then calculated. It is also possible (and a good idea) to apply the fringe pair ratio measurements to scale each frame before combining them, which allows frames of similar pattern but varying amplitude to be combined. Finally, the output can be smoothed because fringes are usually a medium- to large-scale pattern.

Parameters:

sourcestring [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats.

runstring [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame (‘0’ is not supported).

lastint [if source ends ‘s’ or ‘l’]

last exposure number must be >= first or 0 for the whole lot.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

fliststring [if source ends ‘f’]

name of file list. Assumed that bias, flat-fielding etc have been applied to these already.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame to subtract, ‘none’ to ignore.

flatstr

Name of flat field, ‘none’ to ignore.

fpairstr

FringePair file (see setfringe), or ‘none’ to ignore. If specified if will be used to scale the frames prior to combining.

nhalfint [if fpair is not ‘none’]

When calculating the differences for fringe measurement, a region extending +/-nhalf binned pixels will be used when measuring the amplitudes. Basically helps the stats.

ccdstr

CCD(s) to process, ‘0’ for all, ‘1 3’ for ‘1’ and ‘3’ only, etc.

fwhmfloat

FWHM (unbinned pixels) of gaussian smoothing to apply to the output. 0 to ignore. Should be << smallest scale between fringes. Probably no more than 4.

clobberbool [hidden]

clobber any pre-existing output files

outputstr

output file. Set by default to match the last part of “run” (but it will have a different extension so they won’t clash). The default is

Note

This routine writes the files returned by ‘grab’ to automatically generated files, typically in .hipercam/tmp, to avoid polluting the working directory. These are removed at the end, but may not be if you ctrl-C. You should check .hipercam/tmp for redundant files every so often

hipercam.scripts.makemovie(args=None)

makemovie [source] (run first last | flist) trim ([ncol nrow]) (ccd (nx)) bias flat defect log (targ comp ymin ymax yscales yoffset location fraction lpad) cmap width height dstore ndigit fext msub iset (ilo ihi | plo phi) xlo xhi ylo yhi [dpi (style (ms lw))]

makemovie is for generating stills to combine into a movie for presentations. It can optionally also read a log file from the run to display an evolving light curve. There are lots of fiddly parameters mostly to do with the plot positioning, so try it out on a small number of frames first before going mad.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststr [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame.

lastint [if source ends ‘s’ or ‘l’]

last exposure

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstr

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc. If you plot more than one, then a legend is added to any light curve panel to distinguish the light curves.

nxint [if more than 1 CCD]

number of panels across to display for the image display.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame to subtract, ‘none’ to ignore.

flatstr

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

defectstr

Name of defect file, ‘none’ to ignore.

fmapstr

Name of fringe map (see e.g. makefringe), ‘none’ to ignore.

fpairstr [if fringe is not ‘none’]

Name of fringe pair file (see e.g. setfringe). Required if a fringe map has been specified.

nhalfint [if fringe is not ‘none’, hidden]

When calculating the differences for fringe measurement, a region extending +/-nhalf binned pixels will be used when measuring the amplitudes. Basically helps the stats.

rminfloat [if fringe is not ‘none’, hidden]

Minimum individual ratio to accept prior to calculating the overall median in order to reduce the effect of outliers. Although all ratios should be positive, you might want to set this a little below zero to allow for some statistical fluctuation.

rmaxfloat [if fringe is not ‘none’, hidden]

Maximum individual ratio to accept prior to calculating the overall median in order to reduce the effect of outliers. Probably typically < 1 if fringe map was created from longer exposure data.

logstr

Name of reduce log file for light curve plot, ‘none’ to ignore

targstr [if log defined]

Target aperture

compstr [if log defined]

Comparison aperture

yminfloat [if log defined]

Minimum Y value for light curve plot

ymaxfloat [if log defined]

Maximum Y value for light curve plot

ynormlist(float) [if log defined]

Normalisation factors, one per CCD for light curve plot

yoffsetlist(float) [if log defined]

Offsets, one per CCD for light curve plot

locationstr [if log defined]

Offsets, one per CCD for light curve plot

fractionfloat [if log defined]

Fraction of figure to occupy, by height if location is South, by width if it is East

lpadtuple(float) [if log defined]

padding on left, bottom, right and top of light curve plot as fraction of allocated width and height

cmapstr

The matplotlib colour map to use. “Greys” gives the usual greyscale. “none” will give whatever the current default is. Many other choices: “viridis”, “jet”, “hot”, “Oranges”, etc. Enter an invalid one and the program will fail but return a huge list of possibles in the process.

widthfloat

plot width in inches.

heightfloat

plot height in inches.

dstorestr

root directory for plot files. Will get names like dstore/run003_001.png.

ndigitint

number of digits in frame counter, i.e. the ‘001’ of the previous section.

fextstr

file extension ‘png’, ‘jpeg’ for images generated

msubbool

subtract the median from each window before scaling for the image display or not. This happens after any bias subtraction.

isetstr [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilolist(float) [if iset=’d’]

lower intensity level, one per image

ihilist(float) [if iset=’d’]

upper intensity level, one per image

plofloat [if iset=’p’]

lower percentile level

phifloat [if iset=’p’]

upper percentile level

xlofloat

left-hand X-limit for plot, initially at least since it is possible to re-size. For iset=’p’ these limits also set the region of the frame over which the percentil will be calculated. You will usually want yhi-ylo ~ xhi-xlo in magnitude because the aspect ratio is preserved.

xhifloat

right-hand X-limit for plot (can be < xlo to invert the display)

ylofloat

lower Y-limit for plot

yhifloat

upper Y-limit for plot (can be < ylo)

dpiint [hidden]

dots per inch of output. Default 72. Allows control over font size versus image size, in combination with width and height.

stylestr [hidden, if log defined]

style for light curves ‘dots’, ‘line’, ‘both’. The line will be grey for the ‘both’ option.

msfloat [hidden, if log defined and style==dots or both]

markersize. Controls dot size which is useful when fiddling with dpi

lwfloat [hidden, if log defined and style==line or both]

line width

hipercam.scripts.mstats(args=None)

mstats [source] run [temp] (ndigit) first last [twait tmax] bias [dtype]

This downloads a sequence of images from a raw data file and writes out stats (min, max, mean, median, rms) for each window to a file

Parameters:

sourcestring [hidden]

Data source, four options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

runstring

run name to access

firstint

First frame to access

lastint

Last frame to access, 0 for the lot

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds.

biasstring

Name of bias frame to subtract, ‘none’ to ignore.

formatstring

output format for numbers. e.g. the default ‘9.3f’ might give 12345.678 (9 characters, 3 digits after d.p.)

outfilestring

file for output (extension .stats)

hipercam.scripts.mul(args=None)

mul input1 input2 [ccd win] output

Multiplies two hcm frames and outputs the result. Can be applied only to particular CCDs and windows if wanted.

Parameters:

input1string

first input hcm file

input2string

second input hcm file to multiply into the first.

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as either input1 or input2 in which case the input file will be over-written.

hipercam.scripts.ncal(args=None)

ncal [source] (run first last [twait tmax] | flist) trim ([ncol nrow]) (ccd) bias dark flat xybox read gain grain

Calibrates noise characteristics of CCDs by plotting estimator of RMS vs signal level from a series of frames. The estimate is the mean of the absolute difference between each pixel and the mean of its 8 near-neighbours. This is very local and fairly robust. Assuming gaussian noise, the RMS is sqrt(4*Pi/9) times this value, and this is what is plotted as the RMS by this routine. ncal is best to applied to a series of frames with a large dynamic range, ideally starting from bias-like frames to well exposed sky flats. A long flat-field run going to low levels, or a run into twilight at the end of the night could be good places to start.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststr [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed). For data from the HiPERCAM server, a negative number tries to get a frame not quite at the end. i.e. -10 will try to get 10 from the last frame. This is mainly to sidestep a difficult bug with the acquisition system.

lastint [if source ends ‘s’ or ‘l’]

Last frame to access, 0 for the lot

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstr

The CCD to analyse.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame to subtract, ‘none’ to ignore

flatstr

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

xyboxint

the stats will be taken over boxes of xybox-squared pixels to keep the number of points and scatter under control.

readfloat

readout noise, RMS ADU, for overplotting a model

gainfloat

gain, e-/count, for overploting a model

grainfloat

fractional RMS variation due to flat-field variations, if you didn’t include a flat field.

hipercam.scripts.nrtplot(args=None)

nrtplot [source] (run first [twait tmax] | flist) trim ([ncol nrow]) (ccd (nx)) [imwidth pause tupdate plotall] bias [lowlevel highlevel] dark flat fmap (fpair [nhalf rmin rmax]) defect setup [drurl cmap imwidth imheight memory] msub iset (ilo ihi | plo phi) xlo xhi ylo yhi profit [method beta fwhm fwhm_min shbox smooth fhbox hmin read gain thresh (fwnmax fwymax fwwidth fwheight)]

This is ‘nrtplot’ “new” rtplot, a matplotlib-based replacement for the current PGPLOT one. Under development.

Plots a sequence of images as a movie in near ‘real time’, hence ‘rt’. Designed to be used to look at images coming in while at the telescope, ‘nrtplot’ comes with many options, a large number of which are hidden by default, and many of which are only prompted if other arguments are set correctly. If you want to see them all, invoke as ‘nrtplot prompt’. This is worth doing once to know nrtplot’s capabilities.

nrtplot can source data from both the ULTRACAM and HiPERCAM servers, from local ‘raw’ ULTRACAM and HiPERCAM files (i.e. .xml + .dat for ULTRACAM, 3D FITS files for HiPERCAM) and from lists of HiPERCAM ‘.hcm’ files.

nrtplot optionally allows the selection of targets to be fitted with gaussian or moffat profiles, and, if successful, will plot circles of 2x the measured FWHM in green over the selected targets. In this case it can also plot the history of the FWHMs as an aid to focussing. You can also click on targets on the fly to get one-off measurement which are just printed to the screen. See below for a note on this.

You can re-size and zoom & pan during plotting, although it takes some getting used owing to the way the program blocks. For re-sizing, you may need to keep the left-button of the mouse down for a while for anything to happen. See the parameter ‘memory’ below which can help.

When you first run it, try to get the initial fit parameters about right; it doesn’t like to be miles out at the start for some reason and will refuse to fit.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststr [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed). For data from the HiPERCAM server, a negative number tries to get a frame not quite at the end. i.e. -10 will try to get 10 from the last frame. This is mainly to sidestep a difficult bug with the acquisition system.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstr

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

nxint [if more than 1 CCD]

number of panels across to display for the image display.

pausefloat [hidden]

seconds to pause between frames (defaults to 0). This is in addition to any time taken per frame to read and display it, so is just to slow things not to set an exact rate.

tupdatefloat [hidden]

target number of seconds between plot updates. This is to make the plots more responsive and get round issues with plots freezing until the next frame comes through which can be especially annoying with virtual desktops. Experiments with ULTRACAM@NTT indicated 0.2 to 0.6 secs or so works OK. Main impact seems to be on drpc CPU.

plotallbool [hidden]

plot all frames regardless of status (i.e. including blank frames when nskips are enabled (defaults to False). The profile fitting will still be disabled for bad frames.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

lowlevelfloat [hidden]

Level below which a warning about low bias levels is warned. Set=0 to ignore. Applied to first window of first CCD. 2000 about right for ULTRACAM.

highlevelfloat [hidden]

Level above which a warning about high bias levels is warned. Set=0 to ignore. Applied to first window of first CCD. 3500 about right for ULTRACAM.

darkstr

Name of dark frame to subtract, ‘none’ to ignore

flatstr

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

fmapstr

Name of fringe map (see e.g. makefringe), ‘none’ to ignore.

fpairstr [if fmap is not ‘none’]

Name of fringe pair file (see e.g. setfringe). Required if a fringe map has been specified.

nhalfint [if fmap is not ‘none’, hidden]

When calculating the differences for fringe measurement, a region extending +/-nhalf binned pixels will be used when measuring the amplitudes. Basically helps the stats.

rminfloat [if fmap is not ‘none’, hidden]

Minimum individual ratio to accept prior to calculating the overall median in order to reduce the effect of outliers. Although all ratios should be positive, you might want to set this a little below zero to allow for some statistical fluctuation.

rmaxfloat [if fmap is not ‘none’, hidden]

Maximum individual ratio to accept prior to calculating the overall median in order to reduce the effect of outliers. Probably typically < 1 if fringe map was created from longer exposure data.

defectstr

Name of defect file, ‘none’ to ignore.

setupbool

True/yes to access the current windows from hdriver. Useful during observing when setting up windows, but not normally otherwise. Next argument (hidden) is the URL to get to hdriver. Once setup, you should probably turn this off to avoid overloading hdriver, especially if in drift mode as it makes a request for the windows for every frame.

drurlstr [if setup; hidden]

URL needed to access window setting from the camera driver (ultracam, ultraspec, hipercam). The internal server in the camera driver must be switched on which can be done from the GUI.

cmapstr [hidden]

The matplotlib colour map to use. “Greys” gives the usual greyscale. “none” will give whatever the current default is. Many other choices: “viridis”, “jet”, “hot”, “Oranges”, etc. Enter an invalid one and the program will fail but return a huge list of possibles in the process. Note that some will not work well with the colours used to plot annotations.

imwidthfloat [hidden]

image display plot width in inches (0 for default, and note that this will also make the height go to its default)

imheightfloat [hidden]

image display plot height in inches (0 for default, and note that this will also make the width go to its default)

memorybool [hidden]

stores the image display dimensions at the end as defaults for imwidth and imheight next time (or not). This helps when you are repeatedly re-starting nrtplot for the same sort of display.

msubbool

subtract the median from each window before scaling for the image display or not. This happens after any bias subtraction.

isetstr [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilofloat [if iset=’d’]

lower intensity level

ihifloat [if iset=’d’]

upper intensity level

plofloat [if iset=’p’]

lower percentile level

phifloat [if iset=’p’]

upper percentile level

xlofloat

left-hand X-limit for plot, initially at least since it is possible to re-size. For iset=’p’ these limits also set the region of the frame over which the percentil will be calculated. You will usually want yhi-ylo ~ xhi-xlo in magnitude because the aspect ratio is preserved.

xhifloat

right-hand X-limit for plot (can be < xlo to invert the display)

ylofloat

lower Y-limit for plot

yhifloat

upper Y-limit for plot (can be < ylo)

profitbool

If profit=True, you say yes, then you will get the option of selecting a fixed set of objects with a cursor, and the program will then attempt to track these from frame to frame, and fit their profile. You may need to adjust ‘first’ to see anything. Depending on ‘fnmax’, it will also then plot a history of the FWHM measurements, which is useful for focussing. The parameters used for profile fits are hidden and you may want to invoke the command with ‘prompt’ the first time you try profile fitting. NB There is also an option for “on-the-fly” fits of any target: see the note below.

methodstr [hidden]

this defines the profile fitting method, either a gaussian or a moffat profile. The latter is usually best.

betafloat [method == ‘m’; hidden]

default Moffat exponent

fwhmfloat [hidden]

default FWHM, unbinned pixels. Do try to get this about right as it affects whether the profile can be fitted at all.

fwhm_minfloat [hidden]

minimum FWHM to allow, unbinned pixels.

shboxfloat [hidden]

half width of box for searching for a star, unbinned pixels. The above-threshold target closest to the centre of the box in a region +/- shbox around an intial position will be selected. It may not be the brightest, depending upon your threshold settings, so use those to filter faint objects. If profit=True, ‘shbox’ should be large enough to allow for likely changes in position from frame to frame, but not too large to avoid jumping to brighter targets or possibly cosmic rays.

smoothfloat [hidden]

FWHM for gaussian smoothing, binned pixels. The initial position for fitting is determined by finding the maximum flux in a smoothed version of the image in a box of width +/- shbox around the starter position. Typically should be comparable to the stellar width. Its main purpose is to combat cosmic rays which tend only to occupy a single pixel.

fhboxfloat [hidden]

half width of box for profile fit, unbinned pixels. The fit box is centred on the position located by the initial search. It should normally be > ~2x the expected FWHM, and usually smaller than shbox

hminfloat [hidden]

height threshold to accept a fit. If the height is below this value, the position will not be updated. This is to help in cloudy conditions. The limit is applied to the image after it has been smoothed to make less vulnerable to seeing fluctuations. This can mean it can be quite small.

readfloat [hidden]

readout noise, RMS ADU, for assigning uncertainties. Set to a -ve value to try to ascertain on the fly; this is advisable either if you don’t have a bias or you apply ‘msub’, and it will default to this if not specified in this case. The value returned in this case includes sky noise, i.e it should be roughly sqrt(R**2+S/G) where R is the true read noise, S are the sky counts per pixel, and G the gain. If read is set -ve, two fits are carried out per target. The second of these should usually be pretty fast. The first is carried out with an assumed large read noise of 20 in order to soften the weights. The results reported apply to the second fit.

gainfloat [hidden]

gain, ADU/count, for assigning uncertainties.

threshfloat [hidden]

sigma rejection threshold for fits

fwnmaxint [if profit; hidden]

maximum number of frames to buffer the FWHM plot (< 2 ==> no plot). It allows the FWHMs of multiple stars to be tracked which might be useful for focussing. The plot starts to “slide” once the buffers are filled.

fwymaxfloat [if profit; hidden]

maximum FWHM to plot on Y-axis [unbinned pixels]. It is possible to alter the value of the fly by zooming and panning the plot.

fwwidthfloat [if profit; hidden]

FWHM display plot width in inches (0 for default, which will also cause the height to go to its default value)

fwheightfloat [if profit; hidden]

FWHM display plot height in inches (0 for default, which will also cause the width to go to its default value)

Note

To help with dithered long exposures especially, for which ‘profit’ performs very poorly, clicking on any object will attempt a one-off profile fit, with results that are reported to the terminal. It’s a little cludgy in that the profile fits are only carried out and reported after the next frame has been plotted, but it’s better than nothing.

If you don’t apply bias subtraction and/or you do subtract the median, then the usual method to estimate the read noise fails. In that case, setting ‘read’ < 0 will force an attempt to measure the read noise from fluctuations in the sky. You need to ensure that fhbox is large enough to encompass sky regions well away from the target in this case.

hipercam.scripts.pbands(args=None)

pbands log [device width height] norm zero [plo phi increment] pgram (flo fhi over) title

Plots a HiPERCAM reduce log as a single light curve in one panel per CCD. Can optionally be normalised and the y-axis origin set to zero, and can also include periodogram plots if wanted. The purpose of the routine is to facilitate quick-look plots during observing runs.

Parameters:

logstring

name of reduce ASCII log file (text file with loads of columns)

devicestring [hidden, defaults to ‘term’]

‘term’ for interactive plot, file name such as ‘plot.pdf’ or ‘object.png’ for a hardcopy.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

normbool

normalise each light curve to a unit median (or not)

zerobool

set y-axis origin to zero

plofloat [hidden]

percentile to use to set the lower limit of the lightcurve plots (e.g. 5). Takes account of error bars.

phifloat [hidden]

percentile to use to set the upper limit of the lightcurve plot (e.g. 95). Must be > plo. Takes account of error bars.

incrementfloat [hidden]

multiplier to increment the raw numbers derived via plo and phi. If they are y1 and y2, then each is corrected up and down by increment*(y2-y1).

pgrambool

plot periodogram panels (or not). They appear on the right-hand side if yes.

flofloat (if pgram)

If pgram, the mimimum frequency for the periodogram plots, cycles per day. Typically there is little point pushing flo very far below 1 cycle over the timespan of the data.

fhifloat (if pgram)

If pgram, the maximum frequency for the periodogram plots cycles per day. The maximum useful is set by the Nyquist limit (1 cycle per 2 exposures). e.g. For 10 second cadence, this would be be 86400/10/2 = 4320 cycles/day.

overfloat (if pgram)

If pgram, the oversampling factor to determine the frequency grid. Corresponds to the “sample_per_peak” parameter of the astropy LombScargle routine.

titlestr

plot title. Defaults to name of log file

colourslist of strings

colours for each CCD, space or comma-separated. e.g. “r,g,b” or “r g b” (without quotes).

hipercam.scripts.plog(args=None)

plog log [device width height] ccd1 aper1 param1 ccd2 (aper2 param2 scheme) [title]

Provides quick-look plots of HiPERCAM reduce logs.

Parameters:

logstring

name of reduce ASCII log file (text file with loads of columns)

devicestring [hidden, defaults to ‘term’]

‘term’ for interactive plot, file name such as ‘plot.pdf’ for a hardcopy.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

ccd1str

first CCD to consider, e.g. ‘1’

aper1str

first aperture to consider

param1str

first parameter to consider. Choices are ‘x’ = X position, ‘y’ = Y position, ‘f’ = FWHM, ‘b’ = Moffat beta, ‘s’ = sky, ‘m’ = maximum counts in aperture

ccd2str

second CCD to consider; ‘!’ to ignore. Can be (and typically would be) the same as ccd1.

aper2str [if ccd2 != ‘!’]

second aperture to consider

param2str [if ccd2 != ‘!’]

second parameter. See param1 for choices

schemestr [if ccd2 != ‘!’]

how to plot if both apertures are chosen. Choices:

‘d’ = difference, i.e. plot 1-2

‘b’ = both plotted on same panel

‘r’ = ratio, i.e. 1 / 2, good for relative photom

‘s’ = scatter plot, 2 vs 1.

titlestr [hidden]

plot title. Defaults to the run number if not specified

Note

Points marked bad, or flagged as having bad times or junk are ignored. i.e. bitmask = BAD_TIME | JUNK is passed to every invocation of Tseries.mplot. Be careful with linked apertures where all x, y, FWHM, beta automatically have negative errors since they are not fitted. See flagcloud for how to flag up junk data.

hipercam.scripts.redanal(args=None)

redanal aper log

This provides some basic stats on a log file such as the fraction of duff points for each aperture of each CCD, how much targets have moved and the like. The aim is to help with setting parameters in the reduce file in difficult cases and to diagnose problems. It will probably be added to with time.

Parameters:

aperstring

the aperture file used for the reduction.

logstring

the log file.

hipercam.scripts.reduce(args=None)

reduce [source] rfile (run first last [twait tmax]| flist) trim ([ncol nrow]) log lplot implot (ccd nx msub xlo xhi ylo yhi iset (ilo ihi | plo phi))

Reduces a sequence of multi-CCD images, plotting lightcurves as images come in. It can extract with either simple aperture photometry or Tim Naylor’s optimal photometry, on specific targets defined in an aperture file using setaper.

reduce can source data from both the ULTRACAM and HiPERCAM servers, from local ‘raw’ ULTRACAM and HiPERCAM files (i.e. .xml + .dat for ULTRACAM, 3D FITS files for HiPERCAM) and from lists of HiPERCAM ‘.hcm’ files. If you have data from a different instrument you should convert into the FITS-based hcm format.

reduce is primarily configured from a file with extension “.red”. This contains a series of directives, e.g. to say how to re-position and re-size the apertures. An initial reduce file is best generated with the script genred after you have created an aperture file. This contains lots of help on what to do. genred tries to get something usable fast for observing purposes, but the parameters can nearly always be tweaked to produce better outcomes.

A reduce run can be terminated at any point with ctrl-C without doing any harm. You may often want to do this at the start in order to adjust parameters of the reduce file.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’: HiPERCAM server

‘hl’: local HiPERCAM FITS file

‘us’: ULTRACAM server

‘ul’: local ULTRACAM .xml/.dat files

‘hf’: list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’ or a file list. If a run, then reduce and log below will be set to have the same name by default.

firstint [if source ends ‘s’ or ‘l’]

first frame to reduce. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed).

lastint [if source ends ‘s’ or ‘l’, hidden]

last frame to reduce. 0 to just continue until the end. This is not prompted for by default and must be set explicitly. It defaults to 0 if not set. Its purpose is to allow accurate profiling tests.

twaitfloat [if source ends ‘s’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

fliststring [if source ends ‘f’]

name of file list

trimbool

True to trim columns and/or rows off the edges of windows nearest the readout. Particularly useful with ULTRACAM windowed data where the first few rows and columns can contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

rfilestr

the “reduce” file, i.e. ASCII text file suitable for reading by ConfigParser. Best seen by example as it has many parts. If you are reducing a run, this will be set to have the same root name by default (but a different extension to avoid name clashes).

logstr

log file for the results. If you are reducing a run, this will be set to have the same root name by default (but a different extension to avoid name clashes)

tkeepfloat

maximum number of minutes of data to store in internal buffers, 0 for the lot. When large numbers of frames are stored, performance can be slowed (although I am not entirely clear why) in which case it makes sense to lose the earlier points (without affecting the saving to disk). This parameter also gives operation similar to that of “max_xrange” parameter in the ULTRACAM pipeline whereby just the last few minutes are shown.

lplotbool

flag to indicate you want to plot the light curve. Saves time not to especially in high-speed runs.

implotbool

flag to indicate you want to plot images.

ccdstring [if implot]

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

nxint [if implot]

number of panels across to display.

msubbool [if implot]

subtract the median from each window before scaling for the image display or not. This happens after any bias subtraction.

xlofloat [if implot]

left-hand X-limit for plot

xhifloat [if implot]

right-hand X-limit for plot (can actually be < xlo)

ylofloat [if implot]

lower Y-limit for plot

yhifloat [if implot]

upper Y-limit for plot (can be < ylo)

isetstring [if implot]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilofloat [if implot and iset=’d’]

lower intensity level

ihifloat [if implot and iset=’d’]

upper intensity level

plofloat [if implot and iset=’p’]

lower percentile level

phifloat [if implot and iset=’p’]

upper percentile level

Warning

The transmission plot generated with reduce is not reliable in the case of optimal photometry since it is highly correlated with the seeing. If you are worried about the transmission during observing, you should always use normal aperture photometry.

hipercam.scripts.rtplot(args=None)

rtplot [source device width height] (run first [twait tmax] | flist) trim ([ncol nrow]) (ccd (nx)) [pause plotall] bias [lowlevel highlevel] flat defect setup [drurl] msub iset (ilo ihi | plo phi) xlo xhi ylo yhi (profit [fdevice fwidth fheight method beta fwhm fwhm_min shbox smooth splot fhbox hmin read gain thresh])

Plots a sequence of images as a movie in near ‘real time’, hence ‘rt’. Designed to be used to look at images coming in while at the telescope, ‘rtplot’ comes with many options, a large number of which are hidden by default, and many of which are only prompted if other arguments are set correctly. If you want to see them all, invoke as ‘rtplot prompt’. This is worth doing once to know rtplot’s capabilities.

rtplot can source data from both the ULTRACAM and HiPERCAM servers, from local ‘raw’ ULTRACAM and HiPERCAM files (i.e. .xml + .dat for ULTRACAM, 3D FITS files for HiPERCAM) and from lists of HiPERCAM ‘.hcm’ files.

rtplot optionally allows the selection of targets to be fitted with gaussian or moffat profiles, and, if successful, will plot circles of 2x the measured FWHM in green over the selected targets. This option only works if a single CCD is being plotted.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server

‘hl’ : local HiPERCAM FITS file

‘us’ : ULTRACAM server

‘ul’ : local ULTRACAM .xml/.dat files

‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for source can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

devicestr [hidden]

Plot device. PGPLOT is used so this should be a PGPLOT-style name, e.g. ‘/xs’, ‘1/xs’ etc. At the moment only ones ending /xs are supported.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststr [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed). For data from the HiPERCAM server, a negative number tries to get a frame not quite at the end. i.e. -10 will try to get 10 from the last frame. This is mainly to sidestep a difficult bug with the acquisition system.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstr

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

nxint [if more than 1 CCD]

number of panels across to display.

pausefloat [hidden]

seconds to pause between frames (defaults to 0)

plotallbool [hidden]

plot all frames regardless of status (i.e. including blank frames when nskips are enabled (defaults to False). The profile fitting will still be disabled for bad frames.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

lowlevelfloat [hidden]

Level below which a warning about low bias levels is warned. Set=0 to ignore. Applied to first window of first CCD. 2000 about right for ULTRACAM.

highlevelfloat [hidden]

Level above which a warning about high bias levels is warned. Set=0 to ignore. Applied to first window of first CCD. 3500 about right for ULTRACAM.

flatstr

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

defectstr

Name of defect file, ‘none’ to ignore.

setupbool

True/yes to access the current windows from hdriver. Useful during observing when seeting up windows, but not normally otherwise. Next argument (hidden) is the URL to get to hdriver. Once setup, you should probably turn this off to avoid overloading hdriver, especially if in drift mode as it makes a request for the windows for every frame.

drurlstr [if setup; hidden]

URL needed to access window setting from the camera driver (ultracam, ultraspec, hipercam). The internal server in the camera driver must be switched on which can be done from the GUI.

msubbool

subtract the median from each window before scaling for the image display or not. This happens after any bias subtraction.

isetstr [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilofloat [if iset=’d’]

lower intensity level

ihifloat [if iset=’d’]

upper intensity level

plofloat [if iset=’p’]

lower percentile level

phifloat [if iset=’p’]

upper percentile level

xlofloat

left-hand X-limit for plot

xhifloat

right-hand X-limit for plot (can actually be < xlo)

ylofloat

lower Y-limit for plot

yhifloat

upper Y-limit for plot (can be < ylo)

profitbool [if plotting a single CCD only]

carry out profile fits or not. If you say yes, then on the first plot, you will have the option to pick objects with a cursor. The program will then attempt to track these from frame to frame, and fit their profile. You may need to adjust ‘first’ to see anything. The parameters used for profile fits are hidden and you may want to invoke the command with ‘prompt’ the first time you try profile fitting.

fdevicestr [if profit; hidden]

plot device for profile fits, PGPLOT-style name. e.g. ‘/xs’, ‘2/xs’ etc.

fwidthfloat [if profit; hidden]

fit plot width (inches). Set = 0 to let the program choose.

fheightfloat [if profit; hidden]

fit plot height (inches). Set = 0 to let the program choose. BOTH fwidth AND fheight must be non-zero to have any effect

methodstr [if profit; hidden]

this defines the profile fitting method, either a gaussian or a moffat profile. The latter is usually best.

betafloat [if profit and method == ‘m’; hidden]

default Moffat exponent

fwhmfloat [if profit; hidden]

default FWHM, unbinned pixels.

fwhm_minfloat [if profit; hidden]

minimum FWHM to allow, unbinned pixels.

shboxfloat [if profit; hidden]

half width of box for searching for a star, unbinned pixels. The brightest target in a region +/- shbox around an intial position will be found. ‘shbox’ should be large enough to allow for likely changes in position from frame to frame, but try to keep it as small as you can to avoid jumping to different targets and to reduce the chances of interference by cosmic rays.

smoothfloat [if profit; hidden]

FWHM for gaussian smoothing, binned pixels. The initial position for fitting is determined by finding the maximum flux in a smoothed version of the image in a box of width +/- shbox around the starter position. Typically should be comparable to the stellar width. Its main purpose is to combat cosmi rays which tend only to occupy a single pixel.

splotbool [if profit; hidden]

Controls whether an outline of the search box and a target number is plotted (in red) or not.

fhboxfloat [if profit; hidden]

half width of box for profile fit, unbinned pixels. The fit box is centred on the position located by the initial search. It should normally be > ~2x the expected FWHM.

hminfloat [if profit; hidden]

height threshold to accept a fit. If the height is below this value, the position will not be updated. This is to help in cloudy conditions.

readfloat [if profit; hidden]

readout noise, RMS ADU, for assigning uncertainties

gainfloat [if profit; hidden]

gain, ADU/count, for assigning uncertainties

threshfloat [if profit; hidden]

sigma rejection threshold for fits

hipercam.scripts.rupdate(args=None)

rupdate rfile

As changes are made to ‘reduce’, old reduce files can become obsolete. This script tries to bring old reduce files up to date by adding in the new options but in a way that should give the old behaviour. The file is modified in place.

There are some cases where a full update might not be possible and a further manual edit of the reduce file might be required.

Parameters:

rfilestring

the output reduce file created using setaper. This will be read for the targets. The main target will be assumed to have been called ‘1’, the main comparison ‘2’. If there is a ‘3’ it will be plotted relative to ‘2’; all others will be ignored for plotting purposes.

hipercam.scripts.setaper(args=None)

setaper mccd aper ccd [linput width height] rtarg rsky1 rsky2 xlo xhi ylo yhi nx msub [cmap] iset (ilo ihi | plo phi) [profit method (beta betafix betamax) fwhm fwfix (fwmin) shbox smooth splot fhbox read gain thresh]

Interactive definition of photometric extraction apertures. This is a matplotlib-based routine allowing you to place apertures on targets using the cursor. It is an essential prerequisite to running reduce. You are advised at the minimum to have subtracted the bias from the frame of interest since that makes the assignment of uncertainties to the data used when making profile fits much more reliable.

Parameters:

mccdstring

name of an MCCD file, as produced by e.g. ‘grab’

aperstring

the name of an aperture file. If it exists it will be read so that apertures can be added to it. If it does not exist, it will be created on exiting the routine. The aperture files are is a fairly readable / editiable text format

ccdstring

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes). ‘3 4’ will plot CCD ‘3’ and CCD ‘4’. If you want to plot more than one CCD, then you will be prompted for the number of panels in the X direction. This parameter will not be prompted if there is only one CCD in the file.

linputstring [hidden]

sets the way in which the apertures are labelled. ‘n’ = numerical input, with the program just incrementing by 1 for each successive aperture; ‘s’ = single character (without requiring the user to hit hit <CR>); ‘m’ = multi-character, ending with <CR>. Allowed characters are 0-9, a-z, A-Z, no spaces or punctuation, but a single ‘0’ on its own is not permitted.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

rtargfloat [unbinned pixels]

radius of target aperture. The exact value of this does not matter too much since it is normally overridden in ‘reduce’, but typically one aims for 1.5 to 2.5 x FWHM, seeing, depending upon the target brightness.

rsky1float [unbinned pixels]

inner radius of sky aperture.

rsky2float [unbinned pixels]

radius of target aperture

xlofloat

left X-limit to restrict region used to compute percentile limits. This allows you to only consider useful data if for example there are bias strips, and makes setting the percentile limits less fiddly.

xhifloat

right X-limit. See comments for xlo as well.

ylofloat

bottom Y-limit. See comments for xlo as well.

yhifloat

top Y-limit. See comments for xlo as well.

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

msubbool

True/False to subtract median from each window before scaling

cmapstr [hidden]

Colour map to use. “Greys” is usual greyscale. “none” to get matplotlib default.

isetstring [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD bais.

ilofloat [if iset==’d’]

lower intensity level

ihifloat [if iset==’d’]

upper intensity level

plofloat [if iset==’p’]

lower percentile level

phifloat [if iset==’p’]

upper percentile level

profitbool [hidden]

start aperture selection with profile fitting-based position refinement (or not). Can be toggled at any point.

methodstring [hidden]

this defines the profile fitting method, if profile fitting is used to refine the aperture position. Either a gaussian or a moffat profile, ‘g’ or ‘m’. The latter should usually be best.

betafloat [if method == ‘m’; hidden]

default Moffat exponent

betafixbool [if method == ‘m’; hidden]

fix beta or not

beta_maxbool [if method == ‘m’ and not betafix; hidden]

maximum value of beta. Moffat profiles are degenerate with gaussians at large beta so the idea is to avoid wandering to huge beta.

fwhmfloat [hidden]

default FWHM, unbinned pixels. It helps to get this of the right order of mag at the start; too large is better than too small.

fwfix: bool [hidden]

don’t fit the FWHM. Can be more robust; the position is still fitted.

fwminfloat [if not fwfix; hidden]

minimum FWHM to allow, unbinned pixels.

shboxfloat [hidden]

half width of box for searching for a star, unbinned pixels. The brightest target in a region +/- shbox around an intial position will be found. ‘shbox’ should be large enough to allow for likely changes in position from frame to frame, but try to keep it as small as you can to avoid jumping to different targets and to reduce the chances of interference by cosmic rays.

smoothfloat [hidden]

FWHM for gaussian smoothing, binned pixels. The initial position for fitting is determined by finding the maximum flux in a smoothed version of the image in a box of width +/- shbox around the starter position. Typically should be comparable to the stellar width. Its main purpose is to combat cosmic rays which tend only to occupy a single pixel.

splotbool [hidden]

Controls whether an outline of the search box and a target number is plotted (in red) or not.

fhboxfloat [hidden]

half width of box for profile fit, unbinned pixels. The fit box is centred on the position located by the initial search. It should normally be > ~2x the expected FWHM.

readfloat [hidden]

readout noise, RMS ADU, for assigning uncertainties

gainfloat [hidden]

gain, ADU/count, for assigning uncertainties

threshfloat [hidden]

thresh rejection threshold. Be careful. Bad combinations of this and the initial FWHM can cause wholessale rejection of good pixels, so don’t be overzealous.

ndivint [hidden]

sub-division factor for fits. 0=no sub-division. Otherwise it will used ndiv*ndiv sub-evaluations per unbinned pixel

There are a few conveniences to make setaper easier:

1. The plot is initialised in pan mode whereby you can move around and scale using the left and right mouse buttons.

2. All input is accomplished with the keyboard; the mouse buttons are only for navigating the image.

3. The label input can be switched between sequential numerical, single- and multi-character input (‘linput’).

Various standard keyboard shortcuts (e.g. ‘s’ to save) are disabled as they just confuse things and are of limited use in setaper in any case.

Some aspects of the usage of matplotlib in setaper are tricky. It is possible that particular ‘backends’ will cause problems. I have tested this with Qt4Agg, Qt5agg and GTK3Agg. One aspect is the cursor icon in pan mode is a rather indistinct hand where one can’t tell what is being pointed at. I have therefore suppressed this, but only for the tested backends. Others would need require further investigation.

When in setaper, help is always available by hitting ‘h’. Most of the options are self-evident. A few which may not be are:

e(xtra) : add extra pixels to the target aperture. This allows you

to sculpt the shape of the extraction aperure to perhaps

include the flux of a blended star.

l(ink) : link one aperture to another in the same CCD for

re-positioning. If a target is very faint or will

disappear, this will make sure that its position

is defined relative to another, brighter star. Best

to choose one close by because of potential refractive

distortion.

r(eference): toggle whether an aperture is a reference aperture. The

re-positioning can work in two steps: first position

reference stars, then the other stars, using the reference

stars to provide a first cut at the position shift of the

rest.

C(opy) : copy apertures of the CCD the cursor is in to all others.

This basically clones apertures across the CCDs. You will

need to re-centre each on afterwards.

Warning

The aperture positions immediately after a copy reflect the origin CCD, and may be somewhat off if there are significant offsets between CCDs.

hipercam.scripts.setdefect(args=None)

setdefect mccd defect ccd [width height] nx msub [cmap] ffield (auto (hlo hhi severity)) hsbox iset (ilo ihi | plo phi)

Interactive definition of CCD defects. This is a matplotlib-based routine allowing you to define defects using the cursor. Various routines allow you to overplot defects, above all rtplot / nrtplot. Point-like and line-like defects of two different colours (orange for moderate, red for severe can be marked). Hot pixels can be marked by their count rate as measured in dark frames.

You should apply this in stages using bias frames and flat fields to set the point and line defects, and a darkframe from makedark to set hot pixels. The latter can be set semi-automatically by flagging pixels between pre-set rate values.

Parameters:

mccdstr

name of an MCCD file, as produced by e.g. ‘grab’

defectstr

the name of a defect file. If it exists it will be read so that defects can be added to it. If it does not exist, it will be created on exiting the routine. The defect files are in a fairly readable / editiable text format

ccdstr

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes). ‘3 4’ will plot CCD ‘3’ and CCD ‘4’. If you want to plot more than one CCD, then you will be prompted for the number of panels in the X direction. This parameter will not be prompted if there is only one CCD in the file.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

msubbool

True/False to subtract median from each window before scaling

cmapstr [hidden]

The colour map to use. “Greys” is the usual; “Greys_r” reverses it. There are many others; typing an incorrect one will give a list. “none” for matplotlib default.

ffieldbool

If True, all defects will be assumed to be flat-field or poor charge transfer defects as opposed to hot pixels. The latter should be set from dark frames, and have a different impact than the first two types in that they are worst for faint targets. Hot pixels and flat-field defects are shown with the same colours for moderate and severe, but different symbols (filled circles for flat-field defects, stars for hot pixels). If you say “no” in order to add hot pixels, the line defect option is not available.

autobool [if ffield==False]

Hot pixels can be set automatically if wanted. If so then a few extra parameters are needed. This only makes sense if you are feeding setdefect with a dark frame produced by makedark so that the intensity levels mean something.

hlofloat [if auto]

lower limit to flag as a hot pixel as a count rate per second

hhifloat [if auto]

upper limit to flag as a hot pixel as a count rate per second

severitystr [if auto]

‘m’ for moderate, ‘n’ for nasty. Controls colour used to plot the hot pixel; it will be labelled with a rate as well.

hsboxint

half-width in binned pixels of stats box as offset from central pixel hsbox = 1 gives a 3x3 box; hsbox = 2 gives 5x5 etc. This is used by the “show” option when setting defects.

isetstr [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD bais.

ilofloat [if iset==’d’]

lower intensity level

ihifloat [if iset==’d’]

upper intensity level

plofloat [if iset==’p’]

lower percentile level

phifloat [if iset==’p’]

upper percentile level

There are a few conveniences to make setdefect easier:

1. The plot is initialised in pan mode whereby you can move around and scale using the left and right mouse buttons.

2. All input is accomplished with the keyboard; the mouse buttons are only for navigating the image.

3. The label input can be switched between sequential numerical, single- and multi-character input (‘linput’).

Various standard keyboard shortcuts (e.g. ‘s’ to save) are disabled as they just confuse things and are of limited use in setdefect in any case.

Some aspects of the usage of matplotlib in setdefect are tricky. It is possible that particular ‘backends’ will cause problems. I have tested this with Qt4Agg, Qt5agg and GTK3Agg. One aspect is the cursor icon in pan mode is a rather indistinct hand where one can’t tell what is being pointed at. I have therefore suppressed this, but only for the tested backends. Others would need require further investigation.

NB At the end of this routine, it re-orders the defects so that the severe ones follows the moderates. This helps emphasize the severe ones over the moderates when running rtplot.

hipercam.scripts.setfringe(args=None)

setfringe fmap fringe ccd [width height] nx [cmap nhalf] iset (ilo ihi | plo phi)

Interactive definition of CCD fringe pairs. The idea is to place a large number of such pairs at the peaks and valleys of a fringe map file. To defringe a file, the differences of intensity for each pair will be evaluated in the data and the fringe map and the ratio of these differences taken. The median of these ratios will be used to scale the fringe map which will then be subtracted. The use of a median allows the odd ratio to be ruined by stars, as long as a large number of pairs are defined.

Parameters:

fmapstr

name of an MCCD file showing fringes with objects removed, e.g. as produced by ‘makefringe’

fringestr

the name of a file of fringe pairs. If it exists it will be read so that more fringe pairs can be added to it. If it does not exist, it will be created on exiting the routine. The files are in a fairly readable / editable text format

ccdstr

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes). ‘3 4’ will plot CCD ‘3’ and CCD ‘4’. If you want to plot more than one CCD, then you will be prompted for the number of panels in the X direction. This parameter will not be prompted if there is only one CCD in the file.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

cmapstr [hidden]

The colour map to use. “Greys” is the usual; “Greys_r” reverses it. There are many others; typing an incorrect one will give a list. “none” for matplotlib default.

nhalfint [hidden]

half-width in binned pixels of stats box as offset from central pixel nhalf = 1 gives a 3x3 box; hsbox = 2 gives 5x5 etc. This is used by the “show” option when setting FringePair and when reporting FringePair differences.

isetstr [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD bais.

ilofloat [if iset==’d’]

lower intensity level

ihifloat [if iset==’d’]

upper intensity level

plofloat [if iset==’p’]

lower percentile level

phifloat [if iset==’p’]

upper percentile level

hipercam.scripts.splice(args=None)

splice input1 input2 ccd [win] output

Merges two multi-CCD images so that parts of input1 are replaced by equivalent parts of input2. This can be useful e.g. to create a combined flat field out of different frames.

Parameters:

input1str

first input frame

input2str

second input frame

ccdstr

the CCD(s) to transfer, ‘0’ for all of them. ‘1 3’, i.e. separate with spaces for more than one CCD.

winstr [hidden]

the window(s) to transfer, ‘0’ for all of them (default). The same windows must exist in the selected CCDs of both inputs

outputstr

output file.

hipercam.scripts.stats(args=None)

stats input [format]

Lists basic stats of a multi-CCD image, i.e. the minimum, maximum, mean, median and standard deviation of each window of each CCD. The output format can be altered to suit preference.

Parameters:

inputstring

name of the MCCD file

formatstring [hidden, defaults to 9.3f]

C-style format code as used in Python format statements for output of the numerical values. e.g. ‘300.00’ is ‘6.2f’ (6 characters toal, 2 after the decimal point), ‘1.22e24’ is ‘.2e’ (as many characters as needed, 2 after the decimal point)

hipercam.scripts.sub(args=None)

sub input1 input2 [ccd win] output

Subtracts two hcm frames and outputs the result. Can be applied only to particular CCDs and windows if wanted.

Parameters:

input1string

first input hcm file

input2string

second input hcm file to subtract from the first.

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as either input1 or input2 in which case the input file will be over-written.

hipercam.scripts.uls(args=None)

Gives an ‘ls’-like listing of the runs available on the ULTRACAM file server. Just invoke as ‘uls’ with no arguments.

See hls for the HiPERCAM equivalent.

1

Several other commands (aligntool, atanalysis, atbytes, calsearch, harchive, hlogger, hmeta, logsearch, makefield, makedata, pfolder, redplt, tanalysis, tbytes) are not documented here as they are of specialist usage. Information on these is however available via pydoc.