Data logging

The HiPERCAM pipeline includes several commands associated with the creation and mantenance of data logs. They are not of general interest, but this section gives a blow-by-blow account of how one deals with new data. All the commands should be run from a directory called “raw_data”, which itself is a sub-directory of a directory named after the instrument, i.e. hipercam, ultracam or ultraspec. The sequence of steps is then as follows:

  1. When some new data have been taken, say on 2017-04-03 to 2017-04-05, define a name for the run (i.e. a bunch of nights), which could be “2017-04” for a run in April 2017. Create a sub-directory of this name inside raw_data. cd to it, and create a file called “telescope” containing the name of the telescope (WHT, NTT, TNT or VLT).

  2. Next download the data saved by Paul Kerry’s end_of_night_tasks into this directory. The nights should have dates of the form 2017_04_03 etc.

  3. cd up the raw_data directory. Run the command harchive with no arguments. (If you add further nights later, you should run it with the switch ‘-f’ or it will skip the run.) This checks for a set of expected files, changes the directory structure a bit and makes links. There is one special case run called “Others” for which the checks are lighter touch since they are not core data of the vikcam team.

  4. Run redplt which looks for reduction log files and makes png plots of them.

  5. Run hlogger. This works through all runs, extracting header and timing data (which will be written to a sub-directory ‘meta’ of each night directory. They also attempt to identify the target using the name in the header. This is often rather tricky as there are a fair few errors in the logs, both human and instrumental. For the vikcam team data, it is essential that all errors are fixed at this stage through a combination of editing header data and updating the various files TARGETS, AUTO_TARGETS, SKIP_TARGETS and FAILED_TARGETS. The switch ‘-n’ is useful for quickly re-doing a single night, but once done a final complete run of hlogger will be needed to get all files correct.

  6. If you kill hlogger, then delete whatever file it was in the midst of creating, i.e. delete ‘meta/times’ and ‘meta/posdata’ in the night it had reached, otherwise you will end with partial files. You will normally need to kill using ctrl-Z followed by kill, since ctrl-C gets trapped (something I should probably change).