Getting the software

HiPERCAM README file

Here is the HiPERCAM README file which is written in the same markup language as the rest of these pages.

hipercam

hipercam is the reduction package for the multi-CCD camera HiPERCAM built through an ERC Advanced Grant awarded to Vik Dhillon. HiPERCAM is a multi-CCD high-speed camera for astrophysical research. The hipercam pipeline can also be used to reduce data from the ULTRACAM and ULTRASPEC high-speed cameras, and other instruments too after a little data format wrangling.

Installation

hipercam is written in Python3; it does not support Python2.x It relies on multiple third-party packages, as described in the next section. At minimum, you should ensure that Cython and trm.pgplot are installed. Once you have, get the hipercam pipeline software itself using:

git clone https://github.com/HiPERCAM/hipercam.git

This will create a directory; if you are updating then a simple “git pull” from within this directory should suffice. Change to this directory and install with:

pip install . --user

If you have multiple versions of python, then you might need to specify pip3. If you want a self-contained install, I suggest looking into using “pipenv”.

I use the Qt5Agg backend to ensure that the important command setaper works. Others may work too of course. I have this set as the default in my .config/matplotlib/matplotlibrc configuartion file with the line backend: Qt5Agg

Third-Party Modules

Apart from Cython and trm.pgplot, I hope that most of the extras will get automatically installed if necessary by pip. So, if you have Cython and PGPLOT ready, you might as well try pip install . --user here and now.

If something seems amiss, here are details of the third-party packages which you can either install via pip or by looking for the packages in your O/S package manager. e.g. under fedora, Cython appears as python3-Cython.

astropy :

astronomical Python package with lots of useful stuff.

Cython :

C-extensions for Python. Widely used package used to interface to C-libraries and to enable faster code when critical. It is needed at the setup stage so it might have to be installed first rather than relying on pip finding it, although I could be wrong.

fitsio :

Provides fairly direct access to FITS through the cfitsio library. Has some features that the pure python implementation in astropy lacks.

keyring:

Used in ‘logsearch’ to store some passwords safely.

matplotlib :

standard plotting package for Python. You will also need a Qt backend. In opensuse for instance the package python3-matplotib-qt5 enables the Qt5Agg backend.

numba :

Another package along with Cython to support faster numerics. Uses “just-in-time” compilation of selected routines.

numpy :

Python’s numerical data package. Highly likely you will have it.

pandas :

Widely used table-handling module, based on numpy, and again quite likely installed already. It is used for some routines to do with generation of spreadsheets and logs in the pipeline (see also xlsxwriter below).

requests :

http request module. It may well be installed already.

scipy :

Scientific software for Python, closely linked to numpy.

sep :

astronomical source extractor based on Bertin’s source extractor.

setuptools_scm :

package to manage version numbers from the git repository

trm.pgplot :

Cython-ised wrapper for PGPLOT which I wrote specifically for hipercam although it is entirely independent of it. PGPLOT is needed for some of the faster plots as matplotlib is way too slow (at least as standardly used; I am hoping to move towards matplotlib-only). PGPLOT itself (F77/C library) must be installed for this to work. Once you have PGPLOT, you can get trm.pgplot from my github site with:

git clone https://github.com/trmrsh/trm-pgplot

trm.pgplot is not to be confused with “ppgplot” which, although very similar, is a more hand-crafted version with some differences in the calls. Once you have cloned it, you can enter trm-pgplot and install with pip, but make sure to set the enviroment variable PGPLOT_PNG inside setup.py to “true” or “false” first, according to whether you installed the PNG drivers with PGPLOT.

websocket-client :

for talking to the hipercam server.

xlsxwriter :

if you want to use the logging scripts hlogger, the object search script logsearch, or build log database tools that output xlsx files. Since these are unusual, the software is designed to build without insisting on this module.

Further Information

For more information see:

Tom Marsh

Updating

You should keep the directory ‘hipercam’ created by the first git clone command because then it makes updating simple. It becomes a matter of going to the hipercam directory, pulling the changes from github, and re-installing:

cd XXX/YYY/hipercam
git pull
python setup.py install --user

(the last line may vary according to how you install it).

Setting up

In order to use the HiPERCAM pipeline for normal reduction, you just need the commands in your path. Try typing hplot (a command) to see if they are. Apart from this there are a few environment variables to know about.

Connecting to the server

If using the pipeline in conjunction with the HiPERCAM instrument, and you wish to use your own laptop rather than the dat reduction PC, then you need to tell it where the server is which you do by setting an environment variable called HIPERCAM_DEFAULT_URL. Typically I do this in my .cshrc file (C-shell, use export with bash) with:

setenv HIPERCAM_DEFAULT_URL 192.168.1.2:8007/

The value here is appropriate if you are connected to the HiPERCAM private network via ethernet at the telescope. Note that there is no leading ‘http://’ unlike the equivalent in the ULTRACAM pipeline because this address is used with both http and web sockets and the appropriate prefix is tacked on by whichever script is being used. Also note the final ‘/’ on the port number as in ‘8007/’, which must be there. When setting up for the first time, assuming that the HiPERCAM file server is going, then the command hls is a good one to start with to see that you are set up correctly. Correspondingly, try uls if running the HiPERCAM pipeline with the ULTRACAM server.

Matplotlib-based plots

I have sometimes found matplotlib plots simply do not appear, e.g. the mean level plot in makebias. If you encounter this problem, have a look at your matplotlibrc file, probably located in .config/matplotlib in your home directory. Mine say backend : Qt4Agg at the top, and you may need something similar, depending on your matplotlib distribution.

Environment variables associated with HiPERCAM

Here is a summary table of all environment variables associated with HiPERCAM and their purpose. If you are simply reducing data at your home institute, you probably won’t need to set any of these.

Environment variable

Purpose

HIPERCAM_DEFAULT_SOURCE

Sets the default “source”. Primarily useful for the instrument control computer.

HIPERCAM_DEFAULT_URL

URL of the HiPERCAM FileServer on the rack PC to access data

HIPERCAM_ENV

Directory for storage of default parameter files (~/.hipercam by default). Temporary files are stored in a sub-drectory of this called “tmp”. Although attempts are made to delete such files, it is worth checking every now an again that it does not contain too much junk.

ULTRACAM_DEFAULT_URL

URL of the ULTRACAM FileServer on the rack PC to access data