Getting the software¶
HiPERCAM README file¶
Here is the HiPERCAM README file which is written in the same markup language as the rest of these pages.
hipercam is the reduction package for the multi-CCD camera HiPERCAM built through an ERC Advanced Grant awarded to Vik Dhillon. HiPERCAM is a multi-CCD high-speed camera for astrophysical research. The hipercam pipeline can also be used to reduce data from the ULTRACAM and ULTRASPEC high-speed cameras, and other instruments too after a little data format wrangling.
hipercam is written in Python3; it does not support Python2.x It relies on multiple third-party packages, as described in the next section. At minimum, you should ensure that Cython and trm.pgplot are installed. Once you have, get the hipercam pipeline software itself using:
git clone https://github.com/HiPERCAM/hipercam.git
This will create a directory; if you are updating then a simple “git pull” from within this directory should suffice. Change to this directory and install with:
pip install . --user
If you have multiple versions of python, then you might need to specify pip3. If you want a self-contained install, I suggest looking into using “pipenv”.
I use the Qt5Agg backend to ensure that the important command
setaper works. Others may work too of course. I have this set as the
default in my .config/matplotlib/matplotlibrc configuartion file with
the line backend: Qt5Agg
Apart from Cython and trm.pgplot, I hope that most of the extras will
get automatically installed if necessary by pip. So, if you have
Cython and PGPLOT ready, you might as well try
. --user here and now.
If something seems amiss, here are details of the third-party packages
which you can either install via pip or by looking for the packages in
your O/S package manager. e.g. under fedora, Cython appears as
- astropy :
astronomical Python package with lots of useful stuff.
- Cython :
C-extensions for Python. Widely used package used to interface to C-libraries and to enable faster code when critical. It is needed at the setup stage so it might have to be installed first rather than relying on pip finding it, although I could be wrong.
- fitsio :
Provides fairly direct access to FITS through the cfitsio library. Has some features that the pure python implementation in astropy lacks.
Used in ‘logsearch’ to store some passwords safely.
- matplotlib :
standard plotting package for Python. You will also need a Qt backend. In opensuse for instance the package python3-matplotib-qt5 enables the Qt5Agg backend.
- numba :
Another package along with Cython to support faster numerics. Uses “just-in-time” compilation of selected routines.
- numpy :
Python’s numerical data package. Highly likely you will have it.
- pandas :
Widely used table-handling module, based on numpy, and again quite likely installed already. It is used for some routines to do with generation of spreadsheets and logs in the pipeline (see also xlsxwriter below).
- requests :
http request module. It may well be installed already.
- scipy :
Scientific software for Python, closely linked to numpy.
- sep :
astronomical source extractor based on Bertin’s source extractor.
- setuptools_scm :
package to manage version numbers from the git repository
- trm.pgplot :
Cython-ised wrapper for PGPLOT which I wrote specifically for hipercam although it is entirely independent of it. PGPLOT is needed for some of the faster plots as matplotlib is way too slow (at least as standardly used; I am hoping to move towards matplotlib-only). PGPLOT itself (F77/C library) must be installed for this to work. Once you have PGPLOT, you can get trm.pgplot from my github site with:git clone https://github.com/trmrsh/trm-pgplot
trm.pgplot is not to be confused with “ppgplot” which, although very similar, is a more hand-crafted version with some differences in the calls. Once you have cloned it, you can enter trm-pgplot and install with pip, but make sure to set the enviroment variable PGPLOT_PNG inside setup.py to “true” or “false” first, according to whether you installed the PNG drivers with PGPLOT.
- websocket-client :
for talking to the hipercam server.
- xlsxwriter :
if you want to use the logging scripts hlogger, the object search script logsearch, or build log database tools that output xlsx files. Since these are unusual, the software is designed to build without insisting on this module.
For more information see:
You should keep the directory ‘hipercam’ created by the first
command because then it makes updating simple. It becomes a matter of going
to the hipercam directory, pulling the changes from github, and
cd XXX/YYY/hipercam git pull python setup.py install --user
(the last line may vary according to how you install it).
In order to use the HiPERCAM pipeline for normal reduction, you just need the
commands in your path. Try typing
hplot (a command) to see if they
are. Apart from this there are a few environment variables to know about.
Connecting to the server¶
If using the pipeline in conjunction with the HiPERCAM instrument, and you wish to use your own laptop rather than the dat reduction PC, then you need to tell it where the server is which you do by setting an environment variable called HIPERCAM_DEFAULT_URL. Typically I do this in my .cshrc file (C-shell, use export with bash) with:
setenv HIPERCAM_DEFAULT_URL 192.168.1.2:8007/
The value here is appropriate if you are connected to the HiPERCAM private
network via ethernet at the telescope. Note that there is no leading ‘http://’
unlike the equivalent in the ULTRACAM pipeline because this address is used
with both http and web sockets and the appropriate prefix is tacked on by
whichever script is being used. Also note the final ‘/’ on the port number as
in ‘8007/’, which must be there. When setting up for the first time, assuming
that the HiPERCAM file server is going, then the command
hls is a good one to start with to see that you are
set up correctly. Correspondingly, try
uls if running the HiPERCAM pipeline
with the ULTRACAM server.
I have sometimes found matplotlib plots simply do not appear, e.g. the mean
level plot in
makebias. If you encounter this problem, have a look at your
matplotlibrc file, probably located in .config/matplotlib in your home
directory. Mine say
backend : Qt4Agg at the top, and you may need
something similar, depending on your matplotlib distribution.
Environment variables associated with HiPERCAM¶
Here is a summary table of all environment variables associated with HiPERCAM and their purpose. If you are simply reducing data at your home institute, you probably won’t need to set any of these.
Sets the default “source”. Primarily useful for the instrument control computer.
URL of the HiPERCAM FileServer on the rack PC to access data
Directory for storage of default parameter files (~/.hipercam by default). Temporary files are stored in a sub-drectory of this called “tmp”. Although attempts are made to delete such files, it is worth checking every now an again that it does not contain too much junk.
URL of the ULTRACAM FileServer on the rack PC to access data