Coverage for python/lsst/images/ndf/__init__.py: 71%

1# This file is part of lsst-images. 

2# 

3# Developed for the LSST Data Management System. 

4# This product includes software developed by the LSST Project 

5# (https://www.lsst.org). 

6# See the COPYRIGHT file at the top-level directory of this distribution 

7# for details of code ownership. 

8# 

9# Use of this source code is governed by a 3-clause BSD-style 

10# license that can be found in the LICENSE file. 

11 

12 

13"""Archive implementations that write HDS-on-HDF5 files compatible with the 

14Starlink N-Dimensional Data Format (NDF) data model. 

15 

16Files written by this archive are valid NDF files readable by 

17applications and libraries from the 

18`Starlink Software Collection <https://starlink.eao.hawaii.edu>`_ 

19(KAPPA, ``hdstrace``, CUPID, etc.) although the LSST data model adds 

20extensions that are not known to the Starlink tooling. 

21 

22The NDF data model is described in 

23`Jenness et al (2015) <https://doi.org/10.1016/j.ascom.2014.11.001>`_. 

24 

25For a `lsst.images.MaskedImage` the data and variance arrays map to the 

26standard NDF equivalents. 

27Any FITS headers are stored in the standard ``.MORE.FITS`` extension and 

28the WCS is stored as a serialized AST FrameSet, modified to include a ``GRID`` 

29frame in addition to a ``PIXEL`` frame, in the ``.WCS`` component. 

30Starlink tools only allow 8-bit masks so the mask is stored directly as a 

31``QUALITY`` component when only 8 bits are needed but if more bits are 

32required the full mask is stored in the ``.MORE.LSST.MASK`` extension as 

33a 3-D unsigned byte NDF where the third dimension matches the internal 

34representation of the 3-D mask. The ``QUALITY`` component includes a collapsed 

35version of the full mask to enable Starlink software to apply a coarse mask. 

36In the future the collapsing is intending to be more granular to allow mask 

37planes to be combined based on their related concepts. 

38Since the NDF data model has a subset that is compatible with the 

39`~lsst.images.MaskedImage` data model, an NDF can be read as a 

40`~lsst.images.MaskedImage` with full data, variance, mask, WCS, and FITS 

41header support, although of course any extensions will be ignored. 

42 

43For more complex image types such as `lsst.images.VisitImage` the LSST 

44components are stored in the ``.MORE.LSST`` extension, including the ``JSON`` 

45component containing the Pydantic models. Data arrays are represented as 

46NDFs in the ``.MORE.LSST`` extension and referenced by path from the JSON. 

47Tabular data is not supported by the NDF data model and is not currently 

48representable. It is expected that they would become individual columns 

49in an extension, all with the same length. 

50 

51These files use HDF5 format and can be read by any HDF5 tooling such as 

52``h5dump`` or the Python ``h5py``, although the data model includes 

53group attributes to allow it to be read by the Starlink libraries 

54that add additional semantic meaning to data structures. 

55The HDS-on-HDF5 format is described in 

56`Jenness (2015) <https://doi.org/10.1016/j.ascom.2015.02.003>`_. 

57 

58HDF5 files are not generally usable for remote access of components through 

59cloud storage. There is no facility to store byte offsets to components in 

60any of the data structures. 

61""" 

62 

63try: 

64 import h5py # noqa: F401 

65except ImportError as e: 

66 raise ImportError( 

67 "lsst.images.ndf requires the optional 'h5py' package. " 

68 "Install it directly or via 'pip install lsst-images[ndf]'." 

69 ) from e 

70 

71from ._common import * 

72from ._input_archive import * 

73from ._output_archive import *