Tutorial

Table of Contents

• Introduction
• Setup
• Data classes
  • Record
  • Raster
  • Column
• Open (and close) a MEF file
• Write a MEF file
  • Create extensions
  • Write records
  • Write image data
  • Write binary table data
• Read a MEF file
  • Access HDUs
  • Parse records
  • Read image data
  • Read binary table data

Introduction

In this tutorial, we will show how to read and write multi-extension FITS (MEF) files. This means understanding the usage of the following service classes:

• MefFile at file level,
• ImageHdu and BintableHdu at HDU level,
• ImageRaster and BintableColumns at data unit level;

as well as Record, Raster and Column data classes.

We strongly recommend reading first What's a FITS file?, even if you're familiar with the FITS format, because it introduces EleFits-specific wording.

At the end of the tutorial, you will be able to create a MEF file from scratch with unlimited number of various extensions, and to read the metadata and data back!

The tutorial is built together with an example program: EleFitsTutorial.cpp. We've embedded the calls to the logger in the code snippets below, so that you can easily map the execution log to the following explanations. You can already run the program and watch the resulting file:

EleFitsTutorial --output tuto.fits
fv tuto.fits

We'll first discover the data classes, then use them to create a MEF file, and finally read the file and values back.

See also

To go further, at the end of each section, a link points to the reference page for the introduced topic.

Setup

First things first, we have to declare the dependency to EleFits and to use the right headers and namespaces. For the first part, head to the Installation and configuration guide. For the headers and namespace, here's the only thing you'll have to do:

#include "EleFits/MefFile.h"
 
using namespace Euclid;
// EleFits API is in the Euclid::Fits namespace.
// We could have be using namespace Euclid::Fits instead,
// but things would have been less obvious in the snippets.

Data classes

Data classes are the classes which hold the pieces of data read from and written to a FITS file. There are three main classes:

• Record for header units,
• Raster for data units of image HDUs,
• Column for data units of binary table HDUs.

Record

A keyword record as defined in the FITS standard is a triplet of keyword, value and optional comment rendered in the file as:

KEYWORD = VALUE / comment

A unit can be included in the comment as follows:

KEYWORD = VALUE / [unit] comment

The value can be a Boolean, an integer, real or complex number, or a string.

Record is a simple template class which merely stores those fields. For the purpose of the tutorial, we define a structure to hold our records:

struct TutoRecords {
  Fits::Record<std::string> string_record;
  Fits::Record<int> int_record;
  Fits::Record<float> float_record;
  Fits::Record<std::complex<double>> complex_record;
};

Here's a how they can be built:

 
  /* Create a record with unit and comment */
 
  Fits::Record<std::string> string_record("STRING", "VALUE", "unit", "comment");
 
  /* Create a record with keyword and value only */
 
  Fits::Record<int> int_record("INT", 0);
 
  /* Create a record from an initialization list */
 
  Fits::Record<float> float_record {"FLOAT", 3.14F, "", "A piece of Pi"};
  // This is often used as a shortcut to create records as function parameters.
 
  /* Generate a random record */
 
  auto complex_record = Fits::Test::generate_random_record<std::complex<double>>("COMPLEX");
 

See also

Header data classes

Raster

Images in FITS are n-dimensional arrays. Raster is a class which represents this kind of data and provides constant time pixel accessors. Template arguments are the pixel type and number of axes. There are two kinds of rasters:

• VecRaster owns the data as an std::vector;
• PtrRaster points to the data owned by another class, as a raw pointer (be careful not to destroy the data while the raster is alive).

The rasters of this tutorial are stored in a dedicated structure:

struct TutoRasters {
  Fits::VecRaster<std::int16_t, 2> int16_raster2d;
  Fits::VecRaster<std::int32_t, 3> int32_raster3d;
  Fits::VecRaster<std::int64_t, 4> int64_raster4d;
};

Again, let's show an example of how to create rasters:

 
  /* Initialize and later fill a raster */
 
  Fits::VecRaster<std::int16_t, 2> int16_raster2d({4, 3});
  for (const auto& position : int16_raster2d.domain()) {
    int16_raster2d[position] = position[0] + position[1];
  }
  // This demonstrates the iteration over positions;
  // It is possible to use two nested loops instead.
 
  /* Create a raster from a vector */
 
  std::vector<std::int32_t> int32_vec(16 * 9 * 3, 0);
  // ... do what you have to do with the vector, and then move it to the raster ...
  Fits::VecRaster<std::int32_t, 3> int32_raster3d({16, 9, 3}, std::move(int32_vec));
  // Instead of moving a vector, it's also possible to work with
  // a raw pointer with the PtrRaster class.
 
  /* Generate a random raster */
 
  auto int64_raster4d = Fits::Test::RandomRaster<std::int64_t, 4>({17, 9, 3, 24});
 

See also

Image data classes

Column

A binary table is made of columns which can be either scalar (each cell contains a value) or vector (each cell contains several values). In the latter case, the number of values per cell is named repeat count.

For string columns, the repeat count must be greater than the longest string value.

A Column is a simple structure which holds the column name, repeat count (=1 for scalar columns, >1 for vector columns), unit, and owns or references some data, respectively as a VecColumn or as a PtrColumn.

The following structure stores the tutorial columns:

struct TutoColumns {
  Fits::VecColumn<std::string> string_column;
  Fits::VecColumn<std::int32_t> int32_column;
  Fits::VecColumn<float> float_column;
};

Here's a set of examples on how create them:

 
  /* Initialize and later fill a column */
 
  Fits::VecColumn<std::string> string_column({"STRING", "unit", 3}, 100);
  // String columns must be wide-enough to hold each character.
  for (long i = 0; i < string_column.row_count(); ++i) {
    string_column(i) = std::to_string(i);
  }
  // operator() takes two parameters: the row index, and repeat index (=0 by default)
 
  /* Create a column from a vector */
 
  std::vector<std::int32_t> int32_vec(100);
  // ... do what you have to do with the vector, and then move it to the column ...
  Fits::VecColumn<std::int32_t> int32_column({"INT32", "", 1}, std::move(int32_vec));
  // Analogously to rasters, columns can be managed with the lightweight PtrColumn classe.
 
  /* Generate a random column */
 
  auto float_column = Fits::Test::RandomVectorColumn<float>(8, 100);
 

See also

Binary table data classes

Open (and close) a MEF file

Now that we have learned about the data classes, we can move to the service classes, or handlers. The first thing we can do is to create a new file.

The MefFile class represents multi-extension FITS (MEF) files. It extends the FitsFile class, which is also the base class of SifFile for single-image FITS (SIF) files.

Creating (or opening) a file is simply done with the constructor of MefFile (or SifFile for SIF files!):

 
  Fits::MefFile f(filename, Fits::FileMode::Create);
 

A newly created FITS file consists in an empty Primary, which can then be accessed and modified, but is never created by hand.

The mode parameter controls the access rights on the file. For example, an existing file can be opened with read-only mode:

 
  Fits::MefFile f(filename, Fits::FileMode::Read);
 

The file is closed when the destructor of MefFile is called (although a FitsFile::close() method is provided for convenience).

See also

File handlers

Write a MEF file

Create extensions

MefFile provides services to create and access extensions. Conceptually, MefFile is a FitsFile with a vector of HDUs (in contrast, SifFile is a FitsFile with a single HDU).

HDUs are represented by two classes:

• ImageHdu to access image HDUs,
• BintableHdu to access binary table HDUs.

Both derive from parent class Hdu.

To sum up, a MefFile is a vector of Hdus, which can be a mix of ImageHdus and BintableHdus.

There are two kinds of services in MefFile for creating extensions: They can be either initialized with header only or assigned directly with data. Here's an example of creating image HDUs:

 
  /* Fill the header and data units */
 
  const auto& image1 = f.append_image("IMAGE1", {}, rasters.int32_raster3d);
 
  /* Fill the header only (for now) */
 
  const auto& image2 = f.append_null_image<std::int16_t>("IMAGE2", {}, rasters.int16_raster2d.shape());
 

And here's one of creating binary table HDUs:

 
  /* Fill the header and data units */
 
  const auto& table1 =
      f.append_bintable("TABLE1", {}, columns.string_column, columns.int32_column, columns.float_column);
 
  /* Fill the header unit only (for now) */
 
  const auto& table2 = f.append_bintable_header(
      "TABLE2",
      {},
      columns.string_column.info(),
      columns.int32_column.info(),
      columns.float_column.info());
 

See also

File handlers

Write records

Records are read and written through a Header object, which is accessible from an ImageHdu or a BintableHdu as header(). They can be written and updated one-by-one or by sequences.

An optional template parameter controls the write policy, e.g. what to do if the given keyword already exists. By default, a record is created if the keyword doesn't exists, or updated if the keyword already exists.

Record writing functions have different overloards to allow calling them in different ways. Here are a few examples:

 
  /* Write a single record */
 
  h.write(records.string_record);
 
  /* Write several records */
 
  h.write_n(records.int_record, records.float_record, records.complex_record);
 
  /* Update using initialization lists */
 
  h.write_n<Fits::RecordMode::UpdateExisting>(
      Fits::Record<int>("INT", 1),
      Fits::Record<float>("FLOAT", 3.14159F, "", "A larger piece of Pi"),
      Fits::Record<std::complex<double>>("COMPLEX", {180., 90.}));
 

See also

Header unit handlers

Write image data

Remember that we have left IMAGE2 extension without data? Filling it is done through the ImageRaster class, accessed as ImageHdu::raster(). For example, here is how to write all the pixels at once:

 
  image2.raster().write(rasters.int16_raster2d);
 

Many more options are available to write data region-wise. They are described in details in Image data unit handlers.

See also

Image data unit handlers

Write binary table data

Analogously to ImageRaster for image HDUs, BintableColumns provides read/write services for the data unit of the binary table extension.

As the name implies, data is stored, read and written column-wise, as opposed to the row-major ordering of the FITS file. This is to avoid working with very complex structures like tuples with holes, and rather rely on Column, std::vector, arrays... Yet, this also means that I/Os could be very inefficient! To workaround this, several columns can be written (and read) at a time. In this case, I/Os are internally performed chunk-wise, using some buffer, and performances drammatically improve.

 
  /* Write a single column */
 
  table2.columns().write(columns.string_column);
 
  /* Write several columns */
 
  table2.columns().write_n(columns.int32_column, columns.float_column);
 

Like for image HDUs, it is possible to write the data unit only partially, e.g. to append rows, using richer methods of BintableColumns.

See also

Binary table data unit handlers

Read a MEF file

Access HDUs

HDUs can be accessed with a set of methods, templated with the type of HDU: BintableHdu, ImageHdu, or Hdu. For metadata work, we don't need to know the type of HDU: whether this is an image or binary table HDU has no impact, and a Hdu will be returned by default.

HDUs are accessed either by their name (first HDU whose name matches argument is returned) or by their index, and a shortcut is provided for the primary HDU (which has index 0):

 
  /* Access the Primary HDU */
 
  const auto& primary = f.primary();
  const auto primary_index = primary.index();
  // Indices are 0-based.
 
  /* Access an HDU by its index */
 
  const auto& image2 = f.access<Fits::ImageHdu>(2);
  const auto image_name = image2.read_name();
 
  /* Access an HDU by its name */
 
  const auto& table1 = f.find<Fits::BintableHdu>("TABLE1");
  const auto table_index = table1.index();
  // If several HDUs have the same name, the first one is returned.
 

You've probably noticed that we use references to constant HDU handlers. Indeed, HDU handlers are not modified by reading and writing services, only the MefFile is. Actually, HDU-level and data unit-level handlers are very light classes which aim at providing a clear interface.

See also

File handlers

Parse records

Records are parsed using Header services. Like for writing, you can parse several records at once:

 
  /* Read a single record */
 
  const auto int_record = h.parse<int>("INT");
 
  // Records can be sliced as their value for immediate use:
  const int int_value = h.parse<int>("INT");
 
  /* Read several records */
 
  const auto some_records = h.parse_n(
      Fits::as<std::string>("STRING"),
      Fits::as<int>("INT"),
      Fits::as<float>("FLOAT"),
      Fits::as<std::complex<double>>("COMPLEX"));
  const auto& third_record = std::get<2>(some_records);
 
  /* Read as VariantValue */
 
  const auto variant_records = h.parse_n<>({"INT", "COMPLEX"});
  const auto complex_record = variant_records.as<std::complex<double>>("COMPLEX");
 
  /* Read as a user-defined structure */
 
  const auto tuto_records = h.parse_struct<TutoRecords>(
      Fits::as<std::string>("STRING"),
      Fits::as<int>("INT"),
      Fits::as<float>("FLOAT"),
      Fits::as<std::complex<double>>("COMPLEX"));
  const auto& string_record = tuto_records.string_record;
 

See also

Header unit handlers

Read image data

Like for writing, reading images is ensured by ImageRaster. A VecRaster is generally instantiated, and pixel values can be accessed with the subscript operator []:

 
  const auto image = du.read<std::int16_t, 2>();
 
  const auto& first_pixel = image[{0, 0}];
  const auto& last_pixel = image.at({-1, -1});
  // `operator[]` performs no bound checking, while `at` does and enables backward indexing.
 

Again, regions can be read.

See also

Image data unit handlers

Read binary table data

Column reading is provided by BintableColumns. Columns are generally read as VecColumns, values of which are accessed with the call operator ():

 
  /* Read a single column */
 
  const auto vector_column = du.read<double>("VECTOR");
 
  /* Read several columns by their name */
 
  const auto by_name = du.read_n(Fits::as<std::string>("STRING"), Fits::as<std::int32_t>("INT32"));
  const auto& string_column = std::get<0>(by_name);
 
  /* Read several columns by their index */
 
  const auto by_index = du.read_n(Fits::as<std::string>(0), Fits::as<std::int32_t>(1));
  const auto& int_column = std::get<1>(by_index);
 
  /* Use values */
 
  const auto& first_string = string_column(0);
  const auto& first_int = int_column(0);
  const auto& last_float = vector_column.at(-1, -1);
  // There is no operator[]() for columns, because vector columns require 2 indices (row and repeat).
  // operator()() performs no bound checking, while at() does and enables backward indexing.
 

See also

Binary table data unit handlers