spacer link to MAST page spacer logo image spacer
 
link to STScI page


MAST HLSP Time Series Data Delivery Standards

Summer 2013

This page describes MAST's guidelines and standards for time series deliverables. Also please read the general HLSP guidelines page. Time series data can include light curves, photon event lists, or radial velocities: any parameter that has "time" as the dependent variable can be considered a "time series" deliverable. These data are almost always in FITS format. more | less

MAST has put together the following information because it is essential for data deliveries. If the archive cannot extract the information they need from dataset headers, MAST will attempt to calculate it based on original exposures when possible. If MAST is not able to extract or calculate what it needs based on the delivered dataset headers, then some of the data will not be searchable and will not be displayed across multi-mission search results.

[General Header Information]   [File Formats]   [File Names]   [Data Columns]   [FITS Standards]  

[ASCII Standards]   [Time Standards]   [Keyword Nomenclature]   [Units]   [Missing Values]  

[Required Keywords]   [Recommended Keywords]  [Optional Keywords]  [Time Series Examples

General Header Information

MAST requires that FITS headers contain the following information (see Required keywords below):
  • Some sort of ID information (program ID, proposal ID, PI name, etc) if appropriate
  • Time coverage (observation start, duration, reference frame and units)
  • List of exposures (origin of data, only required for MAST Missions)
  • Pointing location and World Coordinate System (WCS) if applicable
  • Wavelength coverage, if spectra

File Formats

The preferred file format is FITS. There must be at least one file per unique object/field. There may be more than one file per object if there is an expected or known offset between them, for example, if there are multiple time series for an object taken with different instruments, or if there is a known source of systematic error between different time series for a given object. Time series should be specified as a FITS BINARY table file: more | less

The FITS BINARY file (rows and columns of data in binary representation) should abide by the material in FITS Standard v3.0 (Pence et al. 2010) paper. The required keywords for a FITS BINARY table can be viewed here: FITS Binary Table Extension.

Expand for non-FITS (ASCII, CSV) details | Retract/Close details

Though strongly discouraged, we will accept some time series data as ASCII text tables in extenuating circumstances. This format equates to the FITS Binary format, but in straight ASCII files. More information is provided below.

Lastly, we will also accept CSV format, where the first line is always the list of column names, or otherwise listed in documentation.

File Names

The names of the files should follow MAST's HLSP file name definitions. These definitions are in place to make your contributed HLSP able to be integrated with the rest of MAST's holdings, including the Hubble Legacy Archive. The file naming convention is given by:
hlsp_project_mission_instrument(-resolution)_field-name_filter_version_product.extension     more | less

For full details, visit the Filenames section in the HLSP main page, and expand for full details of how each field in the file name should be defined.

Data Columns

The time series data arrays should be stored as a FITS BINARY table in the first extension (HDU1). A minimum of two data arrays are required. The first data array should always be "time", and should be labeled as "TIME". The second data array must be the measurement at each time interval. If the accompanying measurements are fluxes, then the label should be "FLUX"; we highly recommend additionally providing "FLUX_ERR" for the flux errors. The guideline is to use the name of the measurement as the column name and append the "_ERR" suffix for the column name of the measurement error. more | less

We suggest using the following commonly used column names:

Column NameDescription
TIMETime
FLUXFlux
FLUX_ERRFlux Error
FLUX_FLAGFlux Flag values; Flag mapping should be in the README file
FLUX_COMMENTSFlux Comments regarding Flux column
RVRadial Velocity
RV_ERRRadial Velocity Error
BKG_FLUXBackground Flux
BKG_FLUX_ERRBackground Flux Error

For any other data, we encourage users to email the MAST team early in the process to determine the best labels to use in the binary table.

Note that undefined (NaN) or blank values within the "time" array are not allowed; all rows must be filled for the primary x-axis. See examples below for FITS extension header of a flux time-series HLSP dataset.

FITS Standards

The above requirements translate into FITS header keywords below. These keywords and files themselves must abide by the FITS standard, and therefore cannot have errors when being checked by tools like 'fverify' or 'fitsverify'. more | less

For multi-extension FITS files (i.e. FITS BINARY Tabular data), please note that many of the keywords cannot be in the 0th extension or HDU, they need to be within the corresponding extension(s) which contain the data.

ASCII Standards

ASCII table deliveries are discouraged, but accepted as long as the data are well-defined and documented. Header keywords must be provided at the top of the ASCII file or as an associated data pair (one file contains the header information and the other contains the data themselves). Please abide by the FITS keyword naming scheme as described below; all required keywords must be present.

Expand for ASCII formatting and header details | Retract/Close details

The data format (columns) must be documented within the file itself and/or the supporting README file. For tabular, time series data in either FITS or ASCII format, please follow the Data Columns guidelines provided above. All columns and rows must contain data values; blanks are not allowed. For blank values, please use value 'NaN' for numeric blanks or 'NULL' for string blanks, in the ASCII file. Note that undefined (NaN) values within the "time" array are not allowed; all rows must be filled with valid values for the primary x-axis.

Within a data line, fields are separated by one or more whitespace characters (space or tab). A field is either a sequence of non-whitespace characters or a sequence of characters between two matching quote characters (single (') or double (") quotes) - spaces are therefore allowed in quoted fields.

Column names should only contain a letter, digit and underscore; punctuation and special characters are not allowed; names must start with a letter. We also advise against using dash ("-") within a column name which results in a FITS warning message during FITS validation.

Specification of keywords and description of the column names in the ASCII file should be done as follows:

  • Use the '#' symbol to precede all keyword text lines (non-data text lines)
  • The next 8 characters are used for FITS keyword name; buffer the 8-chars name with spaces (see "NAXIS" keyword in example below)
  • Keyword names can contain letters, digits, underscores or dashes, and must start with a letter
  • The 9th character is an equal sign '=' followed by a at least a single space
  • The keyword value can be a number or string; strings must be in single quotes
  • Keyword comments are written after a "/" symbol
  • The FITS 80-character line length limit applies here: no more than 79 characters in total can be used on any given line of text; the 80th char is the end of line/return char
  • The FITS "COMMENT = " keyword should be used to describe any additional information
  • The "COMMENT = " keyword value is always a string within single quotes

The following is an ASCII header example of keyword, value pairs, including the 3 data column descriptions for time, flux and flux error. Note that these map directly onto FITS header keywords so that MAST can build FITS files from ASCII tabular time-series deliveries.

                                                           
#TELESCOP= 'GALEX'              / telescope used to acquire data                 
#INSTRUME= 'FUV'                / instrument used to acquire data                              
#RA_TARG =            82.586460 / right ascension of target (deg) (J2000)        
#DEC_TARG=            -7.434805 / declination of target (deg) (J2000)               
#DATE-OBS= '2006-04-28'         / UT date of start of first exposure             
#TIME-OBS= '09:20:38'           / UT start time of first exposure                
#TIMESYS = 'UTC'                / Coordinated Universal Time; defined since 1972.                                     
#
#XTENSION= 'BINTABLE'           /Written by IDL:  Mon May 25 12:26:32 2009       
#BITPIX  =                    8 /                                                
#NAXIS   =                    2 /Binary table                                    
#NAXIS1  =              1152000 /Number of bytes per row                         
#NAXIS2  =                    1 /Number of rows                                  
#PCOUNT  =                    0 /Random parameter count                          
#GCOUNT  =                    1 /Group count                                     
#TFIELDS =                    3 /Number of columns                               
#EXTNAME = 'LIGHTCURVE'         /Extension name                           
#EXTNO   =                    1 /Extension number                                
#TFORM1  = '64000D  '           /Real*8 (double precision)                       
#TTYPE1  = 'TIME    '           /Column 1: Time                            
#TUNIT1  = 'Seconds'            /Units of column 1                               
#TFORM2  = '64000E  '           /Real*4 (floating point)                         
#TTYPE2  = 'FLUX    '           /Column 2: Flux                           
#TUNIT2  = 'erg/s/cm^2'         /Units of column 2                               
#TFORM3  = '64000E  '           /Real*4 (floating point)                         
#TTYPE3  = 'FLUX_ERR   '        /Column 3: Flux Error                     
#TUNIT3  = 'erg/s/cm^2'         /Units of column 3                               
#
#END                                                                             

Time Standards

The most important column in any time series is the array of timestamps themselves. As such, it is CRITICAL that the exact time format and time standard be well-documented in a provided README file. Are your times in JD, HJD, BJD, RJD, MJD? Are you using UTC or TDB time standard? We highly recommend using the TIMESYS FITS keyword, discussed in Appendix B in the FITS Standard v3.0 (Pence et al. 2010) paper. more | less

More information on TIMESYS can be found here. The convention suggested in Appendix B of the FITS Standard v3.0 is part of the mission-specific FITS conventions adopted for, and used in, the RXTE absolute timing accuracy, building on existing High Energy Astrophysics FITS conventions.

Please be sure to specify the precision level of your timestamps; number of significant digits in your data files must reflect this.

Time-related software suggestions:

  • Jason Eastman's IDL Time Utilities contains routines to convert a UTC to BJD in TDB, convert a BJD in TDB to UTC, convert an HJD in TT or TDB to BJD in TDB, convert a JD in UTC to JD in TDB, as well as other time-conversions in IDL.
  • The IDL Astronomy User's Library is available from Goddard. A listing of other Astronomy and IDL related sites is also useful.
  • Russell Owen's Python RO utility package includes RO.Astro for Astronomical coordinates and time conversions, and RO.DS9 for displaying images in the ds9 image viewer.
  • Additional Astronomical Python tools are listed on the AstroPy website.

Header Keyword Nomenclature

In order to capture some keywords which is a composite, that keyword can contain the value of "MULTI", followed by a list of N similar keywords which encompass all the values needed to explain the composite data using a series of related keywords. more | less

For example, a catalog can be built from a single observation with FITS keyword 'INSTRUME' = "COS". For a composite catalog made extracting fluxes from several images, say STIS and COS data, the keys would be as follows:

  • INSTRUME = "MULTI"
  • INSTRU01 = "COS"
  • INSTRU02 = "STIS"

To designate multiple keywords, we denote this as [nn].

Keyword Value Units

Some keywords have standard units and do not need to be explicitly specified in the headers. In cases where you need to specify units for header keywords, this can be done in one of two ways: more | less

  1. KEYWORD + KEYWORD_UNIT pair
  2. You can specify the keyword units using a second, similar keyword where the keyword name contains the string "_UNIT" or "UNIT". Please remember to stay within the FITS 8-character keyword name limit. E.g.:

    WAVECENT = 1688 / central wavelength
    WAVEUNIT = angstrom / central wavelength units

  3. KEYWORD = value / [unit] in brackets within the FITS comment
  4. Comments can contain the keyword unit within brackets; this should be the first text following the standard fits comment delimiter "/", the single slash. Please remember to stay within the FITS 80-character line length limit, which includes the comments. E.g.:

    RA_TARG = 2.0952854139 / [deg] right ascension of target

Missing / Flagged / Mixed-Typed Values

Whether providing your data files in FITS or ASCII format, it is important that every timestamp have some value for every row. In the event that a particular row is missing a measurement value, that table cell must be filled in with a default missing value. For numbers (integers, floats, double precision, etc.) use the special value NaN. For columns that are strings, use the special value "NULL". more | less

In ASCII tables, you cannot mix numbers and string types in the same column (this is fundamentally not allowed in FITS tables). For example, a column called "FLUX" should not contain the float values 1.5, 56.98, 324.987 in some rows and the strings "None", "Bad" in other rows. We would encourage you to create a separate column called "FLUX_COMMENTS" with a string description of the data quality. Or, you can use a flag column ("FLUX_FLAG"); the mapping of the flag values can be described in the README file.

In order to denote an upper or lower limit within an ASCII or FITS table, you need to use two columns. There are two options available to implement upper/lower limits:

  • One column should contain numeric values and the other a string of the limit (e.g. FLUX=1.0, FLUX_LIMIT=">1").
  • One column should contain numeric values and the other a flag indicating the limit (e.g. FLUX=1.0, FLUX_FLAG=2, where the flag values are described in the README).

Back to top



KeywordDescription
  
             // DATA DESCRIPTION KEYWORDS
TELESCOP observatory (e.g. HST, IUE, ISO, VLT) [Expand TELESCOP values List | Retract/Close List]

INSTRUME single instrument alone, or instrument/detector names, or instrument/detector-subdetector names (e.g. "STIS" or "IUE" or "FOS" or "STIS/FUV-MAMA") or "MULTI" for composite [View Current List]
INSTRU[nn] if INSTRUME="MULTI", then an element from allowed values of INSTRUME [View Current List]
TARGNAMEtarget name (according to raw data or catalog); NULL allowed for multi-object case
RA_TARGright ascension of the target [deg] (J2000)*
DEC_TARGdeclination of the target [deg] (J2000)*
EPOCH Epoch of the observation (considered as deprecated; use EQUINOX when possible)
EQUINOX Equinox of celestial coord. system

* RA_TARG and DEC_TARG can vary for moving targets; put the position mean values in header and include the varying positions within data table.

  
             // DATE AND TIME KEYWORDS
DATE-OBS date and time of first observation in the ISO standard 8601 format: YYYY-MM-DDThh:mm:ss.sss
(DATE-OBS) Alternative, accepted date format (yyyy-mm-dd); must be paired with TIME-OBS
(TIME-OBS) Alternative, accepted time format (hh:mm:ss); must be paired with DATE-OBS (yyyy-mm-dd)
EXPTIME effective exposure time (seconds) for each measurement if constant, or "0" for variable (put values in data table) (which are typically in units of counts/second).
EXPSTART start time of first observation (same units as the time-series column of the data)
EXPEND end time of last observation (same units as the time-series column of the data)
  
             // For Tabular Data: FITS BINARY TABLE EXTENSION KEYWORDS
SIMPLE T / FITS standard
XTENSION Type of extension: BINTABLE or ASCII table
EXTNAME Extension name (single word)
(EXTNO) Extension number - recommended keyword
BITPIX number of bits that represent a data value
NAXIS 2 / denoting that the included data array is two-dimensional: rows and columns.
NAXIS1 the number of 8-bit bytes in each row of the table
NAXIS2 number of rows in the table
PCOUNT Random parameter count
GCOUNT Group count
TFIELDS Number of columns
TFORM[n] Format of column (valid data types)
TTYPE[n] Name of column (e.g. TIME, FLUX, FLUX_ERR)
TUNIT[n] Column units (e.g. Seconds, erg/s/cm^2/A, unitless)
  

For multi-extension FITS files (i.e. FITS Tabular data, time-series data), please note that many of the keywords cannot be in the 0th extension or HDU, they need to be within the corresponding extension(s) which contain the data.


RECOMMENDED FITS HEADER KEYWORDS


KeywordDescription
  
PROPOSID Proposal ID (when available)
HLSPLEAD Full name of the lead of HLSP project (Firstname M. Lastname)
PR_INV_L Last name of principal investigator
PR_INV_F First name of principal investigator
PR_INV_M Middle name or initial of principal investigator


OPTIONAL FITS HEADER KEYWORDS


KeywordDescription
  
HLSPNAME High Level Science Product (HLSP) project name, long form
HLSPACRN High Level Science Product (HLSP) project acronym, short form
CITATION citation reference of this project, for publications (else in HLSP README file)
DETECTOR detector of the instrument (same order as INTRUME) or "MULTI" for composite exposure
DETECT[nn] list of detectors of the instrument of composite exposure
APERTURE for HST, the name of the aperture of the instrument (i.e. STIS apertures)
FILTER filter for single exposure or "MULTI" for composite exposure [View Current List]
FILTER[nn] if FILTER="MULTI", filters list when composite exposure [View Current List]
POSANGL position angle
  

HLSP Time-Series Header Examples

Expand for Examples | Retract/Close Examples

MAST has been ingesting and distributing HLSP data products for over 10 years. During this period, the requirements for HLSP data deliveries have expanded in order to help unify all datasets housed at MAST for ease of multi-mission searching. The example HLSP headers may not abide by all the requirements listed above because they were delivered prior to some requirements being written. We encourage the data delivery teams to provide data sample so that all header and data issues can be worked out prior to the actual delivery for ingestion into the archive.

The following projects contain many examples of time-series, lightcurve and event-list file headers, which can be used as a guideline for your datasets:

  • FITS BINARY Table, Time-Series Lightcurve Examples with TIME, FLUX, ERROR (and additional columns)
    From the Kepler HLSP Project:

  • ASCII Table, Time-Series Lightcurve Example
    From the Kepler HLSP Project:

Back to general HLSP guidelines

Back to top