link to MAST page MAST star image
MAST STScI
Tools
Tutorial Site Search
FAQ
High-Level Science Products
Software
FITS
Archive Manual
Related Sites
NASA Datacenters
MAST Services
MAST and the VO
Newsletters & Reports
Data Use Policy
Dataset Identifiers
Acknowledgments
 
link to STScI page


MAST Web Services

HTTP GET Requests
    - Script Examples
    - Mission Searches
    - Simple Cone Search
    - Simple Image Access Protocol
    - Simple Spectral Access Protocol
RSS News Feed
Soap Services

HTTP GET Requests

MAST data holdings can be searched not only using HTML search forms and applications such as Aladin, Datascope, Specview, etc., but also using HTTP GET requests. The GET request allows the search parameters to be included in the URL. As such, they can be called from within programs to automate data searches. The results can be returned in a variety of formats including HTML, VOTable XML format, excel spreadsheet, and comma-separated values which can simplify ingesting results into user-written programs. In addition, submitting GET requests can bypass restrictions currently placed on the web search forms (e.g., restrictions on the max_records value). Unfortunately, there has been little documentation on how the GET requests are formulated. This document will summarize the services currently available from MAST and give examples showing how they can be used.

Remember, when creating URLs, special characters must be encoded. URLs may only use alphanumerics [0-9a-zA-Z], the characters "$-_.+!*'()," and special reserved characters. The characters "<" and ">" must be specified as "%3C" and "%3E" respectively. For example, to specify a parameter such as RA > 120, the GET request parameter would be RA=%3E120. Blanks can be specified using the "+" character.


Script Examples

Scripts for retrieving MAST search results can be written in many programming languages. Basically you just need to download and extract information from a web page. A few simple examples are shown below. We welcome more examples from users.


Mission Searches

In general, mission searches are specified in the form: http://archive.stsci.edu/[data set]/search.php?action=Search&params
where [data set] is the MAST-defined acronym for a particular set of data, and params is the list of search parameters as defined below. Note that "action=Search" is a required parameter for mission search requests. For the pointings search however, action may equal either "Search+Pointings" or "Search+Exposures" The current list of data sets that can be searched in this manner include the following:
  • HST - Hubble Space Telescope
  • Kepler - Kepler Extrasolar planet search mission
  • IUE - International Ultraviolet Explorer
  • HUT - Hopkins Ultraviolet Telescope
  • EUVE - Extreme Ultraviolet Explorer
  • FUSE - Far-UV Spectroscopic Explorer
  • UIT - Ultraviolet Imageing Telescope
  • WUPPE - Wisconsin UV Photo-Polarimeter Explorer
  • BEFS - Berkeley Extreme and Far-UV Spectrometer
  • TUES - Tübingen Echelle Spectrograph
  • IMAPS - Interstellar Medium Absorption Profile Spectrograph
  • HLSP - High Level Science Product
  • Pointings - HST Image Data grouped by position
  • Copernicus - Copernicus Satellite
  • HPOL - ground based spetropolarimater
  • VLA-FIRST - VLA Faint Images of the Radio Sky (21-cm)
  • XMM-OM - X-ray Multi-Mirror Telescope Optical Monitor
The search parameters include both general and mission-specific. and are described in the following pages:
In general, any parameter listed on the HTML search form can be specified in a GET request. If you are parsing the results, it would be easier and probably faster, to request the search results in CSV or VOTable format using the "outputformat" parameter.

Mission search examples:

  • Search the HST archive for observations within the specified radius (in arcminutes), centered on the given RA and Dec values (in decimal degrees), allow a maximum of 1000 records, and output the default columns as comma-separated values: (Try it)
    http://archive.stsci.edu/hst/search.php?RA=53.084&DEC=-27.873&radius=100. &max_records=1000&outputformat=CSV&action=Search

  • Search the Kepler archive for the first 1000 files (sorted by Kepler ID) that have become public in 2009, that was part of the EX program, and output comma-separated values with RA in decimal hours: (try it)
    http://archive.stsci.edu/kepler/data_search/search.php? ads_release_date=%3C2010&ktc_investigation_id=EX&max_records=1000&coordformat=dechr&outputformat=CSV&action=Search

  • Do similiar cone search for XMM-OM data with a search radius of 5 arcminutes show sql query, sexigesimal coordinates, and output results in HTML (this type of request is good for testing) (Try it)
    http://archive.stsci.edu/xmm-om/search.php?RA=53.084&DEC=-27.873 &radius=5.&showquery=on&action=Search

  • Search the HST archive for pep_id 9293, allow a maximum of 1000 records, and output results as comma-separated values: (Try it)
    http://archive.stsci.edu/hst/search.php?sci_pep_id=9293 &max_records=1000&outputformat=CSV&action=Search

  • Search HST as above, but return results in VOTable format (Try it) http://archive.stsci.edu/hst/search.php?sci_pep_id=9293 &max_records=1000&outputformat=VOTable&action=Search

  • Search HUT database for observations of M81 and return only columns for target name and data id: (Try it)
    http://archive.stsci.edu/hut/search.php?action=Search&target=m+81 &selectedColumnsCsv=hut_target_name,hut_data_id

  • Search the ACS Pointings table for the target "M 81" with output format = "CSV", and greater than 2 exposures in the U band: (Try it) http://archive.stsci.edu/pointings/search.php?pnt_ucount=%3E2&target=m+81 &primary=ACS&outputformat=CSV&action=Search+Pointings
Notes:
  1. Be sure to include "action=Search" (with capital S) in the mission searches, but not for SCS or SIAP requests.
  2. Parameter order is not important.
  3. For HST searches, sci_aec is set to "S" by default. This will return Science (S) but not Calibration (C) data. If both are desired, add the param sci_aec=%.
  4. Use a comma separated list for selected columns with the selectedColumnsCsv parameter. Don't use selectedColumnsList as seen from the search form.
  5. Parameters such as resolver=don'tresolve should probably be encoded to be resolve=don%27tresolve. It seems to work without it though. (See PHP command called urlencode).
  6. There is currently a 300 second execution time limit on archive.stsci.edu which can cause large search requests to abort. Contact MAST if you run into this problem.

Simple Cone Search (SCS)

MAST missions can also be searched using the NVO Simple Cone Search protocol (SCS). The base url is the same as for MAST mission searches (i.e., http://archive.stsci.edu/[data_set]/search.php) without the "action=Search" parameter. The search radius parameter "SR" must be included in the URL for the cgi script to recognize a simple cone search is being requested. Besides the search radius in decimal degrees, RA and DEC parameters are also required to perform a cone search. The results are returned in VOTable format. The SCS protocol can be used with any MAST mission and the general MAST parameters can be included as well. Examples:
  • This example will return about 50 HST observations (in VOTable format) at RA,Dec = 53.084,-27.873 and a search radius = 0.01 degrees http://archive.stsci.edu/hst/search.php?RA=53.084&DEC=-27.873&SR=0.01

  • Same as above except use a larger search radius (0.02) but only allow a maximum of 100 records to be returned http://archive.stsci.edu/hst/search.php?RA=53.084&DEC=-27.873&SR=0.02&max_records=100

  • To find all high level science products at the same coordinates: http://archive.stsci.edu/hlsp/search.php?RA=53.084&DEC=-27.873&SR=0.01

Simple Image Access Protocol (SIAP)

MAST missions can also be searched using the NVO Simple Image Access protocol (SIAP). The base URL is http://archive.stsci.edu/siap/search.php
followed by the SIAP parameters. Like the Simple Cone search, the base url is similiar to that for mission searches however only missions with image data are searchable. By default, up to 300 entries will be returned but specifying max_records=nnn can override the default. Adding the parameter "representative=y" will return only "representative" images as defined for the MAST scrapbook tool. Currently the list of instruments, and High Level Science Products (HLSPs) that can be queried using the SIAP include:
  • WFPC - HST Wide Field Planetary Camera
  • WFPC2 - HST Wide Field Planetary Camera 2
  • NICMOS - HST Near Infrared Camera and Multi Object Spectrometer
  • STIS - Space Telescope Imaging Spectrograph
  • FOC - HST Faint Object Camera
  • ACS - HST Advance Camera Survey
  • UIT - Ultraviolet Imageing Telescope
  • UDF - HLSP Ultra Deep Field
  • HDF - HLSP Hubble Deep Field
  • HDF_SOUTH - HLSP Hubble Deep Field South
  • GOODS - HLSP The Great Observatories Origins Deep Survey
  • TNO - HLSP trans-Neptunian objects
  • HELIX - HLSP HST Helix Observations
  • MAOZ_ATLAS - HLSP HST Atlas of Ultraviolet Images of Nearby Galaxies
SIAP Examples:
  • Search for MAST images at RA+151.0, Dec=+69.0 and a search area of 0.5 degrees: http://archive.stsci.edu/siap/search.php?POS=151.0,69.0&SIZE=0.5

  • Display supported SIAP parameters:
    http://archive.stsci.edu/siap/search.php?format=metadata

  • Find WFPC2 observations at RA,Dec = 53.084,-27.873, format = GIF, and a rectangular search area 0,05 degrees wide: http://archive.stsci.edu/siap/search.php?POS=53.08,-27.873 &SIZE=0.05&FORMAT=gif&ID=wfpc2

  • Find "representative" WFPC2 observations as above: http://archive.stsci.edu/siap/search.php?representative=y&POS=53.08,-27.873 &SIZE=0.05&FORMAT=gif&ID=wfpc2


Simple Spectral Access Protocol (SSAP)

Most MAST spectral missions can now be searched using the VO Simple Spectral Access Protocol (SSAP). It is similiar to the SIAP standard in that it returns a VOTable-format file with links to actual spectral data sets.

The SSAP standard has evolved over the last couple of years, and today there are services that follow 2 slightly different standards:

  • The original standard, now known as version 0.4 or 0.5 in the NVO registry, was never officially approved ( see: early proposed SSAP) but was adopted by several data centers before the oficial version was available. Curerntly we have SSAP version 0.5 services available for HUT, FUSE, EUVE, WUPPE, IUE, HPOL, and HST.
  • The first officially-approved standard is now available and is known as version 1.0 (actually the latest version is 1.04). MAST missions supporting version 1.0 currently include only HUT and FUSE but more will be added soon (see SSAP paper and Spectral Data Model papers for description of protocol).

Since we expect the earlier version to eventually go away, the examples below only describe version 1.0 services.

SSAP Examples:

  • Search for FUSE spectra at RA+151.0, Dec=+69.0 and a search area of 2.0 degrees: http://archive.stsci.edu/ssap/search2.php?id=FUSE&POS=151.0,69.0&SIZE=2.0

  • Search for all MAST spectra at RA+151.0, Dec=+69.0 and a search area of 0.5 degrees: http://archive.stsci.edu/ssap/search2.php?POS=151.0,69.0&SIZE=0.5

  • Display supported SSAP parameters:
    http://archive.stsci.edu/ssap/search2.php?format=metadata

  • Search for FOS spectra at RA+151.0, Dec=+69.0 and a search area of 2.0 degrees: http://archive.stsci.edu/ssap/search2.php?id=FOS&POS=151.0,69.0&SIZE=2.0

RSS News Feed

Another MAST web service is our RSS news service. This is a list of our most recent news items which can be requested in HTML or RSS XML-format:
  • http://archive.stsci.edu/mast_news.php?out=html&desc=t&last=10
    shows last 10 entries (by date) in html with descriptions
  • http://archive.stsci.edu/mast_news.php?desc=t&last=10
    shows same in rss xml format
  • http://archive.stsci.edu/mast_news.php?out=html&last=10
    summary in html of last 10 (no descriptions)
  • http://archive.stsci.edu/mast_news.php?out=html&id=25
    show entry 25 in html

SOAP Services

We have one SOAP-based web service which is accessed via the ADS. The web service allows users to enter a data set name and find where it is archived. The service can be accessed from the web form at:
http://ads.harvard.edu/ws/DataVerifier, or, using the latest ADEC naming conventions:
http://vo.ads.harvard.edu/dv/DataVerifier.cgi


Last Modified: Nov 17, 2009 12:03