The Multimission Archive at STScI Newsletter
Table of Contents
The archive and data processing server infrastructure underwent a long overdue upgrade recently. Since 2003, the HST Data Archive and Distribution System (HST-DADS), and Operational Pipeline Unified System (OPUS) servers have been running on a SunFire cluster with Solaris and utilizing Sybase database servers. This configuration was also used for Kepler. This upgrade consisted of deploying new DELL PowerEdge servers running Red Hat Enterprise Linux and utilizing MicroSoft SQL Database Servers. The migration of HST-DADS and OPUS to this new configuration was a complicated task because it involved migrating the 20+ years of HST data and verifying that the new software produced products identical to the old configuration, taking into account the new software/hardware architecture. This involved months of testing with the instruments teams. Another large effort involved the verification and validation of the database servers. This involved verifying data content and query performance between the two systems. Finally, we ran the two systems in parallel ops for approximately 6 months, verifying functionality between the two systems and minimizing impact to our customers. In Sep 2011, the old HST system was shut off and decommissioned, with the Kepler system following shortly after, finishing up in Oct 2011.
To date the new systems are performing very well, distribution times have fallen to about 8 minutes on average for HST data. The new configuration provides us with a stable environment and room to grow in terms of storage and processing capacity. The figure to the right plots the Median Retrieval Times for each processing type since January 2011. The Y-Axis is the median time (ranging from 0-2 hours). The blue line represents the median retrieval time for data processed "On the Fly" or OTFR. The green line non-OTFR data. The red line is for all data. The dramatic drop in OTFR processing times in June 2011 corresponds with the migration of the production processing activities to the new infrastructure. Note that retrieval times include transfer time to the user in addition to the time required to process the data, so large requests transfered to a relatively slow node may skew the statistics somewhat. Click on the image to see a larger version of the plot.
A new algorithm that performs a Pixel-based charge transfer effect (CTE) correction is expected in the production pipeline in late Spring. Initially the algorithm will be available only for ACS data. When CTE is activated, the pipeline will produce two sets of data products for each set of ACS observations. In addition to the standard data products (_CRJ, _FLT and _DRZ files), there will be three new files (_CRC, _FLC and _DRC), which will contain the CTE-corrected data products. Both sets of products will be delivered to the users by the archive. For more details, see the ACS team page, http://www.stsci.edu/hst/acs/performance/calacs_cte/calacs_cte.html.
Most infrared arrays show after-images, known as persistence, following exposures that significantly saturate the detector. The mercury-cadmium-telluride detectors on the Wide Field Camera 3 (WFC3) infra-red channel (IR) are no exception. The after -images can last for several hours. As a result, the after-images can masquerade as stars or nebulae in exposures that follow IR observations of bright objects. Although after-glows that can affect the science quality of images are rare, observers need to account for them when analyzing WFC3/IR images.
The after-image contained in the image to the right is exceptional. Had it not been recognized as an after-image and publis hed, it would have certainly been embarrassing. Had the observation been an important part of the observer's program, a repeat observation of the field would have been needed. However, persistence at this level is rare. In 98% of the WFC3/IR image s taken over the six-month period from January through June 2010, less than 2% of the pixels were affected by persistence above 0.03 electrons s-1 per pixel. For 95% of WFC3/IR images, less than 0.5% of pixels were affected as this level.
To the right is an example of persistence. This image from the WFC3/IR was obtained in parallel with a spectrum from the Cos mic Origins Spectrograph of the active galactic nucleus Ton 580. The diffuse region in the center of the image is unlike any thing found near Ton 580. It is the after-image, due to the persistence of an image of the nearby spiral galaxy NGC 2841, wh ich the WFC3/IR had observed two hours earlier.
One way to eliminate the problem of persistence would be to space out WFC3/IR observations sufficiently to ensure that after-images had time to decay away before the next observation. Indeed, instrument scientists do attempt to identify potential "bad actors" — sources that could result in after-images—using information provided by observers or implicit in Phase II proposals. Typically observations with bad actors are observations of crowded star fields, such as globular clusters the Galactic plane, or the central regions of nearby galaxies. Further WFC3 observations are inhibited for two orbits after an observation on the bad-actor list is scheduled.
We have incorporated the persistence model into prototype software and have used the model to produce after-images for all existing WFC3/IR data. A more detailed description of persistence as it applies to the WFC3/IR channel, including advice on how to minimize its effects when planning observations, is available at http://www.stsci.edu/hst/wfc3/ins_performance/persistence/.
Although after-images are not part of the standard calibration products for WFC3/IR observations, they—and various related files—are available through the Multimission Archive at Space Telescope (MAST), at http://archive.stsci.edu/prepds/persist.
Observers can use persistence files in several ways. First, and most importantly, they can use the files to identify regions of the detector that are affected by persistence, so these can be excluded from the analysis. (Ancillary information on the MAST page allows users to see whether there is anything to worry about in the way of persistence.) Secondly, with caution, they can use the information to actually correct the data for persistence. The figure below shows an example of persistence correction.
Since the last Newsletter, there have been several changes and additions to the Kepler Archive.
Light curves have a new format. An APERTURE extension, showing the pixels observed and the pixels used in the photometry, was added. Some existing columns were renamed, including those for the fluxes. The names now reflect the content of the columns. Quality information and centroiding information are given in additional columns. The Archive Manual contains a description of the new format light curves.
Co-trending Basis Vectors, CBV, http://archive.stsci.edu/kepler/cbv.html are now available. These data products can be used to remove instrumental systematics from the Kepler light curves. There is one FITS file per quarter. See the Kepler Archive Manual for a description of the CBV file format, and see Data Release Notes 12 for more information on the CBV characteristics and how to use them. For a tool to fit and apply CBVs to your Kepler light curve, see http://keplergo.arc.nasa.gov/ContributedSoftwareKepcotrend.shtml. Users should download the November 30, 2011 version of these files.
The pixel response function, PRF, is now available at MAST on the Focal Plan Characterization Model page, http://archive.stsci.edu/kepler/fpc.html. This is the first of the Focal Plane Characterization Model files. The PRF model is comprised of five images per channel. These images describe the PRF in the four corners and the center of the channel. Each PRF was determined by using 10 stars near the corners (or at the center) of the CCD. For computational simplicity, each PRF may be assumed to be located exactly at the active CCD corner (or center). This approximation has negligible impact on the interpolated PRF across the chip. See the Kepler Archive Manual for more information.
FFIs delivered after November 2011 have improved WCS. More recent FFIs also have an uncertainty file. When present, uncertainty files are now included in the ftp directory. MAST expects earlier FFIs will be reprocessed and delivered at some future date.
The Kepler Spacecraft ephemeris, also known as a spice kernel, is being made available to users via anonymous ftp or via the browser at http://archive.stsci.edu/pub/kepler/spice . Since the ephemeris is accumulative, only the most recent file is available. See the kepler Ancillary Files page, http://archive.stsci.edu/kepler/ancillary.html, for more information and links.
The Kepler Project has accelerated the release of the Key Project (Planetary Search and Cluster observations) data from Kepler to the public. Please note that Guest Observer (GO) and Kepler Asteroseismological Science Consortium (KASC) data will continue to have a one year exclusive use period for data taken through Q13. The table below gives the new release dates for most Key Project data. Data for Key Project targets shared with a GO or KASC proposal will have a one year exclusive use period, which starts upon ingest of the data into the archive at MAST.
Various ground-based observing groups have surveyed the Kepler sky region and have made catalogs available to MAST that provide new filter magnitudes for objects in the Kepler field of view. A great many objects are not listed in the Kepler Input Catalog.
The new magnitudes can be accessed by requests to the Kepler Enhanced Target Search page, http://archive.stsci.edu/kepler/kepler_fov_enh/search.php Users should note that the addition of deep blue to far UV (GALEX) magnituders enable the selection of blue objects with greater reliability. The addition of the UK IRT J-band survey by Phil Lucas permits the selection of redder and fainter (chiefly extragalactic) objects.
Generally the Enhanced Target Search form behaves just like the current Target Search page, but it adds new filter magnitudes and colors. However, by clicking the "KIC targets only" button off users can extend their searches to a total of 10 million objects on the Kepler field rather than the KIC's 4.4 million. Within a few months MAST expects that the Enhanced form will supercede the current Target Search one. Note also that for more advanced queries a version of our "KeplerColors" catalog is available on MAST's Casjobs pages (http://mastweb.stsci.edu/kplrcasjobs/).
At this writing MAST is ingesting a UBV catalog of the Kepler field by M. Everett, S. Howell, and K. Kinemuchi. MAST will be receiving another catalog from the European "INT/KIS" Survey group this spring that lists magnitudes in u, g, r, i, and Halpha. The availability of magnitudes from these two surveys will be announced on the MAST news corner.
MAST has added a condition flag to the Kepler Target Search and Kepler Data and Retrieval Search interfaces. The new field, with a pull down menu, currently allows users to search for targets with one of seven conditions: Eclipsing Binary, Exoplanet Host Star, False Positive, Null Kepmag, Possible Artifact, Red Giant or Released Candidate. See http://archive.stsci.edu/kepler/condition_flag.html for more details.
The conditions are target-based, not observation-based, and may not apply equally to all entries in the Data Search interface.
A word of caution: The pull down menu contains an "All Targets" option. This is the default value used for queries and returns all targets regardless of the condition flag. A user wanting a list of all targets with an assigned condition flag can add the condition flag as a "User specified field" and put an "*" or "%" in the field.
Mission Closure (Maybe): As noted in last year's newsletter NASA has terminated operations of GALEX; the actual NASA termination date was deferred until mid-February. As of this writing the satellite is in officially in standby mode http://www.galex.caltech.edu/newsroom/glx2012-01f.html. However, there are hopes to keep its operations via another support mechanism. A further announcement can be expected very shortly; stay tuned to http://www.galex.caltech.edu/newsroom/releases.html. As part of the closure activities all scientific data products and documentation will be transferred to MAST.
GR7 deliveries: A GR7 release is coming to MAST in 2012, as usual in several stages. It will cover near-UV observations made in 2011. The GR7 areal coverage will include the Galactic cap, much of the low Galactic latitudes, almost all of the Small and Large Magellanic Clouds, additional regions in the Kepler field, and new visits to the M31. Observations made in calendar 2012 (until final shutdown) will most likely appear under a different name from GR7. The GR7 deliveries will proceed in the normal course of mission surveys, e.g. with the Medium Imaging and Nearby Galaxy surveys.
As the initial GR7 delivery, the project has delivered the so-called "raw6" survey. MAST plans to convert these files to photon list files early this year. This will set the stage for an on-line tool that will enable users to extract light curves for arbitrarily chosen sky regions or objects.
The GCAT delivery: Seibert et al. have announced the creation of two "GCAT" source catalogs (2012AAS...21934001S) called the All Sky Survey Source Catalog (GASC) and the Medium Imaging Survey Source Catalog (GMSC) These will be delivered to MAST in early 2012 and will form the final and unique identifications for sources detected by GALEX. As the source identifications will be (finally) unique, they will become the official reference for cross-lists for objects of other missions). Sources are included in the GASC or GMSC depending on their exposure times (<800 sec and 800-10000 sec, respectively). The Catalogs will extend to near-UV magnitudes of 21st and 23rd magnitudes, respectively, and the numbers of identified sources are 40 million (GASC) and 22 million (GMSC). A supplement to these catalogs based on GR7 products is currently planned.
MAST has constructed a tool that associates "best" spectral types in the literature with targets observed in MAST's spectroscopic missions. Users can access this tool by clicking "Spectral Classifications" in the Tools tab at the top of the MAST home page or by navigating directly to http://archive.stsci.edu/spec_class. The tool is an interface form called "Spectral Classes of Like Stars." It allows queries for stars' spectral types, luminosity classes, and as many as 4 of the many spectral peculiarities appropriate to the queried spectral type. Ranges in spectral type are also permitted. The retrieval page gives the precise spectral classification for a star matching the query, a bibliographic reference for the classification, and details on how the match was made from the input star name (or coordinates) to classification catalogs. In addition, MAST now offers a tool that retrieves spectral types from the latest version of Brian Skiff's catalog of spectral classifications.
The Spectral Classes form comes with Tips, field descriptions, and help pages that enable users to customize and expand the usefulness of queries with minimal effort. One particular use case is to ask for all spectra in the MAST database for a given spectral classification. Although this tool facilitates research on individual stars, perhaps its forte is for use in compiling lists of stars and spectra for use in spectral atlases, or as a source of what is still missing in our archives - and therefore as a reference source supporting proposals for new UV/optical observations.
MAST and the HST Office of Public Outreach have collaborated to create a database of HST press release data. The images included now contain Astronomy Visualization Metatags including the World Coordinate System. The database includes observatory and science observation information and links to related information and the original hubblesite.org press release. You will find full resolution tiff and jpeg format images containing metadata headers.
You can find them through the following links:
This service provides easy access for:
MAST is developing a new web application for finding and retrieving data from all the MAST missions. This Data Discovery Portal will provide the following features through a single web interface:
In order to make this tool as useful as possible, MAST is requesting your feedback on our initial progress. Please watch the video or try out the tool and send us suggestions for improvements and new features. Note that this is an early release for demonstration purposes only. Some links may not be active, and the ability to download data products has not been enabled.
As new features are developed, we would greatly appreciate continued feedback. If you are interested in helping to guide this project as a beta tester and/or with feature requests, please contact us at email@example.com.
MAST welcomes contribution of high-level science products (HLSP) from the astronomical community. HLSP are community contributed, fully processed (reduced, co-added, cosmic-ray cleaned etc.) images and spectra that are ready for scientific analysis. HLSP also include files such as object catalogs, spectral atlases, and README files describing a given set of data. To find data two types of searches are provided: a Project search and a Product Search
In the past few months several other teams have contributed science ready high-level science products for retrieval through the archive.
Other sets of HLSP that have been archived at MAST include:
The MUG met on September 23, 2011. The presentations and the report from the MUG are also posted online.
To Subscribe or Unsubscribe to the NewsletterIf you would like remove yourself from the mailing list send mail to firstname.lastname@example.org and put the single word unsubscribe in the BODY of the message. Information on the subject line will be ignored.
If you would like to subscribe to the mailing list to receive notification of future Archive Newsletters, send mail to email@example.com and put the single word subscribe in the BODY of the message. Information on the subject line will be ignored. You will be asked to confirm the subscription.