The Zwicky Transient Facility: Data Processing, Products, and Archive

The ZTF camera consists of 16 CCDs, each with ∼6000² pixels. Furthermore, each CCD consists of four ∼3000² pixel amplifier channels with overscan (bias) strips. We shall refer to these amplifiers as CCD-quadrants from hereon. Therefore, each camera exposure consists of 64 CCD-quadrants. The raw data files received at IPAC are CCD-based, where each file contains the already-split CCD-quadrants and corresponding overscans, each packaged into separate extensions of a compressed multi-extension FITS file (MEF; Section 3.3).

A CCD quadrant defines the basic image unit for concurrent (parallel) processing in the ZSDS and from which all data products are derived. This includes all calibrations, processed science images, difference images, reference (co-added) images, source catalogs, and light-curve files. All file products are CCD-quadrant based. There are no cross-quadrant or cross-CCD-dependent processing steps. All quadrants are instrumentally calibrated and treated independently end-to-end.

This CCD-quadrant-based processing approach is primarily driven by our experience with PTF. The ZTF CCD quadrants are of approximately the same size (in number of pixels and areal coverage) as the PTF CCDs. First, as was the case for PTF, most ZTF data products are distributed as flat data files (Section 4). Images this size generate file products (which include source catalog tables) that are easier to manage, analyze, and distribute using existing software and technologies. Second, there are processing steps that are determined to be more robust and accurate (in terms of reducing systematics from spatial variations) if performed as local as possible (spatially) on the P48's 7° × 7° focal plane. For example, astrometric calibration, PSF derivation, photometric-zero-point derivation (Section 3.5), and PSF-matching prior to image differencing (Section 3.6). The CCD-quadrants are large enough to admit sufficient numbers of sources to optimize these steps (in the signal-to-noise (S/N) sense), but small enough to keep memory use, file product sizes, and hence processing throughput at a manageable level all over the sky during processing.

Following splitting of the incoming raw data MEF files into CCD-quadrant images (Section 3.3), they are processed on a 66-node compute cluster. The cluster is composed of commodity 1U dual Xeon servers; 64 outfitted with 128 GB RAM and two high-memory nodes with 768 GB each. The latter are intended for executing ZTF's Moving Object Discovery Engine (ZMODE pipeline; Section 3.9), which performs memory intensive moving-object detection and track generation. The specifications of the compute cluster are summarized in Table 1. We have 1192 (16 × 32 + 34 × 20) physical processor cores across all compute nodes.

Due to data IO load limitations on the databases and archive file servers, we found that eight pipeline instances per node (or concurrent threads, where one thread represents the processing of one CCD quadrant) is optimal and sufficient for processing the incoming data rate in real time, with no latency in the delivery of products (see Section 7.1). By "optimal", we mean maximizing the product generation rate. Therefore, we have in total 528 (8 × 66) concurrent threads processing the incoming data, which arrives at a rate of one exposure/40 sec or equivalently, 64 quadrants/40 sec or 96 quadrants/minute on average. A single quadrant image can be processed in under five minutes on average in terms of wall-clock time (Section 7.1). Therefore, a naïve estimate for the number of processor cores needed to match the incoming data rate is ≃${inp}\_{rate}/{proc}\_{rate}=96/(1/5)=480$. This is really a lower limit, as the processing time is not purely CPU, i.e., disk and network IO are not negligible. Given our good network bandwidth and disk speeds however, we have more than enough cores to handle the incoming data rate.

It is important to note that we are currently underutilizing our compute resources and hence have the capacity to do more. We have a total of 1192 physical cores, but we are currently using less than half. Additional nodes were purchased early in the survey to support ad hoc processing tasks, such as reference-image generation and moving-object detection. These tasks run asynchronously and not during real-time (nightly operations). Requirements on real-time processing and product delivery timescales are being met; however, there is a desire to increase the overall throughput. We are in the process of increasing the number of processor cores for real-time processing. This exercise entails exploring the additional load on the database and archive ingestion servers to ensure the higher production rates can be sustained throughout a night.

The compute cluster runs CentOS7 and is managed using the SLURM²⁵ resource manager (packaged by the OpenHPC²⁶ project). The cluster runs a diverse set of workloads, and is configured with a number of partitions having different polices to maximize throughput. The nightly real-time processing runs in one of the highest-priority partitions, before almost all other pending work.

The Kafka (see footnote 6) cluster consists of an additional set of three compute nodes, each configured with ≈3 TB of local storage. This cluster facilitates the distribution of transient-alert packets to other institutions and eventually the community (the ZADS framework; Section 4.1). Currently, the IPAC Kafka cluster is mirrored close to real time with Kafka running at the University of Washington (UW) over the Internet.

ZTF-Depot is a private webserver to facilitate access to pipeline QA metrics, survey performance statistics, and survey depth-of-coverage maps (both cumulative and nightly, and movies thereof). It also serves as a portal to support the retrieval of products related to solar-system science and NEA discovery in near-real-time: primarily the outputs from streak detection (Section 3.6) and ZMODE (Section 3.9). All other pipeline products destined for public release are copied to the long-term archive at IRSA. These will be accessible through a separate webserver using either a graphical user interface (GUI) or application programming interface (API; Section 5).

The ZSDS pipelines are heavily reliant on a relational database for querying and storing the state (usability) of all intermediate and final file products. This includes QA metrics and observing system diagnostics for long-term trending and analysis. All DB servers in the ZSDS run PostgreSQL 9.6²⁷ , and the primary pipeline operations DB server is equipped with 384 GB of memory.

Because the operations DB is not easily scalable in terms of IO or computing load, careful design and tuning of both the hardware and software were necessary. Database tables and indexes are arranged so that heavily accessed data are stored on solid-state devices (SSDs) as opposed to spinning disks. The higher memory of the operations DB server ensures a relatively high cache-hit rate. The largest DB table stores the raw transient candidates. This table is partitioned into child tables, with each pointing to a specific observing date and CCD quadrant. A set-up process that creates DB views over the correct candidate partitions is executed daily.

Database snapshots are copied to a network-attached file system, and snapshots covering four weeks of recent content are kept and available at any given time. Tape backups of the snapshotted data are separately made on a regular schedule, and moved to an off-site repository. The primary operations DB is also continuously replicated into a separate secondary DB for offline use. This secondary DB allows for analyses to be performed without impacting the performance of the primary DB during operations.

A separate relational DB keeps track of distilled archive product metadata on the IRSA (archive) side. This DB is used for user queries to search for specific archived file products (Sections 4 and 5.1). Another relational DB on the IRSA side is used to store source photometry and metadata extracted from the reference image co-adds (Section 3.7) with additional statistics computed by the source-matching pipeline (Section 3.8). This is referred to as the ZTF Objects Database and is queried to support the retrieval of light curves (multi-epoch photometry) using a file-based look-up (Section 5.3). There is no separate DB table that stores the contents of the epochal source catalog files (Section 4). This is because such a table will approach of order a trillion rows at the end of the nominal three-year survey. This was determined to be too costly to develop and maintain given the allocated resources.

Most of nightly processing and end-of-night tasks are orchestrated by an automated "watchdog" script referred to as the Virtual Pipeline Operator (VPO). This script checks for incoming raw data, creates all date-dependent paths in the operations, and ZTF-Depot filesystems, executes the Ingest pipeline (Section 3.2), and then the relevant calibration-generation or real-time science pipelines in accordance with the priorities programmed into the VPO. The VPO also accumulates QA statistics and number of transient-alerts extracted at regular intervals throughout the night and posts them to ZTF-Depot.

At the end of a night, the VPO performs accountability checks on the number of raw data files sent from the P48, received, successfully ingested (with all required metadata), split, processed, and archived. All statistics and metadata are accumulated in an end-of-night report and emailed to members of the project. Following nightly processing, the VPO triggers the ZMODE (moving-object) pipeline (Section 3.9); checks which fields and CCD quadrants have enough good-quality exposures to warrant reference image creation (if a reference image does not already exist) and, if so, triggers the reference-image pipeline (Section 3.7) and updates the survey all-sky depth-of-coverage maps. The VPO then prepares the processing system for the forthcoming night by copying and partitioning the contents of the transient-candidates DB table into files to facilitate faster source matching with historical detections in real time when generating alert packets.

Other automated daily maintenance tasks involve monitoring disk space, removing old intermediate products from the operations file system, and database "vacuuming." Manual tasks involve monitoring all ZSDS hardware, archive access loads, throughput performance, tending to ad hoc reprocessing requests, and communicating any fatal system errors or outages to the project. Longer-term manual tasks involve executing the light-curve (source-matching) pipeline (Section 3.8) on typically monthly timescales, and performing synopses and quality checks of the reference-image holdings.

The ZTF camera generates 16 CCD-based raw data files per exposure of the focal plane (productID 1 in Section 4; see also Section 2.1). Prior to packaging by the camera software, the data associated with each exposure (all CCDs) are ∼1.3 GB in size. This includes all exposure, telescope, CCD-image, and overscan metadata. The camera software digitizes its native output into 16-bits per pixel. Furthermore, exposures are separated by typically 40 sec (30 sec integration +≃8 sec readout +≃2 sec overhead).

The individual CCD-based data files are first transmitted from the P48 to the San Diego Supercomputing Center (SDSC) via the HPWREN²⁸ microwave network. This link is designed to connect remote areas that are not well served by other technologies. Palomar Observatory is the largest user of bandwidth on this network. At the start of commissioning, the HPWREN bandwidth was rated at ≲200 Mbit/s following overheads and variability (see below). If the raw camera data were immediately transmitted following acquisition (see above), a bandwidth of at least 1.3 GB/40 sec ≃ 260 Mbit/s would be required to transfer all data bits in real time. Furthermore, this bandwidth would need to be sustained over the course of a night. Therefore, to accomodate the HPWREN bandwidth with some margin for variability, the raw CCD data files are first compressed by factors of ≈2 to 2.5 at the P48. This compression is slightly lossy and based on the Rice method as implemented in the fpack²⁹ utility. The loss in information due to compression is negligible.³⁰ Following transmission to the SDSC, the files are pushed to IPAC over the internet.

The performance of the microwave link varies with other usage of the network and weather, particularly if there is an inversion (cloud) layer below the altitude of the P48. Long-term monitoring has shown a median transfer time of a complete exposure of ≃20 sec with 90th percentile around 24 sec and 99th percentile at 40 sec. Therefore, exposures are typically transferred to IPAC within the observing cadence without any backlog.

Upon reception of the raw CCD-based files at IPAC, their filenames and headers are checked for correct formatting and that all the required metadata to facilitate downstream processing are present and within range. The CHECKSUM and DATASUM keywords are also verified in all FITS HDUs. If any of these checks fail, the CCD data file is tagged using a special status flag for future traceability and declared unusable. Such files are not copied to the archive. If a file passes all checks, it is copied to the long-term archive, additional image/exposure identifiers computed, and relevant metadata from the primary header, along with its archived path/filename are stored in the operations database.

At the time of writing, the ingest failure rate is 16 CCD files (or one exposure worth of image data) per night. These are due to timing glitches in the observation start time relative to the "shutter open" time. This is understood and a solution is in progress.

Following ingestion into the archive and operations database, the raw CCD file is decompressed using the funpack utility. This creates a multi-extension FITS (MEF) file with pixel values converted from 16-bit integers to 32-bit floating point. The MEF file is then split to extract the four light-sensitive quadrant images together with their headers. The primary HDU containing exposure-level information is prepended to the header of each quadrant image so it can be propagated to all derived image products downstream.

The four corresponding bias-overscan images are also extracted from the MEF file. Each overscan image is used to compute a floating-bias correction using a second-order polynomial fit. This correction is then applied to the respective light-sensitive quadrant image. Statistics are computed on the split quadrant images, their bias-overscans, and polynomial-fit residuals. These metrics are stored in the operations database and the overscan-corrected quadrant images are copied to the pipeline operations file system for later retrieval. All raw CCD files received at IPAC, which includes data intended for calibrations (i.e., biases and flat fields) and engineering-related testing are split into individual quadrant images and overscan-corrected. These preprocessed intermediate quadrant images are fed into downstream pipelines and not copied to the long-term archive.

There are two separate pipelines that generate calibration image products for use downstream: one to process and stack (zero-exposure) bias images, and another to process and stack exposures of an illuminated "flat screen" to generate relative pixel-to-pixel responsivity maps (high-spatial frequency flat-field images). All processing and stacking occurs at the CCD-quadrant level.

3.4.1. Bias-image Generation

3.4.2. High-frequency Flat-field Image Generation

Instrumental calibration is the first phase of the real-time processing pipeline. Inputs are: a single overscan-corrected CCD-quadrant image in a given filter (Section 3.3); a prior bad-pixel mask; closest-in-time (most recent) calibration images (Section 3.4); lists of prefiltered calibrator sources to support astrometric and photometric calibration covering the CCD-quadrant footprint; and processing parameters. The primary products from this pipeline are productIDs 2, 3, 4, 14 in Table 2 (Section 4).

The processing steps are depicted in Figure 3. In summary, this pipeline applies the calibration image corrections (biases and flats; Section 3.4); it corrects for detector non-linearity; it masks aircraft and satellite streaks (using the CreateTrackImage software described in Laher et al. 2014); it performs astrometric calibration (see Section 3.5.1); it masks ghosts from bright sources; it derives PSFs and generates a PSF-fit photometry catalog using a version of DAOPhot (Stetson 1987) optimized for ZTF. This catalog includes the added benefit of de-blending by fitting to multiple detections simultaneously in a two-pass PSF fit. The pipeline then performs photometric calibration (see Section 3.5.2); generates an aperture-based photometry catalog using SExtractor (Bertin & Arnouts 1996); detects and masks cosmic rays; generates an image of pixel uncertainties; computes image QA metrics and statistics on the source catalogs; assigns an overall quality flag with a bit-string recording the details of processing, and stores these in the operations DB; archives the science image product, accompanying bit mask, and source catalogs.

Zoom In Zoom Out Reset image size

These products can be immediately used as input to the image-differencing (second) phase of the real-time pipeline if other inputs are available (see Section 3.6). Otherwise, the real-time pipeline will terminate and only the science image and catalog products are archived. Having the instrumental calibration steps in their own standalone pipeline also makes reprocessing or testing flexible, i.e., if only the epochal science products are to be replaced.

3.5.1. Astrometric Calibration

3.5.2. Photometric Calibration

Image differencing and event extraction (defined as either candidates for point-source flux transients, SSOs, or streaks from fast(er) moving SSOs) are performed in the second phase of the real-time pipeline, following instrumental calibration (Section 3.5). Image differencing uses as input the epochal CCD-quadrant-based outputs from instrumental calibration, f as well as processing and configuration parameters. It is automatically triggered if a usable (survey-ready) reference image with accompanying PSF-fit photometry catalog (Section 3.7) is available. The primary products from this pipeline are productIDs 5, 9, 12, 14 in Table 2 (Section 4). Alert packets (productID 9) were described in Section 4.1.

The processing steps are depicted in Figure 4. The first stage of this pipeline consists of preparing the image inputs prior to generating the difference image products. In summary, this step involves using a subset of sources from the input reference and epochal (science) PSF-fit photometry catalogs to match the photometric throughputs of the corresponding images; resamples and interpolates the reference image onto the science image using SWarp (Bertin et al. 2002); masks all bad pixels propagated from the science and reference images; computes a smoothly varying differential background image and subtracts this from the science image. Pixel-uncertainty images and PSFs for the science and reference images are then generated. PSF-matching and image differencing are then performed using the ZOGY algorithm (Zackay et al. 2016). The primary products from this step are the actual difference image; an accompanying match-filtered S/N image optimized for point-source detection; and an effective PSF for the difference image.³² QA metrics are then derived to assign a usability flag for the difference image. These metrics are thresholded to determine if point-source events should be extracted. That is, the difference image is of sufficient quality as to not strain downstream processing of the event stream due to too many unreliable extractions.

Zoom In Zoom Out Reset image size

If event extraction is to proceed, the second step of this pipeline detects events from the point-source match-filtered S/N images where detection is performed on both the positive (science minus reference) and negative (reference minus science) images. Note that the negative images are simply −1× the positive images generated by a single run of the ZOGY software. Events are extracted with both aperture and PSF-fit photometry, and additional source features are computed to support filtering and automated ML-vetting downstream. The events are then lightly filtered to remove obvious false positives, the point-source ML classifier is executed to assign RealBogus reliability scores, and image cutouts are generated. Positional association with the nearest known SSO within some specific radius is then performed using the astcheck³³ utility and associated metadata for that object is stored. Association with the nearest reference-image-detected source is also performed.

Processing then proceeds with the detection of linear streaks in the positive difference image using the FindStreaks software (Waszczak et al. 2017). A simple streak model consisting of a 2D Gaussian representation for the PSF convolved with a line is then fit to each streak (Vereš et al. 2012). This fitting returns estimates for the streak endpoint positions; length; position angle; integrated flux; corresponding uncertainties and covariances; and goodness-of-fit metrics. These metrics are used by the streak ML classifier to assign RealBogus reliability scores. These scores are then thresholded to retain likely real candidates from which image cutouts are also generated. The automated ML-vetting steps for both point sources and streaks are described in more detail in Mahabal et al. (2018).

All point-source and streak events, their features, and image QA metrics are stored in the operations DB. The difference image and its PSF are archived and streak metadata with cutouts are pushed to ZTF-Depot for human scanning. A post-processing step gathers all the the point-source event metadata and image cutouts, queries the DB for additional metadata, including cross-matching to the nearest PS1 sources, and then generates an alert packet for each thresholded event (see Section 4.1 for details).

A reference image provides a static representation of the sky, or more specifically, a historical snapshot as defined by the state of the sky recorded in previous image exposures for a given field, CCD quadrant, and filter. The reference image is a benchmark against which future exposures can be compared (i.e., differenced; Section 3.6) to assist with the discovery of flux transients or moving objects. A catalog of sources extracted therefrom also provides a "seed" for source matching and light-curve generation (see Section 3.8). The primary products from this pipeline are productIDs 6, 7, 14 in Table 2 (Section 4).

A reference image is a co-add (a stack-average; see below) of ≥15 (minimum) to 40 (maximum) high quality historical CCD-quadrant images. The minimum of 15 was driven by the need to produce as many reference images as possible over the sky early (≈two weeks) into the survey to trigger image differencing and validate the transient-discovery science programs. This minimum was optimal in that it accounted for the availability of "good" quality input images (see below), gave maximal sky coverage, and yielded good quality co-adds usable for image differencing with depths ≃1.5 mag deeper than the single-exposures. The maximum of 40 was picked to match the image depths from longer camera exposures (up to 300 sec) required by some of the partnership science programs. A large fraction of the reference sky is likely to acquire this maximum depth following bulk regeneration of the reference image library, which has not yet occurred. The maximum of 40 is still somewhat arbitrary and may be higher following a reanalysis of the quality of the current single-exposure image holdings, the quality of the difference-images produced, and transients extracted.

The input images are selected to satisfy a number of quality criteria, primarily images with good astrometric and photometric calibrations, good image quality (point-source FWHM), and robust estimates of pixel noise and background levels falling within specific ranges for each filter. The photometric quality check includes thresholding the calibration zero-point, color term, and limiting magnitude (Section 3.5). Initially, only images satisfying a derived image quality of 20 ≤ FWHM ≤  35 are selected. This ensures that most of the inputs are better than Nyquist-sampled (given the ≃1'' pixel scale). The input images are then ordered according to ascending FWHM and the N^th best quality images (where 15 ≤ N ≤ 40; see above) are retained for co-addition. Due to the large number of epochal images from which to draw from, this operation results in most input images spanning a relatively narrow range in FWHM; typically 20 ≤ FWHM ≲ 24. This restriction in PSF size mitigates inadvertent clipping of pixels containing source signal on and around point-sources in the outlier-trimming step downstream due to the variable PSF across epochs (see below). For more details on the selection criteria for each filter, see the ZSDS Explanatory Supplement (referenced in Section 1). Other inputs include an accompanying list of bad-pixel masks and processing parameters.

Reference image generation is automated by the pipeline executive (VPO; see Section 2.3) and performed at the end of each observing night. A database query is executed to check which distinct combinations of field, CCD quadrant, and filter which were observed the preceding night, and that do not already have an archived reference image, warrant one. That is, do enough good-quality images exist for the given field, CCD quadrant, and filter, accounting for historical images as well? If the minimum number (=15; see above) is satisfied, the reference image pipeline is triggered. The reference image products therefrom are then archived and "locked", i.e., not regenerated unless determined to be of too-low quality for survey use, or until the next round of bulk reference-image regeneration. The input maximum of 40 (see above) pertains to possible future regeneration of reference images by considering all archived historical images.

The first step in the reference-image pipeline consists of regularizing (preconditioning) the input images and creating the co-add image. The science images are first used to determine an optimal co-add footprint geometry and orientation, which defines the final astrometric World Coordinate System (WCS) for the reference image; global median background levels are then subtracted from each respective science image. They are then mapped and resampled onto the co-add footprint using SWarp (Bertin et al. 2002), where distortion is corrected in the process. The science images are then gain-matched because they are very likely to have different photometric throughputs due to variations in atmospheric transparency and the optical system. This is accomplished by rescaling the images using their photometric zeropoints to a fixed target zero-point, one for each filter.³⁴ The filter-specific target zeropoints become the final zeropoints for the co-add images in each filter, and are stored as MAGZP in their FITS headers.

The resampled and gain-matched images are then combined using outlier-trimmed averaging (nσ clipping) on the pixel stacks, with bad pixels (known a priori) removed. The nσ clipping is performed on each individual pixel stack where σ is a robust estimate of the dispersion in the stack based on percentiles. This process removes the most discrepant outliers, e.g., detector transient artifacts and cosmic rays. Astrophysical transients will also be removed or suppressed, depending on their persistence (longevity) across the input images. Additional pixels in the co-add image are then masked due to possible incomplete (seeing-dependent) saturation masking in the science images upstream. For example, a star with a given magnitude may fluctuate between being saturated at its core when the seeing is good (i.e., most photoelectrons accumulate within a few pixels where the wells have exceeded their capacity), and not saturated when the seeing is bad (photoelectrons are accumulated over a larger region where pixels at/near the source peak have not yet reached full well). QA metrics are then computed on the co-add products and written to their FITS headers. Photometric calibration information is assembled and also written to the headers.

The second step involves constructing source catalogs from the reference image. A spatially varying PSF is first derived for the reference image using a version of DAOPhot (Stetson 1987) optimized³⁵ for ZTF; sources are detected and PSF-fit photometry is performed (also using DAOPhot) to generate the PSF-fit catalog; aperture and isophotal photometry is then performed using SExtractor (Bertin & Arnouts 1996) and measurements are stored in a separate catalog; aperture (curve-of-growth) corrections relative to the PSF-fit photometry are then computed for the fixed aperture measurements using a subset of filtered sources; these corrections are stored in the header of the aperture catalog table; QA metrics are computed from the catalogs and photometric calibration information is written to their headers.

An overall quality status flag is assigned to the reference-image (and associated products) by thresholding on specific QA values and stored in the operations DB for bookkeeping. The primary reference image, depth-of-coverage map, pixel-uncertainty image, and catalog table files are copied to the archive with paths/filenames registered in the operations DB for later retrieval by the image differencing (Section 3.6) and source-matching (Section 3.8) pipelines.

It is important to note that reference images are not recalibrated against PS1 using its PSF-fit catalog to assign new zeropoints and color terms, i.e., as done for epochal images in the instrumental calibration pipeline (Section 3.5). This is because we rely on the accuracy of the photometric calibration solutions from the input images, which were nonetheless preselected to be of high quality. Throughput matching of the input images to a common fixed zero-point handles this implicitly. Furthermore, the final reference-image WCS (defining the co-add footprint; see above) is not astrometrically recalibrated or refined. The input images were also preselected to have good astrometric quality. No relative astrometric refinements are needed prior to co-addition.

The purpose of the source-matching pipeline is to construct light curves from the multi-epoch photometry catalogs generated by the instrumental calibration pipeline (Section 3.5). Only the PSF-fit photometry catalogs corresponding to a single field, CCD quadrant, and filter are positionally cross-matched across epochs, going back to the start of the science survey. The starting epoch is configurable. The primary product from this pipeline is productID 11 in Table 2 (Section 4), a product referred to as a matchfile. This is a file in HDF5 format³⁶ containing all the light curves and their statistics for all matched sources in the footprint defined by a specific field, CCD quadrant, and filter. The contents of these matchfiles are extracted by the light-curve query service (Section 5.3 or productID 10 in Table 2).

Matchfile creation and updating of existing versions as data are accumulated is performed on timescales of one month or longer because it is contingent on having enough (new) epochal data in a specific field, CCD quadrant, and filter. A good-quality (survey-ready) reference image source catalog (Section 3.7) for this footprint is also required. This is used to seed the source-matching process (see below). The source-matching pipeline is scheduled at times when no other pipelines are running; typically during daylight hours.

In summary, the steps used to create a matchfile are as follows. The HDF5-structured tables and columns referenced below are defined in the ZSDS Explanatory Supplement (referenced in Section 1). If a matchfile is to be created for the first time, a PSF-fit source catalog corresponding to the reference image is stored. PSF-fit source catalogs corresponding to the epochal images are also stored. The reference image catalog is first used to seed the creation of a sources table in the matchfile. The following empty HDF5 tables are then created in the matchfile: exposures, sourcedata, transients, and transientdata. A positional match to the coordinates in the sources table is performed and for each epochal catalog, a single entry is made in the exposures table. Each source that matches a reference source is assigned the matchid from the sources table. Photometry for that source is then entered in the sourcedata table. Any input detection that is not matched to a reference source is considered a transient. A position match is made against coordinates in the transients table. Transients receive their own matchid and their photometry is recorded in the transientdata table. If alternatively a pre-existing matchfile is to be updated with new data, its path/filename is required on input. In this case, the new epochal catalogs are positionally matched to the existing sources and their measurements appended to the matchfile.

Following the source-matching, the epochal photometry can be optionally refined in a relative sense. The default is to perform relative photometry prior to delivery of a matchfile to the archive to support light-curve queries (Section 5.3). Relative corrections to the input epochal photometric zeropoints (Section 3.5) are computed based on a maximum of 5000 stable (least-variable) sources in the given CCD-quadrant footprint. These corrections are computed using the algorithm described in Ofek et al. (2011). See also Levitan (2013). A delta-zero-point correction is computed for each single-epoch exposure and stored in the exposures table. Both corrected and uncorrected magnitudes are stored for each source in the matchfile product. This relative photometric refinement uses all the epochal measurements in the matchfile, going back to the earliest epoch. Therefore, as new data are matched and appended, subsequent reruns of relative photometry will cause all delta-zero-point corrections to slightly shift. Lightcurve measurements will therefore also shift following application of the new corrections. The performance of relative photometric refinement is presented in Section 7.3.

Following source-matching and (optionally) relative photometric refinement, collapsed-light-curve statistics for sources and transients are computed, and stored in their respective HDF5 tables in the matchfile. These statistics include median and mean magnitudes, dispersion measures (RMS and percentile-based), skewness, kurtosis, chi-square, minimum and maximum, mean pairwise slope, number of measurements below/above various RMS cuts, Stetson indices, and more. These statistics enable users to query light curves exhibiting specific properties or shapes.

The matchfiles are copied from the pipeline operations file system to the archive. The archive ingestion process also reads the matchids, associated time-collapsed statistics, metadata, and reference-position information for all light curves and stores these in an "objects" database. Actual light curves (the single-epoch photometry points) are not stored in this database. The objects database interfaces with a GUI to enable spatial searches and range queries on the pre-computed statistics (Section 5.3). The returned object IDs are used to retrieve the light curves from the associated matchfile(s). The light curves are then packaged and delivered to the user.

A moving-object track is a collection of linked point-source events extracted by the image-differencing pipeline (Section 3.6) corresponding potentially to a SSO. This includes NEAs. Only difference-image point-source events detected across multiple exposures are linked, not streaks produced by faster moving objects. The primary software used to construct candidate moving-object tracks is called ZMODE: ZTF's Moving Object Discovery Engine. The products from ZMODE were summarized under productID 13 in Table 2 (Section 4). A detailed listing of the specific file products is given in the ZSDS Explanatory Supplement (referenced in Section 1).

Below are the high-level steps carried out by the ZMODE pipeline executive script (managed by the VPO; see Section 2.3). The actual track-generation process is described in Appendix B. The ZMODE executive is triggered at the end of each observing night using point-source extractions spanning at most the last four continuous observing nights if available. If four continuous observing nights are not available, detections from either the most recent three or two nights spanning a four day interval are used as input. The primary inputs are a list of the night dates from which the difference-image-based point-source extractions should be queried, thresholds for the input detections (e.g., ML-RealBogus score, S/N, and magnitude limit), velocity-matching thresholds, and thresholds for filtering the orbit-fit quality metrics. A database query is executed to retrieve the point sources satisfying the input criteria. The ZMODE software is then executed on this list of sources to construct candidate moving-object tracks (see Appendix B). Orbit fitting is then performed on each candidate track using the find_orb³⁷ software. The output metrics from find_orb are thresholded to construct a list of likely real moving-objects, i.e., worthy for downstream human vetting prior to submission to the Minor Planet Center (MPC). An orbital elements file is also generated.

Following orbit fitting and filtering, the following products are generated: a provisional MPC report with object ephemerides in the traditional 80-character format; a report following the new Astrometry Data Exchange Standard (ADES³⁸ ) format; a report containing scores for the probable orbit classes for each object generated by the digest2³⁹ software; science and reference image cutouts from the images used in image differencing, centered on every detection used to construct each new track (i.e., not associated with a known SSO). The MPC report files include matches to both previously known objects and potentially new objects not recovered from cross-matching to the MPC archive. All products are copied to ZTF-Depot for human vetting.

Alert packets are binary files containing metadata and contextual information for a single "event" extracted from either a science minus reference difference image or its negative, a reference minus science difference image (see Section 3.6). Such an event may have been triggered from a flux-transient, a reoccurring flux-variable, or moving object. These raw events are lightly filtered to remove obvious false positives, and only those remaining are packetized for distribution. This filtering is discussed in the ZSDS Explanatory Supplement (referenced in Section 1) and in Mahabal et al. (2018).

Alert packet files are in the Apache Avro^TM format⁴² , a binary serialization format for efficient distribution. Alert packets are the basic atomic units of the ZTF Alert Distribution System (ZADS; Patterson et al. 2019). This system comprises staging of alert packets in the ZSDS Kafka (see footnote 6) cluster at IPAC (the producer), mirrors with Kafka clusters located elsewhere, in this case UW, interfaces with other consumers in the consortium, and interfaces with the "alert brokers" (e.g., Narayan et al. 2018).

The information in an alert packet is structured using JSON-based schemas. A description of all the metadata can be found in the ZSDS Explanatory Supplement (referenced in Section 1). Here is a summary of the content:

• 1.  
  An alert name (or objectId). The same objectId can correspond to multiple alert packets, for instance, that are triggered on the same astrophysical object detected across distinct observational epochs. That is, a previously assigned objectId (going back to the beginning of the survey) is reused on any new alert falling within 1.5 arcsec of that same sky position.
• 2.  
  Source-specific features (metrics) for the difference-image-detected event that triggered the alert. These features facilitate additional ML-vetting downstream.
• 3.  
  Image-specific metadata (also features) for the corresponding science, reference, and difference images. These comprise quality metrics from image differencing, observational span of the inputs used to construct the reference image, limiting magnitudes, and image identifiers to facilitate retrieval of ancillary products from the archive. The metrics also include a count on the number of previous candidate events detected near/on the alert position and the number of epochal images covering (touching) that position. This metadata can also be used to guide additional ML-vetting.
• 4.  
  The closest known solar-system object falling within a specific radius. If found, its angular separation, name, and V-band magnitude (if available) are reported.
• 5.  
  The closest, second closest, and third closest PS1 source falling within a specific radius. If found, angular separations, magnitudes, PS1 catalog IDs, and ML-predicted star-galaxy classification scores for all PS1 matches are reported. The PS1 star-galaxy classification process is described in Tachibana & Miller (2018).
• 6.  
  The closest source from the Gaia DR1 catalog, including the brightest source below some magnitude limit within a specific radius. If found, angular separations and Gaia G-band magnitudes are reported.
• 7.  
  Previous (historical) events falling within 1.5 arcsec of the alert position (if any) with source features and image-based metrics for each event. The search span for possible associated historical events is currently 30 days. These historical events have more relaxed extraction criteria than those used to trigger the initial alert. These include detections from all filters and from either positive (science minus reference) or negative (reference minus science) difference images. If the alert position touched a previous image but was not associated with a detected event (from either positive or negative difference), a flux upper-limit and "null" entries for the other metadata are reported instead. The flux upper-limit is based on a global difference image estimate.
• 8.  
  Three 63 × 63 native pixel image cutouts centered on the alert position from the science, reference, and difference images. These cutouts are in lossless .fits.gz format with pixel values retaining the full 32-bit floating point precision inherited from the full input images.

Details on how to access the alert packets in near-real-time and supporting documentation are available on the ZTF public website.¹⁸ Furthermore, all alert packets corresponding to a CCD-quadrant are packaged into a tar-gzipped file. These files are then copied to the long-term archive at IRSA along with other quadrant-based science products. See Section 5 for access details.

There are two ways to access the image and catalog file products from the ZTF archive.

The first and most common access method is via the GUI.⁴³ This allows a user to supply a sky position or a list of positions in any coordinate system. Alternatively, one can specify the names of astrophysical objects resolved by either NED⁴⁴ or SIMBAD.⁴⁵ Searches can also be performed by survey field ID and optionally CCD ID therein. The query returns a list of the CCD-quadrants touching those positions with associated observational metadata. This metadata (including observation timestamps) can be filtered to retain a subset of products for later packaging and downloading. All or a subset of the ancillary products per CCD-quadrant can be downloaded, including pre-packaged alert packets (Section 4.1). The results page also displays image previews for selected CCD-quadrants. These can be explored and manipulated interactively in the browser. This includes the ability to compute image statistics within sub-regions, perform cutouts and save them, or overlay sources from catalogs available in IRSA, from a VO⁴⁶ registry, or from a custom upload.

The second access method is via an API.⁴⁷ This method is non-interactive and useful for executing queries and downloads from within user-customized scripts using either wget or curl commands. The process involves first issuing a query using constraints on the available searchable quadrant-image metadata (seeded by sky position). This returns a list of the CCD-quadrants and their metadata in a table. The user can then construct the archive product URL-paths and filenames using the contents of this table for subsequent downloading using scripted wget or curl calls. Cutouts can also be performed on each image instead.

The GUI⁴³ described in Section 5.1 also allows one to specify the name of a SSO, either its official numbered designation, its NAIF ID⁴⁸ , or its MPC-formatted ephemeris. Alternatively, orbital parameters with a search timespan can be supplied. This returns a list of the CCD-quadrants containing apparitions of the moving object, their metadata, and image previews. The same exploratory services discussed above can then be used.

A more customized tool for exploring and characterizing known SSOs for ZTF is the Moving Object Search Tool⁴⁹ (MOST). This tool returns more metadata for the object, diagnostic orbit plots, orbital elements, ephemeris information, and links to scripts for downloading archived images containing the object. An accompanying API⁵⁰ to perform MOST queries using either wget or curl is also available. This API returns a table of metadata from which the URL-paths and filenames of archive products can be constructed for subsequent downloading. Image cutouts can also be performed on the positions of interest in each epochal image.

A separate custom GUI is available to query photometrically refined light curves from pre-positionally matched sources. This is productID 10 in Table 2. In essence, this service will allow cone-searches and filtering on any of the available pre-computed light-curve-collapsed metrics. A database of the (reference-image extracted) objects used to seed each lightcurve during source matching (Section 3.8) along with their metrics can be queried standalone. This service will enable one to get an estimate of the number of retrievable light curves over regions of sky that satisfy any filters before querying and downloading them. APIs to perform these tasks are also available.

Connected to the ZTF lightcurve GUI is a separate generic time-series viewer and analysis tool.⁵¹ If the light curves were saved in table files, e.g., from an API call, these can be uploaded to the time-series tool. This tool will ingest the light curves and allow one to display zoomed-in images at each timepoint, interactively manipulate them, compute periodograms, periods, other statistics, edit/remove photometry points, and more.

Figure 5 shows a runtime versus elapsed-time density plot for processing a full night's worth of data (specifically night 2018-06-17UT). This plot includes 38,904 independent pipeline instances processed using the 66-node cluster with eight instances per node (see Section 2.1). An instance corresponds to the processing of one CCD-quadrant image through the real-time pipeline, which consists of two sub-pipelines: the instrumental calibration pipeline (phase 1; Section 3.5) and the image-differencing/event-extraction pipeline (phase 2; Section 3.5).

Zoom In Zoom Out Reset image size

As seen in Figure 5, approximately 60% of the images from this night went through the image-differencing pipeline. This is because not all fields and CCD-quadrants observed had pre-existing reference images in the archive to trigger image differencing. Therefore, this plot compares timing results for phase 1 of the real-time pipeline, which is ∼1.5 to 2 minutes per CCD-quadrant, and phases 1 + 2 (full pipeline, including alert packet generation and streak detection), which amounts to ≲5.8 minutes per quadrant for most images observed out of the Galactic plane. The longer runtimes (>360 seconds or >6 minutes) between the elapsed hours of ∼4 and 6.3 are due to processing of images in the Galactic plane (at $| b| \lesssim 3.5^\circ$, 3.4° ≲ l ≲ 80°) where the source density is substantially higher (see below). The runtime here can exceed 12 minutes per quadrant (95th percentile). This night spanned ∼7 hr 30 min and was processed in approximately the same time, i.e., with no accumulated backlog.

The primary factor that affects pipeline runtime is source density, e.g., in the Galactic plane. This puts a strain on source extraction, in particular PSF-fitting and de-blending, and also on event extraction and alert packet generation downstream. The latter is also taxing in the ecliptic plane due to the large number of SSOs that are repeatedly detected.

The astrometric calibration process was summarized in Section 3.5. The accuracy in pipeline-reconstructed astrometry in the g and r filters at airmass ≤1.1 is typically 45 to 65 milliarcsec, measured as the mode in the distribution of RMS errors per axis. Figure 6 shows distributions of the RMS per filter for 190,641 g-filter and 321,420 r-filter quadrant images acquired from 2018-02-05UT to 2018-04-01UT, and observed only at airmass ≤1.1. The astrometric accuracy is slightly better in the g-filter due to its better image quality in general (effective point-source FWHM); i.e., the accuracy of source-centroids depends on the overall image quality. For a 2D Gaussian distribution, the RMS in Figure 6 is equivalent to the median radial separation.

Zoom In Zoom Out Reset image size

It is important to note that the RMS values in Figure 6 are based on the astrometric residuals of extracted source positions with respect to Gaia DR1 using ZTF sources with photometric S/N ≥ 10. At this level, errors in background estimation and Poisson noise will dominate the positional uncertainties. Hence, the RMS values reported above overestimate the true achievable astrometric precision. Figure 7 shows the RMS as a function of g-magnitude for a single CCD-quadrant. From photometric repeatability estimates (Section 7.3), S/N ≥ 10 and S/N ≥ 40 for example correspond to magnitudes of ≲20 and ≲18 respectively. For magnitudes ≲18, the RMS is ≲30 milliarcsec per axis at low airmass (≤1.1) and more suitably represents the limiting astrometric precision.

Zoom In Zoom Out Reset image size

It is also worth noting that the (S/N ≥ 10) modal RMS values reported above are two to four times larger than the positional accuracy of sources in the Gaia DR1 catalog (Gaia Collaboration et al. 2016) and those expected from atmospheric scintillation noise alone (Osborn et al. 2015).

Figure 8 shows the median robust astrometric RMS as a function of airmass range, for the g and r filters using all fields observed from 2017-12-01UT to 2018-03-04UT. This includes data from the instrument commissioning period to sample data from the high-airmass experiments. Above airmass = 3, there is significant degradation in the astrometry. This degradation is due to a combination of the decrease in image quality (increase in FWHM) with increasing airmass causing source-centroid estimates to be less accurate in general and source-color dependent biases due to Differential Chromatic Refraction (DCR). Given ZTF's resolution, DCR starts to dominate above airmass ≃1.8. Currently, the primary survey is limited to airmass ≤2 where the astrometric residuals are <65 and <85 milliarcsec RMS per-axis in g and r respectively. This is tolerable for most ZTF science programs. Airmass ≤2 corresponds to altitudes of ≳30° above the horizon and gives the requisite 3π sky coverage for the ZTF survey.

Zoom In Zoom Out Reset image size

Figure 9 quantifies the photometric repeatability as a function of magnitude by positionally matching sources from the quadrant-based PSF-fit catalogs extracted from ∼25 to 95 image epochs. The overlapping images were observed at airmass ≲1.3 and span 2018-02-04UT to 2018-03-04UT. Results are shown for four different quadrants selected from three different fields on the sky varying in source density. To mitigate variations in throughput (both atmospheric and instrumental), source fluxes in each catalog were globally rescaled to match those in the overlapping reference image (co-add) PSF-fit catalog. That is, each input epochal catalog was rescaled to a common zero-point equal to that of the reference image before the time-collapsed statistics were computed.

Zoom In Zoom Out Reset image size

In general, the internal photometric precision derived from PSF-fit photometry at bright unsaturated fluxes typically lies between 8 and 25 millimag—the range encountered from examining about one hundred quadrant-based catalogs, independent of filter. The primary limiting factor that determines the resulting photometric precision for a CCD-quadrant is the accuracy of flat fields derived from the internal flat-screen exposures (Section 3.4.2). This was verified by constructing low-spatial frequency responsivity maps by binning the photometric residuals from source photometry and examining their spatial distribution.⁵⁵ These residuals are of approximately the correct magnitude to reduce systematics from inaccurate flat-fielding upstream and improve the photometric precision to consistently ≲10 millimag at low to moderate airmass. Flat-fielding residuals have not yet been fully characterized for all CCD-quadrants and filters as a function of other factors (e.g., scattered light). There is a plan to augment the flat-screen high-spatial frequency responsivity maps and minimize the photometric residuals in general.

Cases with lower photometric precision in general (≳20 millimag) mostly occur at higher airmass where image quality and astrometric accuracy are generally lower (Section 7.2). Furthermore, there is no significant deterioration in photometric precision in high-density source regions such as the Galactic plane (top two panels in Figure 9) where source de-blending is more prevalent.

Photometric repeatability metrics are readily available in the source matchfile products (Section 3.8), obviating the need for explicit matching of catalogs across epochs. These metrics can also be retrieved through the light-curve query service (Section 5). At the time of writing however, an insufficient number of matchfiles were available for sampling different source-density regions for use in quantifying the photometric precision (Figure 9). For comparison, example photometric repeatability plots from two matchfiles for the same CCD-quadrant are shown in Figure 10. Each was created in a slightly different way. The added benefit of matchfile creation is that it includes a post-processing step to mutually refine the individual photometric zeropoints in a relative sense (Section 3.8). Figure 10 shows the relative flux RMS with and without relative photometric refinement turned on. As a reminder, the results in Figure 9 (that matched raw catalog-sources directly) included no refinement of zeropoints. The improvement in relative precision from the refinement of photometric zeropoints is typically 10% to 20%. Preliminary matchfile products show precision levels as low as ∼8 millimag per CCD-quadrant in either g or r. The achievable precision is limited by flat-fielding accuracy (see above) and other processing details that depend on sky location.

Zoom In Zoom Out Reset image size

The photometric repeatability versus magnitude plots in Figure 9 can also be used to determine the Nσ magnitude limit for a specific survey field and CCD-quadrant. For instance, 5σ corresponds to a 20% flux RMS and from Figure 9, occurs at r ≃ 20.5 to 21.0 mag for the quadrants shown.

For comparison, Figure 11 shows the relative (model-based) flux uncertainty as a function of magnitude from two quadrant-based PSF-fit catalogs. This model uncertainty is based on propagating pixel uncertainties from a model involving the electronic gain, pixel RMS, PSF-estimation error, through to PSF-fitting. The 5σ (20% flux uncertainty) magnitude limits in Figure 11 are shown by the vertical intersecting lines. These are consistent with estimates from repeatability, implying the model photometric uncertainties are plausible. Furthermore, these estimates were validated in a global sense using matches to PS1 sources and their reported photometric errors.

Zoom In Zoom Out Reset image size

The model-based flux-uncertainty versus magnitude from PSF-fit photometry (e.g., Figure 11) is used in general to estimate an approximate 5σ magnitude limit for each processed quadrant. This is based on computing the median magnitude of sources with S/N values falling within 4.5 ≤ S/N ≤ 5.5. This magnitude limit is stored as the MAGLIM keyword in the FITS header of each processed CCD-quadrant image. Figure 12 shows the spatial variation in magnitude limit for each filter by medianing the values from several thousand quadrants in g and r, and a few hundred quadrants in i across 75 photometric nights spanning 2018-04-01UT to 2018-06-30UT, or ≃3 lunations. All are at airmass ≤1.05 (i.e., near zenith) and all are from 30 sec exposures, the nominal integration time for ZTF.⁵⁶ The overall 5σ median magnitude limits across all quadrants and epochs used in these maps are g = 21.067 ± 0.003, r = 21.012 ± 0.002, and i = 20.51 ± 0.01 mag. The 1σ uncertainties are based on $\sqrt{N}$ statistics.

Zoom In Zoom Out Reset image size

The spatial variations across CCD-quadrants in Figure 12 are consistent with variations in the electronic gain (e−/ADU) for each amplifier.⁵⁷ These are also consistent with variations in photometric throughput inferred from calibrated zeropoints. The drop in sensitivity in the corners is due to vignetting.

The photometric calibration process was summarized in Section 3.5. Figure 13 shows the difference between PS1 and calibrated ZTF magnitudes for three quadrant-based PSF-fit catalogs in filters g, r, and i. The top panels show the residual magnitudes and the bottom panels are their corresponding RMS in flux space as a function of magnitude. The purpose here is to quantify the level of any photometric biases. Preliminary results show that there exist photometric biases in the PSF-fit catalogs of up to 0.02 mag for predominately bright sources. This represents the maximum median deviation measured at magnitude <15 in any filter from examining a few hundred PSF-fit catalogs (red circles in top panels of Figure 13). These biases are also field-dependent and analyses are underway to understand them. Similar biases are seen in the aperture-photometry-based catalogs; however, these are generally larger in high source-density regions due to the effects of source crowding. PSF-fitting is more immune to this effect and hence, we recommend using the PSF-fit catalogs if one is interested in point-source photometry. The PSF-fit catalogs are more optimal for faint source photometry. For bright sources however, aperture photometry is generally superior, provided there is no contamination.

Zoom In Zoom Out Reset image size

The bottom three panels in Figure 13 show the relative flux RMS-deviation with respect to PS1 in 0.5-magnitude bins. The scatter is dominated by the measured instrumental photometry, e.g., local background estimation, PSF estimation, flat-fielding, source confusion, and errors in global astrometry and centroiding. There is also some scatter in the PS1 measurements themselves. The RMS scatter with respect to PS1 is field-dependent and varies from ∼1% to ∼2% for bright, unsaturated sources. A global analysis of the overall photometric calibration accuracy as a function of all survey parameters is pending.

The extraction of point-source events from difference images was described in Section 3.6 and alert packets generated therefrom in Section 4.1. Figure 14 shows the number of raw (non-ML-vetted and non-human-vetted) point-source events to 5σ as a function of several image metrics at the CCD-quadrant level. These metrics refer to the instrumentally calibrated science image, whose photometric and astrometric solutions also apply to the derived subtraction images. Only quadrants containing at least one event from either their positive (science minus reference) or negative (reference minus science) subtraction image are plotted. The events were collected from eight continuous observing nights: 2018-05-23UT to 2018-05-30UT using exposures observed to an airmass of ≈1.5 in g and ≈2.5 in r. The observations include a mixture of integration times: 30 sec (nominal), 60 sec, and 90 sec. About 8% of the data are from fields observed at Galactic latitudes $| b| \lesssim 12^\circ$.

Zoom In Zoom Out Reset image size

Overall, the median number of raw point-source alerts per quadrant and per filter is ≈4 with a 95th percentile of ≈26 (g) and ≈43 (r). It is important to note that a significant fraction of these are false positives (from image artifacts; see Mahabal et al. 2018). We expect at most a few to several real astrophysical alerts per-image quadrant depending on sky location. For example, the alert density will be dominated by asteroids in the ecliptic plane and variable stars in the Galactic plane (see also the summary statistics in Section 6).

The clustering of points in the photometric-zero-point (ZP), median background, and to a lesser extent the magnitude-limit plot is due to different exposure times and observations in the Galactic plane. The ZP plot marks the median ZP values (as vertical lines) at the three exposure times: 30, 60, and 90 sec. The slope about each vertical line is due to varying airmass. Furthermore, values with ZP ≲ 25.7 mag in the ZP plot correspond to low atmospheric transparency (non-photometric conditions). The plot showing point-source FWHM (derived image quality) reflects the distribution of seeing and PSF sizes delivered by the P48 optics, convolved with ≃1 arcsec square pixels. A fraction of the images are under-sampled (FWHM < 2 arcsec). These images do not yield an excess of false events relative to over-sampled images. Such images are appropriately handled by our image subtraction and event-extraction algorithms (Section 3.6). The plot showing median radial separation from Gaia DR1 is based on PSF-fit extractions with photometric S/N ≥ 10 (see Section 7.2). This median radial separation is approximately equal to the RMS deviation per axis.

Figure 15 shows the typical timescales on which alert packets become available for querying at IPAC to support the ZADS (Section 4.1) following observation at the P48. Additional overheads from queries by downstream consumers, including community brokers amount to typically several seconds for ∼1000 packets. This is further discussed in Patterson et al. (2019). Given that exposures are processed at the CCD-quadrant level (Section 2.1), the latency is tied to the processing of an individual quadrant image where difference-image-detected events are extracted and alert packets generated in one instance (single thread) of the real-time pipeline. This histogram was generated from a random sample of ≃10,000 CCD-quadrants observed throughout June 2018. These yielded ≃114,600 alert packets or ≃11.5 alerts per CCD-quadrant on average.

Zoom In Zoom Out Reset image size

The 5th, 50th, and 95th percentiles in the alert production latency are ≃6.7, 8.5, and 12.6 minutes respectively. The tail is due to a combination of observations in ultra-dense regions of the galactic plane and additional overhead incurred from imperfect image differencing causing excessive numbers of events to be processed and filtered prior to packetizing. Approximately 25% of the exposures in this set were on dense regions of the galactic plane. The latency is dominated by pipeline processing (≲5.8 minutes per CCD-quadrant; e.g., see Figure 5). The remaining time is due to data transfer from the P48, raw data ingestion and staging in the SLURM queue (Section 2.1), CCD splitting (Section 3.3), and further queueing prior to triggering of the real-time pipeline (Section 3.5, Section 3.6).