Two Procedures to Flag Radio Frequency Interference in the UV Plane

At low radio frequencies, below 1 GHz, the radio frequency interference (RFI) environment is quite active and can cause severe degradation in image quality. RFI can increase image noise by up to an order of magnitude above the thermal noise expected for the telescope. The images also usually exhibit widespread systematics in the form of multiple ripples, which increase the detection thresholds of faint objects.

There has been much effort directed toward RFI mitigation and flagging strategies in both hardware (pre-correlation) and software (post-correlation) regimes. Hardware-based techniques typically use either some sort of a reference signal to measure the interference and subtract it from the data (Barnbaum & Bradley 1998; Briggs et al. 2000; Fridman & Baan 2001; Hellbourg et al. 2014) or use specialized hardware such as additional antennas (or antenna arrays) to null sources of interference along certain directions (Van Der Veen & Boonstra 2004; Kocz et al. 2010). Among the software-based tools there are techniques that attempt to excise the RFI from the data i.e., recover the uncorrupted visibilities (Golap et al. 2005; Athreya 2009; Pen et al. 2009; Offringa et al. 2012a) and methods to remove—i.e., flag—the affected data (Bhat et al. 2005; Middelberg 2006; Winkel et al. 2007; Offringa et al. 2010, 2012b). The software methods have an advantage in that they can be applied to both new as well as archival observations.

In general, a single mitigation strategy has not proved to be very effective, as different sources of RFI leave different signatures in the data. Persistent RFI appears as strong fringes in a baseline, and the amplitude and phase of these fringes can change as a function of both time and frequency. Broadband RFI can affect the entire baseline and cause fluctuations in antenna gain and are much more difficult to characterize and eliminate. Intermittent RFI, localized in time and frequency, look like "hotspots" in the visibility data and cause large-scale ripples in the image plane.

We describe here two new methods to identify and flag intermittent RFI, which have consistently yielded high-sensitivity images when applied to a variety of GMRT observations.

The measured visibilities in any polarization in the presence of RFI signal can be written as

$\begin{eqnarray}&&{V}^{o}={G}_{i}{G}_{j}^{* }({V}_{{ij}}^{\mathrm{sky}}+{V}_{{ij}}^{\mathrm{RFI}}{e}^{{{if}}_{{ij}}})+{\eta }_{{ij}}\end{eqnarray} \tag{ 1 }$

where ij are antenna indices, ${V}_{{ij}}^{\mathrm{sky}}$ are the visibilities due to sky emission, ${V}_{{ij}}^{\mathrm{RFI}}$ is correlated RFI, ${e}^{{{if}}_{{ij}}}$ is the fringe stopping function (and the only polarization independent factor), and η_ij is additive noise in the system. G_i and G_j are the complex antenna gains, which can be affected by strong RFI even if the RFI is uncorrelated. ${e}^{{{if}}_{{ij}}}$ will stop the fringe of the cosmic source at the phase center but introduces a corresponding fringe on a stationary (terrestrial) source, like RFI. Athreya (2009) used the form of ${e}^{{{if}}_{{ij}}}$ to excise RFI while recovering the visibilities.

V^RFI can be orders of magnitude larger than V^sky and vary with time, frequency, and baseline. The second term in the above equation causes poorer solutions during self-calibration and introduce systematic errors in the image.

In this paper, we focus on intermittent RFI, which are localized in the visibility space. These localized hotspots will result in large-scale ripples across the image. We explore two different visibility spaces, viz. the binned UV plane of the entire interferometer and the time-channel (TC) plane of a single baseline, to locate and flag corrupted visibilities. The two algorithms are individually called GRIDflag and TCflag, respectively, and are combined into an integrated RFI flagging package called IPFLAG.

2.1. RFI Flagging in the Gridded UV Plane—GRIDflag

2.2. RFI Flagging in the TC Plane—TCflag

2.3. Observations and Parameters

All of the observations were done with a bandwidth of 16 MHz in the 150 MHz band and a spectral resolution of 62.5 or 125 kHz. The data was recorded with an integration time of 2s. The flow of analysis is shown in Figure 1.

Zoom In Zoom Out Reset image size

We used the following parameters in IPFLAG:

• 1.  
  UV-bin size (GRIDflag): 10λ, from the field of view at 150 MHz.
• 2.  
  Smoothing window for median visibility background (GRIDflag): 5 × 5 bins.
• 3.  
  UV-bin annuli width (GRIDflag): 3, 3 and 6.5 kλ
• 4.  
  Fourier transform window size (TCflag): bandwidth × 10 minutes.
• 5.  
  Fourier peak detection threshold (TCflag): 3 × RMS noise
• 6.  
  RFI threshold (both): 3 × RMS noise

We transferred the RFI flags from IPFLAG to the raw data, and repeated the entire process of imaging, self-calibration, residual flagging, and flag transfer. Finally, the data was again self-calibrated and imaged for the final result (see Figure 1). The final image covers 625 × 625 with a pixel size of 45. The sources were extracted from the images using the PyBDSF source finder (Mohan & Rafferty 2015), running with identical parameters across all images.

Stand-alone calibration using the standard CASA/AIPS recipes resulted in flux density scale errors of up to 15%. This becomes important while comparing the absolute noise in an image, though it does not affect the relative change in noise with the application of RFI flagging procedures. We therefore anchored our flux density scale to TGSS-ADR (Intema et al. 2017) using the sources in common with the two observations. The fractional discrepancy in source flux densities is shown in Figure 4. For reasons we do not understand, but which may have to do with something similar to CLEAN bias (Condon et al. 1998; Cohen et al. 2007), we find a flux-dependent fractional discrepancy between TGSS and our flux densities. The fractional discrepancy changes between −0.5% and −9% between sources above and below 100 mJy.

We applied IPFLAG to real data from the GMRT at 150 MHz and compared the results with and without the same. The 150 MHz band of the GMRT is important for a variety of astrophysical phenomena but is under-utilized because of the presence of strong RFI. We felt that this comparison using real data would be a more realistic appraisal of the algorithms than simulations with well-behaved noise.

We targeted five fields—VIRMOSC (GMRT observation code: 14RAA01), J1453+3308 (27_063), J1158+2621 (27_063), A2163 (16_259), and 3C286 (TGSS data, Intema et al. 2017).

3C286—A commonly used flux density calibrator source, which is compact and has a flux of 26 Jy at 150 MHz. The field is dominated by point sources with almost no extended emission. However, for reasons that are not understood, this field showed a reduced flux density in TGSS-ADR by ∼25% (see Intema et al. 2017).

VIRMOSC—A field dominated by point sources. The strongest point source in the field is ∼1.7 Jy while it has a single diffuse (4 arcmin) source of 300 mJy.

J1453+3308 and J1158+2621—Double-double radio galaxies, with diffuse outer lobes spanning 4'–7'.

A2163—This is a galaxy cluster with a 14' low surface brightness radio halo.

Figure 2 shows examples of the elimination of the RFI hotspots from the median binned UV plane. Only the upper half of the UV plane is shown in the plot; the lower half is simply the Hermitian conjugate. The band of higher intensity seen at U ≡ [−100 λ, 100 λ] (e.g., J1453+3308 in Figure 2) arises from the inability of the RfiX algorithm to mitigate RFI in regions where the fringe-stop frequency is close to zero.

Zoom In Zoom Out Reset image size

Figure 3 shows a comparison between TCflag (described here) and existing flagging tools RFlag, TFCrop (both implemented in CASA), and AOFlagger (Offringa et al. 2010, 2012b). Our algorithm TCflag is competitive with the others while using a different procedure to estimate the true noise background.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

The total data flagged by IPFLAG was 2.4%–15.7%, and the corresponding loss in UV-bins was 1.2%–3.9%. Table 1 shows that the change in the dirty beam parameters is small before and after IPFLAG, confirming that our procedure did not change the UV coverage despite the loss of a substantial amount of data. We used the same restoring beam before and after IPFLAG in all of our targets. The post imaging comparisons are plotted in Figure 5.

Zoom In Zoom Out Reset image size

Point source fluxes—The plots in Figure 5(a) show the fractional change in flux density (FCF) as a function of the original (unflagged) flux density. These flux densities were compared before primary beam correction. We analyzed the FCF in three flux density regimes, viz. S > 100 mJy, 25σ < S < 100 mJy, and S < 25σ. 25σ is the flux density above which source counts are typically ∼90% complete (e.g., Intema et al. 2017) and as such defines the limit of the reliability of source catalogs for statistical studies. The FCF ranges from −1% to +6% for the strongest sources. The increase in flux may be expected due to an improvement in calibration after the RFI has been flagged. The mean FCF for intermediate sources is smaller than that for the strong sources by −6% to +1%. This trend in reduction of flux density continues to the faintest sources detected. We saw this reduction with all of the other flaggers as well (AOFlagger, RFlag, and TFCrop), and both of the source detection algorithms used (PyBDSF and aegean; Hancock et al. 2012; Mohan & Rafferty 2015). Suspecting that this discrepancy may be a result of a systematic shift in the background level, we analyzed the residuals in the immediate vicinity of the detected sources before and after flagging but did not detect any such systematic offsets. In summary, this effect is independent of the RFI flagging algorithm used; it does not seem to affect the flux density scale, as the brightest sources are not affected; it must have something to do with the CLEAN algorithm that we use, which comes with CASA. The change in flux density of the intermediate sources is much smaller than the reduction in image noise. At the moment, we have no explanation for this effect, but it may be something similar to the CLEAN bias whose impact is most obvious for the faintest sources (Condon et al. 1998; Cohen et al. 2007). We recall that a similar effect was observed while comparing our flux densities with that of TGSS-ADR in Figure 4.

Image noise as a function of radial distance—In general, the image noise is known to reduce with radial distance. A 200 × 200 pixel box was used to calculate the robust RMS at 625 locations across the image. The plots in column (b) of Figure 5 show both the scatter and the trendline for the unflagged and IPFLAG data; the AOFlagger data is only represented by the trendline in the interest of clarity. The plots show that IPFLAG has reduced the noise all across the image.

Image noise as a function of source flux—Column (c) of Figure 5 shows a plot of the image noise plotted against the cumulative flux density within each of the 625 boxes mentioned earlier. Normally, a higher level of local artefacts is expected in the presence of strong point sources.

Noise Histogram—Column (d) of Figure 5 shows the histogram of the noise across the field. There is a clear shift in the distribution of RMS noise to lower values.

Source Counts—The ultimate metric for image improvement is an increase in source detection at lower flux levels. Figure 5(e) shows that IPFLAG passes this criterion by detecting faint sources to the expected depth. The plot shows the cumulative histogram of the number of sources detected as a function of flux density, i.e., N(> S), which shows clearly that the excess detections come from lower flux densities. Table 1 provides the number of sources detected for each field. In all cases, IPFLAG has resulted in the detection of more sources.

Artefacts—Figure 6 shows the reduction in artefacts in the field after the application of IPFLAG.

Zoom In Zoom Out Reset image size

The IPFLAG procedures have worked well across a variety of data sets, substantially reducing the image noise and detecting fainter sources. While we tested the algorithms on GMRT data, they should be as effective for any interferometric array in which the UV plane contains a large number of visibility samples.

We are unaware of any other algorithm that approaches RFI flagging in the manner that GRIDflag does. TCflag is similar to several other algorithms like AOflagger, TFcrop, Rflag, etc., which flag localized RFI in the TC plane of an individual baseline while differing in the manner of estimating the background. The combined application of GRIDflag+TCflag, which are components of our RFI pipeline IPFLAG, outperforms the other algorithms. AOFlagger was the closest in terms of performance, and we recognize that a more experienced user of AOflagger may be able to obtain better results by optimizing its many tuneable parameters for GMRT data.

On the other hand, IPFLAG has only six tuneable parameters in total. These values did not require much optimization, and the same values were used for all of the fields tested. In fact, of the six, the UV-bin size selects itself from the field of view, and the three noise thresholds were set to "universal" default values (at 3σ). The width of the UV-annuli was set by simply distributing the visibilities approximately equally across the three annuli used. While the values may require some changes for other frequencies and interferometers, we believe that they will not need to be tuned for different fields.

Upon application of IPFLAG, our images have consistently reached an RMS noise <1 mJy beam⁻¹, with a corresponding increase in source detection; typical GMRT 150 MHz images have hitherto reached an RMS noise of 1.5–5 mJy beam⁻¹, with exceptional effort yielding 0.7 mJy beam⁻¹ (e.g., Ishwara-Chandra et al. 2010). In three of the five images, we have reached an image noise of 0.38–0.55 mJy beam⁻¹, which is only a factor of ∼2 above the theoretical confusion limit. Even in the case of 3C286, an exceptionally bright flux density calibrator, we have reached 1.0 mJy beam⁻¹, and the peak to RMS noise ratio is in excess of 21000, which is unprecedented for GMRT 150 MHz images.

The fact that GRIDflag reduces the loss of UV coverage in the gridded UV plane means that the reduction in RFI is not offset by a corresponding increase in the sidelobes of the synthesized beam. The efficacy of the algorithm depends on the level of redundancy in the gridded visibility plane. The GMRT has good coverage of the shorter spacings, particularly at low frequencies (Swarup et al. 1991), and this coverage is due to get better with the upgraded GMRT (Gupta et al. 2017).

The algorithms also worked at higher frequencies (325 and 610 MHz), but the improvement was not as substantial. We think that this is because of the weaker RFI environment at these frequencies. However, even these weak RFI environments could substantially impact the performance of ultra-deep imaging projects like the MIGHTEE (Jarvis et al. 2017). Furthermore, we believe that GRIDflag is particularly tailored toward multi-epoch observations such as the MIGHTEE, wherein the same UV grids are sampled day after day.

These algorithms will not work in the presence of persistent, broadband RFI. However, other algorithms are available for these situations (e.g., Athreya 2009, used in this paper). We think that the issues that remain to be addressed are the nonisoplanatic ionosphere and an asymmetric antenna primary beam. Applying our algorithms in conjunction with schemes like SPAM (Intema et al. 2009) should yield further improvement of image quality.

We thank the staff of the GMRT who have made these observations possible. GMRT is run by the National Centre for Radio Astrophysics of the Tata Institute of Fundamental Research. Discussions with Drs. Sanjay Bhatnagar and Ishwara-Chandra contributed to this work. We thank the referee for many suggestions that have considerably improved this manuscript.

Facility: GMRT - Giant Meter-wave Radio Telescope.