obstools.scrub_deveny_pickup module

Scrubber for DeVeny Pickup Noise Module

LDTObserverTools contains python ports of various LDT Observer Tools

Lowell Discovery Telescope (Lowell Observatory: Flagstaff, AZ) https://lowell.edu

This module cleans the RF pickup noise from DeVeny spectral 2D images by fitting a sinusoid to and then subtracting it from each row of the image.

The frequency of the sinusoid tends to vary from frame to frame but is roughly CONSTANT within each frame. Between the non-integer wavelength and the slight delay between row read-out (caused by the advancement of the parallel register) the phase shifts from row to row, yielding an ever-changing pattern in the output images. In order to identify the frequency in a given image, this module fist taken the FFT of the flattened (i.e., time series) image to identify the strongest frequency. (The function that performs the FFT also modifies a copy of the data in various ways to remove spurious signals in the FFT.) From there, it uses a tightly bounded curve fitting (from scipy.optimize) to identify the sinusoid in each line of the image.

Cosmic rays and other impulsive features in the 2D image can cause spurious matches for the sinusoid. So to guard against introducing artifacts into the cleaned image from the fitting, the row-by-row fit coefficients are each considered as a function and smoothed with an appropriate median window to ensure the final subtracted pattern is a smooth function, matching the underlying pickup noise.

The output of the module is a multi-extension FITS file containing [1] the cleaned image, [2] the original image, [3,4] the pattern subtracted from the original image, and [5] a FITS BinTable containing the sinusoid fit parameters for each row of the image.

class obstools.scrub_deveny_pickup.ScrubDevenyPickup

Bases: ScriptBase

Script class for scrub_deveny_pickup tool

Script structure borrowed from pypeit.scripts.scriptbase.ScriptBase.

classmethod get_parser(width=None)

Construct the command-line argument parser.

Parameters:

• description (str, optional) – A short description of the purpose of the script.

• width (int, optional) – Restrict the width of the formatted help output to be no longer than this number of characters, if possible given the help formatter. If None, the width is the same as the terminal width.

• formatter (HelpFormatter) – Class used to format the help output.

Returns:

ArgumentParser – Command-line interpreter.

static main(args)

Main Driver

Simple function that takes the input file list and calls the cleaning function for each one.

obstools.scrub_deveny_pickup.clean_pickup(filename: Path, use_hann: bool = False, sine_plot: bool = True, fft_plot: bool = False)

Clean the Pickup Noise

Warning

Deprecated – do not use. Only included for historical purposes. Use iterative_pypeit_clean() instead.

This is the first version of the cleaning code, which operates directly on the raw DeVeny image. It suffers from confusion from bright night-sky lines and brighter objects. These deficiencies led to the development of the iterative scrubber. This code is kept here for historical purposes.

Parameters:

• filename (Path) – The name of the file to clean

• use_hann (bool) – Use a Hann window when cleaning the image for FFT (Default: False)

• sine_plot (bool) – Create a plot of the sinusoid pattern fits for this image? (Default: True)

• fft_plot (bool) – Create a plot of the FFT analysis for this image? (Default: False)

obstools.scrub_deveny_pickup.create_and_apply_pattern(input_array: ndarray, fit_coeffs: Table) → tuple[numpy.ndarray, numpy.ndarray]

Construct the pattern from the fit coefficients

Note

The returned pattern_array has a mean = 0 (i.e., the pattern is constructed from the pure sinusoid w/o offset or other additive terms).

Parameters:

• input_array (ndarray) – The input array to be cleaned

• fit_coeffs (Table) – The fit coefficient table produced by fit_lines()

Returns:

• cleaned_array (ndarray) – The cleaned sience array

• pattern_array (ndarray) – The pattern removed from input_array – mean = 0

obstools.scrub_deveny_pickup.create_fft_plot(filename: Path, flat_array: ndarray, y_fft: ndarray, x_fft: ndarray, peak_freq: float, qa_dir: Path = None, extra_graphics: bool = False)

Create the FFT analysis plot

This function creates a multipanel plot saved as fft_analysis.pdf in the current working directory. The panels are:

1. The flattened time-like array, where pixels are in the order in which they were read out by the CCD electronics.

2. The real part of the FFT of the flattened array, where the abscissa is shown as the sinusoid period in pixels.

3. The imaginary part of the FFT of the flattened array, where the abscissa is shown as the sinusoid period in pixels.

4. The absolute value squared of the FFT of the flattened array, to show the power as a function of frequency.

Parameters:

• filename (Path) – The filename of the raw data frame

• flat_array (ndarray) – The flattened (1D) pixel array

• y_fft (ndarray) – The (complex) FFT values

• x_fft (ndarray) – The FFT frequencies for the values in y_fft

• peak_freq (float) – The peak frequency measured from the FFT

• qa_dir (Path, optional) – The QA directory into which to place the optional plots. If None, the plots will be placed in the current working directory. (Default: None)

• extra_graphics (bool, optional) – Produce extra graphics formats for documentation? (Default: False)

obstools.scrub_deveny_pickup.fit_lines(data_array: ndarray, pixel_period: float, orig_fitc: Table = None, refit_thresh_sig: float = 3.0, trim_ends: bool = True, objmodel_check: bool = False, show_diagnostic: bool = False, fixed_sinusoid: bool = False) → Table

Fit a sinusoid to each line in the image

This is like a mini-driver function that fits a sinusoid to each line in the image.

It also can refit lines that have excessive RMS residual as compared to the bulk of the image. To enable the refitting, pass in the fit coefficient table from the first fitting.

Parameters:

• data_array (ndarray) – The image array to be processed. This should be HORIZONTAL and in the same orientation as images written out by lois directly from the instrument.

• pixel_period (float) – The predicted pixel period for this image array, as computed by flatten_clean_fft().

• orig_fitc (Table, optional) – If refitting, the first-pass fit coefficient table. If absent, then a full fitting of all lines will be performed. (Default: None)

• refit_thresh_sig (float, optional) – Threshold for RMS deviation (in sigma-clipped standard deviation units) from the mean for a line to be refit. (Default: 3.0)

• trim_ends (bool, optional) – Trim the 5 pixels off the ends of the line before fitting? (Default: True)

• objmodel_check (bool, optional) – Is this image the spec2d objmodel? (Default: False)

• show_diagnostic (bool, optional) – Display a diagnotic plot of several lines w/ fit? (Default: False)

• fixed_sinusoid (bool, optional) – Determine the (roughly) constant sinusoid amplitude and period and fit the lines for phase and secular drift only? (Default: False)

Returns:

Table – A table object containing the fit coefficients, one table row per row of the input data_array.

obstools.scrub_deveny_pickup.flatten_clean_fft(data_array: ndarray, use_hann: bool = False, fft_plot: bool = False, qa_dir: Path = None, filename: Path = None, extra_graphics: bool = False) → float

Find the peak frequency of sinusoidal noise

The function name describes what it does:

1. Flatten the 2D array into a 1D timeseries

2. Clean the ends of the CCD for smoother transition

3. Take the FFT to find the proper frequency to return

In order to more effectively find the frequencies of persistent signal across the detector, start by flattening out the data array into a one- dimensional timeseries-like object representing the order in which the pixels are read out.

According to information in Chapter 5 of Scientific Charge-Coupled Devices by James R. Janesick (2001), the time to advance a row along the parallel register is small compared to the individual pixel digitization time because the latter requires a capacitive stabilization time to minimize readnoise, whereas parallel register moves simply move the charge without trying to measure it. Therefore, there is no spacing in between the rows of the flattened arrays to represent the time required for the parallel charge transfer.

Once the array is flattened, we deal with row-to-row edge effects by modifying the flattened array to replace the 10 pixels at the end of one row and the 10 pixels at the start of the next row with a linearly varying series of values. The first 10 and last 10 pixels of the flattened array are set to the next adjacent value. This removes some of the “ringing” power at integer frequencies of an entire row and places it into the fundamental row-length frequency. There are still sharp edges in the flattened array caused by night sky lines, but most of the power from these lines should be at the fundamental frequency (whole-row).

The detected pickup noise has a period in the range 100-300 pixels, so peaks in the power spectrum at lower frequencies than this (e.g., the fundamental row-length frequency) are masked when choosing the return value.

Additionally, the assumption of continuous readout is close but not entriely accurate. The slight pause in the readout caused by the parallel register readout causes the continuous readout amplifier pickup signal to manifest itself as a Gaussian packet of related frequencies in the FFT. By smoothing the power spectrum ( abs(FFT)^2 ) with a Gaussian kernel, narrow spikes in the FFT (caused by autocorrelation of night sky lines, cosmic rays, edge effects) are diluted compared to the sought signal. The returned period is that of the maximum of the Gaussian-smoothed absolute value squared.

Parameters:

• data_array (ndarray) – _description_

• use_hann (bool) – Use a Hann window on each row in addition to subtracting the row mean (Default: False)

• fft_plot (bool) – Create a debugging plot of the FFT analysis? (Default: False)

• qa_dir (Path, optional) – The QA directory into which to place the optional plots. If None, the plots will be placed in the current working directory. (Default: None)

• filename (Path) – The filename of the raw data frame for the optional plot. (Default: None)

• extra_graphics (bool, optional) – Produce extra graphics formats for documentation? (Default: False)

Returns:

float – The pixel period of the sinusoidal oscillation to be removed

obstools.scrub_deveny_pickup.iterative_pypeit_clean(filename: Path, proc_dir: Path = None, overwrite_raw: bool = False, diagnostics: bool = False, no_refit: bool = False, extra_graphics: bool = False, rerun_fft: bool = False)

Iteratively use PypeIt to clean pickup noise

As part of the data-reduction process, PypeIt fits and extracts images as well as the sky background. We can use the reduced spec2d files to get the “source-free” noise background, which makes identifying the sinusoidal noise more straightforward.

Parameters:

• filename (Path) – The name of the file to clean

• proc_dir (Path, optional) – The location of the PypeIt-processed images, if not ./ldt_deveny_?/ (Default: None)

• overwrite_raw (bool, optional) – Overwrite the raw file rather than create a new file with the ‘_scrub’ suffix (Default: False)

• diagnostics (bool, optional) – Output additional information and plots during the image analysis? (Default: False)

• no_refit (bool, optional) – Do not refit lines with “bad” RMS (Default: False)

• extra_graphics (bool, optional) – Produce extra graphics formats for documentation? (Default: False)

• rereun_fft (bool, optional) – Pass the fitted pattern through the FFT analysis? (Default: False)

obstools.scrub_deveny_pickup.make_image_comparison_plots(filename: Path, spec2d: Spec2DObj, resid: ndarray, pattern: ndarray, qa_dir: Path = None, with_obj: bool = False, extra_graphics: bool = False)

Make a set of Image Comparison plots

This function creates a multipanel plot. The panels are:

1. The PypeIt-reduced 2-dimensional spectrum (spec2d.sciimg)

2. The PypeIt sky spectrum model (spec2d.skymodel).

3. The PypeIt-reduced residual image (spec2d.sciimg - spec2d.skymodel - spec2d.objmodel) modified by the good pixel mask

4. The noise pattern image generated here from the sinusoidal fits to each row of the residual image

5. The cleaned residual image (resid - pattern)

6. The cleaned 2-dimensional spectrum (spec2d.sciimg - pattern)

Parameters:

• filename (Path) – Filename of theraw image from which all this comes

• spec2d (Spec2DObj) – The 2D spectral image class from PypeIt

• resid (ndarray) – The PypeIt-produced residual image (the starting point for the fitting)

• pattern (ndarray) – The constructed pattern from the sinusoidal fits (mean = 0)

• qa_dir (Path, optional) – The QA directory into which to place the optional plots. If None, the plots will be placed in the current working directory. (Default: None)

• with_obj (bool, optional) – Does resid contain the object model? (Default: False)

• extra_graphics (bool, optional) – Produce extra graphics formats for documentation? (Default: False)

obstools.scrub_deveny_pickup.make_sine_label(popt: ndarray, infodict: dict, mesg: str, ier: int) → str

Make a sensible label for the sinusoid example plots

Parameters:

• popt (ndarray) – The array of fit coefficients from scipy.optimize.curve_fit()

• infodict (dict) – Dictionary containing fit information from scipy.optimize.curve_fit()

• mesg (str) – Message about completion from scipy.optimize.curve_fit()

• ier (int) – Integer describing fir completion from scipy.optimize.curve_fit()

Returns:

str – The legend label thusly constructed

obstools.scrub_deveny_pickup.make_sinusoid_fit_plots(filename: Path, fit_coeffs: Table, pixel_period: float, qa_dir: Path = None, extra_graphics: bool = False, fixed_fitc: Table = None, smoothing: dict = None)

Create a set of diagnostic plots from the set of sinusoid fits

This function creates a multipanel plot saved as sinusoid_fits.pdf in the current working directory. The panels are:

1. The row-by-row sinusoid amplitude

2. The row-by-row sinusoid period

3. The row-by-row sinusoid phase shift

4. The row-by-row rms residual when the sinusoidal fit is subtracted from the original line

Parameters:

• filename (Path) – The filename of the original file, used in the plot title

• fit_coeffs (Table) – The fit coefficients table

• pixel_period (float) – The estimated pixel period of the sinusoidal noise from the FFT

• qa_dir (Path, optional) – The QA directory into which to place the optional plots. If None, the plots will be placed in the current working directory. (Default: None)

• extra_graphics (bool, optional) – Produce extra graphics formats for documentation? (Default: False)

• fixed_fitc (Table, optional) – The fit coefficients table with fixed sinusoid (Default: None)

• smoothing (dict, optional) – Polynomial fit coefficients for amplitude and period (Default: None)

obstools.scrub_deveny_pickup.package_into_fits(filename: Path, ccd: CCDData, cleaned_array: ndarray, pattern_array: ndarray, fit_coeffs: dict, pixel_period: float, overwrite_raw: bool = False)

Package the cleaned data into a FITS file

Package everything into a multiextension FITS file:

0. Primary HDU – contains the raw file’s FITS header

1. Cleaned Image HDU – raw - pattern

2. Original Image HDU – raw

3. Pattern Image HDU – pattern

4. Pattern about Raw Mean Image HDU – pattern + mean(raw)

5. Fit Coefficients BinTable HDU – fit_coeffs

The output filename is the same (including path) as the input raw file, but with “_scrub” appended before “.fits”.

Parameters:

• filename (Path) – The filename of the raw data frame

• ccd (CCDData) – The raw data object

• cleaned_array (ndarray) – The cleaned raw data array (raw data - pattern) in ADU

• pattern_array (ndarray) – The computed pattern array in ADU

• fit_coeffs (dict) – The fit coefficients dictionary from fit_lines()

• pixel_period (float) – The FFT-computed pixel period from flatten_clean_fft()

• overwrite_raw (bool, optional) – Overwrite the raw file rather than create a new file with the ‘_scrub’ suffix (Default: False)

obstools.scrub_deveny_pickup.pixper_tofrom_hz(val: ndarray) → ndarray

Convert to/from pixel period and Hertz

_extended_summary_

Parameters:

val (ndarray) – Input value(s) to convert

Returns:

ndarray – Converted output(s)

obstools.scrub_deveny_pickup.smooth_array(array: ndarray, kernel_size: int = 21, ftype: str = 'median') → ndarray

Smooth out an array with a given filter

This may be used for smoothing a line or attempting to smooth fit coefficients from row to row in order to ameliorate the effects of cosmic rays and strong sources.

Note

scipy.signal.medfilt() uses zero-padding for the ends of the signal, therefore subtract the mean and re-add it to eliminate edge weirdness.

Parameters:

• array (ndarray) – The array to be smoothed.

• kernel_size (int, optional) – Kernel or window size for the smoothing function (Default: 21)

• ftype (str, optional) – Smoothing function to use. Options are “median” and “savgol”. (Default: “median”)

Returns:

ndarray – The smoothed coefficients