I looked at several images of Dark exposures taken at various dates. The uploaded files (dark-202Y-MM-DD.gif) should be in animated GIF format, displaying images taken on different days in chronological order. The image scale is fixed between -5 ADU and 50 ADU. The affected area has expanded significantly in the most recent run (August 2024). Additionally, even on the same day, the brightness flickers like a failing fluorescent light. I'm afraid this chip may become unusable shortly same as one of HSC CCDs. I will try to apply PCA analysis to these data.
 

The code currently solves for the amplitude, but if the form changes (as it appears to do!) we'll need a new set of darks.   And the PCA analysis will tell us how many components we need.

I ran the PCA code from the sklearn library on 25 dark frames taken on 2024-08-25. Before running PCA, I removed cosmic rays using the default parameters of lsst.pipe.tasks.repair.RepairTask. The resulting PCA_Cumulative_Variance.png shows the cumulative explained variance, while PCA_components.png displays each principal component. It was determined that 10 components are needed to explain 95% of the variance. 
 

In the previous analysis, I forgot to standardize the data before PCA. Now I have reviewed the procedure as follows.

1. Flatten 2D dark images to 1D vectors
2. Standardize vectors pixel by pixel using sample mean and standard variance
3. Apply PCA on standardized vectors
4. Transform standardized vectors to PCA components
5. Using the first k components, transform back to the original space.

This time I have cut a 128x256 region around the peak of glow from dark frames of run 19 as follows.

The resulting PCA components are the following. In this figure, component 1 is the original image but the images after component 2 are differences from component 1. This shows overall structure shows up only in component 1 and other variations are seen only around the peak of the glow where counts are very large and saturation is happening.

The following figure shows the difference between the original image and the reconstructed image for one visit when a different number of components are used. Beyond 3 components, it looks the same.

From this analysis, 1 component. i.e., scaling the mean image may work well if the dark data is taken within the run and that data is used for the run. When we use more components, 3 components may be good enough.

This time I have used the data of run19. I will check the same analysis on other runs once they are ingested into new datastore.

This time I have tried to remove the frames for which cosmic ray-like structures seem to remain and masked (set to zero value) the pixels flagged as saturated in any input frames. 

The PCA components will be the following. These images are row PCA components before inverse_transformed to the original space. As can be seen, only the 1st component has a meaningful structure and this component explains 93% of variance.

By changing the number of components, the difference between the original and reconstructed image for 1 visit becomes the following. There is little or no improvement even if including more than 2 components.

The frame shown in the figure above is part of the PCA components calculation. I have verified the reconstruction for a frame that is not included in the PCA components calculation. The number of components is only 1 but it seems to be good enough (less than 1% of the original frame)

I have checked the same thing for a frame taken in run18 (PCA components are calculated by the frames of run19). In this case, residual structure can be seen. This implies we have to use the dark frame taken on the same observing run.

I have done the same analysis for run18 data.

Here is PCA components.

This is the difference between the original image and the reconstructed image with various numbers of components for a visit (113513).

In this case, 1 or 4 components are preferable.

When the data from run18 and run19 are combined, the PCA components will become this.

In this case, ~3 components may be needed.

However, there is a case we cannot reconstruct the image properly even if we use more components.

This visit is not used for the calculation of PCA.

I'm trying to use all the data from previous runs. I have selected 5 frames each from the previous 5 runs (run12 20230719, run14 20231216, run16 20240502, run18 20240825, run19 20241017), for a total of 25 frames. Then I did the PCA analysis. However, as shown in the figure showing the first 20 PCA components, I'm not sure yet what's happening. Components 11, 12, and 15 look more important than 6-10. Residual CRs or some other features may contribute to components 6-10 but I need more investigation to confirm.

The following figure shows how StDev of residual images for each dark frame varies as a function of the number of PCA components used. The frames with the same color were taken on the same night. As can be seen, except for the first few components, each PCA component affects only the data taken on a specific night and seems to trace noise-level details. Something seems to be wrong. ???

I have analyzed the dark data taken during the January 2025 commissioning run (run20). Specifically I have used the following visits. 

{20250123: [119786, 119792, 119790, 119787, 119789],
 20250125: [120106, 120112, 120111, 120105, 120093],
 20250127: [120581, 120579, 120580],
 20250128: [120727, 120726, 120721, 120723, 120722]}

Only two PCA components are significant, with the first component explaining 99.8% of the variance. This suggests that the amplifier glow is stable within each run. While applying just the first component is sufficient, the second component can also be included. Despite applying the RepairTask on each frame and using the Robust PCA algorithm to isolate a low-rank matrix from noisy features (in this case, cosmic rays), some cosmic ray contamination remains. To include the second PCA component for dark subtraction, Robert's new cosmic ray remover may need to be applied.