This ticket is based on PIPE2D-80 which is based on PIPE2D-90 as it needs to read the pfsFiberTrace file. Once PIPE2D-90 and PIPE2D-80 are reviewed and merged into master, I will rebase this ticket on master and submit for review

It appears that we looked at the wrong file when we created this ticket. The width of the normalized Flat (point 1.) was actually correct, and so was the variation (point 2.). However concerning the width we decided to not use the proposed SNR cut nor cut the normalized Flat FiberTrace to the width of the spectral FiberTraces for diagnostic reasons. We might need to rethink not using the SNR cut though as it will probably add noise to the adjacent FiberTraces.
We found that in the center of the FiberTraces the spatial profile is slightly too low (of the order of 0.03%). The explanation for this seems to be in the way the spatial profiles are calculated. To avoid negative values for the wings of the spatial profiles aritter set all negative pixel values in the original FiberFlat exposures to zero, what causes the wings to be slightly too high and therefore the centers of the spatial profiles (normalized to a sum of 1.0) slightly too low. We decided that this systematic is within the acceptable errors and that no fix (code changes) is needed at least at this stage.

rhl, are you happy with the text above? If so, we'll close this.

The explanation for this seems to be in the way the spatial profiles are calculated

Please check/demonstrate this by not setting all negative pixels to zero and making sure that the profile is no longer too low. Should be quick to do this: just hack away at the source.

Turns out that I don't set negative pixel values to zero (anymore), only negative profile values. I'm going to check the weights for the most outer pixels next.

Per our meeting of 2016-12-19, please add:

• A description of what's going on here;
• How we can be sure it's not going to be a problem in dithered flats.

Per our meeting of 2017-01-09, aritter believes that this problem may have vanished (or be much reduced?) with newer simulations and/or the current version of his code. He will check this, and update this ticket with material to convince us that that is the case. We can then close this (a post-mortem of bugs in old code or simulations is not necessary).

I have produced plots comparing the original and reconstructed (from the spatial profile and the extracted spectrum) pixel values in all FiberTraces in one Flat, sorted by their distance to the center of the FiberTrace. The profile was calculated with 10 times oversampling (bin width 1/10th pixel). The bin width for which the mean and standard deviation of the difference (original pixel value) / (reconstructed pixel value) were calculated is 1/100th pixel. We originally saw strong systematics in the most outer 1/20th (half a bin width of the profile calculation) of a pixel which seems resolved now:




This looks reasonable to me.

price — in case it's not obvious, I've just been talking to aritter and he points out that the review of this ticket involves not just understanding his figures and agreeing that they indicate that there's nothing to worry about here, but also reviewing the outstanding PR on GitHub.

Please ignore the ticket branch for obs_pfs as this is superseeded by PIPE2D-80. I have filed a ticket PIPE2D-151 to go over this branch and look for code changes worth keeping.

As for what is going with this ticket in general: I used the Flat with visit number 104 to generate a normalized Flat. This is done by calculating the spatial profile, optimally extract the spectrum by scaling the spatial profile to each row in the FiberTrace, reconstruct the (smooth) FiberTrace image by multiplying the spatial profile with the spectrum, and dividing the original FiberTrace image by the reconstructed one. The spatial profile is calculated from the Flat rows in a swath containing 500 rows. The swaths overlap (swath 1: rows 0-499, swath 2: rows 255-754,...). Within each swath the rows are (noisy) dithered representations of the spatial profile, scaled with the flux in the dispersion element. These dithered representations are normalized to a sum of 1 and binned with a bin width of 1/oversample (with oversample=10). For each bin center, the mean value (with sigma rejection iterations) is calculated, and fitted with a cubic spline.
The profile calculated for each swath is assumed true for the center of the swath. In between the swath's centers the profile is interpolated from the 2 adjacent profiles.
More comments to follow...

I've made some comments about the code on the GitHub PR. I still need to grok what it is you're doing because I'm still getting my head around the algorithms. It might be helpful if you could make some "before" plots to match the "after" ones you've already posted, so we can see the problem along with its solution.

Here are some images which illustrate the problems we saw:

• systematics in the pixels which are the farthest away from the center of the FiberTrace, just before a curved FiberTrace crosses over to another column:
  before:
  after:
  The solution was to make the FiberTrace wider. I also improved the profile determination for the most outer pixels by extrapolating the cubic spline outside the limits where the spline was defined. These limits are set by the bin centers of the oversampled profile. Originally I just used the values of the most outer bins.
• systematics in the outer pixels of the normalized Flat, especially in the straight FiberTraces. I only just discovered this when I prepared the images. I will move PIPE2D-77 back into progress and put the fix into a new commit:
  before:
  after:
  The problem was that the dithered FiberTraces were extracted using the pfsFiberTrace from the non-dithered Flat. The fix was to trace the FiberTraces for all (dithered) Flats individually.

RHL suggests:
Add 2 more pixels far out on both sides and set them to 0 to constrain the spline extrapolation.
Calculate ChiSquared, show residual divided by variance ("Chi")

Some more comments about the strategy for the construction of the master Flat from the dithered Flats:

• Only FiberTrace s with zero shift in x are extracted using the pfsFiberTrace, all others are traced before the extraction.
• For each FiberTrace, all dithered realizations are added into an empty image to construct a single, wide FiberTrace representation.
• The same is done for their reconstructions, meaning that each dithered realization is reconstructed individually.
• The sum of the original dithered FiberTrace is divided by the sum of the reconstructed ones to create a normalized wide FiberTrace Flat.
• Pixels with a SNR less than minSNR (config parameter) are set to Unity
• The normalized FiberTrace is extracted from the image using the pfsFiberTrace, and put into the final master Flat

I was thinking that maybe it's better/faster, to trace, extract, and reconstruct the wide sum of the dithered FiberTrace representations, instead of reconstructing each dithered FiberTrace individually. This version is implemented in a different branch now (obs_pfs tickets/PIPE2D-77_1profile). Both versions are working fine, but I didn't exactly compare and profile them yet. I filed another ticket to compare both versions for quality and speed (PIPE2D-154).

A last (hopefully) systematic I forgot to mention earlier in the wings of the spatial profile, where the model was ~10% too low. The fix was to turn up the oversampling factor (config parameter config.profile.overSample) from 10 to 50-100, reducing the bin size for the profile calculation to 1/50-1/100 of a pixel:

• before:
  
• after (with 100 times oversampling):
  

Please review the latest commits in drp_stella and the branch tickets/PIPE2D-77 in obs_pfs

I added review comments to the PR on obs_pfs. Nothing further to add to drp_stella.

Merged into master