See obs_pfs tickets/PIPE2D-80. To test you will need drp_stella tickets/PIPE2D-10 as this ticket contains the functions needed in the namespace pfs.drp.stella(.math). On tiger obs_pfs, drp_stella_data, and drp_stella are in /tigress/HSC/PFS. The data repository is in /tigress/HSC/PFS/PFS. To test run
/tigress/HSC/PFS/obs_pfs/bin/constructNormFlat.py /tigress/HSC/PFS/PFS --rerun azuri/tmp --id visit=29..53 --calibId calibVersion=flat calibDate=2016-08-11 arm=r spectrograph=2 --clobber-config
The normalised Flat will be in /tigress/HSC/PFS/PFS/CALIB/FLAT/r/flat/pfsFlat-2016-08-11-0-2r.fits.

Great!

But: I'll be travelling a lot over the next month, and will have even less time than usual available to do code reviews. Since you're already being blocked by my failure to review other tickets, maybe it would be smarter to have somebody else look at this? Perhaps cloomis could help? (Sorry Craig...)

We believe this has broken reading & writing flats because it hasn't been done with PIPE2D-81 in mind. aritter will check.

Please provide instructions for how to use this without any reference to azuri/tmp reruns. Probably the best way to do this would be to update the install.text file (or did John rewrite it in rst? If so, please update that)

did John rewrite it in rst?

He did, but it should still be straightforward to edit: https://github.com/Subaru-PFS/drp_stella/blob/master/sphinx/user/getting_started.rst.

As the construction of the normalized Flat and persisting it according to the datamodel broke the FiberTrace extraction, we first need to persist the FiberTraceSets before we can merge this issue with master.

This ticket is based on PIPE2D-90 as it needs to read the pfsFiberTrace file. Once PIPE2D-90 is reviewed and merged into master, I will rebase this ticket on master and submit for review

Rebased onto master. However to include it in the quick-start guide I need a consistent data set (I think we have Biases and Darks now, but 1 arc and dithered flats with fiber traces in the same positions, preferably no 2 adjacent ones, are needed). Craig says he can give me an Arc with the same fiber traces as the dithered Flats quickly what would be enough for now to test PIPE2D-80. Should we aim for new dithered Flats in the long term (would take a long time to produce)?

I don't quite understand. You can use the better biases and darks that Craig pointed you at (we agree). So you're asking about the fibre flats; what's the issue here? Whether Craig needs to include the full imaging flats in the simulations?

The fiber traces in the arc I have are not in the same positions as the fiber traces in the dithered flats. Craig said there is an Arc on tiger (which was down yesterday) I can use.

I'm still lost; the dithered flats have a range of trace positions. You mean that there is no flat that has the same trace as the arcs? If so, it sounds as if you know what to do.

In the old arc different fibers were illuminated compared to the dithered Flats. I now got a new Arc with the same fibers as the dithered Flats. I will add it to drp_stella_data so we can flatfield the arc before the extraction.

With the new Arc tests/Spectra.py fails. Both reduceArc versions (one using Craig's file 'RedFiberPixels.fits.gz' with the predicted wavelengths for each pixel in each fiber to create the initial line list, and the one using a previously identified reference spectrum) fail for different reasons. Using the predicted wavelengths fails because something has changed in the simulations and the file 'RedFiberPixels.fits.gz' is not up-to-date anymore. A ticket has been filed to recreate this file (SIM2D-68). The reduceArc version using the reference spectrum fails because of cosmic-rays which irritate the wavelength calibration procedure. While cosmic rays are not new to the simulated arc, in this particular case there is 1 cosmic ray (see attached image) which made it through the cosmic-ray detection steps in the extraction procedure and badly affects the RMS of the fitted lines. There are 2 ways to avoid this in the future.

1. Write a CalibTask (constructArc.py) to construct a master Arc from a number of input Arcs and mask cosmic rays (even if there is only 1 input Arc). I have done this already (PIPE2D-108) and confirmed that it does fix the problem, at least for that particular cosmic ray. However, now another emission line in a different spectrum gets mis-identified due to the fact that most pixels of that PSF are affected by a cosmic ray and set to background values close to 0. So, additionally to masking cosmic rays we also need to exclude emission lines which have too many pixels masked as being problematic (See PIPE2D-110).
2. Include sigma clipping in the fitting procedure to remove outliers (PIPE2D-109). This requires minor changes to Spectra::identify which I am testing now.
I personally think that both ways should be implemented to improve the wavelength solution. First of all we did want to include the 'detrend' procedure in the reduceArc... task (PIPE2D-106 = former INFRA-55 [just moved it to PIPE2D where it belongs]). Since we have 2 independent wavelength-calibration tasks (one using the predicted wavelengths from the simulator [reduceArc.py], one using a reference spectrum [reduceArcRefSpec.py] to create the initial line list), I think it makes sense to first create a master Arc and then run the wavelength calibration (reduceArc...). Also, I believe that detrend.py does not support cosmic-ray detection, so a CalibTask is needed here. Re the sigma clipping I think that this should definitely be included in order to remove outliers caused by remaining cosmic rays, misidentifications, and blends.

Changes are in tickets/PIPE2D-80 in drp_stella, drp_stella_data, and obs_pfs.
Tests in drp_stella/tests/Spectra.py are failing due to cosmics (see PIPE2D-108 [in review], PIPE2D-109 [in review], and PIPE2D-110 [open]) and the still missing new file from Craig containing the predicted wavelengths. Submitting anyway as we know why the tests are failing and that it will be fixed next week.
drp_stella/examples/runPipeline tested and working

See comments on github; https://github.com/Subaru-PFS/obs_pfs/pull/4

We can merge this now, but only after filing tickets to remove the issues identified in the review.

Next time, you need to clean up the commits prior to submitting the ticket for review. It should be arranged into a tidy set of changesets that the reviewer can understand (e.g. all the white space changes in one changset; all the removed print statements; one changeset shouldn't remove or modify one that appeared earlier — the reviewer shouldn't have to comment on something, only to find that it was resolved later).

Git history cleaned up. Merged drp_stella, obs_pfs, and drp_stella_data with master.