PFS-JIRA

PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us

8.3.4

803005

13-09-2019

[PIPE2D-80] Task-ify code for creating normalized flats https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80 DRP 2-D Pipeline <p>Currently, it exists only in an IPython notebook.</p> <p>Task name: <tt>ConstructDitheredFlat</tt>.</p> PIPE2D-80

Task-ify code for creating normalized flats

Sub-task PIPE2D-10 Major Done Done aritter swinbank Fri, 9 Sep 2016 17:35:10 +0000 Tue, 24 Jan 2017 22:08:00 +0000 Tue, 15 Nov 2016 08:13:58 +0000 0 2 <p>See obs_pfs tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a>. To test you will need drp_stella tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-10" title="Flat field the data" class="issue-link" data-issue-key="PIPE2D-10"><del>PIPE2D-10</del></a> as this ticket contains the functions needed in the namespace pfs.drp.stella(.math). On tiger obs_pfs, drp_stella_data, and drp_stella are in /tigress/HSC/PFS. The data repository is in /tigress/HSC/PFS/PFS. To test run<br/> /tigress/HSC/PFS/obs_pfs/bin/constructNormFlat.py /tigress/HSC/PFS/PFS --rerun azuri/tmp --id visit=29..53 --calibId calibVersion=flat calibDate=2016-08-11 arm=r spectrograph=2 --clobber-config<br/> The normalised Flat will be in /tigress/HSC/PFS/PFS/CALIB/FLAT/r/flat/pfsFlat-2016-08-11-0-2r.fits.</p> <p>Great!</p> <p>But: I'll be travelling a lot over the next month, and will have even less time than usual available to do code reviews. Since you're already being blocked by my failure to review other tickets, maybe it would be smarter to have somebody else look at this? Perhaps <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> could help? (Sorry Craig...)</p> <p>We believe this has broken reading & writing flats because it hasn't been done with <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-81" title="Design access mechanism, dataset types for accessing flats from Butler" class="issue-link" data-issue-key="PIPE2D-81"><del>PIPE2D-81</del></a> in mind. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=aritter" class="user-hover" rel="aritter">aritter</a> will check.</p> <p>Please provide instructions for how to use this without any reference to azuri/tmp reruns. Probably the best way to do this would be to update the install.text file (or did John rewrite it in rst? If so, please update that)</p> <blockquote> <p>did John rewrite it in rst?</p></blockquote> <p>He did, but it should still be straightforward to edit: <a href="https://github.com/Subaru-PFS/drp_stella/blob/master/sphinx/user/getting_started.rst" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/blob/master/sphinx/user/getting_started.rst</a>.</p> <p>As the construction of the normalized Flat and persisting it according to the datamodel broke the FiberTrace extraction, we first need to persist the FiberTraceSets before we can merge this issue with master.</p> <p>This ticket is based on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a> as it needs to read the pfsFiberTrace file. Once <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a> is reviewed and merged into master, I will rebase this ticket on master and submit for review</p> <p>Rebased onto master. However to include it in the quick-start guide I need a consistent data set (I think we have Biases and Darks now, but 1 arc and dithered flats with fiber traces in the same positions, preferably no 2 adjacent ones, are needed). Craig says he can give me an Arc with the same fiber traces as the dithered Flats quickly what would be enough for now to test <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a>. Should we aim for new dithered Flats in the long term (would take a long time to produce)?</p> <p>I don't quite understand. You can use the better biases and darks that Craig pointed you at (we agree). So you're asking about the fibre flats; what's the issue here? Whether Craig needs to include the full imaging flats in the simulations?</p> <p>The fiber traces in the arc I have are not in the same positions as the fiber traces in the dithered flats. Craig said there is an Arc on tiger (which was down yesterday) I can use.</p> <p>I'm still lost; the dithered flats have a range of trace positions. You mean that there is no flat that has the same trace as the arcs? If so, it sounds as if you know what to do.</p> <p>In the old arc different fibers were illuminated compared to the dithered Flats. I now got a new Arc with the same fibers as the dithered Flats. I will add it to drp_stella_data so we can flatfield the arc before the extraction.</p> <p>With the new Arc tests/Spectra.py fails. Both reduceArc versions (one using Craig's file 'RedFiberPixels.fits.gz' with the predicted wavelengths for each pixel in each fiber to create the initial line list, and the one using a previously identified reference spectrum) fail for different reasons. Using the predicted wavelengths fails because something has changed in the simulations and the file 'RedFiberPixels.fits.gz' is not up-to-date anymore. A ticket has been filed to recreate this file (<a href="https://pfspipe.ipmu.jp/jira/browse/SIM2D-68" title="Please recreate file containing predicted wavelengths for each pixel in each fiber trace" class="issue-link" data-issue-key="SIM2D-68"><del>SIM2D-68</del></a>). The reduceArc version using the reference spectrum fails because of cosmic-rays which irritate the wavelength calibration procedure. While cosmic rays are not new to the simulated arc, in this particular case there is 1 cosmic ray (see attached image) which made it through the cosmic-ray detection steps in the extraction procedure and badly affects the RMS of the fitted lines. There are 2 ways to avoid this in the future.<br/> <span class="image-wrap" style=""><a id="10301_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10301/10301_Screen+Shot+2016-10-20+at+3.36.38+PM.png" title="Screen Shot 2016-10-20 at 3.36.38 PM.png" file-preview-type="image" file-preview-id="10301" file-preview-title="Screen Shot 2016-10-20 at 3.36.38 PM.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10301/_thumb_10301.png" style="border: 0px solid black" /></a></span> <br/> 1. Write a CalibTask (constructArc.py) to construct a master Arc from a number of input Arcs and mask cosmic rays (even if there is only 1 input Arc). I have done this already (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-108" title="Create CalibTask to construct master Arc" class="issue-link" data-issue-key="PIPE2D-108"><del>PIPE2D-108</del></a>) and confirmed that it does fix the problem, at least for that particular cosmic ray. However, now another emission line in a different spectrum gets mis-identified due to the fact that most pixels of that PSF are affected by a cosmic ray and set to background values close to 0. So, additionally to masking cosmic rays we also need to exclude emission lines which have too many pixels masked as being problematic (See <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-110" title="Please ignore emission lines with too many masked pixels in wavelength calib" class="issue-link" data-issue-key="PIPE2D-110"><del>PIPE2D-110</del></a>).<br/> 2. Include sigma clipping in the fitting procedure to remove outliers (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-109" title="Include sigma clipping in wavelength calibration" class="issue-link" data-issue-key="PIPE2D-109"><del>PIPE2D-109</del></a>). This requires minor changes to Spectra::identify which I am testing now.<br/> I personally think that both ways should be implemented to improve the wavelength solution. First of all we did want to include the 'detrend' procedure in the reduceArc... task (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-106" title="change reduceArc(RefSpec).py to include detrend" class="issue-link" data-issue-key="PIPE2D-106"><del>PIPE2D-106</del></a> = former <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-106" title="change reduceArc(RefSpec).py to include detrend" class="issue-link" data-issue-key="PIPE2D-106"><del>INFRA-55</del></a> <span class="error">[just moved it to PIPE2D where it belongs]</span>). Since we have 2 independent wavelength-calibration tasks (one using the predicted wavelengths from the simulator <span class="error">[reduceArc.py]</span>, one using a reference spectrum <span class="error">[reduceArcRefSpec.py]</span> to create the initial line list), I think it makes sense to first create a master Arc and then run the wavelength calibration (reduceArc...). Also, I believe that detrend.py does not support cosmic-ray detection, so a CalibTask is needed here. Re the sigma clipping I think that this should definitely be included in order to remove outliers caused by remaining cosmic rays, misidentifications, and blends.</p> <p>Changes are in tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a> in drp_stella, drp_stella_data, and obs_pfs.<br/> Tests in drp_stella/tests/Spectra.py are failing due to cosmics (see <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-108" title="Create CalibTask to construct master Arc" class="issue-link" data-issue-key="PIPE2D-108"><del>PIPE2D-108</del></a> <span class="error">[in review]</span>, <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-109" title="Include sigma clipping in wavelength calibration" class="issue-link" data-issue-key="PIPE2D-109"><del>PIPE2D-109</del></a> <span class="error">[in review]</span>, and <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-110" title="Please ignore emission lines with too many masked pixels in wavelength calib" class="issue-link" data-issue-key="PIPE2D-110"><del>PIPE2D-110</del></a> <span class="error">[open]</span>) and the still missing new file from Craig containing the predicted wavelengths. Submitting anyway as we know why the tests are failing and that it will be fixed next week.<br/> drp_stella/examples/runPipeline tested and working</p> <p>See comments on github; <a href="https://github.com/Subaru-PFS/obs_pfs/pull/4" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/obs_pfs/pull/4</a></p> <p>We can merge this now, but only after filing tickets to remove the issues identified in the review.</p> <p>Next time, you need to clean up the commits prior to submitting the ticket for review. It should be arranged into a tidy set of changesets that the reviewer can understand (e.g. all the white space changes in one changset; all the removed print statements; one changeset shouldn't remove or modify one that appeared earlier — the reviewer shouldn't have to comment on something, only to find that it was resolved later). </p> <p>Git history cleaned up. Merged drp_stella, obs_pfs, and drp_stella_data with master.</p> Blocks PIPE2D-47 PIPE2D-90 PIPE2D-108 PIPE2D-109 SIM2D-68 Relates PIPE2D-122 PIPE2D-123 PIPE2D-151 Development Rank 0|ii00pr: Reviewers rhl Sprint 2014-17