https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-109 DRP 2-D Pipeline <p>We should reject outlier emission lines which are affected by cosmics, misidentification, or blends. Please include sigma clipping of lines used for the wavelength calibration during the fitting procedure.</p> PIPE2D-109

Include sigma clipping in wavelength calibration

Story Major Done Done aritter aritter Fri, 21 Oct 2016 03:46:54 +0000 Fri, 3 Mar 2017 05:43:35 +0000 Thu, 9 Feb 2017 05:41:20 +0000 0 2 <p>Code is in drp_stella tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-109" title="Include sigma clipping in wavelength calibration" class="issue-link" data-issue-key="PIPE2D-109"><del>PIPE2D-109</del></a>.<br/> Tests are in drp_stella/tests/Math.py, part of the tests in Utils.cc as I couldn't figure out how to pass PolyFit keywords and keyword values from Python to C++. To test use master obs_pfs and master drp_stella_data and run </p> <p>python tests/Math.py</p> <ul class="alternate" type="square"> <li>Please clean up history. If possible, remove commits which are unrelated to this ticket (all the cleanups) and put them on another ticket.</li> <li>Please set mask bits to indicate where clipping has taken place.</li> </ul> <p>Moved all clean-up commits to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-114" title="Apply clean-ups in PIPE2D-109" class="issue-link" data-issue-key="PIPE2D-114"><del>PIPE2D-114</del></a> and set mask bits where clipping has taken place.</p> <p>I added a bunch of comments on <a href="https://github.com/Subaru-PFS/drp_stella/pull/4" class="external-link" rel="nofollow">this PR</a>. However, I screwed up, and some of the comments seem to have appeared in odd places. I hope that everything is clear enough, but do ask if anything isn't or if something seems to be missing.</p> <p>One thing that I found rather odd is the structure of the <tt>PolyFit</tt> tests. I see you referred to these above, but I didn't notice that until after I'd looked at the code. I think it would be worth spending a bit more effort to figure out how things are supposed to work.</p> <p>I hope that after you've acted on my comments the code will be rather easier to read, and, at that point, I'd like to take another look through before we merge.</p> <p>I believe I addressed all comments from the previous review.</p> <p>This is a significant improvement over the previous version of the code: thank you! Particularly welcome are the improved commit messages — it's much easier to understand what you've actually done.</p> <p>Some more comments on the PR. Most of these are quite minor, though; while I encourage you to read them, I think you can ignore most of them. Here are things I'd specifically like you to act on:</p> <ul class="alternate" type="square"> <li>If I understand correctly, you always perform sigma clipping, even if the user specifies <tt>nIterReject=0</tt>. That seems like pretty pathological behaviour. At a minimum, I think you need to make this clear in the documentation for the <tt>nIterReject</tt> option and print out a big warning when the users asks for no iterations but you give them one anyway.</li> <li>Take out the <tt>if True</tt> in <tt>testPolyFit</tt>.</li> <li>Squash 4962901 into 43f0228.</li> <li>Expand the commit message of 43f0228 to explain why this change was necessary.</li> <li>40b75e0b should clearly be squashed into <b>something</b>: it's not a well-formed commit on its own.</li> <li>Based on the commit messages, f9aa3554 seems to be addressing the same issue as 077f70ad7. Please clarify what's going on here — if these commits really are degenerate, remove one (or squash them); if not, change the messages so they can't be confused.</li> <li>12ed6b53 seems to be entirely superseded by 1778b3b2. Remove it.</li> <li>I don't understand the motivation for 2c01217. If the measurement errors aren't used, why are you initializing them or allocating space for them at all? I don't see why one set of useless numbers is "cleaner" than another set of useless numbers.</li> <li>c0726b254 is entirely degenerate with 03b9440ae. Remove it.</li> <li>There are several separate commits which initialize various arrays or variables to 0. I don't understand why these have suddenly become necessary: are they fixing bugs (if so, where are the tests?) or are you just being neat? Can we at least squash all these various commits into one?</li> <li>1778b3b2 definitely requires a unit test. I'm worried that the tests added in 9cf52a3 didn't catch this problem — why not?</li> <li>The changes in 09058d0 are very very extensive. I suggest: <ul class="alternate" type="square"> <li>Rather than making this change last in the history of the ticket, you add the relevant headers so you can use logging as the first commit. Then never add any <tt>#ifdef __DEBUG...</tt> statements at all during this work: all of the output you are adding on this ticket can be a call to the logger, and you will not need this final commit to convert them all.</li> <li>File another ticket to convert any output which was not added on this ticket to use the logger.</li> </ul> </li> </ul> <p>I'd like to take another look at this before we regard it as finally done, so I'm setting it back to "in progress" for now.</p> <ul> <li>I changed the PolyFit function with sigma rejection to not perform the sigma rejection at all if <tt>nIterReject=0</tt>.</li> <li>I don't see an <tt>if True</tt> in <tt>testPolyFit</tt>...?</li> <li>Regarding 2c01217 I now only create a ndarray of length 1, added comments to the code, and changed the commit message</li> <li>Regarding the initializing commits, the change in 03b9440 regarding <tt>sigma</tt> and <tt>covar</tt> is purely cosmetical, while the change in ee5215c (removed 2 initializations as they were unnecessary) was a bug fix (test added)</li> <li>Regarding 1778b3b2 there are tests in the source code. The issue itself is very difficult to test from outside the function.</li> <li>I applied all other suggested changes, except for the last one as it was too late for that, but will keep it in mind next time...</li> </ul> <ul> <li>I think 188710084 is new since I looked at this previously. If these are purely cleanups, it seems like they'd be better suited on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-114" title="Apply clean-ups in PIPE2D-109" class="issue-link" data-issue-key="PIPE2D-114"><del>PIPE2D-114</del></a>.</li> <li>Similar story for f050bcd9c: you should not add new trailing spaces in <b>any</b> commit — where that's happened on this ticket, you need to edit the commit to take them out again. You should not insert them on one commit and then add another to take them out again. A commit to go through old work and remove spaces that were incorrectly added <b>on other tickets</b> is pure cleanup, and belongs on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-114" title="Apply clean-ups in PIPE2D-109" class="issue-link" data-issue-key="PIPE2D-114"><del>PIPE2D-114</del></a>.</li> <li>Here is the <tt>if True</tt> I object to: <a href="https://github.com/Subaru-PFS/drp_stella/blob/f3ed39943d32bcbfaf0f128d290fb979281fa7b9/tests/Spectra.py#L517" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/blob/f3ed39943d32bcbfaf0f128d290fb979281fa7b9/tests/Spectra.py#L517</a></li> <li>Please file a ticket to request that <tt>testPolyFit()</tt> in <tt>Utils.cc</tt> (ie, the C++ code, not the Python) should be split into multiple separate functions.</li> <li>Please file another ticket to request that the C++ test code be replaced with (or augmented by) Python tests when either we figure out how to wrap the necessary functionality with Swig or we replace it with pybind11.</li> </ul> <p>Otherwise, go ahead and merge.</p> <p>Fixed commits by removing freshly introduced trailing spaces.<br/> Moved clean-up commits to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-114" class="external-link" rel="nofollow">PIPE2D-114</a>.<br/> Added tickets <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-157" class="external-link" rel="nofollow">PIPE2D-157</a> and <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-158" class="external-link" rel="nofollow">PIPE2D-158</a>.</p> <p>Merged into master</p> Blocks PIPE2D-80 Relates PIPE2D-114 PIPE2D-110 PIPE2D-157 PIPE2D-158 PIPE2D-163 Development Rank 0|ii00uj: Reviewers swinbank Sprint 2014-16 Story Points 2.0