https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-378 DRP 2-D Pipeline <p>I should have post a ticket earlier on this, but anyway to be able to detrend blue and red images I had to "isolate" the detrend folder (repo or target). Meaning that I have created one detrend folder with calib product in each. then I call the detrend with the correct target folder.</p> <p>But it seems that it is not supposed to work like that.</p> <p> </p> PIPE2D-378

detrend of the different arm images at LAM

Bug Normal Won't Fix Won't Fix price fmadec Wed, 27 Feb 2019 15:26:24 +0000 Wed, 12 Feb 2020 19:04:25 +0000 Wed, 12 Feb 2020 19:04:25 +0000 0 4 <p>I have a bunch of changes associated with the handling of the blue arm from <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-316" title="Demonstrate arm merging" class="issue-link" data-issue-key="PIPE2D-316"><del>PIPE2D-316</del></a> awaiting the new review/merge policy.</p> <p>I didn'tt expect Paul to need to change anything to detrend r/b as I thought that the data model was strong enough – but maybe his changes will reveal stupidity on my part.</p> <p> </p> <p>I believe the data model was fine; it was just boring butler implementation stuff.</p> <p>Here's a question from the ICS side: should we always indicate the normal/low res "arm" for bias and darks? For filenames, etc.</p> <p>I don't think the pipeline cares, but that might make sense for humans.</p> <p>The datamodel says:</p> <blockquote><p>(While theoretically we don't need to distinguish between the r and m chips, it's simpler to be<br/> consistent, so we will have 16, not 12, biases/darks for the complete spectrograph)</p></blockquote> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=fmadec" class="user-hover" rel="fmadec">fmadec</a>, can you point me to some files that I can test on?</p> <p>I am not sure I was clear enough last time, so let me (try to) clarify and be sure we are on the same page:</p> <p>First, I am not using 2 arms at the same time (at least for now) but one at a time , so my point was that to be able to detrend the blue (but for the med that the same) while I previously detrented Red data, I cannot do it because it gave me an error (not able to find biases ...) so I had to create a separate drp folder to be able to process them.</p> <p> </p> <p>so to reproduce that, make calib product for the red, process any red data, then make calib product for the blue and try to process either blue or red and that should not work anymore.</p> <p>hopefully I was clear enough this time !</p> <p>biases for blue: 12548..12562</p> <p>arcs: 12593..12597</p> <p> </p> <p> </p> <p> </p> <p> </p> <p>Using the current master branch of datamodel, obs_pfs and drp_stella, and the <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-359" title="Set up regular pipeline runs using real (LAM) data" class="issue-link" data-issue-key="PIPE2D-359"><del>PIPE2D-359</del></a></tt> branch of pfs_pipe2d, I ran:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pprice@tiger2-sumire:/tigress/pprice/pipe2d-378 $ mkdir CALIBS pprice@tiger2-sumire:/tigress/pprice/pipe2d-378 $ ingestCalibs.py /tigress/HSC/PFS/LAM --calib CALIBS ../drp_stella_data/raw/detectorMap-sim-*.fits --validity 1000 --mode=copy pprice@tiger2-sumire:/tigress/pprice/pipe2d-378 $ pfs_build_calibs.sh -r price/pipe2d-378 -c 10 -C /tigress/pprice/pipe2d-378/CALIBS -n -b visit=12548..12562 -d visit=12578..12592 -f visit=12623..12751 -F visit=12618..12622 -a visit=12593..12597 /tigress/HSC/PFS/LAM </pre> </div></div> <p>This failed at the <tt>reduceArc</tt> stage:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lsst.daf.persistence.butlerExceptions.NoResults: No locations <span class="code-keyword">for</span> get: datasetType:pfsConfig dataId:DataId(initialdata={<span class="code-quote">'visit'</span>: 12595, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2019-02-26'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'expId'</span>: 12595, <span class="code-quote">'arm'</span>: <span class="code-quote">'b'</span>, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'ARC'</span>, <span class="code-quote">'ccd'</span>: 0, <span class="code-quote">'filter'</span>: <span class="code-quote">'b'</span>, <span class="code-quote">'expTime'</span>: 2.01, <span class="code-quote">'dataType'</span>: <span class="code-quote">'arc'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2019-02-26'</span>, <span class="code-quote">'pfiDesignId'</span>: 4296015889, <span class="code-quote">'slitOffset'</span>: 0.0}, tag=set()) </pre> </div></div> <p>This is because it couldn't find a <tt>pfsConfig</tt> for the exposure. I don't know what fibers were used, so I don't know what dummy pfsConfig to create. But I expect that if I created one, it would work fine.</p> <p>I conclude that <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-316" title="Demonstrate arm merging" class="issue-link" data-issue-key="PIPE2D-316"><del>PIPE2D-316</del></a> fixed the problems with producing blue arm calibs. If you have any problems using the current master branch, please send me instructions on how to reproduce the problem (including a list of commands to run, and the error messages you get), and I will investigate further.</p> <p>Does this data pre-date the header key that specifies the configuration?  If so, can we find some b + r data that allows Paul to check that this really does all work?</p> <p>This doesn't appear to be an issue any more, and the pipeline has developed since it was filed. If it's still a problem, please open a new ticket with instructions on how to reproduce.</p> Development Rank 0|02qpt6:508yl0chzzy0000r Sprint 2DDRP-2019 C 2DDRP-2019 D 2DDRP-2019 E 2DDRP-2019 F 2DDRP-2019 G 2DDRP-2019 H Story Points 3.0