https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90 DRP 2-D Pipeline <p>Please write a Calib Task to create and write FiberTraces. Currently the 'Flat' used for the FiberTrace creation is read from the pipeline output CALIB/FLAT. The new task ConstructNormFlat now writes the CALIB/FLAT which cannot be used anymore to trace the apertures, so we need a new task to create the FiberTraces starting with raw Flats (xOffset=0.0).</p> PIPE2D-90

Create Calib Task to create and write FiberTraces

Story Major Done Done aritter aritter Thu, 29 Sep 2016 21:32:39 +0000 Tue, 11 Oct 2016 10:57:20 +0000 Tue, 11 Oct 2016 10:57:20 +0000 0 3 <p>Is it necessary to expand the scope of the current sprint to include this? If possible, we should try to finish the tickets we have in play before we move on to anything else.</p> <p><a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a> broke the existing code for tracing the apertures. Before we can merge <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a> into master we need to make the FiberTraceSets persistable and change the existing code to include this</p> <p>Ok, thanks for clarifying.</p> <p>See obs_pfs and drp_stella tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a>.<br/> Quick-start guide has been updated to include constructFiberTrace.<br/> Once this ticket is reviewed and merged with master I will rebase <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-48" title="Change extraction tasks to read FiberTrace parameters from disk" class="issue-link" data-issue-key="PIPE2D-48"><del>PIPE2D-48</del></a>, <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a>, and <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-77" title="Make sure there are no systematics in the normalized flat" class="issue-link" data-issue-key="PIPE2D-77"><del>PIPE2D-77</del></a> on master and put them in review as well.</p> <p>The instructions include the line</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> constructFiberTrace.py /Users/azuri/spectra/pfs/PFS --rerun azuri/tmp --id field=FLAT dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibDate=2015-12-22 arm=r spectrograph=2 visit=4 </pre> </div></div> <p>This is inconsistent with the other entries in the file. Do we also need to ingest the fibreTraces?</p> <p>Also, why do you specify <tt>field=FLAT</tt> not an explicit visit number, and why is the visit in the calibId?</p> <p>Is constructFiberTrace pushed to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a>?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ git pull remote: Counting objects: 1, done. remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (1/1), done. From github.com:Subaru-PFS/drp_stella fdcf7c9..20927fb master -&gt; origin/master Current branch tickets/PIPE2D-90 is up to date. $ find . -name constructFiberTrace.py $ </pre> </div></div> <p>It's in obs_pfs. (RHL adds: how did I miss that??)</p> <p> Trying to test the line in the quick-start guide now. 'visit' definitely needs to be replaced with 'calibVersion' since Friday. Fixed the path to $PFS_DATA</p> <p>Did you push the data model changes? I don't see a tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a> branch, and I don't see a ticket for this change to the data model.</p> <p>You didn't reply to my question about ingestion; it seems inconsistent with the other calibration products like flats not to need to ingest the fiberTraces.</p> <p>Fixed the command in the quick-start guide and tested it. You're right - pfsFiberTraces do need to get ingested into the CALIB database. I replaced field=Flat with visit=5 to be consistent with the previous line reduceFlat...<br/> Sorry, the datamodel changes are in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-68" title="Add FiberTrace to data model" class="issue-link" data-issue-key="PIPE2D-68"><del>PIPE2D-68</del></a>. Should have mentioned that...</p> <p>Hmm. Don't I need to specify <tt>calibVersion=fiberTrace</tt> in the --calibId? And if I add that I get<br/> {{<br/> File "/Users/rhl/LSST/obs/pfs/python/lsst/obs/pfs/pfsMapper.py", line 97, in map<br/> location = super(PfsMapper, self).map(datasetType, copyId, write=write)<br/> File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/daf_persistence/python/lsst/daf/persistence/mapper.py", line 172, in map<br/> func = getattr(self, 'map_' + datasetType)<br/> AttributeError: 'PfsMapper' object has no attribute 'map_linearizer'<br/> }}</p> <p>That's <br/> {{<br/> $ cd /Users/rhl/LSST/obs/pfs<br/> $ git pull<br/> Current branch tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a> is up to date.<br/> }}<br/> so I think I have the right mapper.</p> <p>Can you modify the <tt>runDemo</tt> script that I sent you and confirm that it works?</p> <p>Also, I'm surprised to see that it uses `parseAndRun()` not `parseAndSubmit()` like the other tasks (e.g. reduceBias.py). Can you explain?</p> <p>Please pull drp_stella and obs_pfs again (just did some changes). The 'map_linearizer' error normally shows up when the 'config.isr.doLinearize = False' is commented out what I only see in obs_pfs/config/pfs/dark.py.<br/> I replaced 'parseAndRun()' with 'parseAndSubmit()' so now the parameter '--cores 1' needed to get added to the command line for constructFiberTrace... (change pushed in drp_stella)</p> <p>I was unable to confirm that the code is correct (as the full processing relies on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-48" title="Change extraction tasks to read FiberTrace parameters from disk" class="issue-link" data-issue-key="PIPE2D-48"><del>PIPE2D-48</del></a>) but it does run! I'm passing this review for now, but we need to confirm that it's doing the right thing as part of the <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-48" title="Change extraction tasks to read FiberTrace parameters from disk" class="issue-link" data-issue-key="PIPE2D-48"><del>PIPE2D-48</del></a> review.</p> <p>Created issue to write test for pfsFiberTrace (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-98" title="Add test to confirm that a FiberTraceSet reconstructed from pfsFiberTrace is identical to the original one" class="issue-link" data-issue-key="PIPE2D-98"><del>PIPE2D-98</del></a>). Merged into master</p> Blocks PIPE2D-80 PIPE2D-10 Relates PIPE2D-98 Development Epic Link PIPE2D-47 Rank 0|ii00no: Reviewers rhl Sprint 2014-16 Story Points 2.0