https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-338 DRP 2-D Pipeline <p>LAM data are (of course) not coming with pfsConfig file that are needed for the full pipeline to run. At the moment to solve this, I am taking dummy pfsConfig files from drp_stella_data and renaming them to match what my script (modelled after pfs_integration_test.sh) is asking for. This ticket is to either 1. automatize this process or 2. expand the documentation with the swindle that I am doing, described above</p> PIPE2D-338

Handle lack of pfsConfig files for the LAM data

Task Normal Done Done price ncaplar Thu, 7 Feb 2019 14:29:58 +0000 Thu, 7 Mar 2019 15:39:00 +0000 Thu, 7 Mar 2019 15:38:47 +0000 6.0 0 6 <p>Are you really running the "full pipeline"? If you're not running parts of the pipeline that require the pfsConfig, you might be able to use the <tt>ingestImages.py</tt> script instead of <tt>ingestPfsImages.py</tt> (it may need some config tweaks). Or we could add a flag to <tt>ingestPfsImages.py</tt> that disables ingestion of the pfsConfigs.</p> <p>I was trying to run pipeline up to <tt>reduce_Arc.py</tt>, modeled after <tt>pfs_integration_test.sh</tt> and the script that I used to reduce data taken in December 2017. I was not aware of the difference between <tt>ingestImages.py</tt> and <tt>ingestPfsImages.py</tt>. At the moment this is anyhow somewhat mute for me because I am not able to run full pipeline due to DetectorMap difference discussed in slack and I am cutting my stamps manually. As I discussed with <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a>, this a problem that anybody else (e..g, LAM or Japan) who would be trying to follow the manual and/or <tt>pfs_integration_test</tt> script would encounter. </p> <p>One annoyance is that data taken with the LAM sparse slit bundles are <em>precisely</em> when you want/need a <tt>pfsConfig</tt>, both for the pipeline (<tt>constructFiberTrace</tt>) and other consumers. (Would it be hard to make a <tt>pfsConfig</tt> for each of the LAM groups?)</p> <p>I think we need to take responsibility for providing <tt>pfsConfig</tt> files for all versions of cable-A. This includes the current LAM dummy cable-A, for which I suspect that Neven's file is fine.</p> <p>We have a minor problem that there's currently no good place to store these; suggestions welcome — this is the generic boot-strap problem, which also applies to <tt>DetectorMap</tt>.  Given that we are planning to move to separate products for code and camera properties (i.e. splitting the <tt>obs_pfs</tt> package), I propose that we invent <tt>obs_pfs_data</tt> now and put the <tt>pfsConfig</tt> and <tt>pfsDetectorMap</tt> files there. Bootstrapping a new repository would put these into the proper place.</p> <p>One other change that will be needed is to change the actor to write the proper <tt>pfsConfigId</tt> into the headers; this would be a very good thing anyway.</p> <p>Can you glance at <a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-207" title="Create repo(s) for varying instrument characteristics" class="issue-link" data-issue-key="INFRA-207">INFRA-207</a>, which was my bleat for deciding on how to partition data like this before we created a zillion similar repos....</p> <p>I've pushed a script to drp_stella on branch <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-338" title="Handle lack of pfsConfig files for the LAM data" class="issue-link" data-issue-key="PIPE2D-338"><del>PIPE2D-338</del></a></tt>. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=ncaplar" class="user-hover" rel="ncaplar">ncaplar</a>, could you please check this does what you need?</p> <p>Anyone who cares about the implementation is welcome to review <a href="https://github.com/Subaru-PFS/drp_stella/pull/54" class="external-link" rel="nofollow">the GitHub PR</a>.</p> <p>Grrr, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a>,unfortunately I am having problem with installing the ticket at the moment, so I am unable to give feedback (but this is completely on my end). Just looking at it, it seems fine. As mentioned, the most crucial part for me is to be able to run reduce_Arc.py that would create detectorMap with correct solutions.</p> <p>It should be impossible to have "problems installing the ticket" as it's just a <tt>git checkout</tt>. More to the point, where did "<tt>reduce_Arc.py</tt> that would create detectorMap with correct solutions" come from? I assume that that's just a typo for <tt>reduceArc.py</tt>, but the code to update the <tt>DetectorMap</tt> should be a separate one-off to write an updated <tt>DetectorMap</tt> followed by registering it. This is separate from reducing arc data.</p> <p>Ok, sorry for the confusion. <br/> 1. I managed to run it (was working in the wrong directory). Running with fibers currently at lam, i.e., <tt>makeLamDesign.py blue green red3 red6</tt> creates <tt>pfiDesign-0x0000001001000011.fits</tt></p> <p>2. Indeed, I meant <tt>ReduceArc.py</tt>. I agree, my comment about <tt>DetectorMap</tt> was simply wrong. I just wanted to make the point that I wanted to be able to run <tt>ReduceArc.py</tt>, but that is obvious and did not contribute to the discussion here. I apologize.</p> <p>I've got <tt>ingestPfsImages.py</tt> working now so that it if it can't find the <tt>pfsConfig</tt> file, it will look for the <tt>pfiDesign</tt> file and use that to create the <tt>pfsConfig</tt> file, and then ingest it. So I believe this is ready to merge, once people are happy with it.</p> <p>Just a quick comment : There is no 'red' bundle in the dcb and we can at maximum connect 5 fiber bundles at the same time.</p> <p>so having fiber <span class="error">&#91;2, 3, 308, 339, 340, 342, 649, 650&#93;</span> together is not possible i'm afraid.</p> <p>Thanks for letting me know. I've removed the <tt>red</tt> setting.</p> <p>Following discussion with @rhl: will proceed with merging branch in order to make progress. Will address subsequent problems that could arise in future tickets.</p> <p>Merged to master.</p> Relates INSTRM-606 Development Rank 0|02qpt6:3k Reviewers ncaplar Sprint 2019 B Story Points 2.0