https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/DAMD-40 Data Model <p>The outputs from the 2d pipeline have to be passed to the 1d pipelines.  For this, the relative locations of the pipeline outputs have to be defined.  A few constraints:</p> <ul> <li>outputs should be in a rerun directory</li> <li>there will be multiple 1d reruns based on a single 2d rerun (e.g., run lam1d multiple times tweaking params on a single set of data processed by 2d)</li> </ul> <p>There may be other constraints we need to consider.</p> DAMD-40

Define relative locations of the 1d + 2d pipeline outputs

Story Normal Done Done Unassigned Masayuki Tanaka Fri, 11 Jan 2019 03:58:55 +0000 Tue, 15 Sep 2020 06:52:28 +0000 Tue, 15 Sep 2020 06:52:28 +0000 0 8 <p>I first thought that all the 1d outputs can be under a 2d rerun directory, something line this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2d/ ├── BIAS ├── CALIB ├── DARK ├── FLAT ├── OTHER_STUFF └── rerun        ├── dr1        ├── dr2        └── dr3                ├── ga1d                │     ├── STUFF                │     └── rerun                │             ├── test1                │             └── test2                └── lam1d                       ├── STUFF                       └── rerun                              ├── test1                              └── test2</pre> </div></div> <p>But, this may require having 1d repos for each 2d rerun.  Perhaps not a good idea.  Another naive thinking is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>./ ├── 2d │     ├── BIAS │     ├── CALIB │     ├── DARK │     ├── FLAT │     ├── OTHER_STUFF │     └── rerun │      ├── dr1 │     ├── dr2 │     └── dr3 ├── ga1d │     ├── STUFF │     └── rerun │             ├── dr3_test1 │             └── dr3_test2 └── lam1d         ├── STUFF         └── rerun                 ├── dr3_test1                 └── dr3_test2</pre> </div></div> <p>For this, we need to a mapping between the 2d and 1d reruns.  One way to do it is to have 1d rerun name as "2d_rerun_name"_"your_string".   In the above example, "dr3_test1" is based on the "dr3" rerun from 2d.</p> <p>There are probably other (and better) options.</p> <p> </p> <p>Sorry, this got lost. I have no problem with defining the layout of the outputs here, but they should merely be documenting the implementation in the butler templates (for 2-D).</p> <p>In particular, this means that the reruns and calibrations should probable be laid out a bit differently. Is this urgent?</p> <p>No, this is not urgent.  We will probably implement one of the above in the proto-type database v2, but we do not have real 2d outputs there (only simulated psfObject files) and the directory structure will be different anyway.</p> <p><tt><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=sogo.mineo" class="user-hover" rel="sogo.mineo">sogo.mineo</a> suggested that this is also a possible option:</tt></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ROOT +-- BIAS +-- CALIB : : + 1DSTUFF +-- rerun +-- 2d/dr1 +-- 2d/dr2 +-- 2d/dr3 +-- gal1d/dr3_test1 +-- gal1d/dr3_test2 +-- lam1d/dr3_test1 +-- lam1d/dr3_test2</pre> </div></div> <p>Do we prefer to have different root for 2d and 1d pipelines?</p> <p>Need a definitive agreement in the next 3 weeks.</p> <p>Once an agreement has been made, a document capturing that agreement will be prepared and uploaded to PbWorks. Hassan will take that action.</p> <p>I really don't care much, but I don't understand the examples &#8211; what is <tt>BIAS</tt>, for example? </p> <p>The reruns need to be under ROOT, at least via a symbolic link, so that you can point the butler at e.g. <tt>ROOT/rerun/drp2</tt> or <tt>ROOT/rerun/rhl/foo</tt>. I don't know how 1-D will handle reruns, but I'd expect them to be able to be able to process any 2-D rerun, including all the book-keeping so we know what we did. Ideally this would be transparent (as is true on the 2-D side), but this is independent of the logical layout.</p> <p>Until we understand how 1-D will handle reprocessing it's hard to take a decision.</p> <p>If you ingest new data, you'll have directories $(OBJECT) under your root directory.  That is what BIAS etc meant.  Anyway, that is not very important.  Here is how you run the 1d pipeline:<br/> <a href="https://github.com/Subaru-PFS/drp_1dpipe/blob/master/MANUAL.md" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_1dpipe/blob/master/MANUAL.md</a></p> <p>It looks like you will have to tell the pipeline where your spectra are with respect to the root directory.  I am not sure if you have any control on the output directory (it does not look like it).  We could ask the LAM team to handle reruns.  Another option might be to ask Mineo-kun to somehow handle 1d and 2d reruns internally in his code.  I'd prefer the former because we want to have a good level of transparency at the rerun level.</p> <p>The relative locations of the input (where the 2D outputs are stored) and output (where the DRP-1D data products are stored) directories are comand line parameters of the pipeline (–spectra_dir and --output_dir), and default values can be adapted as a function of implementation (e.g the output directory can be named from the input one with a time_stamp). Thus you can produce as many rerun of a given input catalog as wanted.<br/>  <br/> A question came to us from this ticket : does that mean that there will be 2 implementations of the DRP-1D : one for the on-the-fly reduction that will run automatically with a fixed configuration, and a second one related to the science database that will be used for rerun in different configurations ?</p> <p>Thank you!  The ability to specify input/output directories is exactly what we want for our purpose.  I am going to educate myself on how to run the 1d pipeline a little further.</p> <p>I am not sure if we are going to run the 1d pipeline at the summit (with different config).  But, we may change the config and rerun the pipeline for testing purposes or whatever reason and that is what 'rerun' is for.</p> <p>I've run the LAM 1d pipeline on ~60k simulated PFS spectra.  I think any of the options discussed on this ticket will work.  HSC has about 80 reruns for SSP production so far (there should be more for testing purposes), and if I think about &gt;10 years of PFS operations, I would suggest that we have separate root directories for 2ddrp, lam1d, (and probably ga1d as well, although I do not know yet what the ga1d output looks like).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>./ ├── 2d │ ├── BIAS │ ├── CALIB │ ├── DARK │ ├── FLAT │ ├── OTHER_STUFF │ └── rerun │ ├── dr1 │ ├── dr2 │ └── dr3 ├── ga1d │ ├── STUFF │ └── rerun │ ├── dr3_test1 │ └── dr3_test2 └── lam1d ├── STUFF └── rerun ├── dr3_test1 └── dr3_test2</pre> </div></div> <p>The 1d rerun names should inherit the 2d rerun name to make the relationship clear.</p> <p>This is fine, except that I'm not sure what you mean by CALIB. The 2d pipeline has one CALIB directory which contains the biases, flats, fiberTraces, etc. The <b>raw</b> biases etc. are in the same place as all the other raw data, so it's more like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>./ +-- 2d | +-- raw | | +-- data | +-- CALIB | | +-- BIAS | | +-- DARK | | +-- FLAT | | +-- FIBER_TRACE | | +-- OTHER_STUFF | +-- rerun | +-- rhl | | +-- test1 | +-- dr1 | +-- dr2 | +-- dr3 +-- ga1d | +-- STUFF | +-- rerun | +-- dr3_test1 | +-- dr3_test2 +-- lam1d +-- STUFF +-- rerun +-- dr3_test1 +-- dr3_test2</pre> </div></div> <p>Except that I'd expect that we'd need some new CALIBs for every rerun (because the algorithms to build e.g. fibre traces and arc solutions will change, and I have no idea how much smarter we'll get with the H4RGs), so I'd put a CALIB in each rerun (and probably symbolic link it to the top level, but that's a real detail).</p> <p>On the 2-D side the reruns logically include the raw data, but the butler handles that so it's not visible to the file system.</p> <p>Thank you for the useful comment!  This ticket is about relative locations of the 2d and 1d outputs, and I think we can safely close it now.</p> Relates SCIDB-98 Development Rank 0|s0020g: