https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-910 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-905" title="Restrict writing of pfsDesign/pfsConfig subsets" class="issue-link" data-issue-key="PIPE2D-905"><del>PIPE2D-905</del></a> restricts the ability to write subsets of pfsConfig/Design files, but I think that it'd be better to instead update the pfsDesignId and <em>allow</em> them to be written. It seems dangerous to have an incorrect pfsDesignId associated with data, and we do need to be able to write them to support open use.</p> PIPE2D-910

Update pfsDesignIds when subsetting pfsConfigs

Story Normal Open Unresolved price rhl Thu, 30 Sep 2021 20:42:53 +0000 Thu, 13 Jan 2022 14:47:44 +0000 0 5 <p>Changing the <tt>pfsDesignId</tt> would break the connection between the exposure (<tt>W_PFDSGN</tt> header) and the <tt>pfsDesign</tt> file.</p> <p>We've agreed that it's not clear exactly how we should handle all this. We think it might be important to be able to write pfsConfig subsets, e.g., for distributing censored raw data when users share fibers within the same exposure. We need input from NAOJ (e.g., <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=msyktnk" class="user-hover" rel="msyktnk">Masayuki Tanaka</a>) on how they envision data distribution of shared exposures will work.</p> <p>One possibility is that when exporting data for shared exposures, we write a new "raw" exposure (manipulating pixel values to remove proprietary spectra, either by masking or subtracting the extracted spectra) with a new pfsDesignId (<tt>W_PFDSGN</tt> header) that maps to the modified pfsConfig. One thing to consider is flagging censored fibers (e.g., <tt>targetType=CENSORED</tt> and set target information to meaningless values) rather than removing them completely; I'm not sure what that gains, but it seems more honest and helpful about what's actually in the image.</p> <p>No matter the exact method we use, it seems to me that data reduction based on censored raw data is always going to be inferior to the original raw data because of optical crosstalk. I hope that the pipeline-extracted spectra will be the primary output data product rather than the raw images.</p> <p>NAOJ has not decided yet how we distribute raw data.  I think we do not want to modify the raw images.  My current thinking is that we replace the information of censored targets with NULL in pfsDesign and generate multiple pfsDesign files for all PIs in an exposure.  I know this will require modifications to the datamodel.  I have not discussed this yet with the STARS people and I think that is the first thing I should do.  And, what did you mean by optical crosstalk?  Overlapping spatial traces between neigboring fibers?  If so, can we extract all spectra and just not write censored fibers in the output?</p> <p>Yes, optical crosstalk is the overlap between neighbouring fibers in the spatial dimension. That means that the spectrum in fiber <tt>i</tt> is required to extract the spectrum from fiber <tt>i+1</tt>, and vice versa. That means that if users are going to extract spectra themselves, they need access to all spectra on the image in order to extract optimal quality spectra.</p> <p>Created <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-122" title="Determine raw data distribution policy for shared exposures" class="issue-link" data-issue-key="DAMD-122">DAMD-122</a> to track the issue of raw data distribution policy, and made it a blocker.</p> Blocks DAMD-122 Relates PIPE2D-905 Development Rank 0|02qpq3:r3s000001 Sprint 2DDRP-2021 A 10 2DDRP-2021 A11 Story Points 1.0