https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-618 DRP 2-D Pipeline <p>Data have been acquired from SM1 at Subaru for determining the stability of the wavelength solution: See the SM1 logbook <a href="https://people.lam.fr/madec.fabrice/pfs/logbook_opdb_visit_sm1_subaru.html" class="external-link" rel="nofollow">https://people.lam.fr/madec.fabrice/pfs/logbook_opdb_visit_sm1_subaru.html</a> for visit details (look for name='stabilityTest').</p> <p>Please perform a complete processing of bias, dark, quartz and arc data, listed under the name 'stabilityTest' using the 2D DRP. </p> <p>Process as far as <tt>reduceArc.py</tt>. So we should for each arc exposure, have the corresponding wavelength solution and <tt>detectorMap</tt>.</p> <p>In addition, the visit numbers processed should be sent to <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=sogo.mineo" class="user-hover" rel="sogo.mineo">sogo.mineo</a> at NOAJ so that he can look into updating his YAML mechanism to extract those visits from the opDB.</p> <p>Note: the actual analysis of the wavelength solution to characterize instrument breathing modes would be done later and will be the subject of a separate ticket.</p> PIPE2D-618

Process SM1 stability data

Task Normal Done Done price hassan Tue, 21 Jul 2020 15:47:58 +0000 Mon, 4 Jan 2021 20:23:20 +0000 Thu, 24 Sep 2020 21:39:54 +0000 0 4 <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@price-laptop:~/pfs/pipe2d-618 $ sqlite3 obslog.sqlite3 sqlite&gt; .mode csv sqlite&gt; .<span class="code-keyword">import</span> PIPE2D-618.csv obslog sqlite&gt; .schema CREATE TABLE obslog( <span class="code-quote">"cmd_str"</span> TEXT, <span class="code-quote">"pfs_visit_id"</span> TEXT, <span class="code-quote">"visit_set_id"</span> TEXT, <span class="code-quote">"exp_type"</span> TEXT, <span class="code-quote">"time_exp_start"</span> TEXT, <span class="code-quote">"notes"</span> TEXT, <span class="code-quote">"data_flag"</span> TEXT, <span class="code-quote">"comments"</span> TEXT, <span class="code-quote">"name"</span> TEXT, <span class="code-quote">"status"</span> TEXT, <span class="code-quote">"sequence_type"</span> TEXT, <span class="code-quote">"sps_module_id"</span> TEXT, <span class="code-quote">"arm"</span> TEXT, <span class="code-quote">"sps_camera_id"</span> TEXT ); sqlite&gt; select count(*) from obslog; 2596 sqlite&gt; select min(pfs_visit_id), max(pfs_visit_id) from obslog; 18218,19555 Need to: * Generate suitable bias (18218..18227), dark (18188..18202), flat (17839..17982). * Bootstrap a detectorMap (flat: 18231, arc: 18232) * For each set: + <span class="code-object">Process</span> flats as a fiberTrace + <span class="code-object">Process</span> each arc independently (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-618 $ generateCommands.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-618 --blocks=pipe2d_618 -j 20 pipe2d-618.yaml calibs.sh (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-618 $ ./calibs.sh Bootstrap r: bootstrap INFO: Median difference from detectorMap: -1.251848,-1.112205 pixels bootstrap INFO: Fit 116/141 points, rms: x=0.713381 y=0.189416 total=0.458008 pixels bootstrap INFO: Updating detectorMap... bootstrap INFO: Median difference from detectorMap: 0.685743,-1.230428 pixels bootstrap INFO: Fit 77/96 points, rms: x=0.349973 y=0.116337 total=0.121325 pixels Bootstrap b: bootstrap INFO: Median difference from detectorMap: 0.244493,-0.289952 pixels bootstrap INFO: Fit 8/12 points, rms: x=0.088476 y=0.142863 total=0.089313 pixels bootstrap INFO: Updating detectorMap... bootstrap INFO: Median difference from detectorMap: 1.236342,-0.364347 pixels bootstrap INFO: Fit 7/7 points, rms: x=0.258951 y=0.009806 total=0.178473 pixels generateCommands isn't going to work <span class="code-keyword">for</span> the main processing, since we want to run reduceArc on each arc individually (to detect any <span class="code-object">short</span> timescale motion). --&gt; fiberTrace.sh (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-618 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --output=/projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy --config clobber=True register.ignore=True -- /projects/HSC/PFS/Subaru/rerun/price/pipe2d-618/fiberTrace/FIBERTRACE/*.fits The problem now is that the butler doesn't recognise visit0 (can only have a single fiberTrace per day). So we'll hack the calibRegistry when we want to turn the fiberTrace over. --&gt; arcs.sh </pre> </div></div> <p>Data is in <tt>/projects/HSC/PFS/Subaru/rerun/price/pipe2d-618/arc</tt>.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=sogo.mineo" class="user-hover" rel="sogo.mineo">sogo.mineo</a>: Can you look at Paul's YAML file and related scripts please? Does this help in your work in extracting information from the opDB for your DRP processing framework? (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-423" title="Regular DRP processing of LAM and Subaru data" class="issue-link" data-issue-key="PIPE2D-423">PIPE2D-423</a> and related tickets)</p> <p>The yaml file declares that bias is made from "visit=18218..18227", but these are not the only bias exposures in obslog. What is the logic behind the choice of "visit=18218..18227" among others?</p> <p>I wanted a single bias for the whole dataset (my job wasn't to examine the biases for changes, just the arcs) to avoid annoyances from having to modify the calib registry (like I had to for the fiberTraces). That set of biases is close to the dark, which is probably why I preferred it over other sets.</p> <p><tt><a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-618" title="Process SM1 stability data" class="issue-link" data-issue-key="PIPE2D-618"><del>PIPE2D-618</del></a>.csv</tt> does not contain the darks and flats that are specified in the YAML file (though I don't know whether it is problematic or not... I am interested in how to choose files from the opDB in order to automate the choice)</p> <p>I think this work is OK &#8211; Tanaka-san told me that it is not my task to decide which exposures to use. Sorry for my misunderstanding.</p> Relates PIPE2D-423 Development Rank 0|02qps2:k Reviewers sogo.mineo Sprint 2DDRP-2021 A Story Points 10.0