https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-641 DRP 2-D Pipeline <p>The weekly got stuck, apparently running for a very long time attempting to fit a <tt>GlobalDetectorMap</tt> to the full 600 fibers.</p> <p>Investigate, fix and re-enable the weekly.</p> PIPE2D-641

Optimise fitGlobalDetectorMap on full fiber density

Story Normal Done Done price price Fri, 2 Oct 2020 19:50:31 +0000 Mon, 4 Jan 2021 20:23:22 +0000 Mon, 16 Nov 2020 21:09:39 +0000 0 2 <p>The attached script, <tt>fitLines.py</tt>, running on 130k arc line positions from four exposures from the (full fiber density) weekly dataset, uses linear least-squares rather than downhill minimisation, and gets decent results: softening values of 0.036 pixels (0.021 spatial, 0.054 spectral). With the proof of concept validated, I'll implement it in the codebase.</p> <p>Finally got it implemented (C++ always takes longer than I think). I think the fits are ever so slightly better than before, and much faster.</p> <p>Here's what I get from running on the weekly (full fiber density):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-641%] $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -d /projects/HSC/PFS/weekly -r pipe2d-641 -c 10 /scratch/pprice/pipe2d-641 (lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-641%] $ reduceArc.py /scratch/pprice/pipe2d-641 --calib=/scratch/pprice/pipe2d-641/CALIB --rerun=pipe2d-641/arcs --doraise --id visit=39..45:2 | tee brn.log (lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-641%] $ reduceArc.py /scratch/pprice/pipe2d-641 --calib=/scratch/pprice/pipe2d-641/CALIB --rerun=pipe2d-641/arcs --doraise --id visit=40..46:2 arm=m | tee m.log b: reduceArc.fitDetectorMap INFO: Final fit: chi2=296454.633021 xRMS=0.038589 yRMS=0.092824 (0.006306 nm, 3.655250 km/s) from 119697/142132 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=219549.121436 xRMS=0.059681 yRMS=0.185631 (0.012611 nm, 7.309855 km/s) from 15793 lines (10.0%) reduceArc.fitDetectorMap INFO: Softening errors by 0.030189 pixels (0.002051 nm, 1.188785 km/s) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=238319.660268 xRMS=0.044844 yRMS=0.112083 (0.007614 nm, 4.413650 km/s) from 119697/157925 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=151455.364161 xRMS=0.059557 yRMS=0.185325 (0.012590 nm, 7.297814 km/s) from 15793 lines (10.0%) r: reduceArc.fitDetectorMap INFO: Final fit: chi2=208448.545078 xRMS=0.032496 yRMS=0.053421 (0.004602 nm, 1.718851 km/s) from 102209/120699 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=314353.249240 xRMS=0.070175 yRMS=0.126513 (0.010898 nm, 4.070620 km/s) from 13411 lines (10.0%) reduceArc.fitDetectorMap INFO: Softening errors by 0.010789 pixels (0.000929 nm, 0.347150 km/s) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=204322.631086 xRMS=0.033078 yRMS=0.054438 (0.004689 nm, 1.751552 km/s) from 102209/134110 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=287979.832164 xRMS=0.070154 yRMS=0.126540 (0.010901 nm, 4.071493 km/s) from 13411 lines (10.0%) n: reduceArc.fitDetectorMap INFO: Final fit: chi2=105463.539670 xRMS=0.035957 yRMS=0.063009 (0.005107 nm, 1.387816 km/s) from 56019/62746 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=162841.441183 xRMS=0.072830 yRMS=0.127715 (0.010351 nm, 2.813034 km/s) from 6972 lines (10.0%) reduceArc.fitDetectorMap INFO: Softening errors by 0.006646 pixels (0.000539 nm, 0.146384 km/s) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=112340.274352 xRMS=0.034029 yRMS=0.059611 (0.004831 nm, 1.312986 km/s) from 56019/69718 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=196057.234933 xRMS=0.073124 yRMS=0.128263 (0.010395 nm, 2.825090 km/s) from 6972 lines (10.0%) m: reduceArc.fitDetectorMap INFO: Final fit: chi2=124275.154422 xRMS=0.031840 yRMS=0.051163 (0.002404 nm, 0.902800 km/s) from 61373/74706 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=826248.248961 xRMS=0.068049 yRMS=0.120112 (0.005645 nm, 2.119446 km/s) from 8301 lines (10.0%) reduceArc.fitDetectorMap INFO: Softening errors by 0.010383 pixels (0.000488 nm, 0.183208 km/s) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=122688.367620 xRMS=0.032146 yRMS=0.051587 (0.002424 nm, 0.910282 km/s) from 61373/83007 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=789833.129258 xRMS=0.068014 yRMS=0.120107 (0.005644 nm, 2.119350 km/s) from 8301 lines (10.0%) </pre> </div></div> <p>And here's the result of running on a real frame (low fiber density) from Subaru:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-641%] $ reduceArc.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-641 --id visit=18232 arm=r reduceArc.fitDetectorMap INFO: Final fit: chi2=188.197839 xRMS=0.016024 yRMS=0.014622 (0.001260 nm, 0.470436 km/s) from 167/245 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=1106.159286 xRMS=0.143804 yRMS=0.065177 (0.005617 nm, 2.096998 km/s) from 27 lines (9.9%) reduceArc.fitDetectorMap INFO: No softening necessary (Note that the fit includes a 0.01 pixel error softening, to avoid giving too much weight to high-S/N points and causing lots to be rejected.) Previously, <span class="code-keyword">for</span> the same data: reduceArc.fitDetectorMap INFO: Final fit: chi2=563.767972 xRMS=0.098677 yRMS=0.127012 (0.011062 nm, 4.132477 km/s) from 120/245 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=5078.160298 xRMS=0.163361 yRMS=0.119234 (0.010384 nm, 3.879419 km/s) from 27 lines (9.9%) reduceArc.fitDetectorMap INFO: Softening errors by 0.017188 pixels (0.001497 nm, 0.559219 km/s) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=175.235651 xRMS=0.104461 yRMS=0.126252 (0.010934 nm, 4.084716 km/s) from 120/272 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=403.334350 xRMS=0.100842 yRMS=0.056601 (0.004902 nm, 1.831252 km/s) from 27 lines (9.9%) </pre> </div></div> <p>Merged to master, re-enabled the weekly and started a run manually.</p> Blocks PIPE2D-644 Development Rank 0|02qps2:q Reviewers hassan Sprint 2DDRP-2021 A