[PIPE2D-641] Optimise fitGlobalDetectorMap on full fiber density Created: 03/Oct/20  Updated: 05/Jan/21  Resolved: 17/Nov/20

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File fitLines.py    
Issue Links:
Blocks
blocks PIPE2D-644 Create NIR simulated data and run the... Done
Sprint: 2DDRP-2021 A
Reviewers: hassan

 Description   

The weekly got stuck, apparently running for a very long time attempting to fit a GlobalDetectorMap to the full 600 fibers.

Investigate, fix and re-enable the weekly.

 Comments   
Comment by price [ 09/Oct/20 ]

The attached script, fitLines.py, running on 130k arc line positions from four exposures from the (full fiber density) weekly dataset, uses linear least-squares rather than downhill minimisation, and gets decent results: softening values of 0.036 pixels (0.021 spatial, 0.054 spectral). With the proof of concept validated, I'll implement it in the codebase.

Comment by price [ 31/Oct/20 ]

Finally got it implemented (C++ always takes longer than I think). I think the fits are ever so slightly better than before, and much faster.

Here's what I get from running on the weekly (full fiber density):

(lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-641%] $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -d /projects/HSC/PFS/weekly -r pipe2d-641 -c 10 /scratch/pprice/pipe2d-641


(lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-641%] $ reduceArc.py /scratch/pprice/pipe2d-641 --calib=/scratch/pprice/pipe2d-641/CALIB --rerun=pipe2d-641/arcs --doraise --id visit=39..45:2 | tee brn.log
(lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-641%] $ reduceArc.py /scratch/pprice/pipe2d-641 --calib=/scratch/pprice/pipe2d-641/CALIB --rerun=pipe2d-641/arcs --doraise --id visit=40..46:2 arm=m | tee m.log

b:

reduceArc.fitDetectorMap INFO: Final fit: chi2=296454.633021 xRMS=0.038589 yRMS=0.092824 (0.006306 nm, 3.655250 km/s) from 119697/142132 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=219549.121436 xRMS=0.059681 yRMS=0.185631 (0.012611 nm, 7.309855 km/s) from 15793 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.030189 pixels (0.002051 nm, 1.188785 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=238319.660268 xRMS=0.044844 yRMS=0.112083 (0.007614 nm, 4.413650 km/s) from 119697/157925 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=151455.364161 xRMS=0.059557 yRMS=0.185325 (0.012590 nm, 7.297814 km/s) from 15793 lines (10.0%)

r:

reduceArc.fitDetectorMap INFO: Final fit: chi2=208448.545078 xRMS=0.032496 yRMS=0.053421 (0.004602 nm, 1.718851 km/s) from 102209/120699 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=314353.249240 xRMS=0.070175 yRMS=0.126513 (0.010898 nm, 4.070620 km/s) from 13411 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.010789 pixels (0.000929 nm, 0.347150 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=204322.631086 xRMS=0.033078 yRMS=0.054438 (0.004689 nm, 1.751552 km/s) from 102209/134110 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=287979.832164 xRMS=0.070154 yRMS=0.126540 (0.010901 nm, 4.071493 km/s) from 13411 lines (10.0%)

n:

reduceArc.fitDetectorMap INFO: Final fit: chi2=105463.539670 xRMS=0.035957 yRMS=0.063009 (0.005107 nm, 1.387816 km/s) from 56019/62746 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=162841.441183 xRMS=0.072830 yRMS=0.127715 (0.010351 nm, 2.813034 km/s) from 6972 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.006646 pixels (0.000539 nm, 0.146384 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=112340.274352 xRMS=0.034029 yRMS=0.059611 (0.004831 nm, 1.312986 km/s) from 56019/69718 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=196057.234933 xRMS=0.073124 yRMS=0.128263 (0.010395 nm, 2.825090 km/s) from 6972 lines (10.0%)

m:

reduceArc.fitDetectorMap INFO: Final fit: chi2=124275.154422 xRMS=0.031840 yRMS=0.051163 (0.002404 nm, 0.902800 km/s) from 61373/74706 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=826248.248961 xRMS=0.068049 yRMS=0.120112 (0.005645 nm, 2.119446 km/s) from 8301 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.010383 pixels (0.000488 nm, 0.183208 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=122688.367620 xRMS=0.032146 yRMS=0.051587 (0.002424 nm, 0.910282 km/s) from 61373/83007 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=789833.129258 xRMS=0.068014 yRMS=0.120107 (0.005644 nm, 2.119350 km/s) from 8301 lines (10.0%)

And here's the result of running on a real frame (low fiber density) from Subaru:

(lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-641%] $ reduceArc.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-641 --id visit=18232 arm=r

reduceArc.fitDetectorMap INFO: Final fit: chi2=188.197839 xRMS=0.016024 yRMS=0.014622 (0.001260 nm, 0.470436 km/s) from 167/245 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=1106.159286 xRMS=0.143804 yRMS=0.065177 (0.005617 nm, 2.096998 km/s) from 27 lines (9.9%)
reduceArc.fitDetectorMap INFO: No softening necessary

(Note that the fit includes a 0.01 pixel error softening, to avoid giving too much weight to high-S/N points and causing lots to be rejected.)

Previously, for the same data:

reduceArc.fitDetectorMap INFO: Final fit: chi2=563.767972 xRMS=0.098677 yRMS=0.127012 (0.011062 nm, 4.132477 km/s) from 120/245 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=5078.160298 xRMS=0.163361 yRMS=0.119234 (0.010384 nm, 3.879419 km/s) from 27 lines (9.9%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.017188 pixels (0.001497 nm, 0.559219 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=175.235651 xRMS=0.104461 yRMS=0.126252 (0.010934 nm, 4.084716 km/s) from 120/272 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=403.334350 xRMS=0.100842 yRMS=0.056601 (0.004902 nm, 1.831252 km/s) from 27 lines (9.9%)
Comment by price [ 17/Nov/20 ]

Merged to master, re-enabled the weekly and started a run manually.

Generated at Sat Feb 10 15:55:43 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.