https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-891 DRP 2-D Pipeline <p>Determine how repeatable 2D line flux measurements are, and what is limiting this repeatability.</p> PIPE2D-891

Investigate line flux repeatability

Story Normal Done Done price price Wed, 25 Aug 2021 21:04:06 +0000 Thu, 30 Sep 2021 14:55:59 +0000 Thu, 23 Sep 2021 19:27:29 +0000 0 3 <p>I followed multiple ideas to improve the flux measurements, many of which were based in improving the detectorMaps:</p> <ul> <li>add <tt>DoubleDetectorMap</tt></li> <li>deblend when centroiding</li> <li>flag lines with bad pixels when photometering</li> <li>review line list</li> <li>signal-to-noise cut in fitting detectorMap</li> </ul> <p>In the end, I got down to about <tt>xSys=0.016 ySys=0.037</tt> in fitting the detectorMap using sky lines (with <tt>readLineList.exclusionRadius=0.2</tt>):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.adjustDetectorMap INFO: Final fit: chi2=54099.876549 dof=23522 xRMS=0.024463 yRMS=0.037982 (0.003280 nm) from 11773/15601 lines reduceExposure.adjustDetectorMap INFO: Fit quality from reserved lines: chi2=614201.080218 xRMS=0.024387 yRMS=0.064839 (0.005599 nm) from 1733 lines (10.0%) reduceExposure.adjustDetectorMap INFO: Softening errors by x=0.016433, y=0.036511 pixels (0.003153 nm) to yield chi^2/dof=1 reduceExposure.adjustDetectorMap INFO: Softened fit: chi2=23387.268754 dof=23522 xRMS=0.025453 yRMS=0.042447 (0.003665 nm) from 11773 lines reduceExposure.adjustDetectorMap INFO: Softened fit quality from reserved lines: chi2=244358.657181 xRMS=0.024418 yRMS=0.064045 (0.005530 nm) from 1733 lines </pre> </div></div> <p>The difference between <tt>xSys</tt> and <tt>ySys</tt> likely indicates that there's more to be gained, but I'm not sure where yet. (<tt>xSys =~ ySys</tt> on arc lines, so the problem is intrinsic to the sky.)</p> <p>With that detectorMap, I then get much better photometry, though still not ideal: chi^2 for the worst line fluxes is about 30 over five fibers, and that's when I include a 1% systematic floor (although before this work, the worst chi^2 was tens of thousands). It's possible that small inaccuracies in the PSF model are causing extra noise, and I think adding aperture corrections is the next step: if the fidelity of the PSF varies as a function of fiber or position within a fiber (which could be due to the PSF model failing to follow real variation, or the PSF model variation not being accurate), then we need aperture corrections.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">import</span> numpy as np from pfs.drp.stella <span class="code-keyword">import</span> ArcLineSet lines = ArcLineSet.readFits(<span class="code-quote">"arcLines-046588-r1.fits"</span>) select = np.isin(lines.fiberId, (255, 401, 464, 525, 587)) &amp; np.isfinite(lines.intensity) &amp; np.isfinite(lines.intensityErr) &amp; (lines.status == 0) &amp; ~lines.flag intensityErr = np.where(lines.intensityErr &gt; 0.01*lines.intensity, lines.intensityErr, 0.01*lines.intensity) mean = {} chi2 = {} <span class="code-keyword">for</span> wl in set(lines.wavelength[select]): choose = select &amp; (lines.wavelength == wl) weight = 1.0/intensityErr[choose]**2 mean[wl] = np.sum(lines.intensity[choose]*weight)/weight.sum() chi2[wl] = (((lines.intensity[choose] - mean[wl])/intensityErr[choose])**2).sum() worst = sorted(chi2.keys(), key=lambda wl: chi2[wl], reverse=True) print([(wl, chi2[wl]) <span class="code-keyword">for</span> wl in worst[:10]]) </pre> </div></div> <p>yields the result:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [(753.27865, 30.999020699146175), (777.551, 30.257735938302954), (734.2907, 29.114637957334242), (885.22485, 25.031808119535704), (846.76845, 22.07350674889075), (691.45295, 21.767605415662246), (737.13955, 18.922802997897875), (854.10225, 18.811584139994014), (894.58505, 17.468980257836517), (961.0362, 16.9003032349872)] </pre> </div></div> <p>Oh, I should mention that the OH doublets that we combined into single lines in our linelist are typically separated by something like a tenth (e.g., 6923.152+6923.287) to a few tenths (e.g., 7401.688+7402.029) of an Angstrom, which is about a tenth of a pixel to a few tenths of a pixel in <tt>arm=r</tt>, so maybe we shouldn't be surprised that there's an additional 0.03 pixel systematic error in the spectral dimension compared to the spatial. Centroiding on galaxies is noisier than centroiding on stars. Not sure if the measurement errors should account for that or not, but I don't think we need to worry too much.</p> <p>Reviewed the changes in pfs_utils specifically to allow the repair of the integration test. Those specific changes are fine.</p> <p>Now all changes approved. No major comments.</p> <p>Merged.</p> <p>I'm not sure that the blending is to blame. The raw centroid error will be a bit worse, but the code should be handling that (it's proportional to the FWHM/(SN)), but that should be very small for bright lines. The usual problem with galaxies is that the PSF changes, and that that changes the bias for the galaxy centroid. Is there any evidence of this?</p> <p>Adding the deblender made very little difference, if any.</p> <p>I'm not sure that I understand. You mean, simultaneous fitting of a double-line model with known splittings didn't improve the centroid repeatability?</p> <p>No, I mean adding an SDSS-style deblender (removing pixels that don't belong to the peak of interest) didn't help.</p> <p>I don't think I'd do that! This is much closer to forced-photometry part of the the crowded field problem, with low-spatial order tweaks to the "astrometry".</p> Relates PIPE2D-829 Development Rank 0|02qpt6:508yl0chzzy000002jr Reviewers hassan Sprint 2DDRP-2021 A 8 2DDRP-2021 A 9 Story Points 2.0