https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1362 DRP 2-D Pipeline <p>After applying the fix for <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1361" class="external-link" rel="nofollow">PIPD2D-1361</a>, we clearly observe the spatial variation of the flux calibration vector. The variation likely has two components; the global pattern due to the global offset, rotation, and scale difference, and fiber-to-fiber offset. We want to correct for the former with, e.g., low-order polinomials. As the seeing is a function of wavelength, a fiber offset introduces a wavelength-dependent flux loss. We thus need to allow the flux calibration vector to vary both spatially and along the wavelength. The correction can be estimated by comparing the PS1 magnitudes and synthetic PS1 magnitudes of each FLUXSTD.</p> PIPE2D-1362

correct for the focal plane variation of the flux calibration vector

Task Normal Review Complete Unresolved sogo.mineo Masayuki Tanaka EDR EngRun Wed, 31 Jan 2024 00:24:05 +0000 Fri, 9 Feb 2024 19:10:05 +0000 0 4 <p>The figures here show the magnitude and color offsets with respect to the PS1 photometry. The first figure is the i-band magnitude offset and the 2nd one is g-y color offset. Note the similar spatial patterns. This is due to the wavelength dependence of the seeing.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/17100/17100_fig1_nocorr.png" height="430" width="474" style="border: 0px solid black" /></span><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/17101/17101_fig2_nocorr.png" height="427" width="471" style="border: 0px solid black" /></span></p> <p>We have to think about how much the fibre-to-fibre variation is corrected by the fibre profiles (and associated normalisation). That's done using a roughly uniform surface brightness, but it should take out the fibre throughput variation &#8211; although it's hard to separate this from spatial variation. </p> <p>I should have been clearer. What I meant by the fiber-to-fiber variation is the residual fiber offset after we take out the global offset/pattern. I imagine the residual offset is nearly random (but I may be wrong). I agree that the throughput variation between the fibers will be largely taken out by the quartz.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=sogo.mineo" class="user-hover" rel="sogo.mineo">sogo.mineo</a> already implemented a correction function in the flux calibration code (I should have filed this ticket when he started, sorry). He can elaborate on the algorithmic details, but here are the plots after the correction is applied. The first set of the figures is for 3rd order correction and the 2nd set for 4th order correction.<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/17104/17104_fig1_3rd_order.png" height="385" width="425" style="border: 0px solid black" /></span><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/17105/17105_fig2_3rd_order.png" height="383" width="422" style="border: 0px solid black" /></span></p> <p>4th order:</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/17106/17106_fig1_4th_order.png" height="371" width="409" style="border: 0px solid black" /></span><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/17107/17107_fig2_4th_order.png" height="372" width="410" style="border: 0px solid black" /></span></p> <p> </p> <p>The details of fitting <tt>FluxCalib</tt> (a subclass of <tt>FocalPlaneFunction</tt>) is:</p> <p>1. Divide observed spectra of flux standards by reference spectra to get proto-fluxCalib vectors.<br/> 2. Adjust heights of the proto-fluxCalib vectors to each other.<br/> 3. Take their average to get h(lam) (<tt>ConstantFocalPlaneFunction</tt>)<br/> 4. Let g(x, y, lam) be a trivariate polynomial (of order 3 by default).<br/> 5. Let f(x, y, lam) = h(lam) exp(g(x, y, lam)). This is a <tt>FluxCalib</tt>.<br/> 6. Let (calibrated spectrum) = (observed spectrum) / f(x<span class="error">&#91;i&#93;</span>, y<span class="error">&#91;i&#93;</span>, lam) for each flux standard.<br/> 7. Integrate calibrated spectra to get synthetic broadband fluxes.<br/> 8. Let chisq be the sum of ((broadband flux in pfsConfig) - (synthetic broadband flux))^2 / error^2<br/> 9. Minimize chisq.</p> <p>So, I distort the average fluxCalib h(lam) to forcibly make it conform to pfsConfig.psfFlux.<br/> Though I am not sure the resulting fluxCalib is good for spectroscopy, Tanaka-san's images show that the distortion is working well.</p> <p>Could you review PRs of <tt>datamodel</tt> and <tt>drp_stella</tt>, too?</p> <p>The PRs have been approved, but I just found that the integration test would break during <tt>coaddSpectra.py</tt> because an instance of <tt>FluxCalib</tt> class (named "fluxCal" in butler) cannot be created. The reason for this is that <tt>FluxCalib</tt> is defined outside <tt>focalPlaneFunction.py</tt> even though it is a subclass of <tt>FocalPlaneFunction</tt>.</p> <p>I decided to define <tt>FluxCalib</tt> in <tt>fitFluxCal.py</tt> because its overrided method <tt>fitArrays()</tt> has a parameter of type <tt>BroadbandFluxChi2</tt>. This class is not general enough to be known to <tt>focalPlaneFunction.py</tt>.</p> <p>This problem is solved if I revise <tt>PfsMapper.yaml</tt> in <tt>obs_pfs</tt> so that the type of <tt>"fluxCal"</tt> will be <tt>FluxCalib</tt>. But if I do so, the output of <tt>fluxCalibrate.py</tt> will not be able to be read by <tt>coaddSpectra.py</tt>,</p> <p>Does anyone have an idea to solve this problem?</p> <p>I pushed another commit. I moved <tt>FluxCalib</tt> class to <tt>focalPlaneFunction.py</tt>. Its method <tt>fitArrays()</tt> takes a parameter <tt>fitter</tt>, and all that <tt>fitArrays()</tt> does is to call <tt>fitter()</tt>. After this commit, the Gen2 integration test passes.</p> <p>Gen3 test still fails:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 24 quanta successfully, 0 failed and 3 remain out of total 27 quanta. py.warnings WARNING: /data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/utils/polynomialND.py:76: RuntimeWarning: divide by zero encountered in true_divide self._scale = 1.0 / (self._max - self._min) py.warnings WARNING: /data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/utils/polynomialND.py:130: RuntimeWarning: invalid value encountered in multiply x *= scale lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task <span class="code-quote">'fitFluxCal'</span> on quantum {instrument: <span class="code-quote">'PFS-F'</span>, exposure: 24, ...} failed. Exception RuntimeError: No good points <span class="code-object">Process</span> task-{instrument: <span class="code-quote">'PFS-F'</span>, exposure: 24, ...}: Traceback (most recent call last): File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/process.py"</span>, line 315, in _bootstrap self.run() File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/process.py"</span>, line 108, in run self._target(*self._args, **self._kwargs) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/mpGraphExecutor.py"</span>, line 143, in _executeJob quantumExecutor.execute(taskDef, quantum, butler) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/singleQuantumExecutor.py"</span>, line 135, in execute result = self._execute(taskDef, quantum, butler) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/singleQuantumExecutor.py"</span>, line 212, in _execute self.runQuantum(runTask, quantum, taskDef, butler) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/singleQuantumExecutor.py"</span>, line 580, in runQuantum task.runQuantum(butlerQC, inputRefs, outputRefs) File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/fitFluxCal.py"</span>, line 761, in runQuantum outputs = self.run(**inputs, pfsArmList=armInputs.pfsArm, sky1dList=armInputs.sky1d) File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/fitFluxCal.py"</span>, line 703, in run fluxCal = self.calculateCalibrations(pfsConfig, pfsMerged, pfsMergedLsf, references) File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/fitFluxCal.py"</span>, line 900, in calculateCalibrations <span class="code-keyword">return</span> self.fitFocalPlane.run( File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/fitFocalPlane.py"</span>, line 109, in run raise RuntimeError(<span class="code-quote">"No good points"</span>) RuntimeError: No good points lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last): File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/cli/cmd/commands.py"</span>, line 130, in run script.run(qgraphObj=qgraph, **kwargs) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/cli/script/run.py"</span>, line 187, in run f.runPipeline(qgraphObj, taskFactory, args) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/cmdLineFwk.py"</span>, line 740, in runPipeline executor.execute(graph, butler) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/mpGraphExecutor.py"</span>, line 373, in execute self._executeQuantaMP(graph, butler) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/mpGraphExecutor.py"</span>, line 568, in _executeQuantaMP raise MPGraphExecutorError(message) lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: Task &lt;TaskDef(pfs.drp.stella.fitFluxCal.FitFluxCalTask, label=fitFluxCal) dataId={instrument: <span class="code-quote">'PFS-F'</span>, exposure: 24, ...}&gt; failed, exit code=1 </pre> </div></div> <p>Is this error the one that <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> told me a few weeks ago?</p> <p>Yes, I think that's the same one. It would be great if you could also fix that.</p> Relates PIPE2D-1361 Development Epic Link PIPE2D-303 Rank 0|zzt14f: Reviewers price