https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1196 DRP 2-D Pipeline <p>about bootstrapping <b>detectorMap:</b><br/> the recent change of <b>pfsConfig</b> which now include the engineering fibers was not propagated (that's my guess) everywhere.<br/> so the number of fiber really illuminated does not match with pfsConfig</p> <p>output for n2 at LAM:<br/> I used 10 fibers</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bootstrap.profiles WARN: Failed to identify 24 fiberIds: [653, 683, 696, 743, 762, 788, 835, 874, 880, 924, 940, 967, 990, 1033, 1069, 1077, 1122, 1166, 1169, 1211, 1258, 1272, 1301, 1302] bootstrap.profiles INFO: Profiled 10 fibers bootstrap INFO: Found 10 fibers on flat</pre> </div></div> <p>but there is an other issue, because there is 16 engineering fibers so we should have 26 fibers reported by pfsConfig, 2 are missing (652 and 1033)</p> <p>eng. fiberIds are : ([ 652, 696, 743, 788, 835, 880, 924, 967, 987, 1033, 1077, 1122, 1166, 1211, 1258, 1302] for sm2</p> PIPE2D-1196

Bootstrap a detectorMap failed

Bug Blocker Done Done price fmadec Thu, 30 Mar 2023 12:53:33 +0000 Wed, 12 Apr 2023 21:23:11 +0000 Wed, 12 Apr 2023 21:23:11 +0000 0 3 <p>can you post your command ? so we can reproduce the problem ?</p> <p>And the log with backtrace that shows that the command failed.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-python"> (lsst-scipipe-3.0.0) [fmadec@luigi ~]$ bootstrapDetectorMap.py /data/drp/sps --calib=/data/drp/sps/CALIB/ --rerun fmadec/n2/bootstrap --flatId visit=48749 arm=n --arcId visit=48738 arm=n spectrograph=2 --no-versions --clobber-config -c isr.doBias=<span class="code-keyword"><span class="code-object">False</span></span> isr.doDark=<span class="code-keyword"><span class="code-object">False</span></span> isr.doFlat=<span class="code-keyword"><span class="code-object">False</span></span> isr.doLinearize=<span class="code-keyword"><span class="code-object">False</span></span> isr.doIPC=<span class="code-keyword"><span class="code-object">True</span></span> profiles.findThreshold=800 profiles.associationDepth=20 profiles.convolveGrowMask=2 profiles.pruneMinFrac=0.6 findLines.doSubtractContinuum=<span class="code-keyword"><span class="code-object">False</span></span> </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-python"> root INFO: Loading config overrride <span class="code-object">file</span> <span class="code-quote">'/software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2023.09/config/bootstrap.py'</span> /software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:778: <span class="code-object">FutureWarning</span>: Gen2 Butler has been deprecated (PfsButler). It will be removed sometime after v23.0 but no earlier than the end of 2021. namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) /software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:778: <span class="code-object">FutureWarning</span>: Gen2 Butler has been deprecated (PfsMapper). It will be removed sometime after v23.0 but no earlier than the end of 2021. namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) lsst.CameraMapper INFO: Loading exposure registry <span class="code-keyword">from</span> /data/drp/sps/registry.sqlite3 lsst.CameraMapper INFO: Loading calib registry <span class="code-keyword">from</span> /data/drp/sps/CALIB/calibRegistry.sqlite3 lsst.CameraMapper INFO: Loading calib registry <span class="code-keyword">from</span> /data/drp/sps/CALIB/calibRegistry.sqlite3 root INFO: Running: /software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.09/<span class="code-object">bin</span>/bootstrapDetectorMap.py /data/drp/sps --calib=/data/drp/sps/CALIB/ --rerun fmadec/n2/bootstrap --flatId visit=48749 arm=n --arcId visit=48738 arm=n spectrograph=2 --no-versions --clobber-config -c isr.doBias=<span class="code-keyword"><span class="code-object">False</span></span> isr.doDark=<span class="code-keyword"><span class="code-object">False</span></span> isr.doFlat=<span class="code-keyword"><span class="code-object">False</span></span> isr.doLinearize=<span class="code-keyword"><span class="code-object">False</span></span> isr.doIPC=<span class="code-keyword"><span class="code-object">True</span></span> profiles.findThreshold=800 profiles.associationDepth=20 profiles.convolveGrowMask=2 profiles.pruneMinFrac=0.6 findLines.doSubtractContinuum=<span class="code-keyword"><span class="code-object">False</span></span> bootstrap.isr INFO: Performing ISR on sensor {<span class="code-quote">'visit'</span>: 48749, <span class="code-quote">'arm'</span>: <span class="code-quote">'n'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2023-03-27'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'B'</span>, <span class="code-quote">'spectrograph'</span>: 2, <span class="code-quote">'field'</span>: <span class="code-quote">'TEST'</span>, <span class="code-quote">'ccd'</span>: 5, <span class="code-quote">'<span class="code-object">filter</span>'</span>: <span class="code-quote">'n'</span>, <span class="code-quote">'expTime'</span>: 10.0, <span class="code-quote">'dataType'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2023-03-27T15:49:32.097'</span>, <span class="code-quote">'pfsDesignId'</span>: 2743916541777272600, <span class="code-quote">'slitOffset'</span>: 0.0} bootstrap.profiles WARN: Failed to identify 24 fiberIds: [653, 683, 696, 743, 762, 788, 835, 874, 880, 924, 940, 967, 990, 1033, 1069, 1077, 1122, 1166, 1169, 1211, 1258, 1272, 1301, 1302] bootstrap.profiles INFO: Profiled 10 fibers bootstrap INFO: Found 10 fibers on flat bootstrap FATAL: Failed on dataId={<span class="code-quote">'visit'</span>: 48749, <span class="code-quote">'arm'</span>: <span class="code-quote">'n'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2023-03-27'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'B'</span>, <span class="code-quote">'spectrograph'</span>: 2, <span class="code-quote">'field'</span>: <span class="code-quote">'TEST'</span>, <span class="code-quote">'ccd'</span>: 5, <span class="code-quote">'<span class="code-object">filter</span>'</span>: <span class="code-quote">'n'</span>, <span class="code-quote">'expTime'</span>: 10.0, <span class="code-quote">'dataType'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2023-03-27T15:49:32.097'</span>, <span class="code-quote">'pfsDesignId'</span>: 2743916541777272600, <span class="code-quote">'slitOffset'</span>: 0.0}: <span class="code-object">RuntimeError</span>: Insufficient traces (10) found vs expected number of fibers (0) Traceback (most recent call last): File <span class="code-quote">"/software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 436, <span class="code-keyword">in</span> __call__ result = <span class="code-keyword">self</span>.runTask(task, dataRef, kwargs) File <span class="code-quote">"/software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 504, <span class="code-keyword">in</span> runTask <span class="code-keyword">return</span> task.runDataRef(dataRef, **kwargs) File <span class="code-quote">"/software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.09/python/pfs/drp/stella/bootstrap.py"</span>, line 143, <span class="code-keyword">in</span> runDataRef traces = <span class="code-keyword">self</span>.traceFibers(flatRef, flatConfig) File <span class="code-quote">"/software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.09/python/pfs/drp/stella/bootstrap.py"</span>, line 191, <span class="code-keyword">in</span> traceFibers <span class="code-keyword">raise</span> <span class="code-object">RuntimeError</span>(<span class="code-quote">"Insufficient traces (%d) found vs expected number of fibers (%d)"</span> % <span class="code-object">RuntimeError</span>: Insufficient traces (10) found vs expected number of fibers (0) </pre> </div></div> <p>Why is it expecting to find zero fibers? What's in the pfsConfig?</p> <p>it first says:</p> <p>pfsDesignId': 2743916541777272600 :</p> <p> bootstrap.profiles WARN: Failed to identify 24 fiberIds: <span class="error">&#91;653, 683, 696, 743, 762, 788, 835, 874, 880, 924, 940, 967, 990, 1033, 1069, 1077, 1122, 1166, 1169, 1211, 1258, 1272, 1301, 1302&#93;</span></p> <p>why it then says 0 fibers, I don't know...</p> <p> </p> <p>I think it's because of the new targetType DCB</p> <p>BootstrapConfig define the expected targetTypes to allow</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>targetType = ListField(dtype=str, default=["SCIENCE", "SKY", "FLUXSTD", "SUNSS_IMAGING", "SUNSS_DIFFUSE"], doc="Target types to allow") </pre> </div></div> <p>We just need to add "DCB" in there.</p> <p>I build on the branch and the run the command again, this error no longer comes up.<br/> I have another error though : ValueError: On entry to DLASCL parameter number 4 had an illegal value</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(lsst-scipipe-3.0.0) [pfs@mario drp_stella]$ bootstrapDetectorMap.py /data/drp/sps --calib=/data/drp/sps/CALIB/ --rerun alefur/n2/bootstrap --flatId visit=48749 arm=n --arcId visit=48738 arm=n spectrograph=2 --no-versions --clobber-config -c isr.doBias=False isr.doDark=False isr.doFlat=False isr.doLinearize=False isr.doIPC=True profiles.findThreshold=800 profiles.associationDepth=20 profiles.convolveGrowMask=2 profiles.pruneMinFrac=0.6 findLines.doSubtractContinuum=False root INFO: Loading config overrride file '/software/devel/alefur/obs_pfs/config/bootstrap.py' /software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:778: FutureWarning: Gen2 Butler has been deprecated (PfsButler) . It will be removed sometime after v23.0 but no earlier than the end of 2021. namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) /software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:778: FutureWarning: Gen2 Butler has been deprecated (PfsMapper) . It will be removed sometime after v23.0 but no earlier than the end of 2021. namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) lsst.CameraMapper INFO: Loading exposure registry from /data/drp/sps/registry.sqlite3 lsst.CameraMapper INFO: Loading calib registry from /data/drp/sps/CALIB/calibRegistry.sqlite3 lsst.CameraMapper INFO: Loading calib registry from /data/drp/sps/CALIB/calibRegistry.sqlite3 root INFO: Running: /software/devel/alefur/drp_stella/bin/bootstrapDetectorMap.py /data/drp/sps --calib=/data/drp/sps/CALIB/ --rerun alefur/n2/bootstrap --flatId visit=48749 arm=n --arcId visit=48738 arm=n spectrograph=2 --no-versions --clobber-config -c isr.doBias=False isr.doDark=False isr.doFlat=False isr.doLinearize=False isr.doIPC=True profiles.findThreshold=800 profiles.associationDepth=20 profiles.convolveGrowMask=2 profiles.pruneMinFrac=0.6 findLines.doSubtractContinuum=False bootstrap.isr INFO: Performing ISR on sensor {'visit': 48749, 'arm': 'n', 'dateObs': '2023-03-27', 'site': 'L', 'category': 'B', 'spectrograph': 2, 'field': 'TEST', 'ccd': 5, 'filter': 'n', 'expTime': 10.0, 'dataType': 'FLAT', 'taiObs': '2023-03-27T15:49:32.097', 'pfsDesignId': 2743916541777272600, 'slitOffset': 0.0} bootstrap.profiles WARN: Failed to identify 24 fiberIds: [653, 683, 696, 743, 762, 788, 835, 874, 880, 924, 940, 967, 990, 1033, 1069, 1077, 1122, 1166, 1169, 1211, 1258, 1272, 1301, 1302] bootstrap.profiles INFO: Profiled 10 fibers bootstrap INFO: Found 10 fibers on flat bootstrap.isr INFO: Performing ISR on sensor {'visit': 48738, 'arm': 'n', 'spectrograph': 2, 'dateObs': '2023-03-27', 'site': 'L', 'category': 'B', 'field': 'TEST', 'ccd': 5, 'filter': 'n', 'expTime': 18.0, 'dataType': 'COMPARISON', 'taiObs': '2023-03-27T15:10:45.436', 'pfsDesignId': 2743916541777272600, 'slitOffset': 0.0} bootstrap.readLineList INFO: Filtered line lists, keeping species {'KrI'}. bootstrap INFO: Found 2341 lines in 10 traces bootstrap INFO: Matched 629 lines bootstrap INFO: Median difference from detectorMap: nan,nan pixels bootstrap FATAL: Failed on dataId={'visit': 48749, 'arm': 'n', 'dateObs': '2023-03-27', 'site': 'L', 'category': 'B', 'spectrograph': 2, 'field': 'TEST', 'ccd': 5, 'filter': 'n', 'expTime': 10.0, 'dataType': 'FLAT', 'taiObs': '2023-03-27T15:49:32.097', 'pfsDesignId': 2743916541777272600, 'slitOffset': 0.0}: ValueError: On entry to DLASCL parameter number 4 had an illegal value Traceback (most recent call last): File "/software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 436, in __call__ result = self.runTask(task, dataRef, kwargs) File "/software/stack/20220609/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 504, in runTask return task.runDataRef(dataRef, **kwargs) File "/software/devel/alefur/drp_stella/python/pfs/drp/stella/bootstrap.py", line 153, in runDataRef self.fitDetectorMap(matches, lineResults.detectorMap, fiberId) File "/software/devel/alefur/drp_stella/python/pfs/drp/stella/bootstrap.py", line 425, in fitDetectorMap result = fitChebyshev2D(xx, yy, self.config.spatialOrder, self.config.spectralOrder, File "/software/devel/alefur/drp_stella/python/pfs/drp/stella/bootstrap.py", line 609, in fitChebyshev2D xFit = fitter(model, xx[0][good], xx[1][good], yy[0][good]) File "/software/stack/20220609/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/astropy/modeling/fitting.py", line 269, in wrapper return func(self, model, x, y, z=z, **kwargs) File "/software/stack/20220609/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/astropy/modeling/fitting.py", line 771, in __call__ lacoef, resids, rank, sval = np.linalg.lstsq(lhs[good], File "&lt;__array_function__ internals&gt;", line 5, in lstsq File "/software/stack/20220609/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/numpy/linalg/linalg.py", line 2305, in lstsq x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj) ValueError: On entry to DLASCL parameter number 4 had an illegal value </pre> </div></div> <p>NANs were getting into the positions used for the fit, presumably from the detectorMap being given wavelengths slightly off the detector. I've added a change to exclude these.</p> <p>merged to master</p> Development Rank 0|zzswnj: Reviewers price