https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1215 DRP 2-D Pipeline <p>Please check the quality of the detectorMaps generated using the data of the engineering run in 2023 Apr. A script for detectorMap QA developed by Hamano-san is supposed to be used here.</p> <p> </p> <p>Edit (wtg):  As discussed in the 2D-DRP meeting, I will first be checking the quality of the existing detectorMaps (created before the 2023 Apr run) before generating (#1214) and checking new detectorMaps.  See also the comment in 1214.</p> PIPE2D-1215

Update the 2D plot part of detectorMap QA code

Task Normal Open Unresolved Wilfred Gee Kiyoto Yabe EDR EngRun QualityAssurance Thu, 11 May 2023 06:32:38 +0000 Tue, 6 Feb 2024 01:00:35 +0000 0 3 <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> can you let me know the location of Hamano-san's QA scripts, as mentioned in the 2D-DRP meeting? Thanks!</p> <p>The codes are now here: /work/shamano/pfs_qa</p> <p> </p> <p>How does this relate to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1146" title="QA checks: continuum flux and wavelength of atmospheric features" class="issue-link" data-issue-key="PIPE2D-1146">PIPE2D-1146</a></p> <p>I'm just imaging that only residuals in wavelength and spatial direction (and structure over the detector) are checked here. I don't think Hamano-san's script checks <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1146" title="QA checks: continuum flux and wavelength of atmospheric features" class="issue-link" data-issue-key="PIPE2D-1146">PIPE2D-1146</a> yet, so if that ticket is done, we can check it as well.</p> <p>My mistake &#8211; I read <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1146" title="QA checks: continuum flux and wavelength of atmospheric features" class="issue-link" data-issue-key="PIPE2D-1146">PIPE2D-1146</a> too quickly. The OH lines are interesting, but not the absorption. すみません</p> <p>I've just started to look at the script from Hamano-san (and will be doing the initial import into the repo soon), but I think <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> is correct as the script only makes reference to the wavelength and spatial direction (script just says` Xcenter`).</p> <p>I talked to you the other day, but could you provide plots for Apr run here please?</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=wtgee" class="user-hover" rel="wtgee">Wilfred Gee</a> Could you tell me how to setup and use the latest version in the PR? I also would like to use that for the science frames we took recently.</p> <p>Hi Yabe-san,</p> <p>The module can be set up in the usual way. First clone the directory, checkout the branch, build with `scons`, then use `setup` to put into environment (as more or less outlined <a href="https://pipelines.lsst.io/install/package-development.html" class="external-link" rel="nofollow">https://pipelines.lsst.io/install/package-development.html</a>).</p> <p>I've already performed those steps on `pfsa-usr01`, so you could can just use `setup -jr /work/wtg/drp_qa` to have the `detectorMapQa.py` command available. Note that I still might make some future changes to the PR that would affect usage.</p> <p>Let me know if it gives you any trouble!</p> <p>Thank you. I think I could run the code. But it works only if I setup pfs_pipe2d for w.2023.27 (or earlier version) with the following code:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> setup pfs_pipe2d w.2023.30a setup -jr /work/kiyoyabe/erun/run12/src/obs_pfs setup -jr /work/kiyoyabe/erun/run12/src/drp_qa VISITS=97828 REPO=<span class="code-quote">'/work/drp'</span> RERUN=<span class="code-quote">'kiyoyabe/erun/run12/CALIB-2023-07-v3'</span> CALIB=<span class="code-quote">'/work/drp/CALIB-2023-07-v3'</span> detectorMapQa.py $REPO --calib $CALIB --rerun $RERUN --id visit=$VISITS arm=b^r --no-versions </pre> </div></div> <p>Could you confirm that we can use with the latest pipeline? Or maybe I did something wrong. The error message is like this:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:778: FutureWarning: Gen2 Butler has been deprecated (PfsButler). It will be removed sometime after v23.0 but no earlier than the end of 2021. namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) /work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:778: FutureWarning: Gen2 Butler has been deprecated (PfsMapper). It will be removed sometime after v23.0 but no earlier than the end of 2021. namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB-2023-07-v3/calibRegistry.sqlite3 lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB-2023-07-v3/calibRegistry.sqlite3 root INFO: Running: /work/kiyoyabe/erun/run12/src/drp_qa/bin/detectorMapQa.py /work/drp --calib /work/drp/CALIB-2023-07-v3 --rerun kiyoyabe/erun/run12/CALIB-2023-07-v3 --id visit=97828 arm=b^r --no-versions Traceback (most recent call last): File <span class="code-quote">"/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pandas/core/indexes/base.py"</span>, line 3621, in get_loc <span class="code-keyword">return</span> self._engine.get_loc(casted_key) File <span class="code-quote">"pandas/_libs/index.pyx"</span>, line 136, in pandas._libs.index.IndexEngine.get_loc File <span class="code-quote">"pandas/_libs/index.pyx"</span>, line 163, in pandas._libs.index.IndexEngine.get_loc File <span class="code-quote">"pandas/_libs/hashtable_class_helper.pxi"</span>, line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item File <span class="code-quote">"pandas/_libs/hashtable_class_helper.pxi"</span>, line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: <span class="code-quote">'Trace'</span> The above exception was the direct cause of the following exception: Traceback (most recent call last): File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 436, in __call__ result = self.runTask(task, dataRef, kwargs) File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 504, in runTask <span class="code-keyword">return</span> task.runDataRef(dataRef, **kwargs) File <span class="code-quote">"/work/kiyoyabe/erun/run12/src/drp_qa/python/pfs/qa/detectorMapQa.py"</span>, line 665, in runDataRef <span class="code-keyword">return</span> self.run(detectorMapList, arcLinesList, pfsArmList) File <span class="code-quote">"/work/kiyoyabe/erun/run12/src/drp_qa/python/pfs/qa/detectorMapQa.py"</span>, line 692, in run self.plotResidual.run(dd, aa, pp) File <span class="code-quote">"/work/kiyoyabe/erun/run12/src/drp_qa/python/pfs/qa/detectorMapQa.py"</span>, line 81, in run arc_data = stability.getArclineData(arcLines) File <span class="code-quote">"/work/kiyoyabe/erun/run12/src/drp_qa/python/pfs/qa/utils/stability.py"</span>, line 70, in getArclineData arc_data[<span class="code-quote">'Trace'</span>] = arc_data.description.str.get_dummies()[<span class="code-quote">'Trace'</span>] File <span class="code-quote">"/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pandas/core/frame.py"</span>, line 3505, in __getitem__ indexer = self.columns.get_loc(key) File <span class="code-quote">"/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pandas/core/indexes/base.py"</span>, line 3623, in get_loc raise KeyError(key) from err KeyError: <span class="code-quote">'Trace'</span> During handling of the above exception, another exception occurred: Traceback (most recent call last): File <span class="code-quote">"/work/kiyoyabe/erun/run12/src/drp_qa/bin/detectorMapQa.py"</span>, line 4, in &lt;module&gt; DetectorMapQaTask.parseAndRun() File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 688, in parseAndRun resultList = taskRunner.run(parsedCmd) File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 244, in run resultList = list(mapFunc(self, targetList)) File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 450, in __call__ <span class="code-quote">", "</span>.join(str(ref.dataId) <span class="code-keyword">for</span> ref in dataRef), File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 450, in &lt;genexpr&gt; <span class="code-quote">", "</span>.join(str(ref.dataId) <span class="code-keyword">for</span> ref in dataRef), AttributeError: <span class="code-quote">'list'</span> object has no attribute <span class="code-quote">'dataId'</span> </pre> </div></div> <p> </p> <p> </p> <p> </p> <p> </p> <p>Anyway, plots I could make are here: <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/16185/16185_detectorMapQA-097828-20230726.pdf" title="detectorMapQA-097828-20230726.pdf attached to PIPE2D-1215">detectorMapQA-097828-20230726.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>It seems that there are blank panel, so I must have done something wrong. So, I appreciate if you can make the plots for the same specific visit.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> I think I figured out why the one panel is broken, trying to fix now.</p> <p>I was using the pipeline as of last week, so more recent than the `w.2023.27`. I'll see if I can duplicate the error message above.</p> Blocks PIPE2D-1250 PIPE2D-1214 Development Rank 0|02qpjd:00r20060i200186403c14 Sprint EDR+AIT Story Points 1.0