[PIPE2D-1215] Update the 2D plot part of detectorMap QA code Created: 11/May/23 Updated: 06/Feb/24 |
|
| Status: | Open |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Normal |
| Reporter: | Kiyoto Yabe | Assignee: | Wilfred Gee |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | EDR, EngRun, QualityAssurance | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
detectorMapQA-097828-20230726.pdf
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Story Points: | 1 | ||||||||||||
| Sprint: | EDR+AIT | ||||||||||||
| Description |
|
Please check the quality of the detectorMaps generated using the data of the engineering run in 2023 Apr. A script for detectorMap QA developed by Hamano-san is supposed to be used here.
Edit (wtg): As discussed in the 2D-DRP meeting, I will first be checking the quality of the existing detectorMaps (created before the 2023 Apr run) before generating (#1214) and checking new detectorMaps. See also the comment in 1214. |
| Comments |
| Comment by Wilfred Gee [ 12/May/23 ] |
|
Kiyoto Yabe can you let me know the location of Hamano-san's QA scripts, as mentioned in the 2D-DRP meeting? Thanks! |
| Comment by Kiyoto Yabe [ 12/May/23 ] |
|
The codes are now here: /work/shamano/pfs_qa
|
| Comment by rhl [ 13/May/23 ] |
|
How does this relate to PIPE2D-1146 |
| Comment by Kiyoto Yabe [ 13/May/23 ] |
|
I'm just imaging that only residuals in wavelength and spatial direction (and structure over the detector) are checked here. I don't think Hamano-san's script checks PIPE2D-1146 yet, so if that ticket is done, we can check it as well. |
| Comment by rhl [ 13/May/23 ] |
|
My mistake – I read PIPE2D-1146 too quickly. The OH lines are interesting, but not the absorption. すみません |
| Comment by Wilfred Gee [ 13/May/23 ] |
|
I've just started to look at the script from Hamano-san (and will be doing the initial import into the repo soon), but I think Kiyoto Yabe is correct as the script only makes reference to the wavelength and spatial direction (script just says` Xcenter`). |
| Comment by Kiyoto Yabe [ 21/Jul/23 ] |
|
I talked to you the other day, but could you provide plots for Apr run here please? |
| Comment by Kiyoto Yabe [ 26/Jul/23 ] |
|
Wilfred Gee Could you tell me how to setup and use the latest version in the PR? I also would like to use that for the science frames we took recently. |
| Comment by Wilfred Gee [ 26/Jul/23 ] |
|
Hi Yabe-san, The module can be set up in the usual way. First clone the directory, checkout the branch, build with `scons`, then use `setup` to put into environment (as more or less outlined https://pipelines.lsst.io/install/package-development.html). I've already performed those steps on `pfsa-usr01`, so you could can just use `setup -jr /work/wtg/drp_qa` to have the `detectorMapQa.py` command available. Note that I still might make some future changes to the PR that would affect usage. Let me know if it gives you any trouble! |
| Comment by Kiyoto Yabe [ 26/Jul/23 ] |
|
Thank you. I think I could run the code. But it works only if I setup pfs_pipe2d for w.2023.27 (or earlier version) with the following code:
setup pfs_pipe2d w.2023.30a setup -jr /work/kiyoyabe/erun/run12/src/obs_pfs setup -jr /work/kiyoyabe/erun/run12/src/drp_qa VISITS=97828 REPO='/work/drp' RERUN='kiyoyabe/erun/run12/CALIB-2023-07-v3' CALIB='/work/drp/CALIB-2023-07-v3' detectorMapQa.py $REPO --calib $CALIB --rerun $RERUN --id visit=$VISITS arm=b^r --no-versions Could you confirm that we can use with the latest pipeline? Or maybe I did something wrong. The error message is like this:
/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:778: FutureWarning: Gen2 Butler has been deprecated (PfsButler). It will be removed sometime after v23.0 but no earlier than the end of 2021. namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) /work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:778: FutureWarning: Gen2 Butler has been deprecated (PfsMapper). It will be removed sometime after v23.0 but no earlier than the end of 2021. namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs) lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB-2023-07-v3/calibRegistry.sqlite3 lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB-2023-07-v3/calibRegistry.sqlite3 root INFO: Running: /work/kiyoyabe/erun/run12/src/drp_qa/bin/detectorMapQa.py /work/drp --calib /work/drp/CALIB-2023-07-v3 --rerun kiyoyabe/erun/run12/CALIB-2023-07-v3 --id visit=97828 arm=b^r --no-versions Traceback (most recent call last): File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc return self._engine.get_loc(casted_key) File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'Trace' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 436, in __call__ result = self.runTask(task, dataRef, kwargs) File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 504, in runTask return task.runDataRef(dataRef, **kwargs) File "/work/kiyoyabe/erun/run12/src/drp_qa/python/pfs/qa/detectorMapQa.py", line 665, in runDataRef return self.run(detectorMapList, arcLinesList, pfsArmList) File "/work/kiyoyabe/erun/run12/src/drp_qa/python/pfs/qa/detectorMapQa.py", line 692, in run self.plotResidual.run(dd, aa, pp) File "/work/kiyoyabe/erun/run12/src/drp_qa/python/pfs/qa/detectorMapQa.py", line 81, in run arc_data = stability.getArclineData(arcLines) File "/work/kiyoyabe/erun/run12/src/drp_qa/python/pfs/qa/utils/stability.py", line 70, in getArclineData arc_data['Trace'] = arc_data.description.str.get_dummies()['Trace'] File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pandas/core/frame.py", line 3505, in __getitem__ indexer = self.columns.get_loc(key) File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3623, in get_loc raise KeyError(key) from err KeyError: 'Trace' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/work/kiyoyabe/erun/run12/src/drp_qa/bin/detectorMapQa.py", line 4, in <module> DetectorMapQaTask.parseAndRun() File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 688, in parseAndRun resultList = taskRunner.run(parsedCmd) File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 244, in run resultList = list(mapFunc(self, targetList)) File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 450, in __call__ ", ".join(str(ref.dataId) for ref in dataRef), File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 450, in <genexpr> ", ".join(str(ref.dataId) for ref in dataRef), AttributeError: 'list' object has no attribute 'dataId'
|
| Comment by Kiyoto Yabe [ 26/Jul/23 ] |
|
Anyway, plots I could make are here: detectorMapQA-097828-20230726.pdf It seems that there are blank panel, so I must have done something wrong. So, I appreciate if you can make the plots for the same specific visit. |
| Comment by Wilfred Gee [ 27/Jul/23 ] |
|
Kiyoto Yabe I think I figured out why the one panel is broken, trying to fix now. I was using the pipeline as of last week, so more recent than the `w.2023.27`. I'll see if I can duplicate the error message above. |
