[PIPE2D-1365] Make the current detectorMapQa.py workable Created: 31/Jan/24 Updated: 02/Feb/24 |
|
| Status: | Open |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Normal |
| Reporter: | Kiyoto Yabe | Assignee: | Wilfred Gee |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | EDR, EngRun, QualityAssurance | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
The current detectorMapQa.py does not work (Masayuki Tanaka can report the problem here), so please investigate the issue. Also, please make sure that the script provide all plots including 2D and 1D. |
| Comments |
| Comment by Masayuki Tanaka [ 31/Jan/24 ] |
|
On the pfsa server, I setup the current pipeline (w.2023.30a). Following Yabe-kun's suggestion, I did $ setup -jr /work/wtg/obs_pfs Then I ran detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --clobber-versions This resulted in the following error: $ detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --clobber-versions
This pickle file is owned by Yabe-kun and it is entirely possible that I am doing something stupid. For your information, $ eups list | grep setup
|
| Comment by price [ 01/Feb/24 ] |
|
I don't see a problem with packages.pickle~3, but maybe you dealt with that problem. I do see a problem with packages.pickle: -rw-r--r-- 1 tanaka pfs 17231 Jan 29 19:40 packages.pickle This file is missing a group write permission. I think both Masayuki Tanaka and Kiyoto Yabe need to check that they have umask 2 set in their shell startup scripts. |
| Comment by price [ 01/Feb/24 ] |
|
BTW, your use of --clobber-versions is triggering this problem. If you used --no-versions, you wouldn't see it, and that's probably what you want anyway when you're working on a shared rerun. |
| Comment by Kiyoto Yabe [ 02/Feb/24 ] |
|
Now I see an error like this: (lsst-scipipe-3.0.0) detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --no-versions lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3 lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3 root INFO: Running: /work/wtg/drp_qa/bin/detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --no-versions Traceback (most recent call last): File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 436, in __call__ result = self.runTask(task, dataRef, kwargs) File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 504, in runTask return task.runDataRef(dataRef, **kwargs) File "/work/wtg/drp_qa/python/pfs/qa/detectorMapQa.py", line 704, in runDataRef return self.run(detectorMapList, arcLinesList, pfsArmList) File "/work/wtg/drp_qa/python/pfs/qa/detectorMapQa.py", line 731, in run self.plotResidual.run(dd, aa, pp) File "/work/wtg/drp_qa/python/pfs/qa/detectorMapQa.py", line 84, in run arc_data = stability.getArclineData(arcLines, statusTypes=list(), dropNa=True, dropColumns=['xx', 'yy', 'xy']) AttributeError: module 'pfs.qa.utils.stability' has no attribute 'getArclineData' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/work/wtg/drp_qa/bin/detectorMapQa.py", line 4, in <module> DetectorMapQaTask.parseAndRun() File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 688, in parseAndRun resultList = taskRunner.run(parsedCmd) File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 244, in run resultList = list(mapFunc(self, targetList)) File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 450, in __call__ ", ".join(str(ref.dataId) for ref in dataRef), File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 450, in <genexpr> ", ".join(str(ref.dataId) for ref in dataRef), AttributeError: 'list' object has no attribute 'dataId' This is purely a problem of detectorMapQa.py. |
| Comment by Masayuki Tanaka [ 02/Feb/24 ] |
|
Yes, I saw the same problem. price , just for my own education, what is the difference between --clobber-versions and --no-versions? |
| Comment by Wilfred Gee [ 02/Feb/24 ] |
|
Thanks for trying this. Kiyoto Yabe I'll look into that error and get fixed. I think it's just pointing in some wrong places so hopefully an easy fix. |
| Comment by price [ 02/Feb/24 ] |
|
--clobber-versions removes the existing file that records the versions and writes a new one. --no-versions doesn't do any version checking or recording. In general, you should not use either during production. If the stack has upgraded during a production, you could use --clobber-versions to record that. --no-versions is for development. |