PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 [PIPE2D-1365] Make the current detectorMapQa.py workable https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1365 DRP 2-D Pipeline <p>The current detectorMapQa.py does not work (<a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=msyktnk" class="user-hover" rel="msyktnk">Masayuki Tanaka</a> can report the problem here), so please investigate the issue. Also, please make sure that the script provide all plots including 2D and 1D.</p> PIPE2D-1365 Make the current detectorMapQa.py workable Task Normal Open Unresolved Wilfred Gee Kiyoto Yabe Wed, 31 Jan 2024 02:48:25 +0000 Thu, 1 Feb 2024 22:17:02 +0000 0 4 <p>On the pfsa server, I setup the current pipeline (w.2023.30a). Following Yabe-kun's suggestion, I did</p> <p><tt>$ setup -jr /work/wtg/obs_pfs</tt><br/> <tt>$ setup -jr /work/wtg/drp_qa</tt></p> <p>Then I ran</p> <p><tt>detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --clobber-versions</tt></p> <p> This resulted in the following error:</p> <p><tt>$ detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --clobber-versions</tt><br/> <tt>lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql</tt><br/> <tt>lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3</tt><br/> <tt>lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3</tt><br/> <tt>root INFO: Running: /work/wtg/drp_qa/bin/detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --clobber-versions</tt><br/> <tt>conda.core.prefix_data INFO: Package conda-forge/linux-64::psycopg-3.1.8-py38h838c7ec_0 has multiple python anchor files.</tt><br/> {{ Using lib/python3.8/site-packages/psycopg-3.1.8.dist-info/RECORD}}<br/> <tt>detectorMapQa FATAL: Failed in task initialization: <span class="error">&#91;Errno 1&#93;</span> Operation not permitted: '/work/drp/rerun/pfs/internal/edr2-20231203/config/packages.pickle~3'</tt><br/> <tt>Traceback (most recent call last):</tt><br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 377, in precall}}<br/> {{ self._precallImpl(task, parsedCmd)}}<br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 350, in _precallImpl}}<br/> {{ task.writePackageVersions(parsedCmd.butler, clobber=parsedCmd.clobberVersions)}}<br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 877, in writePackageVersions}}<br/> {{ return butler.put(packages, dataset, doBackup=doBackup)}}<br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/butler.py", line 1454, in put}}<br/> {{ location.getRepository().backup(location.datasetType, dataId)}}<br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/repository.py", line 271, in backup}}<br/> {{ self._mapper.backup(*args, **kwargs)}}<br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_base/gc18b5ea8f9+af69d5793a/python/lsst/obs/base/cameraMapper.py", line 726, in backup}}<br/> {{ self.rootStorage.copyFile(oldPath, "%s~%d" % (newPath, n))}}<br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/posixStorage.py", line 365, in copyFile}}<br/> {{ shutil.copy(os.path.join(self.root, fromLocation), os.path.join(self.root, toLocation))}}<br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/shutil.py", line 419, in copy}}<br/> {{ copymode(src, dst, follow_symlinks=follow_symlinks)}}<br/> {{ File "/work/tanaka/pipe2d/26apr2023/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/shutil.py", line 308, in copymode}}<br/> {{ chmod_func(dst, stat.S_IMODE(st.st_mode))}}<br/> <tt>PermissionError: <span class="error">&#91;Errno 1&#93;</span> Operation not permitted: '/work/drp/rerun/pfs/internal/edr2-20231203/config/packages.pickle~3'</tt></p> <p> </p> <p>This pickle file is owned by Yabe-kun and it is entirely possible that I am doing something stupid. For your information,</p> <p><tt>$ eups list | grep setup</tt><br/> <tt>afw g31dc256b76+308b63831b current w_2022_17 setup</tt><br/> <tt>astro_metadata_translator g1ca0471399+c55107240f current w_2022_17 setup</tt><br/> <tt>astshim gb04e0da5b2+ce83317b7e current w_2022_17 setup</tt><br/> <tt>base gc57f72a14a+ce83317b7e current w_2022_17 setup</tt><br/> <tt>coadd_utils g169eefeba3+e480a1db32 current w_2022_17 setup</tt><br/> <tt>cp_pipe ge16c6bda5e+9234ee834c current w_2022_17 setup</tt><br/> <tt>cpputils g1ffd6c4751+50c04d9c73 current w_2022_17 setup</tt><br/> <tt>ctrl_mpexec gb02ad94e9c+e480a1db32 current w_2022_17 setup</tt><br/> <tt>ctrl_pool g6ea9deef35+e480a1db32 current w_2022_17 setup</tt><br/> <tt>daf_base gcc072456be+b6f9ac6643 current w_2022_17 setup</tt><br/> <tt>daf_butler gb7ebdee0ea+6728436f39 current w_2022_17 setup</tt><br/> <tt>daf_persistence g6a31054a6e+4ae6138134 current w_2022_17 setup</tt><br/> <tt>datamodel w.2023.30a current setup</tt><br/> <tt>dax_apdb g7e0c5f63d1+18023899f9 current w_2022_17 setup</tt><br/> <tt>drp_pfs_data w.2023.30-2-g9ad6703 current setup</tt><br/> <tt>drp_stella w.2023.30a current setup</tt><br/> <tt>dustmaps_cachedata g41a3ec361e+ce83317b7e current w_2022_17 setup</tt><br/> <tt>eigen g04a8d4365e current w_2022_17 setup</tt><br/> <tt>geom g491c310abf+35ead245fa current w_2022_17 setup</tt><br/> <tt>ip_diffim g4d7a404405+e7a229012f current w_2022_17 setup</tt><br/> <tt>ip_isr g8142d5ee96+96f555657d current w_2022_17 setup</tt><br/> <tt>kht g14ffe67dc2+c057cea34b current w_2022_17 setup</tt><br/> <tt>log gd27fc4addd+b6f9ac6643 current w_2022_17 setup</tt><br/> <tt>meas_algorithms gf36ae6ace1+1d7ae49801 current w_2022_17 setup</tt><br/> <tt>meas_astrom gaab3d2dbd9+96f555657d current w_2022_17 setup</tt><br/> <tt>meas_base g0129b61b94+2131942eb1 current w_2022_17 setup</tt><br/> <tt>meas_deblender gf9afb8a27f+3d65f423aa current w_2022_17 setup</tt><br/> <tt>meas_extensions_convolved ge96dbbe9aa+56b954ee52 current w_2022_17 setup</tt><br/> <tt>meas_extensions_gaap g8ae2c9d62c+51b14322dc current w_2022_17 setup</tt><br/> <tt>meas_extensions_photometryKron g235fec5482+96f555657d current w_2022_17 setup</tt><br/> <tt>meas_extensions_piff g4882acf283+96f555657d current w_2022_17 setup</tt><br/> <tt>meas_extensions_scarlet g746e97fe6b+efc4113c79 current w_2022_17 setup</tt><br/> <tt>meas_extensions_shapeHSM gfbfb15d729+96f555657d current w_2022_17 setup</tt><br/> <tt>meas_extensions_trailedSources gbacec6188e+a1e9a24ff9 current w_2022_17 setup</tt><br/> <tt>meas_modelfit g8cdf404ddd+53ad3d90d7 current w_2022_17 setup</tt><br/> <tt>obs_base gc18b5ea8f9+af69d5793a current w_2022_17 setup</tt><br/> <tt>obs_pfs LOCAL:/work/wtg/obs_pfs setup</tt><br/> <tt>obs_test g26fc4c0ca3+1efc28e7e0 current w_2022_17 setup</tt><br/> <tt>obs_test_data g86cffb07a8 current w_2022_17 setup</tt><br/> <tt>pex_config g9aff22451c+c55107240f current w_2022_17 setup</tt><br/> <tt>pex_exceptions g958458c828+fc3c6d1a77 current w_2022_17 setup</tt><br/> <tt>pfs_pipe2d w.2023.30a current setup</tt><br/> <tt>pfs_qa LOCAL:/work/wtg/drp_qa setup</tt><br/> <tt>pfs_utils w.2023.30a current setup</tt><br/> <tt>pipe_base g590c34a36e+5da9528084 current w_2022_17 setup</tt><br/> <tt>pipe_drivers gef6abdcebb+582a64c339 current w_2022_17 setup</tt><br/> <tt>pipe_tasks gb1d6de0934+6a8639994f current w_2022_17 setup</tt><br/> <tt>proxmin g33b4157f25 current w_2022_17 setup</tt><br/> <tt>resources gf1bf1f77a0+b6f9ac6643 current w_2022_17 setup</tt><br/> <tt>scarlet g6315d299d3+4083830bf8 current w_2022_17 setup</tt><br/> <tt>scarlet_extensions g9d18589735+0af1cd992e current w_2022_17 setup</tt><br/> <tt>sconsUtils g26c60a276e current w_2022_17 setup</tt><br/> <tt>shapelet gd79a88f3f0+ec8e2bba64 current w_2022_17 setup</tt><br/> <tt>skymap g602c5592b1+bd95241bbd current w_2022_17 setup</tt><br/> <tt>sphgeom g0f31348038+ce83317b7e current w_2022_17 setup</tt><br/> <tt>utils g62e1f4f43a+9633a190c8 current w_2022_17 setup</tt><br/> <tt>verify ga2309f1883+eca4149912 current w_2022_17 setup</tt><br/> <tt>verify_metrics g4cd34326d0+ce83317b7e current w_2022_17 setup</tt></p> <p> </p> <p>I don't see a problem with <tt>packages.pickle~3</tt>, but maybe you dealt with that problem. I do see a problem with <tt>packages.pickle</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> -rw-r--r-- 1 tanaka pfs 17231 Jan 29 19:40 packages.pickle </pre> </div></div> <p>This file is missing a group write permission. I think both <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=msyktnk" class="user-hover" rel="msyktnk">Masayuki Tanaka</a> and <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> need to check that they have <tt>umask 2</tt> set in their shell startup scripts.</p> <p>BTW, your use of <tt>&#45;&#45;clobber-versions</tt> is triggering this problem. If you used <tt>&#45;&#45;no-versions</tt>, you wouldn't see it, and that's probably what you want anyway when you're working on a shared rerun.</p> <p>Now I see an error like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe-3.0.0) detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --no-versions lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3 lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3 root INFO: Running: /work/wtg/drp_qa/bin/detectorMapQa.py /work/drp/ --calib /work/drp/CALIB --rerun pfs/internal/edr2-20231203/ --id visit=92743 --no-versions Traceback (most recent call last): File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 436, in __call__ result = self.runTask(task, dataRef, kwargs) File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 504, in runTask <span class="code-keyword">return</span> task.runDataRef(dataRef, **kwargs) File <span class="code-quote">"/work/wtg/drp_qa/python/pfs/qa/detectorMapQa.py"</span>, line 704, in runDataRef <span class="code-keyword">return</span> self.run(detectorMapList, arcLinesList, pfsArmList) File <span class="code-quote">"/work/wtg/drp_qa/python/pfs/qa/detectorMapQa.py"</span>, line 731, in run self.plotResidual.run(dd, aa, pp) File <span class="code-quote">"/work/wtg/drp_qa/python/pfs/qa/detectorMapQa.py"</span>, line 84, in run arc_data = stability.getArclineData(arcLines, statusTypes=list(), dropNa=True, dropColumns=[<span class="code-quote">'xx'</span>, <span class="code-quote">'yy'</span>, <span class="code-quote">'xy'</span>]) AttributeError: module <span class="code-quote">'pfs.qa.utils.stability'</span> has no attribute <span class="code-quote">'getArclineData'</span> During handling of the above exception, another exception occurred: Traceback (most recent call last): File <span class="code-quote">"/work/wtg/drp_qa/bin/detectorMapQa.py"</span>, line 4, in &lt;module&gt; DetectorMapQaTask.parseAndRun() File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 688, in parseAndRun resultList = taskRunner.run(parsedCmd) File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 244, in run resultList = list(mapFunc(self, targetList)) File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 450, in __call__ <span class="code-quote">", "</span>.join(str(ref.dataId) <span class="code-keyword">for</span> ref in dataRef), File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 450, in &lt;genexpr&gt; <span class="code-quote">", "</span>.join(str(ref.dataId) <span class="code-keyword">for</span> ref in dataRef), AttributeError: <span class="code-quote">'list'</span> object has no attribute <span class="code-quote">'dataId'</span> </pre> </div></div> <p>This is purely a problem of detectorMapQa.py.</p> <p>Yes, I saw the same problem. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> , just for my own education, what is the difference between --clobber-versions and --no-versions?</p> <p>Thanks for trying this. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> I'll look into that error and get fixed. I think it's just pointing in some wrong places so hopefully an easy fix.</p> <p><tt>&#45;&#45;clobber-versions</tt> removes the existing file that records the versions and writes a new one. <tt>&#45;&#45;no-versions</tt> doesn't do any version checking or recording. In general, you should not use either during production. If the stack has upgraded during a production, you could use <tt>&#45;&#45;clobber-versions</tt> to record that. <tt>&#45;&#45;no-versions</tt> is for development.</p> Development Rank 0|zzt153: