https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1131 DRP 2-D Pipeline <p>pfsConfig files are now created under <em>/data/raw/DATE/pfsConfig</em> regardless of the lightSource.<br/> Hence, there should no longer be any magic for dcb or SuNSS, the config files now exist, you just have to use them : </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(rubin3_ics) pfs@actors-ics:/data/raw/2022-12-13/pfsConfig$ ls -ltr total 130 -rw-rw-r-- 1 pfs pfs 23040 Dec 13 08:39 pfsConfig-0x00010001001010-084090.fits -rw-rw-r-- 1 pfs pfs 23040 Dec 13 09:22 pfsConfig-0x00010001001010-084091.fits -rw-rw-r-- 1 pfs pfs 23040 Dec 13 09:28 pfsConfig-0x00010001001010-084092.fits -rw-rw-r-- 1 pfs pfs 46080 Dec 13 09:30 pfsConfig-0x000000deadbeef-084093.fits </pre> </div></div> PIPE2D-1131

ingest pfsConfig from /data/raw/DATE/pfsConfig

Task Normal Done Done price arnaud.lefur Tue, 13 Dec 2022 19:36:54 +0000 Fri, 16 Dec 2022 03:47:32 +0000 Fri, 16 Dec 2022 03:47:32 +0000 obs_pfs 0 4 <p>let me know how/if the syntax of ingestPfsImages.py evolves, I should no longer to need to specify pfsConfigDir right ?</p> <p><tt>ingestPfsImages.py</tt> expects that the <tt>pfsConfig</tt> file is sitting in the same directory as the images, or it's in the <tt>--pfsConfigDir</tt>.</p> <p>Makes sense, can you make it evolve so it expect file in <em>/data/raw/DATE/pfsConfig</em> ?</p> <p>Why do we want to put the pfsConfig files in a separate directory? They are as important as the raw pixels, so why not put them in the same directory? Then a user interested in the data for one night only has to copy files from a single directory.</p> <p>Is it that bad to have "/data/raw/$DATE/{a,few,types}/", where the "/data/raw/$DATE"  is the single directory? Granted that the other directories are mostly for engineering purposes – guider frames, etc. – so not of universal interest.</p> <p>I had been thinking that "/data/raw" is the end of the ICS domain, the DRP root as the beginning of the DRP domain, and the ingest scripts as the only things which need to know about both.</p> <p>If that is not natural, we can still adapt.</p> <p> </p> <p>The ingest script doesn't know about date-based directories: it receives a list of image files, and needs to find the associated pfsConfig files. The current options are that these files are in the same directory as the image files or they are in a separate directory. There is currently no facility to build that separate directory name; while I think that's something we could do (and fairly easily, I think), I don't like treating the pfsConfig files substantially differently from the image files. I can see users the morning after their run copying their image files while neglecting the pfsConfig files.</p> <p>One technical point is that the PFSx-writing processes all run as a distinct "pfs-data" user which is only useable on the fourteen or so DAQ hosts, and those data files (and directories) are not supposed to be group-writeable (hmm to self: the directories are right but the files not). It would be hard (and undesirable) to change that at this point. Long after that decision we added pfsConfig writing for every SPS visit, and we would have to do some splitting and refactoring to allow pfsConfigs to also be written by that "pfs-data" user.</p> <p>That doesn't mean we can't do it if appropriate.</p> <p>Sounds like it's easier for me to change things on the DRP side. I'll try to tackle this tomorrow.</p> <p>I've got an implementation that seems to do the job. Given <tt>--pfsConfigDir /path/to</tt>, it will look for the pfsConfig file in the following places:</p> <ul> <li><tt>/path/to/</tt></li> <li><tt>/path/to/&lt;dateObs&gt;/pfsConfig/</tt></li> <li>Look for <tt>pfsDesign</tt> file in <tt>/path/to</tt> and write pfsConfig to <tt>/path/to/&lt;dateObs&gt;/pfsConfig/</tt></li> </ul> <p>So I expect you'll use <tt>--pfsConfigDir /data/raw</tt>.</p> <p>Changes in pull request look fine. Would prefer a warning if a <tt>pfsConfig</tt> file is created from a <tt>pfsDesign</tt>. See comments in PR.</p> <p>Changed the existing debug logging to warning level.</p> <p>Merged.</p> Development Rank 0|zzsvtr: Reviewers hassan