https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-571 DRP 2-D Pipeline <p>We have been generating data repos individually. Instead, we should use a shared data repo, as we do for HSC. This makes reducing data easier, and also eases collaboration.</p> <ul> <li>Clean up the different raw repos on <tt>/tigress</tt></li> <li>Ingest all data (LAM, Subaru) <ul> <li>Identify and fix data that doesn't ingest</li> </ul> </li> <li>Send <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> the command for ingestion so that his script can run it regularly on new data.</li> </ul> PIPE2D-571

Ingest into a common data repo

Story Normal Done Done price price Tue, 5 May 2020 14:12:26 +0000 Mon, 4 Jan 2021 20:23:17 +0000 Thu, 21 May 2020 19:15:07 +0000 0 2 <p>Cleaning up all the raw files first.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pprice@tiger2-sumire:/projects/HSC/PFS/LAM-raw $ find . | xargs -n 20 -P 10 md5sum &gt; ~/LAM-raw.txt pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw $ find . | xargs -n 20 -P 10 md5sum &gt; ~/raw.txt &gt;&gt;&gt; with open(<span class="code-quote">"raw.txt"</span>) as fd: ... raw = dict(line.strip().split() <span class="code-keyword">for</span> line in fd.readlines()) &gt;&gt;&gt; with open(<span class="code-quote">"LAM-raw.txt"</span>) as fd: ... lamRaw = dict(line.strip().split() <span class="code-keyword">for</span> line in fd.readlines()) &gt;&gt;&gt; len(set(lamRaw.keys()) - set(raw.keys())) 0 &gt;&gt;&gt; len(set(lamRaw.values()) - set(raw.values())) 0 ==&gt; Blew away /projects/HSC/PFS/LAM-raw, as everything is in /projects/HSC/PFS/LAM/raw. pprice@tiger2-sumire:/projects/HSC/PFS/JHU/raw $ find . | xargs -n 100 -P 20 md5sum &gt; ~/jhu.txt pprice@tiger2-sumire:/projects/HSC/PFS/raw $ find . | xargs -n 100 -P 20 md5sum &gt; ~/raw.txt pprice@tiger2-sumire:~ $ diff --<span class="code-keyword">new</span>-line-format=<span class="code-quote">"" --unchanged-line-format="</span>" &lt;(sort raw.txt) &lt;(sort jhu.txt) 761aad7694f64002e40f6b7005eb90ea ./INVENTORY_NOTES.txt c56bfdedd8287bc8cba86b14de520076 ./INVENTORY_NOTES.txt~ &gt;&gt;&gt; with open(<span class="code-quote">"raw.txt"</span>) as fd: ... raw = dict(line.strip().split() <span class="code-keyword">for</span> line in fd.readlines()) ... &gt;&gt;&gt; with open(<span class="code-quote">"jhu.txt"</span>) as fd: ... jhu = dict(line.strip().split() <span class="code-keyword">for</span> line in fd.readlines()) ... &gt;&gt;&gt; len(raw) 1523 &gt;&gt;&gt; len(jhu) 4477 &gt;&gt;&gt; len(set(raw.keys()) - set(jhu.keys())) 1 &gt;&gt;&gt; set(raw.keys()) - set(jhu.keys()) {<span class="code-quote">'c56bfdedd8287bc8cba86b14de520076'</span>} &gt;&gt;&gt; raw[<span class="code-quote">'c56bfdedd8287bc8cba86b14de520076'</span>] <span class="code-quote">'./INVENTORY_NOTES.txt~'</span> Craig Loomis 3:32 PM That `raw` directory and INVENTORY_NOTES.txt represents quite a bit of time. Basically, it selects the very early r1 and b1 which are actually useable. So I’d put in JHU/keep/ ==&gt; Moved /projects/HSC/PFS/raw to /projects/HSC/PFS/JHU/keep pprice@tiger2-sumire:/projects/HSC/PFS/LAM $ <span class="code-keyword">for</span> dd in HgArFeb2019_raw KrFeb2019_raw NeonApr2019_raw NeonFeb2019_raw ; <span class="code-keyword">do</span> find $dd | xargs -n 10 -P 20 md5sum &gt; $dd.txt ; done pprice@tiger2-sumire:/projects/HSC/PFS/LAM $ find raw | xargs -n 100 -P 20 md5sum &gt; raw.txt pprice@tiger2-sumire:/projects/HSC/PFS/LAM $ diff --<span class="code-keyword">new</span>-line-format=<span class="code-quote">"" --unchanged-line-format="</span>" &lt;(awk <span class="code-quote">'{print $1}'</span> HgArFeb2019_raw.txt | sort -u) &lt;(awk <span class="code-quote">'{print $1}'</span> raw.txt | sort -u) pprice@tiger2-sumire:/projects/HSC/PFS/LAM $ diff --<span class="code-keyword">new</span>-line-format=<span class="code-quote">"" --unchanged-line-format="</span>" &lt;(awk <span class="code-quote">'{print $1}'</span> KrFeb2019_raw.txt | sort -u) &lt;(awk <span class="code-quote">'{print $1}'</span> raw.txt | sort -u) pprice@tiger2-sumire:/projects/HSC/PFS/LAM $ diff --<span class="code-keyword">new</span>-line-format=<span class="code-quote">"" --unchanged-line-format="</span>" &lt;(awk <span class="code-quote">'{print $1}'</span> NeonApr2019_raw.txt | sort -u) &lt;(awk <span class="code-quote">'{print $1}'</span> raw.txt | sort -u) pprice@tiger2-sumire:/projects/HSC/PFS/LAM $ diff --<span class="code-keyword">new</span>-line-format=<span class="code-quote">"" --unchanged-line-format="</span>" &lt;(awk <span class="code-quote">'{print $1}'</span> NeonFeb2019_raw.txt | sort -u) &lt;(awk <span class="code-quote">'{print $1}'</span> raw.txt | sort -u) ==&gt; Can blow away HgArFeb2019_raw KrFeb2019_raw NeonApr2019_raw NeonFeb2019_raw. ==&gt; Done. </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Neven Caplar:princeton: 4:56 PM I suggest Sep 09 2018 as starting date (start of https:<span class="code-comment">//people.lam.fr/madec.fabrice/pfs/ait_logbook_SM1.html) (edited) </span> (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw $ ingestPfsImages.py /projects/HSC/PFS/LAM --pfsConfigDir /projects/HSC/PFS/LAM/raw/pfsDesign --config parse.pfsDesignId=1099528409104 clobber=True -- <span class="code-quote">'201[89]-*/PFLA*.fits'</span> RuntimeError: Unable to find PfsConfig or PfsDesign <span class="code-keyword">for</span> pfsDesignId=0x0000010001001000 RuntimeError: Unable to find PfsConfig or PfsDesign <span class="code-keyword">for</span> pfsDesignId=0x0000100000001111 &gt;&gt;&gt; pfs.utils.dummyCableB.DummyCableBDatabase().interpret(0x0000010001001000) [<span class="code-quote">'red1'</span>, <span class="code-quote">'red4'</span>, <span class="code-quote">'red8'</span>] &gt;&gt;&gt; pfs.utils.dummyCableB.DummyCableBDatabase().interpret(0x0000100000001111) [<span class="code-quote">'blue'</span>, <span class="code-quote">'green'</span>, <span class="code-quote">'orange'</span>, <span class="code-quote">'red1'</span>, <span class="code-quote">'yellow'</span>] (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw/pfsDesign $ makeDummyCableBDesign.py red1 red4 red8 (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw/pfsDesign $ makeDummyCableBDesign.py blue green orange red1 yellow ValueError: could not convert string to <span class="code-object">float</span>: <span class="code-quote">'NO CURRENT VALUE'</span> Fixed on tickets/PIPE2D-571 (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw $ ingestPfsImages.py /projects/HSC/PFS/LAM --pfsConfigDir /projects/HSC/PFS/LAM/raw/pfsDesign --config parse.pfsDesignId=1099528409104 -- <span class="code-quote">'201[89]-*/PFLA*.fits'</span> | tee ingest.log (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw $ grep <span class="code-quote">"Unable to find PfsConfig"</span> ingest.log | sed -e <span class="code-quote">'s|^.*pfsDesignId=||'</span> | sort -u 0x0000000000000001 0x0000000000000010 0x0000000001000000 &gt;&gt;&gt; pfs.utils.dummyCableB.DummyCableBDatabase().interpret(0x0000000000000001) [<span class="code-quote">'blue'</span>] &gt;&gt;&gt; pfs.utils.dummyCableB.DummyCableBDatabase().interpret(0x0000000000000010) [<span class="code-quote">'green'</span>] &gt;&gt;&gt; pfs.utils.dummyCableB.DummyCableBDatabase().interpret(0x0000000001000000) [<span class="code-quote">'red4'</span>] (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw/pfsDesign $ makeDummyCableBDesign.py blue Wrote pfsDesign-0x0000000000000001.fits (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw/pfsDesign $ makeDummyCableBDesign.py green Wrote pfsDesign-0x0000000000000010.fits (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/LAM/raw/pfsDesign $ makeDummyCableBDesign.py red4 Wrote pfsDesign-0x0000000001000000.fits 66 files have multiple header problems (e.g., no DATE-OBS, EXPTIME, IMAGETYP) that aren't worth saving <span class="code-keyword">for</span> now. price@price-laptop:~/pfs/obs_pfs (tickets/PIPE2D-571=) $ grep <span class="code-quote">"No locations <span class="code-keyword">for</span> get"</span> ingest.log | sed -e <span class="code-quote">'s|^.*Failed to ingest file \(.*\.fits\):.*$|\1|'</span> 2019-03-01/PFLA01340412.fits 2019-03-01/PFLA01340812.fits 2019-03-01/PFLA01340712.fits 2019-03-01/PFLA01340612.fits 2019-03-01/PFLA01340512.fits 2019-02-22/PFLA01253114.fits 2019-02-22/PFLA01254114.fits 2019-02-22/PFLA01253714.fits 2019-02-22/PFLA01253214.fits 2019-02-22/PFLA01253314.fits 2019-02-22/PFLA01252914.fits 2019-02-22/PFLA01253814.fits 2019-02-22/PFLA01253914.fits 2019-02-22/PFLA01253014.fits 2019-02-22/PFLA01254014.fits 2019-02-28/PFLA01317312.fits 2019-02-28/PFLA01316212.fits 2019-02-28/PFLA01316912.fits 2019-02-28/PFLA01315912.fits 2019-02-28/PFLA01318212.fits 2019-02-28/PFLA01317712.fits 2019-02-28/PFLA01318112.fits 2019-02-28/PFLA01316612.fits 2019-02-28/PFLA01317512.fits 2019-02-28/PFLA01316112.fits 2019-02-28/PFLA01316812.fits 2019-02-28/PFLA01317212.fits 2019-02-28/PFLA01317012.fits 2019-02-28/PFLA01317912.fits 2019-02-28/PFLA01317612.fits 2019-02-28/PFLA01316312.fits 2019-02-28/PFLA01317112.fits 2019-02-28/PFLA01316512.fits 2019-02-28/PFLA01317412.fits 2019-02-28/PFLA01318012.fits 2019-02-28/PFLA01317812.fits 2019-02-28/PFLA01316012.fits 2019-02-28/PFLA01316412.fits 2019-02-28/PFLA01316712.fits 2019-03-05/PFLA01351311.fits 2019-02-27/PFLA01314112.fits 2019-02-27/PFLA01313912.fits 2019-02-27/PFLA01314612.fits 2019-02-27/PFLA01314012.fits 2019-02-27/PFLA01314512.fits 2019-02-27/PFLA01315412.fits 2019-02-27/PFLA01313812.fits 2019-02-27/PFLA01292014.fits 2019-02-27/PFLA01315712.fits 2019-02-27/PFLA01315612.fits 2019-02-27/PFLA01292114.fits 2019-02-27/PFLA01315312.fits 2019-02-27/PFLA01315212.fits 2019-02-27/PFLA01314712.fits 2019-02-27/PFLA01291714.fits 2019-02-27/PFLA01315812.fits 2019-02-27/PFLA01314312.fits 2019-02-27/PFLA01315512.fits 2019-02-27/PFLA01315112.fits 2019-02-27/PFLA01291814.fits 2019-02-27/PFLA01314812.fits 2019-02-27/PFLA01291914.fits 2019-02-27/PFLA01314212.fits 2019-02-27/PFLA01314412.fits 2019-02-27/PFLA01314912.fits 2019-02-27/PFLA01315012.fits ingestPfs WARN: Failed to ingest file 2019-02-18/PFLA01249512.fits: File <span class="code-quote">"src/PropertySet.cc"</span>, line 486, in void lsst::daf::base::PropertySet::add(<span class="code-keyword">const</span> string&amp;, <span class="code-keyword">const</span> T&amp;) [with T = <span class="code-object">double</span>; std::string = std::basic_string&lt;<span class="code-object">char</span>&gt;] W_ENFCAX has mismatched type {0} lsst::pex::exceptions::TypeError: <span class="code-quote">'W_ENFCAX has mismatched type'</span> Also <span class="code-keyword">for</span> 2019-02-18/PFLA01249612.fits They have LOTS of duplicate header values with different types that kills lsst.afw.fits.readMetadata. Not going to bother fixing them either. (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/Subaru/raw $ gethead */PFSA*.fits W_PFDSGN &gt; pfsDesignId.txt (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/Subaru/raw $ awk <span class="code-quote">'{print $2}'</span> pfsDesignId.txt | sort -u 1099528409104 16 4503599627370496 72057594037927936 -9998 (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/Subaru/raw $ grep -- <span class="code-quote">'-9998'</span> pfsDesignId.txt PFSA00000214.fits -9998 A single file has a bad pfsDesignId (negative!?). Looks like it's one of the first exposures, so don't care. (lsst-scipipe) pprice@tiger2-sumire:/projects/HSC/PFS/Subaru/raw $ ingestPfsImages.py /projects/HSC/PFS/Subaru --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 -- <span class="code-quote">'*/PFSA*.fits'</span> 2&gt;&amp;1 | tee ingest.log </pre> </div></div> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a>, could you please review these changes to obs_pfs to support the ingestion of LAM data?</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>: The command used to ingest was:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ingestPfsImages.py /projects/HSC/PFS/Subaru --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 -- <span class="code-quote">'*/PFSA*.fits'</span> </pre> </div></div> <p>I didn't add the <tt>-c clobber=True register.ignore=True</tt> because this was from scratch. You shouldn't need it if you only run the command on new data.</p> <p>Merged to master.</p> <p>One question, one point:</p> <p>Is that <tt>--config parse.pfsDesignId=1099528409104</tt> <b>required</b>? Or does it usually use the <tt>W_PFDSGN</tt> value? </p> <p>So we need to have a populated <tt>/projects/HSC/PFS/Subaru/drp/pfsDesign/</tt> directory before calling ingest. That will take a bit of attention at the Subaru side.</p> <p>It usually uses the <tt>W_PFDSGN</tt> value. <tt>parse.pfsDesignId</tt> provides a default value in the event <tt>W_PFDSGN</tt> isn't supplied (which was necessary for old LAM data, but probably less necessary for modern Subaru data).</p> Relates INSTRM-823 Development Rank 0|02qps1:2 Reviewers hassan Sprint 2DDRP-2021 A Story Points 3.0