https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1083 DRP 2-D Pipeline <p>Following <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1058" title="Avoid/fix registry sqlite deadlocks" class="issue-link" data-issue-key="PIPE2D-1058"><del>PIPE2D-1058</del></a>, postgresql ingest failed at Hilo:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ ingestPfsImagesPgsql.py /work/drp --ignore-ingested --pfsConfigDir /work/drp/pfsDesign --mode=copy --doraise /data/raw/2022-09-22/sps/PFSA0794131* [...] lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3 lsst.ingestPfs INFO: /data/raw/2022-09-22/sps/PFSA07941311.fits --&lt;copy&gt;--&gt; /work/drp/2022-09-22/PFSA079413b1.fits lsst.ingestPfs WARN: Failed to ingest file /data/raw/2022-09-22/sps/PFSA07941311.fits: Failed to register file /data/raw/2022-09-22/sps/PFSA07941311.fits Traceback (most recent call last): File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/commissioning-2022919/python/lsst/obs/pfs/ingest.py"</span>, line 675, in runFile self.register.addRow(registry, info, dryrun=args.dryrun, create=args.create) File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_tasks/gb1d6de0934+6a8639994f/python/lsst/pipe/tasks/ingest.py"</span>, line 388, in addRow conn.cursor().execute(sql, values) psycopg2.errors.SyntaxError: syntax error at or near <span class="code-quote">"OR"</span> LINE 1: INSERT OR IGNORE INTO raw_visit VALUES (79413,<span class="code-quote">'IQ_TEST'</span>,'202... </pre> </div></div> <p>There have been changes in pipe&#95;tasks that I wasn't aware of, and I must have screwed up the testing of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1058" title="Avoid/fix registry sqlite deadlocks" class="issue-link" data-issue-key="PIPE2D-1058"><del>PIPE2D-1058</del></a>.</p> PIPE2D-1083

Postgresql ingest is broken

Bug Normal Done Done price price EngRun Thu, 22 Sep 2022 02:15:44 +0000 Thu, 29 Sep 2022 20:44:52 +0000 Thu, 29 Sep 2022 20:44:52 +0000 0 2 <p>I have a fix to obs_pfs on branch <tt>u/price/20220921</tt> that I have released as the <tt>commissioning-20220921</tt> version so it can be used for the commissioning run while this goes through review.</p> <p>Word from the commissioning run is that the fix is working. I've put it on the official ticket branch.</p> <p>Adding Paul's analysis as documented in the relevant commit messages:</p> <blockquote> <p>PostgreSQL ingest appears to have been broken on the LSST side<br/> by DM-21869, which hard-coded an 'INSERT OR IGNORE' sql command<br/> that pgsql doesn't support. The 'addVisits' method disappeared<br/> too as part of that. This requires re-implementing the addRow<br/> method on the PFS side, with the visit adding removed and the<br/> unsupported SQL removed.</p></blockquote> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a>: does this mean a separate ticket needs to be filed for the LSST/Rubin people for the bug introduced by <tt>DM-21869</tt>?</p> <p>Code changes look fine, although possible code duplication present in the two <tt>addRow</tt> methods. Is that worth addressing?</p> <p>LSST do not care about that code at all, if it even still exists, as Gen2 is being actively removed.<br/> For the same reason, I'm not worried about the code duplication.</p> <p>Merged to master.</p> Relates PIPE2D-1058 Development Rank 0|zzsuq7: Reviewers hassan