[PIPE2D-1083] Postgresql ingest is broken Created: 22/Sep/22  Updated: 30/Sep/22  Resolved: 30/Sep/22

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: EngRun
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PIPE2D-1058 Avoid/fix registry sqlite deadlocks Done
Reviewers: hassan

 Description   

Following PIPE2D-1058, postgresql ingest failed at Hilo:

$ ingestPfsImagesPgsql.py /work/drp --ignore-ingested --pfsConfigDir /work/drp/pfsDesign --mode=copy --doraise /data/raw/2022-09-22/sps/PFSA0794131*
[...]
lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql
lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3
lsst.ingestPfs INFO: /data/raw/2022-09-22/sps/PFSA07941311.fits --<copy>--> /work/drp/2022-09-22/PFSA079413b1.fits
lsst.ingestPfs WARN: Failed to ingest file /data/raw/2022-09-22/sps/PFSA07941311.fits: Failed to register file /data/raw/2022-09-22/sps/PFSA07941311.fits
Traceback (most recent call last):
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/commissioning-2022919/python/lsst/obs/pfs/ingest.py", line 675, in runFile
    self.register.addRow(registry, info, dryrun=args.dryrun, create=args.create)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_tasks/gb1d6de0934+6a8639994f/python/lsst/pipe/tasks/ingest.py", line 388, in addRow
    conn.cursor().execute(sql, values)
psycopg2.errors.SyntaxError: syntax error at or near "OR"
LINE 1: INSERT OR IGNORE INTO raw_visit VALUES (79413,'IQ_TEST','202...

There have been changes in pipe_tasks that I wasn't aware of, and I must have screwed up the testing of PIPE2D-1058.

 Comments   
Comment by price [ 22/Sep/22 ]

I have a fix to obs_pfs on branch u/price/20220921 that I have released as the commissioning-20220921 version so it can be used for the commissioning run while this goes through review.

Comment by price [ 23/Sep/22 ]

Word from the commissioning run is that the fix is working. I've put it on the official ticket branch.

Comment by hassan [ 23/Sep/22 ]

Adding Paul's analysis as documented in the relevant commit messages:

PostgreSQL ingest appears to have been broken on the LSST side
by DM-21869, which hard-coded an 'INSERT OR IGNORE' sql command
that pgsql doesn't support. The 'addVisits' method disappeared
too as part of that. This requires re-implementing the addRow
method on the PFS side, with the visit adding removed and the
unsupported SQL removed.

price: does this mean a separate ticket needs to be filed for the LSST/Rubin people for the bug introduced by DM-21869?

Comment by hassan [ 23/Sep/22 ]

Code changes look fine, although possible code duplication present in the two addRow methods. Is that worth addressing?

Comment by price [ 23/Sep/22 ]

LSST do not care about that code at all, if it even still exists, as Gen2 is being actively removed.
For the same reason, I'm not worried about the code duplication.

Comment by price [ 30/Sep/22 ]

Merged to master.

Generated at Sat Feb 10 16:02:29 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.