Uploaded image for project: 'DRP 2-D Pipeline'
  1. DRP 2-D Pipeline
  2. PIPE2D-1272

Switch drp summit registry to postgres

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Normal
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      1
    • Sprint:
      Eng12July, Eng13Oct

      Description

      Since summit drp is not heavily used we though we could get away with using sqlite but some ingest failures started to happen few days ago.

      (rubin3_ics) pfs@shell-ics:/data/logs/actors/drp$ cat 2023-*|grep 'Failed to register'
2023-07-22 04:02:35.951Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-22/ramps/PFSB09719733.fits: Failed to register file /data/raw/2023-07-22/ramps/PFSB09719733.fits
2023-07-22 10:23:59.725Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-22/ramps/PFSB09731433.fits: Failed to register file /data/raw/2023-07-22/ramps/PFSB09731433.fits
2023-07-23 06:42:20.056Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-23/ramps/PFSB09748533.fits: Failed to register file /data/raw/2023-07-23/ramps/PFSB09748533.fits
2023-07-23 13:06:58.276Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-23/ramps/PFSB09757133.fits: Failed to register file /data/raw/2023-07-23/ramps/PFSB09757133.fits
2023-07-23 14:10:09.782Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-23/ramps/PFSB09758713.fits: Failed to register file /data/raw/2023-07-23/ramps/PFSB09758713.fits
2023-07-24 12:43:33.564Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-24/ramps/PFSB09781333.fits: Failed to register file /data/raw/2023-07-24/ramps/PFSB09781333.fits
2023-07-24 21:06:02.068Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-24/ramps/PFSB09789733.fits: Failed to register file /data/raw/2023-07-24/ramps/PFSB09789733.fits
2023-07-25 11:05:10.116Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/ramps/PFSB09801433.fits: Failed to register file /data/raw/2023-07-25/ramps/PFSB09801433.fits
2023-07-25 13:24:19.964Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/sps/PFSA09804332.fits: Failed to register file /data/raw/2023-07-25/sps/PFSA09804332.fits
2023-07-25 21:38:30.050Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/ramps/PFSB09813513.fits: Failed to register file /data/raw/2023-07-25/ramps/PFSB09813513.fits
2023-07-25 22:30:51.503Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/ramps/PFSB09814833.fits: Failed to register file /data/raw/2023-07-25/ramps/PFSB09814833.fits
2023-07-25 22:42:22.287Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/ramps/PFSB09815133.fits: Failed to register file /data/raw/2023-07-25/ramps/PFSB09815133.fits
2023-07-26 05:26:15.771Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-26/ramps/PFSB09820413.fits: Failed to register file /data/raw/2023-07-26/ramps/PFSB09820413.fits
2023-07-26 08:42:29.993Z actor.ingestPfs  30 ingest.py:627 Failed to ingest file /data/raw/2023-07-26/sps/PFSA09833631.fits: Failed to register file /data/raw/2023-07-26/sps/PFSA09833631.fits

      My guess is that it's failing because of concurrent ingest.
      price I remember you did the migration on pfsa, if you have some instructions/scripts ready, I'd be happy to do that on shell2-ics.

        Attachments

          Issue Links

          relates to

          Task - A task that needs to be done. PIPE2D-1058 Avoid/fix registry sqlite deadlocks

          • Normal - Normal priorities
          • Done

            Activity

              People

              • Assignee:
                arnaud.lefur arnaud.lefur
                Reporter:
                arnaud.lefur arnaud.lefur
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: