https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1272 DRP 2-D Pipeline <p>Since summit drp is not heavily used we though we could get away with using sqlite but some ingest failures started to happen few days ago.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(rubin3_ics) pfs@shell-ics:/data/logs/actors/drp$ cat 2023-*|grep 'Failed to register' 2023-07-22 04:02:35.951Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-22/ramps/PFSB09719733.fits: Failed to register file /data/raw/2023-07-22/ramps/PFSB09719733.fits 2023-07-22 10:23:59.725Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-22/ramps/PFSB09731433.fits: Failed to register file /data/raw/2023-07-22/ramps/PFSB09731433.fits 2023-07-23 06:42:20.056Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-23/ramps/PFSB09748533.fits: Failed to register file /data/raw/2023-07-23/ramps/PFSB09748533.fits 2023-07-23 13:06:58.276Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-23/ramps/PFSB09757133.fits: Failed to register file /data/raw/2023-07-23/ramps/PFSB09757133.fits 2023-07-23 14:10:09.782Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-23/ramps/PFSB09758713.fits: Failed to register file /data/raw/2023-07-23/ramps/PFSB09758713.fits 2023-07-24 12:43:33.564Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-24/ramps/PFSB09781333.fits: Failed to register file /data/raw/2023-07-24/ramps/PFSB09781333.fits 2023-07-24 21:06:02.068Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-24/ramps/PFSB09789733.fits: Failed to register file /data/raw/2023-07-24/ramps/PFSB09789733.fits 2023-07-25 11:05:10.116Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/ramps/PFSB09801433.fits: Failed to register file /data/raw/2023-07-25/ramps/PFSB09801433.fits 2023-07-25 13:24:19.964Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/sps/PFSA09804332.fits: Failed to register file /data/raw/2023-07-25/sps/PFSA09804332.fits 2023-07-25 21:38:30.050Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/ramps/PFSB09813513.fits: Failed to register file /data/raw/2023-07-25/ramps/PFSB09813513.fits 2023-07-25 22:30:51.503Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/ramps/PFSB09814833.fits: Failed to register file /data/raw/2023-07-25/ramps/PFSB09814833.fits 2023-07-25 22:42:22.287Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-25/ramps/PFSB09815133.fits: Failed to register file /data/raw/2023-07-25/ramps/PFSB09815133.fits 2023-07-26 05:26:15.771Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-26/ramps/PFSB09820413.fits: Failed to register file /data/raw/2023-07-26/ramps/PFSB09820413.fits 2023-07-26 08:42:29.993Z actor.ingestPfs 30 ingest.py:627 Failed to ingest file /data/raw/2023-07-26/sps/PFSA09833631.fits: Failed to register file /data/raw/2023-07-26/sps/PFSA09833631.fits </pre> </div></div> <p>My guess is that it's failing because of concurrent ingest.<br/> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> I remember you did the migration on pfsa, if you have some instructions/scripts ready, I'd be happy to do that on shell2-ics.</p> PIPE2D-1272

Switch drp summit registry to postgres

Bug Normal Open Unresolved arnaud.lefur arnaud.lefur engRun12 Wed, 26 Jul 2023 11:42:50 +0000 Wed, 30 Aug 2023 14:20:07 +0000 0 3 <p>The db name in Hilo is <tt>registry_db</tt>. We need to be sure that either that db on <tt>shell2-ics</tt> does not get replicated to Hilo, or that it gets a different name.....</p> <p>My notes from <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1058" title="Avoid/fix registry sqlite deadlocks" class="issue-link" data-issue-key="PIPE2D-1058"><del>PIPE2D-1058</del></a>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> PIPE2D-1058: Avoid/fix registry sqlite deadlocks We<span class="code-quote">'re going to move the registry to postgresql. I'</span>ve asked CSES <span class="code-keyword">for</span> a <span class="code-keyword">new</span> database on tiger2-sumire. Testing on my laptop <span class="code-keyword">while</span> waiting <span class="code-keyword">for</span> CSES: createuser -h localhost pfs createdb -h localhost --owner=pfs pfs_gen2 sqlite3 registry.sqlite3 .dump | sed -e <span class="code-quote">'s|^PRAGMA.*$||'</span> -e <span class="code-quote">'s|integer primary key autoincrement|serial primary key|g'</span> -e <span class="code-quote">'s|<span class="code-object">double</span>|<span class="code-object">double</span> precision|g'</span> -e <span class="code-quote">'s|^.*sqlite.*$||'</span> -e <span class="code-quote">'s|pfsDesignId <span class="code-object">int</span>|pfsDesignId bigint|g'</span> | tee registry.sql | psql -h localhost -U pfs pfs_gen2 Here's how to delete all the tables in a database in postgresql: DO $$ DECLARE r RECORD; BEGIN FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = current_schema()) LOOP EXECUTE <span class="code-quote">'DROP TABLE IF EXISTS '</span> || quote_ident(r.tablename) || <span class="code-quote">' CASCADE'</span>; END LOOP; END $$; pfs_gen2=&gt; select field, pfsDesignId, count(*) from raw group by field, pfsDesignId order by field, pfsDesignId; Results look identical (at least <span class="code-keyword">for</span> the first page!) between postresql and sqlite. Yabe-san created a database at Hilo: psql -h 133.40.152.102 -p 5433 -U pfs -d registry_gen2 The password is XXX Save that password: price@pfsa-usr01-gb:~ $ cat .pgpass *:*:registry_gen2:pfs:XXX I<span class="code-quote">'m working on pfsa-usr01-gb (via <span class="code-quote">"ssh pfs"</span> with the VPN), which I'</span>m told is at Hilo. Looks like there's a data repo in /work/drp, and raw data in /data/raw, and the stack is /work/stack. price@pfsa-usr01-gb:~ $ sqlite3 /work/drp/registry.sqlite3 .dump | sed -e <span class="code-quote">'s|^PRAGMA.*$||'</span> -e <span class="code-quote">'s|integer primary key autoincrement|serial primary key|g'</span> -e <span class="code-quote">'s|<span class="code-object">double</span>|<span class="code-object">double</span> precision|g'</span> -e <span class="code-quote">'s|^.*sqlite.*$||'</span> -e <span class="code-quote">'s|pfsDesignId <span class="code-object">int</span>|pfsDesignId bigint|g'</span> | tee registry.sql | psql -h 133.40.152.102 -p 5433 -U pfs registry_gen2 Well, that worked just fine. To use it, we need to put a <span class="code-quote">"registry.pgsql"</span> file in the repo: price@pfsa-usr01-gb:~ $ cat /work/drp/registry.pgsql host: <span class="code-quote">"133.40.152.102"</span> port: 5433 user: <span class="code-quote">"pfs"</span> database: <span class="code-quote">"registry_gen2"</span> (Password will be supplied via ~/.pgpass, which is better <span class="code-keyword">protected</span>.) price@pfsa-usr01-gb:~ $ . /work/stack/loadLSST.bash (lsst-scipipe-1172c30) price@pfsa-usr01-gb:~ $ setup pfs_pipe2d (lsst-scipipe-1172c30) price@pfsa-usr01-gb:~ $ reduceExposure.py /work/drp --calib /work/drp/CALIB-20220630 --rerun price/pipe2d-1058 --id visit=72016 arm=r root INFO: Loading config overrride file <span class="code-quote">'/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2022.17/config/reduceExposure.py'</span> CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql CameraMapper INFO: Loading calib registry from /work/drp/CALIB-20220630/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /work/drp/CALIB-20220630/calibRegistry.sqlite3 root WARN: No data found <span class="code-keyword">for</span> dataId=OrderedDict([(<span class="code-quote">'visit'</span>, 72016), (<span class="code-quote">'arm'</span>, <span class="code-quote">'r'</span>)]) root INFO: Running: /work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.17/bin/reduceExposure.py /work/drp --calib /work/drp/CALIB-20220630 --rerun price/pipe2d-1058 --id visit=72016 arm=r root WARN: Not running the task because there is no data to process; you may preview data using <span class="code-quote">"--show data"</span> Well, that worked, sort of. It went to the correct registry, but there's no visit=72016 in there. I guess that means we can test the ingest! (lsst-scipipe-1172c30) price@pfsa-usr01-gb:~ $ ingestPfsImages.py /work/drp <span class="code-quote">"/data/raw/2022-02-22/sps/PFSA07*.fits"</span> --mode=copy CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3 Traceback (most recent call last): File <span class="code-quote">"/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2022.17/bin/ingestPfsImages.py"</span>, line 3, in &lt;module&gt; PfsIngestTask.parseAndRun() File <span class="code-quote">"/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py"</span>, line 416, in parseAndRun task.run(args) File <span class="code-quote">"/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py"</span>, line 542, in run context = self.register.openRegistry(root, create=args.create, dryrun=args.dryrun) File <span class="code-quote">"/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py"</span>, line 298, in openRegistry context = RegistryContext(registryName, self.createTable, create, self.config.permissions) File <span class="code-quote">"/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2022.17/python/lsst/obs/pfs/ingest.py"</span>, line 53, in __init__ os.chmod(registryName, permissions) PermissionError: [Errno 1] Operation not permitted: <span class="code-quote">'/work/drp/registry.sqlite3'</span> So we need to fix the ingest. Basing off w.2022.17 to be consistent with what's in use at Hilo. (lsst-scipipe-1172c30) price@pfsa-usr01-gb:~ $ ingestPfsImages.py /work/drp <span class="code-quote">"/data/raw/2022-02-22/sps/PFSA07*.fits"</span> --mode=copy --pfsConfigDir /work/drp/pfsDesign -c clobber=True Success! But the visits are not in the psql registry... and it's ended up in the sqlite registry! Oh! There's a separate script <span class="code-keyword">for</span> ingesting into a psql registry. It has a different RegisterTask with slightly different SQL commands <span class="code-keyword">for</span> creating the tables, and such. (lsst-scipipe-1172c30) price@pfsa-usr01-gb:~/pfs/obs_pfs[tickets/PIPE2D-1058] $ ingestPfsImagesPgsql.py /work/drp <span class="code-quote">"/data/raw/2022-02-22/sps/PFSA07*.fits"</span> --mode=copy --pfsConfigDir /work/drp/pfsDesign -c clobber=True CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3 ingestPfs INFO: /data/raw/2022-02-22/sps/PFSA07206314.fits --&lt;copy&gt;--&gt; /work/drp/2022-02-22/PFSA072063m1.fits File <span class="code-quote">"/home/price/pfs/obs_pfs/bin/ingestPfsImagesPgsql.py"</span>, line 3, in &lt;module&gt; PfsPgsqlIngestTask.parseAndRun() File <span class="code-quote">"/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py"</span>, line 416, in parseAndRun task.run(args) File <span class="code-quote">"/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py"</span>, line 553, in run self.register.addRow(registry, info, dryrun=args.dryrun, create=args.create) File <span class="code-quote">"/work/stack_INFRA-290/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py"</span>, line 368, in addRow conn.cursor().execute(sql, values) psycopg2.IntegrityError: duplicate key value violates unique constraint <span class="code-quote">"raw_pkey"</span> DETAIL: Key (id)=(2) already exists. We need to reset the auto increment start point. registry_gen2=# select pg_get_serial_sequence(<span class="code-quote">'raw'</span>, <span class="code-quote">'id'</span>); pg_get_serial_sequence ------------------------ <span class="code-keyword">public</span>.raw_id_seq (1 row) registry_gen2=# select max(id) from raw; max ------- 94666 (1 row) registry_gen2=# ALTER SEQUENCE raw_id_seq RESTART WITH 94667; ALTER SEQUENCE (lsst-scipipe-1172c30) price@pfsa-usr01-gb:~/pfs/obs_pfs[tickets/PIPE2D-1058] $ ingestPfsImagesPgsql.py /work/drp <span class="code-quote">"/data/raw/2022-02-22/sps/PFSA07*.fits"</span> --mode=copy --pfsConfigDir /work/drp/pfsDesign -c clobber=True (lsst-scipipe-1172c30) price@pfsa-usr01-gb:~/pfs/obs_pfs[tickets/PIPE2D-1058] $ reduceExposure.py /work/drp --calib /work/drp/CALIB-20220630 --rerun price/pipe2d-1058 --id visit=72016 arm=r -c isr.doFlat=False That works! </pre> </div></div> Relates PIPE2D-1058 Development Rank 0|zzsz13: Sprint Eng12July Eng13Oct Story Points 1.0