https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/OBSPROC-28 PFS observation processing/procedure <p>There seems to be duplication of objId in `pfsDesign.guideStars` of `/data/pfsDesign/pfsDesign-0x74f2c06289c9823d.fits`. The objId is 6034944. This causes `UniqueViolation` error when ingesting to `pfs_design_agc` table. Investigate that and fix if it is real.</p> OBSPROC-28

duplication of objId in pfsDesign.guideStars

Task Normal Done Done Unassigned Kiyoto Yabe Sat, 17 Sep 2022 00:01:49 +0000 Sat, 17 Sep 2022 22:16:26 +0000 Sat, 17 Sep 2022 17:12:48 +0000 0 4 <p>In the datamodel code, objId for guide stars are defined as int32, while it is documented as int64 in datamodel.txt. I think this discrepancy causes an unintended rounding of int64 to int32 when reading a pfsDesign file.</p> <p><a href="https://github.com/Subaru-PFS/datamodel/blob/1264bb21b62120040d1ca7b9c1e1071dbc5e7874/python/pfs/datamodel/guideStars.py#L162" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/datamodel/blob/1264bb21b62120040d1ca7b9c1e1071dbc5e7874/python/pfs/datamodel/guideStars.py#L162</a></p> <p>You are correct <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=monodera" class="user-hover" rel="monodera">monodera</a> that there is a bug in the <tt>datamodel</tt> code. Fix in ticket branch of <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-136" title="Datamodel reads in pfsDesign.guideStar objIds as 32bit not 64bit" class="issue-link" data-issue-key="DAMD-136"><del>DAMD-136</del></a> for review. Test script to verify code copied below:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> from pfs.datamodel.pfsConfig <span class="code-keyword">import</span> PfsDesign a = PfsDesign.read(0x74f2c06289c9823d, dirName=<span class="code-quote">'.'</span>) objId = a.guideStars.objId <span class="code-keyword">assert</span>(len(set(objId)) == len(objId)) print(<span class="code-quote">'ObjectIds are unique'</span>) </pre> </div></div> <p>Fix for <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-136" title="Datamodel reads in pfsDesign.guideStar objIds as 32bit not 64bit" class="issue-link" data-issue-key="DAMD-136"><del>DAMD-136</del></a> now in <tt>datamodel</tt> master.</p> <p>Thank you! I confirmed that the problem resolved. </p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> I think now you can ingest guidestars info into pfs_design_agc table using the latest datamodel package.</p> <p>Thanks ! Yes I just tested and ingestPfsDesign works fine now.<br/> But ingestPfsConfig still doesn't behave properly :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pfs_config_agc_pkey" DETAIL: Key (pfs_design_id, visit0, guide_star_id)=(8427009382214631997, 78607, 6034944) already exists. [SQL: INSERT INTO pfs_config_agc (pfs_design_id, visit0, guide_star_id, agc_camera_id, agc_final_x_pix, agc_final_y_pix) VALUES (%(pfs_design_id)s, %(visit0)s, %(guide_star_id)s, %(agc_camera_id)s, %(agc_final_x_pix)s, %(agc_final_y_pix)s)] [parameters: ({'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 6673024, 'agc_camera_id': 0, 'agc_final_x_pix': 1017.9483642578125, 'agc_final_y_pix': 827.9166259765625}, {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 6034944, 'agc_camera_id': 0, 'agc_final_x_pix': 946.2750244140625, 'agc_final_y_pix': 902.5942993164062}, {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 6033152, 'agc_camera_id': 0, 'agc_final_x_pix': 896.0719604492188, 'agc_final_y_pix': 912.3250122070312}, {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 6031872, 'agc_camera_id': 0, 'agc_final_x_pix': 810.6386108398438, 'agc_final_y_pix': 923.0083618164062}, {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 6663936, 'agc_camera_id': 0, 'agc_final_x_pix': 960.7747192382812, 'agc_final_y_pix': 838.6574096679688}, {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 21225472, 'agc_camera_id': 0, 'agc_final_x_pix': 962.1170654296875, 'agc_final_y_pix': 819.9133911132812}, {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 21219712, 'agc_camera_id': 0, 'agc_final_x_pix': 840.3349609375, 'agc_final_y_pix': 862.7570190429688}, {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 21228416, 'agc_camera_id': 0, 'agc_final_x_pix': 881.2999267578125, 'agc_final_y_pix': 726.2146606445312} ... displaying 10 of 556 total bound parameter sets ... {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 21866112, 'agc_camera_id': 5, 'agc_final_x_pix': 50.762168884277344, 'agc_final_y_pix': 1008.1949462890625}, {'pfs_design_id': 8427009382214631997, 'visit0': 78607, 'guide_star_id': 9773824, 'agc_camera_id': 5, 'agc_final_x_pix': 35.872276306152344, 'agc_final_y_pix': 827.4269409179688})] </pre> </div></div> <p>If you want to reproduce the problem, I've put the pfsConfig file I just generated there :<br/> <em>/data/tests/pfsConfig/pfsConfig-0x74f2c06289c9823d-078607.fits</em></p> <p>Thanks.</p> <p>That <tt>pfsConfig</tt> file you provided does contain a single duplicate <tt>guide_star_id</tt> value of <tt>6034944</tt>. But how was it generated? Was it generated using the fix to <tt>datamodel</tt> in <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-136" title="Datamodel reads in pfsDesign.guideStar objIds as 32bit not 64bit" class="issue-link" data-issue-key="DAMD-136"><del>DAMD-136</del></a>?</p> <p>So, as <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a> pointed out, the pfsConfig file needed to be regenerated.<br/> So I did so, using the latest datamodel and managed to ingest it into opdb successfully. <br/> All good now, thanks for the help.</p> Relates DAMD-136 Development Rank 0|zzsuof: