[DAMD-8] Object IDs need to be 64bits, so 16 hex digits Created: 04/Sep/16  Updated: 24/May/19  Resolved: 24/May/19

Status: Done
Project: Data Model
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Major
Reporter: rhl Assignee: hassan
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
blocks DAMD-57 Implement range of datamodel changes Done
Relates
relates to DAMD-56 Change pfsVisitHash formatting from %... Done
relates to DAMD-58 Ensure that Object ID range covers th... Done

 Description   

We need 64 bit object IDs to encode all the information in an HSC objectID: (tract, patch, filter, objId) or (visit, ccd, objId). The data model talks about 64bit IDs, but only specifies enough hexadecimal digits (8) in filenames to write out 32bits.

Please change the objId filename formats to "%016d"

(and thanks to Shimono-san who pointed out a related problem in DAMD-4)

 Comments   
Comment by hassan [ 20/Apr/19 ]

Instead of implementing a %16d format according to original description, will implement %16x.

Comment by hassan [ 27/Apr/19 ]

Example output filenames from a recent integration test run using the fixed version of

{obs_pfs}

are shown below:

rerun//integration/pipeline/pfsCoadd/000/00000/0,0/pfsCoadd-000-00000-0,0-0000000000000001-004-1eab2e92.fits
rerun//integration/pipeline/pfsCoadd/001/00000/0,0/pfsCoadd-001-00000-0,0-000000000000004c-004-1eab2e92.fits
rerun//integration/pipeline/pfsCoadd/001/00000/0,0/pfsCoadd-001-00000-0,0-000000000000005e-002-f1e1e95a.fits
rerun//integration/pipeline/pfsCoadd/001/00000/0,0/pfsCoadd-001-00000-0,0-0000000000000037-004-1eab2e92.fits
rerun//integration/pipeline/pfsCoadd/001/00000/0,0/pfsCoadd-001-00000-0,0-0000000000000070-004-1eab2e92.fits
rerun//integration/pipeline/pfsCoadd/001/00000/0,0/pfsCoadd-001-00000-0,0-0000000000000012-004-1eab2e92.fits
rerun//integration/pipeline/pfsCoadd/001/00000/0,0/pfsCoadd-001-00000-0,0-0000000000000069-002-2036187a.fits
rerun//integration/pipeline/pfsCoadd/001/00000/0,0/pfsCoadd-001-00000-0,0-0000000000000047-004-1eab2e92.fits
rerun//integration/pipeline/pfsCoadd/001/00000/0,0/pfsCoadd-001-00000-0,0-0000000000000073-004-1eab2e92.fits
rerun//integration/pipeline/pfsObject/000/00000/0,0/pfsObject-000-00000-0,0-0000000000000001-000032.fits
rerun//integration/pipeline/pfsObject/000/00000/0,0/pfsObject-000-00000-0,0-0000000000000001-000033.fits
rerun//integration/pipeline/pfsObject/000/00000/0,0/pfsObject-000-00000-0,0-0000000000000000-000033.fits
rerun//integration/pipeline/pfsObject/000/00000/0,0/pfsObject-000-00000-0,0-0000000000000000-000032.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000070-000032.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000037-000033.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000047-000033.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000069-000033.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-000000000000004c-000032.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-000000000000004c-000033.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000047-000032.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000070-000033.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000037-000032.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000073-000033.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000073-000032.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-000000000000005e-000032.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000012-000033.fits
rerun//integration/pipeline/pfsObject/001/00000/0,0/pfsObject-001-00000-0,0-0000000000000012-000032.fits
Comment by rhl [ 27/Apr/19 ]

It looks as if you forgot to change the format for the pfsVisitHash at the same time.  It needs to be 0x016x (I don't know what happened to the 0x that's mandated in the datamodel):

"pfsObject-%05d-%s-%3d-%016x-%02d-0x%016x.fits" % (tract, patch, catId, objId, nVisit % 100, pfsVisitHash)

Please check all the patterns. Incidentally, I think that the datamodel package has examples of all the files that are generated when you make it, so you should have been able to check the filenames that way. It would be worth checking that it's still true that the examples are complete and correct.

Also, while I know that you're quoting outputs from the code, pfsCoadd is not in the datamodel.

Comment by hassan [ 27/Apr/19 ]

The change in format of pfsVisitHash is not the topic of this particular ticket, so if that is needed I'll raise a separate one to address that.

Comment by rhl [ 27/Apr/19 ]

My comment in DAMD-4 claims that the hash problem is fixed (but I think I may be lying).  Could you check before raising a new ticket? We should make sure that all these formatting tickets are merged at the same time.

Comment by hassan [ 24/May/19 ]

Merged to master (commit 60d300c).

Generated at Sat Feb 10 15:33:17 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.