[PIPE2D-1291] Unable to retrieve calexp after reducing exposure Created: 31/Aug/23  Updated: 31/Aug/23  Resolved: 31/Aug/23

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: arnaud.lefur Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Jared reports that on tiger, as he's able to successfully access the pfsArm and pfsMerged, he's not able to access the calexp even though the file exist, the butler.get fails with :

 NotFoundError: 
  File "src/PropertySet.cc", line 186, in const std::type_info& lsst::daf::base::PropertySet::typeOf(const string&) const
    CALIB not found {0}
lsst::pex::exceptions::NotFoundError: 'CALIB not found'

rerun : /projects/HSC/PFS/Subaru/rerun/edr3-20230829
dataId:  dict(arm='r', visit=98042, spectrograph=1)


 Comments   
Comment by rhl [ 31/Aug/23 ]

Please provide a complete how-to-repeat

Comment by arnaud.lefur [ 31/Aug/23 ]

Just do :
reduceExposure.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB --rerun=pipe2d-1291 -j 8 --id visit=98042 -c isr.doFlat=False

and then : 

import lsst.daf.persistence as dafPersist

butler = dafPersist.Butler('/projects/HSC/PFS/Subaru/rerun/pipe2d-1291')

butler.get('pfsArm', arm='r', spectrograph=1, visit=98042) # OK
butler.get('calexp', arm='r', spectrograph=1, visit=98042) # FAILS
Comment by price [ 31/Aug/23 ]

This is due to the use of a dotted HIERARCH key name (PFS.DRP2D.CALIB) when recording the calib path (PIPE2D-1281). I'm updating it to use underscores instead of dots (PFS_DRP2D_CALIB).

Fix is in obs_pfs, and is currently being tested.

Comment by rhl [ 31/Aug/23 ]

That requires me to regenerate my own rerun, which is more than is needed to replicate this problem.  So

butler = dafPersist.Butler('/projects/HSC/PFS/Subaru/rerun/edr3-20230829')
butler.get('calexp', arm='r', spectrograph=1, visit=98042) # FAILS

would have been the minimal how-to-repeat. The initial ticket didn't tell me if e.g. a CALIB argument was passed to the butler.

But in this case, as so often, Paul's found the real problem

Comment by arnaud.lefur [ 31/Aug/23 ]

Thanks.
Does it mean I need to re-process the data once the fix is in place ?

Comment by price [ 31/Aug/23 ]

Yes, I'm afraid so.

Comment by arnaud.lefur [ 31/Aug/23 ]

No worries, I did reprocess the data and was able to access the calexp file.
So your fix appears to work, thanks !

Comment by price [ 31/Aug/23 ]

Integration test passed.
Merged to master.

Generated at Sat Feb 10 16:05:34 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.