[PIPE2D-48] Change extraction tasks to read FiberTrace parameters from disk Created: 22/Jul/16  Updated: 16/Mar/17  Resolved: 12/Oct/16

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Major
Reporter: aritter Assignee: aritter
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
is blocked by PIPE2D-47 make FiberTrace persistable Done
is blocked by PIPE2D-68 Add FiberTrace to data model Done
Relates
relates to PIPE2D-172 Please remove all if False blocks in ... Done
relates to PIPE2D-173 Use np.empty((N, M)) not np.ndarray(s... Won't Fix
Story Points: 2
Sprint: 2014-16
Reviewers: rhl

 Description   

Currently the Flat FiberTraces are traced each time spectra are to be extracted. Change the extraction tasks to read FiberTrace definitions from disk

 Comments   
Comment by rhl [ 03/Sep/16 ]

This will of course require updating the data model.

Comment by aritter [ 10/Oct/16 ]

This ticket is based on PIPE2D-90 as it needs to read the pfsFiberTrace file. Once PIPE2D-90 is reviewed and merged into master, I will rebase this ticket on master and submit for review

Comment by aritter [ 11/Oct/16 ]

The code changes are in drp_stella tickets/PIPE2D-48. You will need datamodel tickets/PIPE2D-68 and obs_pfs master. I added runPipeline to bin.src. All commands executed independently work fine, however when I try to run runPipeline I get '[Errno 35] Resource temporarily unavailable'. No idea why. Also, when I tried to follow the quick-start guide (drp_stella/sphinx/user/getting_started.rst) on my Linux machine, I got:

(It turns out that the Resource problem occurs while dumping the eups versions. RHL)

conda create --name lsst-v12_1 python=2
Fetching package metadata: .....Error: HTTPError: 403 Client Error: Forbidden: http://conda.lsst.codes/stack/0.12.1/linux-64/

I remember the linux binaries aren't ready yet? Maybe we can get Craig install the lsst-v12_1 binaries following the quick-start guide?

Comment by rhl [ 11/Oct/16 ]

You can't put something into review with a comment like

when I try to run runPipeline I get '[Errno 35] Resource temporarily unavailable'.

It's probably an MPI thing, but it may be something worse and you need to sort this out. If you use v12_1 ctrl_pool do you still have the problem?

Comment by rhl [ 12/Oct/16 ]

When I run the constructFiberTrace example in in getting_started.rst I get an error (I'm using -batch-type none) but the obsolete version -cores 1 gives the same error.

Four questions:

  1. Do you see this?
  2. If not, why not? It can be hidden by disabling dark and bias processing, but that's the wrong solution
  3. Why are the bias/dark images the wrong size?
  4. Is this another silly RHL mistake?
constructFiberTrace.py $PFS_DATA_DIR --rerun $whoami/tmp --id visit=5 dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibVersion=fiberTrace calibDate=2015-12-22 arm=r spectrograph=2 --batch-type none --clobber-versions
root INFO: Loading config overrride file '/Users/rhl/LSST/obs/pfs/config/constructFiberTrace.py'
root INFO: Config override file does not exist: '/Users/rhl/LSST/obs/pfs/config/pfs/constructFiberTrace.py'
root INFO: input=/Users/rhl/PFS/Data
root INFO: calib=None
root INFO: output=/Users/rhl/PFS/Data/rerun/rhl/tmp
CameraMapper INFO: Loading registry registry from /Users/rhl/PFS/Data/rerun/rhl/tmp/_parent/registry.sqlite3
CameraMapper INFO: Loading calibRegistry registry from /Users/rhl/PFS/Data/CALIB/calibRegistry.sqlite3
root INFO: Loading config overrride file '/Users/rhl/LSST/obs/pfs/config/constructFiberTrace.py'
root INFO: Config override file does not exist: '/Users/rhl/LSST/obs/pfs/config/pfs/constructFiberTrace.py'
root INFO: input=/Users/rhl/PFS/Data
root INFO: calib=None
root INFO: output=/Users/rhl/PFS/Data/rerun/rhl/tmp
CameraMapper INFO: Loading registry registry from /Users/rhl/PFS/Data/rerun/rhl/tmp/_parent/registry.sqlite3
CameraMapper INFO: Loading calibRegistry registry from /Users/rhl/PFS/Data/CALIB/calibRegistry.sqlite3
root INFO: Running: /Users/rhl/LSST/obs/pfs/bin/constructFiberTrace.py /Users/rhl/PFS/Data --rerun rhl/tmp --id visit=5 dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibVersion=fiberTrace calibDate=2015-12-22 arm=r spectrograph=2 --clobber-versions
CameraMapper INFO: Loading registry registry from /Users/rhl/PFS/Data/rerun/rhl/tmp/_parent/registry.sqlite3
CameraMapper INFO: Loading calibRegistry registry from /Users/rhl/PFS/Data/CALIB/calibRegistry.sqlite3
constructFiberTrace INFO: Scatter processing
constructFiberTrace INFO: Processing DataId(initialdata={'category': 'A', 'taiObs': '2015-12-22', 'visit': 5, 'dateObs': '2015-12-22', 'site': 'S', 'filter': 'r', 'field': 'FLAT', 'spectrograph': 2, 'ccd': 5, 'arm': 'r'}, tag=set([])) on babayaga:84237
constructFiberTrace.isr INFO: Performing ISR on sensor DataId(initialdata={'category': 'A', 'taiObs': '2015-12-22', 'visit': 5, 'dateObs': '2015-12-22', 'site': 'S', 'filter': 'r', 'field': 'FLAT', 'spectrograph': 2, 'ccd': 5, 'arm': 'r'}, tag=set([]))
constructFiberTrace.isr.assembleCcd WARN: No WCS found in input exposure
constructFiberTrace WARN: Unable to process DataId(initialdata={'category': 'A', 'taiObs': '2015-12-22', 'visit': 5, 'dateObs': '2015-12-22', 'site': 'S', 'filter': 'r', 'field': 'FLAT', 'spectrograph': 2, 'ccd': 5, 'arm': 'r'}, tag=set([])): maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173))
RuntimeError on babayaga:84237 in map: maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173))
Traceback (most recent call last):
  File "/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py", line 112, in wrapper
    return func(*args, **kwargs)
  File "/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py", line 236, in wrapper
    return func(*args, **kwargs)
  File "/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py", line 554, in map
    return self._processQueue(context, func, list(zip(list(range(num)), dataList)), *args, **kwargs)
  File "/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py", line 451, in _processQueue
    return [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
  File "/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py", line 459, in process
    exposure = self.processSingle(sensorRef)
  File "/Users/rhl/LSST/obs/pfs/python/lsst/obs/pfs/constructFiberTraceTask.py", line 152, in processSingle
    exposure = CalibTask.processSingle(self, sensorRef)
  File "/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py", line 477, in processSingle
    return self.isr.runDataRef(dataRef).exposure
  File "/Users/rhl/LSST/pipe/base/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py", line 558, in runDataRef
    result = self.run(ccdExposure, **isrData.getDict())
  File "/Users/rhl/LSST/pipe/base/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py", line 495, in run
    self.biasCorrection(ccdExposure, bias)
  File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py", line 588, in biasCorrection
    isr.biasCorrection(exposure.getMaskedImage(), biasExposure.getMaskedImage())
  File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isr.py", line 224, in biasCorrection
    (maskedImage.getBBox(afwImage.LOCAL), biasMaskedImage.getBBox(afwImage.LOCAL)))
RuntimeError: maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173))
constructFiberTrace FATAL: Failed: maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173))
Traceback (most recent call last):
  File "/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py", line 282, in __call__
    result = task.run(**args)
  File "/Users/rhl/LSST/obs/pfs/python/lsst/obs/pfs/constructFiberTraceTask.py", line 178, in run
    return CalibTask.run(self, newExpRefList, butler, calibId)
  File "/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py", line 353, in run
    data = self.scatterProcess(pool, ccdIdLists)
  File "/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py", line 421, in scatterProcess
    resultList = pool.map(self.process, dataIdList)
  File "/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py", line 112, in wrapper
    return func(*args, **kwargs)
  File "/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py", line 236, in wrapper
    return func(*args, **kwargs)
  File "/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py", line 554, in map
    return self._processQueue(context, func, list(zip(list(range(num)), dataList)), *args, **kwargs)
  File "/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py", line 451, in _processQueue
    return [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
  File "/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py", line 459, in process
    exposure = self.processSingle(sensorRef)
  File "/Users/rhl/LSST/obs/pfs/python/lsst/obs/pfs/constructFiberTraceTask.py", line 152, in processSingle
    exposure = CalibTask.processSingle(self, sensorRef)
  File "/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py", line 477, in processSingle
    return self.isr.runDataRef(dataRef).exposure
  File "/Users/rhl/LSST/pipe/base/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py", line 558, in runDataRef
    result = self.run(ccdExposure, **isrData.getDict())
  File "/Users/rhl/LSST/pipe/base/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py", line 495, in run
    self.biasCorrection(ccdExposure, bias)
  File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py", line 588, in biasCorrection
    isr.biasCorrection(exposure.getMaskedImage(), biasExposure.getMaskedImage())
  File "/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isr.py", line 224, in biasCorrection
    (maskedImage.getBBox(afwImage.LOCAL), biasMaskedImage.getBBox(afwImage.LOCAL)))
RuntimeError: maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173))
Comment by aritter [ 12/Oct/16 ]

I did see and fix it. That's the wrong number in the datamodel.txt which I had included in the DRP blindly. The fix is to rebuild the biases and darks

Comment by rhl [ 12/Oct/16 ]

I did rebuild the darks. Where did you fix the numbers (product and SHA1)?

Comment by aritter [ 12/Oct/16 ]

obs_pfs master commit 713ab1a

Comment by aritter [ 12/Oct/16 ]

Fixed commands by adding '--batch-type none'. Putting back in review.

Comment by rhl [ 12/Oct/16 ]

There's no need to fix row/column ordering issues on this ticket (but please file another if needs be!). The white space can stay this time. Leave the fix output requests to another ticket. Fix the if blocks you want to keep in another ticket; remove the ones you don't.

  1. Please remove personal entries from .gitignore
  2. There should be no config overrides in runPipeline
  3. Please move runPipeline to examples
  4. Please put all white-space only changes into a separate commit, and label it appropriately
  5. Check that the memory is contiguous for all the data in a given fibre. If you need to transpose, please add a note to PIPE2D-35
  6. Please don't comment out templates; remove them
  7. Remove all cout << statements; use logging (including tracing as appropriate)
  8. remove all python print statements to use log
  9. How many info logs should become debug?
  10. Please remove all if False blocks. If they are possibly useful control them with variables (probably set by LsstDebug)
  11. There's no need to specify the wavelength files on the command line.
  12. Please factor the I/O into separate functions, and don't read it for every exposure
  13. Use "not success" not "success == False"
  14. What do comments like "THIS DOESN'T WORK" mean?
  15. Please remove all commented out code
  16. Use np.empty((N, M)) not np.array(shape=(N, M))
  17. Explain shape for wLenTemp
  18. Use k += 1 not k = k + 1
  19. Use # not """ ... """ for comments
  20. Things like getAllFluxes() should not need to make a copy if you get the column/row major right
  21. Split Spectra.py tests into separate tests if you might not want to run them (and then use a skip decorator)
Comment by rhl [ 12/Oct/16 ]

Due to timing, you may merge to master before addressing these code review issues (even the easy ones). Please file a ticket to deal with them though!

Comment by aritter [ 12/Oct/16 ]

PIPE2D-48 depends on PIPE2D-68 which is in review. Should I wait with the merge until PIPE2D-68 is merged with master, or make drp_stella master depend on datamodel's branch tickets/PIPE2D-68 for a short time?

Comment by rhl [ 12/Oct/16 ]

You can't merge data model to master before making the merging the data model as the change to datamodel.txt can only be done via a DAMD ticket.

You may merge PIPE2D-48 now, and I'll review and merge the python parts of PIPE2D-68 tomorrow.

Comment by aritter [ 12/Oct/16 ]

ticket PIPE2D-100 filed to deal with the code review issues. Merged into master

Generated at Sat Feb 10 15:47:43 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.