https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-48 DRP 2-D Pipeline <p>Currently the Flat FiberTraces are traced each time spectra are to be extracted. Change the extraction tasks to read FiberTrace definitions from disk</p> PIPE2D-48

Change extraction tasks to read FiberTrace parameters from disk

Story Major Done Done aritter aritter Thu, 21 Jul 2016 19:42:12 +0000 Thu, 16 Mar 2017 00:02:35 +0000 Wed, 12 Oct 2016 03:35:13 +0000 0 2 <p>This will of course require updating the data model. </p> <p>This ticket is based on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a> as it needs to read the pfsFiberTrace file. Once <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a> is reviewed and merged into master, I will rebase this ticket on master and submit for review</p> <p>The code changes are in drp_stella tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-48" title="Change extraction tasks to read FiberTrace parameters from disk" class="issue-link" data-issue-key="PIPE2D-48"><del>PIPE2D-48</del></a>. You will need datamodel tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-68" title="Add FiberTrace to data model" class="issue-link" data-issue-key="PIPE2D-68"><del>PIPE2D-68</del></a> and obs_pfs master. I added runPipeline to bin.src. All commands executed independently work fine, however when I try to run runPipeline I get '<span class="error">&#91;Errno 35&#93;</span> Resource temporarily unavailable'. No idea why. Also, when I tried to follow the quick-start guide (drp_stella/sphinx/user/getting_started.rst) on my Linux machine, I got:</p> <p>(It turns out that the Resource problem occurs while dumping the eups versions. RHL)</p> <p>conda create --name lsst-v12_1 python=2<br/> Fetching package metadata: .....Error: HTTPError: 403 Client Error: Forbidden: <a href="http://conda.lsst.codes/stack/0.12.1/linux-64/" class="external-link" rel="nofollow">http://conda.lsst.codes/stack/0.12.1/linux-64/</a></p> <p>I remember the linux binaries aren't ready yet? Maybe we can get Craig install the lsst-v12_1 binaries following the quick-start guide?</p> <p>You can't put something into review with a comment like</p> <blockquote><p>when I try to run runPipeline I get '<span class="error">&#91;Errno 35&#93;</span> Resource temporarily unavailable'.</p></blockquote> <p>It's probably an MPI thing, but it may be something worse and you need to sort this out. If you use v12_1 ctrl_pool do you still have the problem?</p> <p>When I run the constructFiberTrace example in in getting_started.rst I get an error (I'm using <tt>-<del>batch-type none</tt>) but the obsolete version <tt></del>-cores 1</tt> gives the same error.</p> <p>Four questions:</p> <ol> <li>Do you see this?</li> <li>If not, why not? It can be hidden by disabling dark and bias processing, but that's the wrong solution</li> <li>Why are the bias/dark images the wrong size?</li> <li>Is this another silly RHL mistake?</li> </ol> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> constructFiberTrace.py $PFS_DATA_DIR --rerun $whoami/tmp --id visit=5 dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibVersion=fiberTrace calibDate=2015-12-22 arm=r spectrograph=2 --batch-type none --clobber-versions root INFO: Loading config overrride file <span class="code-quote">'/Users/rhl/LSST/obs/pfs/config/constructFiberTrace.py'</span> root INFO: Config override file does not exist: <span class="code-quote">'/Users/rhl/LSST/obs/pfs/config/pfs/constructFiberTrace.py'</span> root INFO: input=/Users/rhl/PFS/Data root INFO: calib=None root INFO: output=/Users/rhl/PFS/Data/rerun/rhl/tmp CameraMapper INFO: Loading registry registry from /Users/rhl/PFS/Data/rerun/rhl/tmp/_parent/registry.sqlite3 CameraMapper INFO: Loading calibRegistry registry from /Users/rhl/PFS/Data/CALIB/calibRegistry.sqlite3 root INFO: Loading config overrride file <span class="code-quote">'/Users/rhl/LSST/obs/pfs/config/constructFiberTrace.py'</span> root INFO: Config override file does not exist: <span class="code-quote">'/Users/rhl/LSST/obs/pfs/config/pfs/constructFiberTrace.py'</span> root INFO: input=/Users/rhl/PFS/Data root INFO: calib=None root INFO: output=/Users/rhl/PFS/Data/rerun/rhl/tmp CameraMapper INFO: Loading registry registry from /Users/rhl/PFS/Data/rerun/rhl/tmp/_parent/registry.sqlite3 CameraMapper INFO: Loading calibRegistry registry from /Users/rhl/PFS/Data/CALIB/calibRegistry.sqlite3 root INFO: Running: /Users/rhl/LSST/obs/pfs/bin/constructFiberTrace.py /Users/rhl/PFS/Data --rerun rhl/tmp --id visit=5 dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibVersion=fiberTrace calibDate=2015-12-22 arm=r spectrograph=2 --clobber-versions CameraMapper INFO: Loading registry registry from /Users/rhl/PFS/Data/rerun/rhl/tmp/_parent/registry.sqlite3 CameraMapper INFO: Loading calibRegistry registry from /Users/rhl/PFS/Data/CALIB/calibRegistry.sqlite3 constructFiberTrace INFO: Scatter processing constructFiberTrace INFO: Processing DataId(initialdata={<span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2015-12-22'</span>, <span class="code-quote">'visit'</span>: 5, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2015-12-22'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'S'</span>, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'field'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'spectrograph'</span>: 2, <span class="code-quote">'ccd'</span>: 5, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>}, tag=set([])) on babayaga:84237 constructFiberTrace.isr INFO: Performing ISR on sensor DataId(initialdata={<span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2015-12-22'</span>, <span class="code-quote">'visit'</span>: 5, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2015-12-22'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'S'</span>, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'field'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'spectrograph'</span>: 2, <span class="code-quote">'ccd'</span>: 5, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>}, tag=set([])) constructFiberTrace.isr.assembleCcd WARN: No WCS found in input exposure constructFiberTrace WARN: Unable to process DataId(initialdata={<span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2015-12-22'</span>, <span class="code-quote">'visit'</span>: 5, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2015-12-22'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'S'</span>, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'field'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'spectrograph'</span>: 2, <span class="code-quote">'ccd'</span>: 5, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>}, tag=set([])): maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173)) RuntimeError on babayaga:84237 in map: maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173)) Traceback (most recent call last): File <span class="code-quote">"/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py"</span>, line 112, in wrapper <span class="code-keyword">return</span> func(*args, **kwargs) File <span class="code-quote">"/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py"</span>, line 236, in wrapper <span class="code-keyword">return</span> func(*args, **kwargs) File <span class="code-quote">"/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py"</span>, line 554, in map <span class="code-keyword">return</span> self._processQueue(context, func, list(zip(list(range(num)), dataList)), *args, **kwargs) File <span class="code-quote">"/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py"</span>, line 451, in _processQueue <span class="code-keyword">return</span> [func(self._getCache(context, i), data, *args, **kwargs) <span class="code-keyword">for</span> i, data in queue] File <span class="code-quote">"/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py"</span>, line 459, in process exposure = self.processSingle(sensorRef) File <span class="code-quote">"/Users/rhl/LSST/obs/pfs/python/lsst/obs/pfs/constructFiberTraceTask.py"</span>, line 152, in processSingle exposure = CalibTask.processSingle(self, sensorRef) File <span class="code-quote">"/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py"</span>, line 477, in processSingle <span class="code-keyword">return</span> self.isr.runDataRef(dataRef).exposure File <span class="code-quote">"/Users/rhl/LSST/pipe/base/python/lsst/pipe/base/timer.py"</span>, line 121, in wrapper res = func(self, *args, **keyArgs) File <span class="code-quote">"/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py"</span>, line 558, in runDataRef result = self.run(ccdExposure, **isrData.getDict()) File <span class="code-quote">"/Users/rhl/LSST/pipe/base/python/lsst/pipe/base/timer.py"</span>, line 121, in wrapper res = func(self, *args, **keyArgs) File <span class="code-quote">"/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py"</span>, line 495, in run self.biasCorrection(ccdExposure, bias) File <span class="code-quote">"/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py"</span>, line 588, in biasCorrection isr.biasCorrection(exposure.getMaskedImage(), biasExposure.getMaskedImage()) File <span class="code-quote">"/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isr.py"</span>, line 224, in biasCorrection (maskedImage.getBBox(afwImage.LOCAL), biasMaskedImage.getBBox(afwImage.LOCAL))) RuntimeError: maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173)) constructFiberTrace FATAL: Failed: maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173)) Traceback (most recent call last): File <span class="code-quote">"/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py"</span>, line 282, in __call__ result = task.run(**args) File <span class="code-quote">"/Users/rhl/LSST/obs/pfs/python/lsst/obs/pfs/constructFiberTraceTask.py"</span>, line 178, in run <span class="code-keyword">return</span> CalibTask.run(self, newExpRefList, butler, calibId) File <span class="code-quote">"/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py"</span>, line 353, in run data = self.scatterProcess(pool, ccdIdLists) File <span class="code-quote">"/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py"</span>, line 421, in scatterProcess resultList = pool.map(self.process, dataIdList) File <span class="code-quote">"/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py"</span>, line 112, in wrapper <span class="code-keyword">return</span> func(*args, **kwargs) File <span class="code-quote">"/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py"</span>, line 236, in wrapper <span class="code-keyword">return</span> func(*args, **kwargs) File <span class="code-quote">"/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py"</span>, line 554, in map <span class="code-keyword">return</span> self._processQueue(context, func, list(zip(list(range(num)), dataList)), *args, **kwargs) File <span class="code-quote">"/Users/rhl/LSST/ctrl/pool/python/lsst/ctrl/pool/pool.py"</span>, line 451, in _processQueue <span class="code-keyword">return</span> [func(self._getCache(context, i), data, *args, **kwargs) <span class="code-keyword">for</span> i, data in queue] File <span class="code-quote">"/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py"</span>, line 459, in process exposure = self.processSingle(sensorRef) File <span class="code-quote">"/Users/rhl/LSST/obs/pfs/python/lsst/obs/pfs/constructFiberTraceTask.py"</span>, line 152, in processSingle exposure = CalibTask.processSingle(self, sensorRef) File <span class="code-quote">"/Users/rhl/LSST/pipe/drivers/python/lsst/pipe/drivers/constructCalibs.py"</span>, line 477, in processSingle <span class="code-keyword">return</span> self.isr.runDataRef(dataRef).exposure File <span class="code-quote">"/Users/rhl/LSST/pipe/base/python/lsst/pipe/base/timer.py"</span>, line 121, in wrapper res = func(self, *args, **keyArgs) File <span class="code-quote">"/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py"</span>, line 558, in runDataRef result = self.run(ccdExposure, **isrData.getDict()) File <span class="code-quote">"/Users/rhl/LSST/pipe/base/python/lsst/pipe/base/timer.py"</span>, line 121, in wrapper res = func(self, *args, **keyArgs) File <span class="code-quote">"/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py"</span>, line 495, in run self.biasCorrection(ccdExposure, bias) File <span class="code-quote">"/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isrTask.py"</span>, line 588, in biasCorrection isr.biasCorrection(exposure.getMaskedImage(), biasExposure.getMaskedImage()) File <span class="code-quote">"/Users/lsst/products/DarwinX86/miniconda/latest/envs/lsst-v12_1/opt/lsst/ip_isr/python/lsst/ip/isr/isr.py"</span>, line 224, in biasCorrection (maskedImage.getBBox(afwImage.LOCAL), biasMaskedImage.getBBox(afwImage.LOCAL))) RuntimeError: maskedImage bbox Box2I((0, 0), (4095, 4175)) != biasMaskedImage bbox Box2I((0, 0), (4095, 4173)) </pre> </div></div> <p>I did see and fix it. That's the wrong number in the datamodel.txt which I had included in the DRP blindly. The fix is to rebuild the biases and darks</p> <p>I did rebuild the darks. Where did you fix the numbers (product and SHA1)?</p> <p>obs_pfs master commit 713ab1a</p> <p>Fixed commands by adding '--batch-type none'. Putting back in review.</p> <p>There's no need to fix row/column ordering issues on this ticket (but please file another if needs be!). The white space can stay this time. Leave the fix output requests to another ticket. Fix the if blocks you want to keep in another ticket; remove the ones you don't.</p> <ol> <li>Please remove personal entries from .gitignore</li> <li>There should be no config overrides in runPipeline</li> <li>Please move runPipeline to examples</li> <li>Please put all white-space only changes into a separate commit, and label it appropriately</li> <li>Check that the memory is contiguous for all the data in a given fibre. If you need to transpose, please add a note to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-35" title="Read &amp; write 2-D outputs according to the data model " class="issue-link" data-issue-key="PIPE2D-35"><del>PIPE2D-35</del></a></li> <li>Please don't comment out templates; remove them</li> <li>Remove all cout &lt;&lt; statements; use logging (including tracing as appropriate)</li> <li>remove all python print statements to use log</li> <li>How many info logs should become debug?</li> <li>Please remove all if False blocks. If they are possibly useful control them with variables (probably set by LsstDebug)</li> <li>There's no need to specify the wavelength files on the command line.</li> <li>Please factor the I/O into separate functions, and don't read it for every exposure</li> <li>Use "not success" not "success == False"</li> <li>What do comments like "THIS DOESN'T WORK" mean?</li> <li>Please remove all commented out code</li> <li>Use np.empty((N, M)) not np.array(shape=(N, M))</li> <li>Explain shape for wLenTemp</li> <li>Use k += 1 not k = k + 1</li> <li>Use # not """ ... """ for comments</li> <li>Things like getAllFluxes() should not need to make a copy if you get the column/row major right</li> <li>Split Spectra.py tests into separate tests if you might not want to run them (and then use a skip decorator)</li> </ol> <p>Due to timing, you may merge to master before addressing these code review issues (even the easy ones). Please file a ticket to deal with them though!</p> <p><a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-48" title="Change extraction tasks to read FiberTrace parameters from disk" class="issue-link" data-issue-key="PIPE2D-48"><del>PIPE2D-48</del></a> depends on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-68" title="Add FiberTrace to data model" class="issue-link" data-issue-key="PIPE2D-68"><del>PIPE2D-68</del></a> which is in review. Should I wait with the merge until <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-68" title="Add FiberTrace to data model" class="issue-link" data-issue-key="PIPE2D-68"><del>PIPE2D-68</del></a> is merged with master, or make drp_stella master depend on datamodel's branch tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-68" title="Add FiberTrace to data model" class="issue-link" data-issue-key="PIPE2D-68"><del>PIPE2D-68</del></a> for a short time?</p> <p>You can't merge data model to master before making the merging the data model as the change to datamodel.txt can only be done via a DAMD ticket. </p> <p>You may merge <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-48" title="Change extraction tasks to read FiberTrace parameters from disk" class="issue-link" data-issue-key="PIPE2D-48"><del>PIPE2D-48</del></a> now, and I'll review and merge the python parts of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-68" title="Add FiberTrace to data model" class="issue-link" data-issue-key="PIPE2D-68"><del>PIPE2D-68</del></a> tomorrow.</p> <p>ticket <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-100" title="Please address code review issues with PIPE2D-48" class="issue-link" data-issue-key="PIPE2D-100"><del>PIPE2D-100</del></a> filed to deal with the code review issues. Merged into master</p> Blocks PIPE2D-47 PIPE2D-68 Relates PIPE2D-172 PIPE2D-173 Development Rank 0|ii00np:i Reviewers rhl Sprint 2014-16 Story Points 2.0