-
Type:
Bug
-
Status: Done (View Workflow)
-
Priority:
Blocker
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:3
-
Sprint:2DDRP-2021 A 6
So I did the usual steps to make the best detectorMap we can.
But I tried to construct co-added fiberProfile and this time it worked fine :
(lsst-scipipe) [afur@tiger2-sumire Subaru]$ constructFiberProfiles.py /tigress/HSC/PFS/Subaru/ --calib /tigress/HSC/PFS/Subaru/CALIB-alefur-dcb21/ --rerun alefur/pipe2d-794-dcb21/ --id visit=51437..51448 arm=r --batch-type=none -c isr.doFlat=False --clobber-config
fiberProfiles INFO: Scatter combination
fiberProfiles INFO: Combining 12 inputs {'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'filter': None, 'calibTime': '2021-05-13T19:39:22.590250240', 'visit0': 51437, 'calibDate': '2021-05-13'} on tiger2-sumire.princeton.edu:248851
fiberProfiles.centroidTraces INFO: Measured 87552 centroids for 21 traces
fiberProfiles.adjustDetectorMap INFO: detectorMap parameter adjustment: [-4.86461392e-02 -7.23413176e-04 4.12246307e-02 -4.37391265e+00
1.55668964e+00 1.44867453e+01]
fiberProfiles.adjustDetectorMap INFO: Adjustment chi2: trace=229575.408722 from 78565/87552; x=0.000000 y=0.000000 from 0/0; total=229575.408722 from 78565
fiberProfiles.adjustDetectorMap INFO: Adjustment robust RMS (pixels): trace=0.089716; x=0.000000 y=0.000000
fiberProfiles.adjustDetectorMap INFO: Adjustment weighted RMS (pixels): trace=0.085612; x=0.000000 y=0.000000
fiberProfiles.profiles INFO: Profiled 21 fibers
fiberProfiles INFO: 21 fiber profiles found on combined flat
fiberProfiles INFO: Median relative transmission of fiber 12 is 88080.147133
fiberProfiles INFO: Median relative transmission of fiber 32 is 114510.465595
fiberProfiles INFO: Median relative transmission of fiber 60 is 53196.588825
fiberProfiles INFO: Median relative transmission of fiber 110 is 41931.565493
fiberProfiles INFO: Median relative transmission of fiber 111 is 96943.350955
fiberProfiles INFO: Median relative transmission of fiber 161 is 120261.821369
fiberProfiles INFO: Median relative transmission of fiber 210 is 102462.704979
fiberProfiles INFO: Median relative transmission of fiber 223 is 82062.618103
fiberProfiles INFO: Median relative transmission of fiber 259 is 91472.425965
fiberProfiles INFO: Median relative transmission of fiber 289 is 116851.204934
fiberProfiles INFO: Median relative transmission of fiber 341 is 120401.857669
fiberProfiles INFO: Median relative transmission of fiber 360 is 63771.561047
fiberProfiles INFO: Median relative transmission of fiber 400 is 96077.192405
fiberProfiles INFO: Median relative transmission of fiber 418 is 115958.292795
fiberProfiles INFO: Median relative transmission of fiber 449 is 53005.611072
fiberProfiles INFO: Median relative transmission of fiber 497 is 106138.081933
fiberProfiles INFO: Median relative transmission of fiber 518 is 101344.702511
fiberProfiles INFO: Median relative transmission of fiber 545 is 92336.873732
fiberProfiles INFO: Median relative transmission of fiber 593 is 64579.054335
fiberProfiles INFO: Median relative transmission of fiber 621 is 94532.765159
fiberProfiles INFO: Median relative transmission of fiber 643 is 96358.704685
fiberProfiles INFO: Writing {'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'filter': None, 'calibTime': '2021-05-13T19:39:22.590250240', 'visit0': 51437, 'calibDate': '2021-05-13'} on tiger2-sumire.princeton.edu:248851
Note that I'm still have to use isr.doFlat=False .
Then, I've fitted and ingested a new detectorMap using a set of 36 arcs, everything okay so far.
Then it blew up when I tried to rerun constructFiberProfile(exact same command as before) :
Traceback (most recent call last): [278/1861]
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/18.1.0/python/lsst/pipe/drivers/constructCalibs.py", line 369, in __call__
result = task.runDataRef(**args)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/18.1.0/python/lsst/pipe/drivers/constructCalibs.py", line 476, in runDataRef
calibs = self.scatterCombine(combinePool, outputId, ccdIdLists, scales)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/18.1.0/python/lsst/pipe/drivers/constructCalibs.py", line 711, in scatterCombine
images = pool.map(self.combine, data, outputId)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 680, in map
return self.reduce(context, None, func, dataList, *args, **kwargs)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 113, in wrapper
return func(*args, **kwargs)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 240, in wrapper
return func(*args, **kwargs)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 716, in reduce
*args, **kwargs)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 573, in _reduceQueue
resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 573, in <listcomp>
resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/constructFiberProfilesTask.py", line 148, in combine
detMap = self.adjustDetectorMap.run(detMap, ArcLineSet.empty(), traces).detectorMap
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/adjustDetectorMap.py", line 346, in run
result = fit(params, selectTraces, selectLines)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/adjustDetectorMap.py", line 296, in fit
selectLines=useLines), params, method="Powell")
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/_minimize.py", line 591, in minimize
return _minimize_powell(fun, x0, args, callback, **options)
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 2610, in _minimize_powell
tol=xtol*100)
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 2372, in _linesearch_powell
alpha_min, fret, iter, num = brent(myfunc, full_output=1, tol=tol)
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 2083, in brent
res = _minimize_scalar_brent(func, brack, args, **options)
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 2115, in _minimize_scalar_brent
brent.optimize()
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 1899, in optimize
xa, xb, xc, fa, fb, fc, funcalls = self.get_bracket_info()
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 1873, in get_bracket_info
xa, xb, xc, fa, fb, fc, funcalls = bracket(func, args=args)
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 2349, in bracket
fw = func(*((w,) + args))
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 2371, in myfunc
return func(p + alpha*xi)
File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 300, in function_wrapper
return function(*(wrapper_args + args))
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/adjustDetectorMap.py", line 236, in calculateChi2
traceResid = calculateTraceResiduals(adjusted)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/adjustDetectorMap.py", line 199, in calculateTraceResiduals
return xTraces - detectorMap.getXCenter(fiberIdTraces, yTraces)
lsst.pex.exceptions.wrappers.LengthError:
File "src/spline.cc", line 74, in pfs::drp::stella::math::Spline<T>::Spline(const ndarray::Array<const double, 1, 1>&, const ndarray::Array<const double, 1, 1>&, pfs::drp::stella::math::Spline<T>::Inte
rpolationTypes) [with T = double]
At least 3 points are needed to fit a spline {0}
lsst::pex::exceptions::LengthError: 'At least 3 points are needed to fit a spline'
So I decided to forget about the second iteration and decided to proceed with reduceExposure, but it also failed with :
reduceExposure FATAL: Failed on dataIds=[{'visit': 51449, 'arm': 'r', 'dateObs': '2021-05-13', 'site': 'S', 'category': 'A', 'spectrograph': 1, 'field': 'SEEING_TEST', 'ccd': 1, 'filter': 'r', 'expTime': 8.0, 'dataType': 'COMPARISON', 'taiObs': '2021-05-13T19:46:28.673', 'pfsDesignId': 17592186044673, 'slitOffset': 0.0}]: LengthError:
File "src/image/Image.cc", line 68, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_allocateView(const Extent2I&, ndarray::Manager::Ptr&) [with PixelT = double; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<double, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>; ndarray::Manager::Ptr = boost::intrusive_ptr<ndarray::Manager>]
Image dimensions (2147479549 x 4176) too large; int overflow detected. {0}
lsst::pex::exceptions::LengthError: 'Image dimensions (2147479549 x 4176) too large; int overflow detected.'
Traceback (most recent call last):
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
result = self.runTask(task, dataRef, kwargs)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/reduceExposure.py", line 243, in runDataRef
calibs = self.getSpectralCalibs(sensorRef, exposure, pfsConfig)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/reduceExposure.py", line 450, in getSpectralCalibs
fiberTraces = fiberProfiles.makeFiberTracesFromDetectorMap(detectorMap)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/fiberProfileSet.py", line 135, in makeFiberTracesFromDetectorMap
return self.makeFiberTraces(detectorMap.bbox.getDimensions(), centers)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/fiberProfileSet.py", line 155, in makeFiberTraces
traces.add(self[fiberId].makeFiberTrace(dimensions, centers[fiberId], fiberId))
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.26/python/pfs/drp/stella/fiberProfile.py", line 244, in makeFiberTrace
self.profiles, ~self.profiles.mask, centerFunc(rows), self.norm)
lsst.pex.exceptions.wrappers.LengthError:
File "src/image/Image.cc", line 68, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_allocateView(const Extent2I&, ndarray::Manager::Ptr&) [with PixelT = double; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<double, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>; ndarray::Manager::Ptr = boost::intrusive_ptr<ndarray::Manager>]
Image dimensions (2147479549 x 4176) too large; int overflow detected. {0}
lsst::pex::exceptions::LengthError: 'Image dimensions (2147479549 x 4176) too large; int overflow detected.'
