[PIPE2D-1147] flux calibration fails for m-band data

Wed, 25 Jan 2023 06:41:30 +0000
Reducing data set of GA targets with m-band, flux calibration fails in the middle with the error message below:
INFO  2023-01-16T16:39:45.573-1000 fitPfsFluxReference ({'visit': 83219, 'dateObs': '2022-11-21'})(fitPfsFluxReference.py:290)- Fitting models to spectra (takes some time)...
INFO  2023-01-16T17:18:37.548-1000 fitPfsFluxReference ({'visit': 83219, 'dateObs': '2022-11-21'})(fitPfsFluxReference.py:322)- Making reference spectra by interpolation
WARN  2023-01-16T17:19:56.388-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_ABSENT_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.390-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_BAD_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.390-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_DEFICIENT_BBFLUXES' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.391-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_FITBBSED_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.392-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_ESTIMATERADIALVELOCITY_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.393-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_FITMODELS_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.394-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_MAKEREFERENCESPECTRA_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitReference.py:81: IntegrationWarning: The maximum number of subdivisions (100) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze
  the integrand in order to determine the difficulties.  If the position of a
  local difficulty can be determined (singularity, discontinuity) one will
  probably gain from splitting up the interval and calling the integrator
  on the subranges.  Perhaps a special-purpose integrator should be used.
  return scipy.integrate.quad(function, self.wavelength[0], self.wavelength[-1],
/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitReference.py:81: IntegrationWarning: The maximum number of subdivisions (100) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze
  the integrand in order to determine the difficulties.  If the position of a
  local difficulty can be determined (singularity, discontinuity) one will
  probably gain from splitting up the interval and calling the integrator
  on the subranges.  Perhaps a special-purpose integrator should be used.
  return scipy.integrate.quad(function, self.wavelength[0], self.wavelength[-1],
WARNING: VerifyWarning: Keyword name 'MP_ABSENT_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_BAD_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_DEFICIENT_BBFLUXES' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_FITBBSED_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_ESTIMATERADIALVELOCITY_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_FITMODELS_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_MAKEREFERENCESPECTRA_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py", line 809, in curve_fit
    transform = cholesky(sigma, lower=True)
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/linalg/_decomp_cholesky.py", line 88, in cholesky
    c, lower = _cholesky(a, lower=lower, overwrite_a=overwrite_a, clean=True,
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/linalg/_decomp_cholesky.py", line 37, in _cholesky
    raise LinAlgError("%d-th leading minor of the array is not positive "
numpy.linalg.LinAlgError: 16-th leading minor of the array is not positive definiteThe above exception was the direct cause of the following exception:Traceback (most recent call last):
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 433, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 504, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 174, in runDataRef
    reference = self.run(pfsConfig, merged, mergedLsf)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 275, in run
    radialVelocities = self.getRadialVelocities(pfsConfig, pfsMerged, pfsMergedLsf, bbPdfs)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 423, in getRadialVelocities
    radialVelocities.append(self.estimateRadialVelocity.run(spectrum, modelSpectrum))
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/estimateRadialVelocity.py", line 166, in run
    pfit, pcov = scipy.optimize.curve_fit(
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py", line 811, in curve_fit
    raise ValueError("`sigma` must be positive definite.") from e
ValueError: `sigma` must be positive definite.
"""The above exception was the direct cause of the following exception:Traceback (most recent call last):
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/bin/fitPfsFluxReference.py", line 3, in <module>
    FitPfsFluxReferenceTask.parseAndRun()
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 688, in parseAndRun
    resultList = taskRunner.run(parsedCmd)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 244, in run
    resultList = list(mapFunc(self, targetList))
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 47, in _runPool
    return pool.map_async(function, iterable).get(timeout)
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
ValueError: `sigma` must be positive definite.
Note that I specified like `--id visit=83219 spectrograph=1`.
PFS-JIRA

[PIPE2D-1147] flux calibration fails for m-band data
