[PIPE2D-1147] flux calibration fails for m-band data Created: 18/Jan/23  Updated: 25/Jan/23  Resolved: 25/Jan/23

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: Kiyoto Yabe Assignee: sogo.mineo
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File v83219_fiberId342.png    
Reviewers: price

 Description   

Reducing data set of GA targets with m-band, flux calibration fails in the middle with the error message below:

INFO  2023-01-16T16:39:45.573-1000 fitPfsFluxReference ({'visit': 83219, 'dateObs': '2022-11-21'})(fitPfsFluxReference.py:290)- Fitting models to spectra (takes some time)...
INFO  2023-01-16T17:18:37.548-1000 fitPfsFluxReference ({'visit': 83219, 'dateObs': '2022-11-21'})(fitPfsFluxReference.py:322)- Making reference spectra by interpolation
WARN  2023-01-16T17:19:56.388-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_ABSENT_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.390-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_BAD_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.390-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_DEFICIENT_BBFLUXES' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.391-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_FITBBSED_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.392-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_ESTIMATERADIALVELOCITY_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.393-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_FITMODELS_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
WARN  2023-01-16T17:19:56.394-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_MAKEREFERENCESPECTRA_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitReference.py:81: IntegrationWarning: The maximum number of subdivisions (100) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze
  the integrand in order to determine the difficulties.  If the position of a
  local difficulty can be determined (singularity, discontinuity) one will
  probably gain from splitting up the interval and calling the integrator
  on the subranges.  Perhaps a special-purpose integrator should be used.
  return scipy.integrate.quad(function, self.wavelength[0], self.wavelength[-1],
/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitReference.py:81: IntegrationWarning: The maximum number of subdivisions (100) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze
  the integrand in order to determine the difficulties.  If the position of a
  local difficulty can be determined (singularity, discontinuity) one will
  probably gain from splitting up the interval and calling the integrator
  on the subranges.  Perhaps a special-purpose integrator should be used.
  return scipy.integrate.quad(function, self.wavelength[0], self.wavelength[-1],
WARNING: VerifyWarning: Keyword name 'MP_ABSENT_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_BAD_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_DEFICIENT_BBFLUXES' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_FITBBSED_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_ESTIMATERADIALVELOCITY_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_FITMODELS_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: VerifyWarning: Keyword name 'MP_MAKEREFERENCESPECTRA_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py", line 809, in curve_fit
    transform = cholesky(sigma, lower=True)
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/linalg/_decomp_cholesky.py", line 88, in cholesky
    c, lower = _cholesky(a, lower=lower, overwrite_a=overwrite_a, clean=True,
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/linalg/_decomp_cholesky.py", line 37, in _cholesky
    raise LinAlgError("%d-th leading minor of the array is not positive "
numpy.linalg.LinAlgError: 16-th leading minor of the array is not positive definiteThe above exception was the direct cause of the following exception:Traceback (most recent call last):
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 433, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 504, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 174, in runDataRef
    reference = self.run(pfsConfig, merged, mergedLsf)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 275, in run
    radialVelocities = self.getRadialVelocities(pfsConfig, pfsMerged, pfsMergedLsf, bbPdfs)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 423, in getRadialVelocities
    radialVelocities.append(self.estimateRadialVelocity.run(spectrum, modelSpectrum))
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/estimateRadialVelocity.py", line 166, in run
    pfit, pcov = scipy.optimize.curve_fit(
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py", line 811, in curve_fit
    raise ValueError("`sigma` must be positive definite.") from e
ValueError: `sigma` must be positive definite.
"""The above exception was the direct cause of the following exception:Traceback (most recent call last):
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/bin/fitPfsFluxReference.py", line 3, in <module>
    FitPfsFluxReferenceTask.parseAndRun()
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 688, in parseAndRun
    resultList = taskRunner.run(parsedCmd)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 244, in run
    resultList = list(mapFunc(self, targetList))
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 47, in _runPool
    return pool.map_async(function, iterable).get(timeout)
  File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
ValueError: `sigma` must be positive definite.

Note that I specified like `--id visit=83219 spectrograph=1`.

 Comments   
Comment by Kiyoto Yabe [ 18/Jan/23 ]

FWIW, there is a strange FLUXSTD which may be affected by the fiberProfile issued we discussed in the phonecon.

Comment by sogo.mineo [ 18/Jan/23 ]

Please try --config estimateRadialVelocity.useCovar=False.

The exception was raised by EstimateRadialVelocityTask at the final step of its work. It wanted to find in a fractional accuracy the center of the highest peak in the cross correlation function. It tried to fit a Gaussian to the peak, and failed. I guess the reason is that the peak was fatter than I assumed it to be. Such a situation makes the covariance matrix nearly singular. Using the entire covariance in fitting is extremely more fragile than using its diagonal elements only.

I gave the fitter not only the diagonal errors but also the entire covariance matrix because the error estimate of the peak position (that is, the error of the radial velocity) that was output by the fitter was correct only when I gave the entire covariance matrix. Because it was planned to use the velocity error later, I had to give the entire covariance matrix. Since the velocity error is not used now, you can safely set --config estimateRadialVelocity.useCovar=False.

Comment by Kiyoto Yabe [ 19/Jan/23 ]

Thank you! With that config, the process was successfully completed. So, are we going to remove that option (or set False as default) in future?

Comment by sogo.mineo [ 19/Jan/23 ]

Yamashita-san agreed. I further asked him whether I can change the Gaussian fit to quadratic interpolation, which is more robust. He agreed to it, too. I will implement it in this ticket.

Comment by sogo.mineo [ 24/Jan/23 ]

Could you review this PR?

Comment by sogo.mineo [ 25/Jan/23 ]

Merged. Thank you for the review.

Generated at Sat Feb 10 16:03:23 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.