Uploaded image for project: 'DRP 2-D Pipeline'
  1. DRP 2-D Pipeline
  2. PIPE2D-1147

flux calibration fails for m-band data

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Done (View Workflow)
    • Priority: Normal
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reducing data set of GA targets with m-band, flux calibration fails in the middle with the error message below:

      INFO  2023-01-16T16:39:45.573-1000 fitPfsFluxReference ({'visit': 83219, 'dateObs': '2022-11-21'})(fitPfsFluxReference.py:290)- Fitting models to spectra (takes some time)...
      INFO  2023-01-16T17:18:37.548-1000 fitPfsFluxReference ({'visit': 83219, 'dateObs': '2022-11-21'})(fitPfsFluxReference.py:322)- Making reference spectra by interpolation
      WARN  2023-01-16T17:19:56.388-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_ABSENT_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
      WARN  2023-01-16T17:19:56.390-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_BAD_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
      WARN  2023-01-16T17:19:56.390-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_DEFICIENT_BBFLUXES' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
      WARN  2023-01-16T17:19:56.391-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_FITBBSED_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
      WARN  2023-01-16T17:19:56.392-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_ESTIMATERADIALVELOCITY_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
      WARN  2023-01-16T17:19:56.393-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_FITMODELS_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
      WARN  2023-01-16T17:19:56.394-1000 astropy ({'visit': 83219, 'dateObs': '2022-11-21'})(logger.py:211)- VerifyWarning: Keyword name 'MP_MAKEREFERENCESPECTRA_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created.
      /work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitReference.py:81: IntegrationWarning: The maximum number of subdivisions (100) has been achieved.
        If increasing the limit yields no improvement it is advised to analyze
        the integrand in order to determine the difficulties.  If the position of a
        local difficulty can be determined (singularity, discontinuity) one will
        probably gain from splitting up the interval and calling the integrator
        on the subranges.  Perhaps a special-purpose integrator should be used.
        return scipy.integrate.quad(function, self.wavelength[0], self.wavelength[-1],
      /work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitReference.py:81: IntegrationWarning: The maximum number of subdivisions (100) has been achieved.
        If increasing the limit yields no improvement it is advised to analyze
        the integrand in order to determine the difficulties.  If the position of a
        local difficulty can be determined (singularity, discontinuity) one will
        probably gain from splitting up the interval and calling the integrator
        on the subranges.  Perhaps a special-purpose integrator should be used.
        return scipy.integrate.quad(function, self.wavelength[0], self.wavelength[-1],
      WARNING: VerifyWarning: Keyword name 'MP_ABSENT_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
      WARNING: VerifyWarning: Keyword name 'MP_BAD_FIBER' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
      WARNING: VerifyWarning: Keyword name 'MP_DEFICIENT_BBFLUXES' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
      WARNING: VerifyWarning: Keyword name 'MP_FITBBSED_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
      WARNING: VerifyWarning: Keyword name 'MP_ESTIMATERADIALVELOCITY_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
      WARNING: VerifyWarning: Keyword name 'MP_FITMODELS_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
      WARNING: VerifyWarning: Keyword name 'MP_MAKEREFERENCESPECTRA_FAILED' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
      multiprocessing.pool.RemoteTraceback:
      """
      Traceback (most recent call last):
        File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py", line 809, in curve_fit
          transform = cholesky(sigma, lower=True)
        File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/linalg/_decomp_cholesky.py", line 88, in cholesky
          c, lower = _cholesky(a, lower=lower, overwrite_a=overwrite_a, clean=True,
        File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/linalg/_decomp_cholesky.py", line 37, in _cholesky
          raise LinAlgError("%d-th leading minor of the array is not positive "
      numpy.linalg.LinAlgError: 16-th leading minor of the array is not positive definiteThe above exception was the direct cause of the following exception:Traceback (most recent call last):
        File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 125, in worker
          result = (True, func(*args, **kwds))
        File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
          return list(map(*args))
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 433, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 504, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 174, in runDataRef
          reference = self.run(pfsConfig, merged, mergedLsf)
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 275, in run
          radialVelocities = self.getRadialVelocities(pfsConfig, pfsMerged, pfsMergedLsf, bbPdfs)
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/fitPfsFluxReference.py", line 423, in getRadialVelocities
          radialVelocities.append(self.estimateRadialVelocity.run(spectrum, modelSpectrum))
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/python/pfs/drp/stella/estimateRadialVelocity.py", line 166, in run
          pfit, pcov = scipy.optimize.curve_fit(
        File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py", line 811, in curve_fit
          raise ValueError("`sigma` must be positive definite.") from e
      ValueError: `sigma` must be positive definite.
      """The above exception was the direct cause of the following exception:Traceback (most recent call last):
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2023.02/bin/fitPfsFluxReference.py", line 3, in <module>
          FitPfsFluxReferenceTask.parseAndRun()
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 688, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 244, in run
          resultList = list(mapFunc(self, targetList))
        File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 47, in _runPool
          return pool.map_async(function, iterable).get(timeout)
        File "/work/stack_INFRA-312/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py", line 771, in get
          raise self._value
      ValueError: `sigma` must be positive definite.
      

      Note that I specified like `--id visit=83219 spectrograph=1`.

        Attachments

          Activity

            People

            • Assignee:
              sogo.mineo sogo.mineo
              Reporter:
              kiyoto.yabe Kiyoto Yabe
              Reviewers:
              price
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: