[PIPE2D-1362] correct for the focal plane variation of the flux calibration vector Created: 31/Jan/24  Updated: 15/Feb/24  Resolved: 15/Feb/24

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: Masayuki Tanaka Assignee: sogo.mineo
Resolution: Done Votes: 0
Labels: EDR, EngRun
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File fig1_3rd_order.png     PNG File fig1_4th_order.png     PNG File fig1_nocorr.png     PNG File fig2_3rd_order.png     PNG File fig2_4th_order.png     PNG File fig2_nocorr.png    
Issue Links:
Relates
relates to PIPE2D-1361 fitFluxCal.py: calibration vectors ar... Done
Epic Link: flux calibration
Reviewers: price

 Description   

After applying the fix for PIPD2D-1361, we clearly observe the spatial variation of the flux calibration vector. The variation likely has two components; the global pattern due to the global offset, rotation, and scale difference, and fiber-to-fiber offset. We want to correct for the former with, e.g., low-order polinomials. As the seeing is a function of wavelength, a fiber offset introduces a wavelength-dependent flux loss. We thus need to allow the flux calibration vector to vary both spatially and along the wavelength. The correction can be estimated by comparing the PS1 magnitudes and synthetic PS1 magnitudes of each FLUXSTD.

 Comments   
Comment by Masayuki Tanaka [ 31/Jan/24 ]

The figures here show the magnitude and color offsets with respect to the PS1 photometry. The first figure is the i-band magnitude offset and the 2nd one is g-y color offset. Note the similar spatial patterns. This is due to the wavelength dependence of the seeing.

Comment by rhl [ 31/Jan/24 ]

We have to think about how much the fibre-to-fibre variation is corrected by the fibre profiles (and associated normalisation). That's done using a roughly uniform surface brightness, but it should take out the fibre throughput variation – although it's hard to separate this from spatial variation.

Comment by Masayuki Tanaka [ 31/Jan/24 ]

I should have been clearer. What I meant by the fiber-to-fiber variation is the residual fiber offset after we take out the global offset/pattern. I imagine the residual offset is nearly random (but I may be wrong). I agree that the throughput variation between the fibers will be largely taken out by the quartz.

sogo.mineo already implemented a correction function in the flux calibration code (I should have filed this ticket when he started, sorry). He can elaborate on the algorithmic details, but here are the plots after the correction is applied. The first set of the figures is for 3rd order correction and the 2nd set for 4th order correction.

4th order:

 

Comment by sogo.mineo [ 31/Jan/24 ]

The details of fitting FluxCalib (a subclass of FocalPlaneFunction) is:

1. Divide observed spectra of flux standards by reference spectra to get proto-fluxCalib vectors.
2. Adjust heights of the proto-fluxCalib vectors to each other.
3. Take their average to get h(lam) (ConstantFocalPlaneFunction)
4. Let g(x, y, lam) be a trivariate polynomial (of order 3 by default).
5. Let f(x, y, lam) = h(lam) exp(g(x, y, lam)). This is a FluxCalib.
6. Let (calibrated spectrum) = (observed spectrum) / f(x[i], y[i], lam) for each flux standard.
7. Integrate calibrated spectra to get synthetic broadband fluxes.
8. Let chisq be the sum of ((broadband flux in pfsConfig) - (synthetic broadband flux))^2 / error^2
9. Minimize chisq.

So, I distort the average fluxCalib h(lam) to forcibly make it conform to pfsConfig.psfFlux.
Though I am not sure the resulting fluxCalib is good for spectroscopy, Tanaka-san's images show that the distortion is working well.

Comment by sogo.mineo [ 31/Jan/24 ]

Could you review PRs of datamodel and drp_stella, too?

Comment by sogo.mineo [ 08/Feb/24 ]

The PRs have been approved, but I just found that the integration test would break during coaddSpectra.py because an instance of FluxCalib class (named "fluxCal" in butler) cannot be created. The reason for this is that FluxCalib is defined outside focalPlaneFunction.py even though it is a subclass of FocalPlaneFunction.

I decided to define FluxCalib in fitFluxCal.py because its overrided method fitArrays() has a parameter of type BroadbandFluxChi2. This class is not general enough to be known to focalPlaneFunction.py.

This problem is solved if I revise PfsMapper.yaml in obs_pfs so that the type of "fluxCal" will be FluxCalib. But if I do so, the output of fluxCalibrate.py will not be able to be read by coaddSpectra.py,

Does anyone have an idea to solve this problem?

Comment by sogo.mineo [ 08/Feb/24 ]

I pushed another commit. I moved FluxCalib class to focalPlaneFunction.py. Its method fitArrays() takes a parameter fitter, and all that fitArrays() does is to call fitter(). After this commit, the Gen2 integration test passes.

Gen3 test still fails:

lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 24 quanta successfully, 0 failed and 3 remain out of total 27 quanta.
py.warnings WARNING: /data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/utils/polynomialND.py:76: RuntimeWarning: divide by zero encountered in true_divide
  self._scale = 1.0 / (self._max - self._min)

py.warnings WARNING: /data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/utils/polynomialND.py:130: RuntimeWarning: invalid value encountered in multiply
  x *= scale

lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'fitFluxCal' on quantum {instrument: 'PFS-F', exposure: 24, ...} failed. Exception RuntimeError: No good points
Process task-{instrument: 'PFS-F', exposure: 24, ...}:
Traceback (most recent call last):
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 143, in _executeJob
    quantumExecutor.execute(taskDef, quantum, butler)
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 135, in execute
    result = self._execute(taskDef, quantum, butler)
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 212, in _execute
    self.runQuantum(runTask, quantum, taskDef, butler)
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 580, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/fitFluxCal.py", line 761, in runQuantum
    outputs = self.run(**inputs, pfsArmList=armInputs.pfsArm, sky1dList=armInputs.sky1d)
  File "/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/fitFluxCal.py", line 703, in run
    fluxCal = self.calculateCalibrations(pfsConfig, pfsMerged, pfsMergedLsf, references)
  File "/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/fitFluxCal.py", line 900, in calculateCalibrations
    return self.fitFocalPlane.run(
  File "/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/fitFocalPlane.py", line 109, in run
    raise RuntimeError("No good points")
RuntimeError: No good points
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 130, in run
    script.run(qgraphObj=qgraph, **kwargs)
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/cli/script/run.py", line 187, in run
    f.runPipeline(qgraphObj, taskFactory, args)
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 740, in runPipeline
    executor.execute(graph, butler)
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 373, in execute
    self._executeQuantaMP(graph, butler)
  File "/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/ctrl_mpexec/gb02ad94e9c+e480a1db32/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 568, in _executeQuantaMP
    raise MPGraphExecutorError(message)
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: Task <TaskDef(pfs.drp.stella.fitFluxCal.FitFluxCalTask, label=fitFluxCal) dataId={instrument: 'PFS-F', exposure: 24, ...}> failed, exit code=1

Is this error the one that price told me a few weeks ago?

Comment by price [ 08/Feb/24 ]

Yes, I think that's the same one. It would be great if you could also fix that.

Comment by sogo.mineo [ 13/Feb/24 ]

Interim report: I ran the integration test with the master branch and tickets/PIPE2D-1362 branch of drp_stella, and compared the products. The products started diverging from each other at "sky1d" or "pfsMerged". But every time I ran the integration test with the master branch of drp_stella, different "sky1d" and "pfsMerged" were output (difference of FITS headers are ignored). So, the variation of "sky1d" and "pfsMerged" does not seem to be the cause of the error.

Comment by sogo.mineo [ 14/Feb/24 ]

The problem was that the integration test contains too few flux standards to determine 3rd-order trivariate polynomial. I made a small change to pfs_pipe2d (so that the polynomial order would be 0), and the integration test passed. Could you price review the pull request to pfs_pipe2d?

Comment by sogo.mineo [ 15/Feb/24 ]

Three branches have been merged. Thanks for reviewing.

Generated at Thu Apr 10 19:52:23 JST 2025 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.