pfsCoadd expands when processed with many visits

(lsst-scipipe-9.0.0) [pfs@pfsa-an01-gb run21]$ pipetask --long-log --log-level PFS=INFO run -b /work/datastore -i u/kiyoyabe/processing/run21/20250415a -o u/kiyoyabe/processing/run21/coadd.20250415a -p '$DRP_STELLA_DIR/pipelines/science.yaml#coaddSpectra' -d "combination = 'ssp/run21/co.20250415a' AND cat_id = 10091" --fail-fast -j 12
INFO 2025-04-17T21:38:13.445-10:00 lsst.pipe.base.quantum_graph_builder ()(quantum_graph_builder.py:344) - Processing pipeline subgraph 1 of 1 with 1 task(s).
INFO 2025-04-17T21:38:13.752-10:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:163) - Iterating over query results to associate quanta with datasets.
INFO 2025-04-17T21:38:13.854-10:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:205) - Initial bipartite graph has 1 quanta, 1042 dataset nodes, and 1040 edges from 336 query row(s).
INFO 2025-04-17T21:38:14.552-10:00 lsst.pipe.base.quantum_graph_builder ()(quantum_graph_builder.py:572) - Generated 1 quantum for task coaddSpectra.
INFO 2025-04-17T21:39:06.534-10:00 lsst.ctrl.mpexec.cmdLineFwk ()(cmdLineFwk.py:909) - QuantumGraph contains 1 quantum for 1 task, graph ID: '1744961946.5184138-4659'
Quanta    Tasks
------ ------------
     1 coaddSpectra
INFO 2025-04-17T21:39:14.749-10:00 lsst.ctrl.mpexec.singleQuantumExecutor (coaddSpectra:{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'})(singleQuantumExecutor.py:207) - Preparing execution of quantum for label=coaddSpectra dataId={instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'}.
INFO 2025-04-17T21:39:14.987-10:00 lsst.ctrl.mpexec.singleQuantumExecutor (coaddSpectra:{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'})(singleQuantumExecutor.py:261) - Constructing task and executing quantum for label=coaddSpectra dataId={instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'}.
ERROR 2025-04-17T22:20:00.384-10:00 timer.lsst.daf.butler.datastores.fileDatastore (coaddSpectra:{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'})(fileDatastore.py:1331) - Writing dataset pfsCoadd@{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'} [sc=PfsObjectSpectra] (run=u/kiyoyabe/processing/run21/coadd.20250415a/20250418T073748Z id=1f164cec-ba06-43ec-894e-30a756ce6e6e) with formatter lsst.obs.pfs.formatters.PfsObjectSpectraFormatter: Took 96.0203 seconds
ERROR 2025-04-17T22:20:00.386-10:00 lsst.ctrl.mpexec.singleQuantumExecutor (coaddSpectra:{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'})(singleQuantumExecutor.py:271) - Execution of task 'coaddSpectra' on quantum {instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'} failed. Exception RuntimeError: Failed to serialize dataset pfsCoadd@{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'} [sc=PfsObjectSpectra] (run=u/kiyoyabe/processing/run21/coadd.20250415a/20250418T073748Z id=1f164cec-ba06-43ec-894e-30a756ce6e6e) of type pfs.drp.stella.datamodel.pfsTargetSpectra.PfsObjectSpectra using formatter lsst.obs.pfs.formatters.PfsObjectSpectraFormatter.
Process task-{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'}:
Traceback (most recent call last):
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/daf_butler/g22d49dda5f+264564050d/python/lsst/daf/butler/datastores/fileDatastore.py", line 1333, in _write_in_memory_to_artifact
    formatter_compat.write(inMemoryDataset, cache_manager=self.cacheManager)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/daf_butler/g22d49dda5f+264564050d/python/lsst/daf/butler/_formatter.py", line 900, in write
    self.write_locally_then_move(in_memory_dataset, uri, cache_manager)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/daf_butler/g22d49dda5f+264564050d/python/lsst/daf/butler/_formatter.py", line 1014, in write_locally_then_move
    self.write_local_file(in_memory_dataset, temporary_uri)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/obs_base/g0f054406a4+bf72739bfa/python/lsst/obs/base/formatters/fitsGeneric.py", line 52, in write_local_file
    in_memory_dataset.writeFits(uri.ospath)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/datamodel/w.2025.15/python/pfs/datamodel/pfsTargetSpectra.py", line 478, in writeFits
    fits.writeto(fd)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1042, in writeto
    hdu._prewriteto(checksum=checksum)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/astropy/io/fits/hdu/table.py", line 510, in _prewriteto
    self.data._scale_back(update_heap_pointers=not self._manages_own_heap)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/astropy/io/fits/fitsrec.py", line 1121, in _scale_back
    raise ValueError(
ValueError: The heapsize limit for 'P' format has been reached. Please consider using the 'Q' format for your file.
Failed to serialize dataset pfsCoadd@{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'} [sc=PfsObjectSpectra] (run=u/kiyoyabe/processing/run21/coadd.20250415a/20250418T073748Z id=1f164cec-ba06-43ec-894e-30a756ce6e6e) of type pfs.drp.stella.datamodel.pfsTargetSpectra.PfsObjectSpectra to temporary location file:///work/datastore/u/kiyoyabe/processing/run21/coadd.20250415a/20250418T073748Z/pfsCoadd/rmjlffc2cztgmrxq.fits.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 184, in _executeJob
    _, report = quantumExecutor.execute(task_node, quantum)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 177, in execute
    result = self._execute(task_node, quantum)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 269, in _execute
    self.runQuantum(task, quantum, task_node, limited_butler)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 484, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/lfs/work/pfs/processing/run21/src/drp_stella/python/pfs/drp/stella/coaddSpectra.py", line 234, in runQuantum
    butler.put(PfsCoadd(outputs.pfsCoadd.values()), outputRefs.pfsCoadd)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/pipe_base/g01c6efbd85+e8cc187249/python/lsst/pipe/base/_quantumContext.py", line 392, in put
    self._put(values, dataset)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/pipe_base/g01c6efbd85+e8cc187249/python/lsst/pipe/base/_quantumContext.py", line 225, in _put
    self.__butler.put(value, ref)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/daf_butler/g22d49dda5f+264564050d/python/lsst/daf/butler/utils.py", line 71, in inner
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/daf_butler/g22d49dda5f+264564050d/python/lsst/daf/butler/direct_butler/_direct_butler.py", line 998, in put
    self._datastore.put(obj, datasetRefOrType)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/daf_butler/g22d49dda5f+264564050d/python/lsst/daf/butler/utils.py", line 71, in inner
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/daf_butler/g22d49dda5f+264564050d/python/lsst/daf/butler/datastores/fileDatastore.py", line 2130, in put
    storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/daf_butler/g22d49dda5f+264564050d/python/lsst/daf/butler/datastores/fileDatastore.py", line 1335, in _write_in_memory_to_artifact
    raise RuntimeError(
RuntimeError: Failed to serialize dataset pfsCoadd@{instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'} [sc=PfsObjectSpectra] (run=u/kiyoyabe/processing/run21/coadd.20250415a/20250418T073748Z id=1f164cec-ba06-43ec-894e-30a756ce6e6e) of type pfs.drp.stella.datamodel.pfsTargetSpectra.PfsObjectSpectra using formatter lsst.obs.pfs.formatters.PfsObjectSpectraFormatter.
ERROR 2025-04-17T22:20:02.716-10:00 lsst.daf.butler.cli.utils ()(utils.py:202) - Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 233, in run
    script.run(qgraphObj=qgraph, **kwargs)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/cli/script/run.py", line 239, in run
    f.runPipeline(qgraphObj, taskFactory, args)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 859, in runPipeline
    executor.execute(graph)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 456, in execute
    self._executeQuantaMP(graph, self.report)
  File "/work/stack-2025-02-12/conda/envs/lsst-scipipe-9.0.0/share/eups/Linux64/ctrl_mpexec/g486af8b45a+1d2cae42c0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 684, in _executeQuantaMP
    raise MPGraphExecutorError(message)
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: Task <coaddSpectra dataId={instrument: 'PFS', cat_id: 10091, combination: 'ssp/run21/co.20250415a'}> failed, exit code=1