This ticket should be blocked on another ticket to upgrade the LSST base stack.

Because a ready Gen3 middleware is not yet available, and Gen2 will be supported for at least another year, I suggest we needn't think about this ticket for another six months. The right time to think about this will be when LSST start using Gen3 for their fortnightly RC2 runs.

Thought: the Gen3 butler doesn't care about filename templates. We're going to need a facility to extract files from the butler and write them with the correct filename template for distribution.

I've got the entire calib and science pipelines working under Gen3. There is more work that needs to be done, but having this merged will allow us to work on the below tasks one by one:
1. Add scripts to export products from the Gen3 datastore.
2. Check quality of Gen3 reduction compared to Gen2.
3. Add Gen3 support to bootstrapDetectorMap and new flux calibration tasks from NAOJ.
4. Check that we can run real data with Gen3.
5. Write Gen3 PFS user guide.
6. Port weekly to Gen3.
7. Set up shared Gen3 datastore with Postgresql registry.
8. Regenerate Subaru calibs with Gen3.
9. Investigate bash completion for "butler" and "pipetask" commands.
10. Investigate parsl plugin for processing on cluster.

This effectively replaces PIPE2D-598: there's little point in merging that one if we're going to merge this one soon after, as it requires another change to the base LSST stack, from 23.0.0 to w_2022_17.

Reviewed pull requests related to this ticket. Changes look fine.

Also looked at the pfsMerged output from an example SuNSS visit (71280) processed with the tickets\PIPE2D-506 branches and with a recent weekly w.2022.18. A 'spectral image' (generated using the showAllSpectraAsImage utility but setting the flux as the difference of the flux values between the two processing runs is shown in figures and In general the difference is close to zero, with the difference at the blue end being noiser. A strange feature around 502nm for fiberIds around 338 is seen. As suggested by Paul, and agreed, this should not stop the merging of this ticket. Future work on such anomalies can be addressed as part of PIPE2D-1031.

I'd have expected the results to be equal to close to machine precision. This needs to be investigated, but doing so as part of PIPE2D-1031 is fine.

My first guess is that there are configuration parameters that differ between the pipelines. This is exactly what PIPE2D-1031 was created for. We will not drop Gen2 support until we are convinced the Gen3 pipeline is performing at the level of Gen2.

Merging this work soon is important, since there are a bunch of changes that I don't want to conflict with other work. On the other hand, we don't want to merge such a large change right before the observing run on the off chance that it causes problems. I propose to merge immediately after the observing run, and in the mean time base other non-urgent work off of this ticket branch.

Merged.