Also seen in 97546 with the PIPE2D-1271 fixes in.

fiberId=84,85,86 are known to have bad profiles. I believe those cobras are broken so that they cannot be hidden, which means we cannot measure accurate profiles for them. One thing we could do is replace the profile of bad fibers with the average profile.

arnaud.lefur, could we please have these fibers (and others like them) flagged suitably in the pfsConfig? Do we need an extra FiberStatus setting?

We can surely do better than that? We know that the fibres are broken, so surely we can do no-worse than non-EO traces.

The "non-even/odd traces" have the profiles measured completely independently, and so are subject to contamination by nearby fibers. Because of this, we usually limit it to +/- 2 pixels, compared to the +/- 10 pixels for the simultaneous measurement. I think the above profiles look fairly consistent within the +/- 2 pixels limit, but the model can't tell the difference between the three neighboring fibers that can't be hidden, so you see the other bad fibers as well. So how do you want to fix this? Here are some options:
1. Replace the bad fibers with the average of good nearby fibers. I've implemented this already, and the result (below) looks reasonable. I suggest this could be a suitable interim solution.
2. Replace the bad fibers with the non-simultaneous profile measurement, limited to +/- 2 pixels.
3. Attempt to do something more sophisticated in the simultaneous measurement, like measuring all the profiles relative to the average profile. I like this as a long-term solution.
4. Treat the bad fibers differently in the simultaneous fit, maybe replacing them with the average of good nearby fibers. The difference between this and (1) above is that this replacement is done during the fit rather than after the fact.

Replacing bad fibers with the average of good nearby fibers:

I think I like the first solution too.

One sounds OK for now. I'd have to think if we can do anything better, but there's probably no need as these will always be mostly-sky fibres. We do need to be able to know when broken cobras fall on bright objects – something that e.g. Masayuki Tanaka might want to think about.

However, I'm still not happy with the explanation of the original problem. The extractions should mix the spectra from the three bad fibres, but they shouldn't go negative. The problem is apparently that we don't force the fibre profiles to be non-negative; we absolutely need to do this. It may not play well with the scattered light that we're seeing, but we need to fix that too.

Filed PIPE2D-1273 ("Introduce non-negativity constraint in fiber profile measurement") to tackle once we get the background under control.

How do we want to get this into the Official Calibs? It's not all that critical now you've diagnosed the problem. I've filed DAMD-151 to address the book-keeping problems.

I've already copied the updated profiles into CALIB-2023-07-v3.

Merged changes to master.