Example PSFs from LAM data should be accessible in order to address this ticket.

The example PSFs have been created in PIPE2D-379 and they should be ready to use (I am using then now for my modeling).

Clarification of my earlier comment: Keigo Nakamura needs access to the example PSFs that have been created from LAM data as part of PIPE2D-379, which are situated on the Princeton cluster.

I don't understand this.  The LSST stack that the PFS pipeline is based on has support for `PSFex` built in;  you can start with an arc image and run the PSF modelling code.

When we come to see if piff works better we will have to think about interfaces – LSST is planning to integrate piff at some point, and we can either wait for this work or contribute it;  it may be reasonable to test piff on arc data before doing the integration work.

Key LSST task to be used that makes use of PSFex is CharacterizeImageTask: https://pipelines.lsst.io/modules/lsst.pipe.tasks/tasks/lsst.pipe.tasks.characterizeImage.CharacterizeImageTask.html#lsst-task-lsst-pipe-tasks-characterizeimage-characterizeimagetask

Documentation providing examples are missing. An LSST ticket has been raised to address this:
https://jira.lsstcorp.org/browse/DM-18600

However a useful starting point for using the CharacterizeImageTask is available here: https://github.com/RobertLuptonTheGood/notebooks/blob/master/Demos/Image%20Processing.ipynb

I attached the output image of PSFEX. I tried simulated image of ARC in integration test (visit =31).
At the top region, PSF has some fluctuation.

The very initial results:

Modeled PSF with PSFEx at each position of the detector and spots used for the calculation (small circle; redder color means brighter). Ne arc PostISRCCD image (visit=17244) is used. `config.repair.doCosmicRay = False`  I'm not sure why very bright lines in the lower area of the image are not used by PSFEx, but this may be able to be controlled by tuning parameters. Current set of parameter is like this:

lsst.meas.extensions.psfex.psfexPsfDeterminer.PsfexPsfDeterminerConfig(kernelSize=41.0, kernelSizeMin=31, kernelSizeMax=51, PsfexPsfDeterminerConfignEigenComponents=4, spatialOrder=2, sizeCellX=256, sizeCellY=256, _PsfexPsfDeterminerConfignStarPerCell=3, samplingSize=1.0, badMaskBits=['INTRP', 'SAT'], psfexBasis='PIXEL_AUTO', _PsfexPsfDeterminerConfigborderWidth=0, _PsfexPsfDeterminerConfignStarPerCellSpatialFit=5, _PsfexPsfDeterminerConfigconstantWeight=True, _PsfexPsfDeterminerConfig_nIterForPsf=3, tolerance=0.01, lam=0.05, reducedChi2ForPsfCandidates=2.0, spatialReject=3.0, recentroid=False)

lsst.meas.algorithms.objectSizeStarSelector.ObjectSizeStarSelectorConfig(doFluxLimit=False, fluxMin=12500.0, fluxMax=0.0, doSignalToNoiseLimit=True, signalToNoiseMin=20.0, signalToNoiseMax=0.0, widthMin=0.0, widthMax=20.0, sourceFluxField='base_GaussianFlux_instFlux', widthStdAllowed=0.5, nSigmaClip=3.0, badFlags=['base_PixelFlags_flag_edge', 'base_PixelFlags_flag_interpolatedCenter', 'base_PixelFlags_flag_saturatedCenter', 'base_PixelFlags_flag_crCenter', 'base_PixelFlags_flag_bad', 'base_PixelFlags_flag_interpolated'])

This is a 1D profile of each image along X direction in each location (+/-15 pix stack in Y direction).

This is a 1D profile of each image along Y direction in each location (+/-15 pix stack in X direction).

In each plot, the X location is shown at the top of each sub-panel, and color shows the Y location (bluer means lower Y).

I'm perhaps doing stupid things, and I'm very happy to know a proper set of parameters.

Anyway, I don't see a large change of the line profile in Y direction, but see slight change (1-2 percent level) in the wing of the profile.

This is an example of 2D (top) and 1D (bottom) residual of a fairly bright spot at the center of the image. You see 1-2 percent level of residual (compared to the peak). Note that this is actually a good case, and there is worse case indeed. 

This is a map of RMS of 1D residual divided by the peak of the original line. Most of the spots show 2-3% level of residual, but some spots show >5% level residual. Bad spots in lower part of the image are fairly bright but the positional information of the detection result looks bad for some reason and this causes the large residual.

In plots like arc_ne_17244_psfex_xstack.png what are the units? The y-axis is labelled "counts" – are they in fractions of the peak, or the total flux, or something else?

Also, the PSF images are showing ringing. Did you turn on oversampling, and (if so) are you using the version of psfEx in the LSST that I fixed/hacked?

Yes, it is a normalized value so that the total count is unity. 

samplingSize=1.0, so I think oversampling is OFF.

I'm using ver. 18.1.0 for meas_extensions_psfex but I'm not sure if this is a version you fixed. I actually asked naoki.yasuda to install the package on a server at IPMU, but some steps described in README (such as checking out your subversion repo) may have been skipped.

It has been installed with `eups distrib install meas_extensions_psfex v18_1_0` on top of installed LSST stack for pfs_pipe.

The data is undersampled (as is normal for fibre spectrographs), so I think you need to turn on oversampling.

As reported in previous telecon, turning on oversampling causes a strange result as shown below:

This is an image of the local PSF modeling near the FoV center for 5x oversampling. In the processing, 192 out of 237 PSF candidate objects (arc spots) are used for the modeling.

A set of configuration is shown below:

// code placeholder

config = CharacterizeImageTask.ConfigClass()
config.psfIterations = 1
config.useSimplePsf = False
config.repair.doCosmicRay = False
config.doApCorr = False

config.measurePsf.psfDeterminer.name = 'psfex'
config.measurePsf.psfDeterminer['psfex'].badMaskBits=['INTRP', 'SAT']
config.measurePsf.psfDeterminer['psfex'].kernelSize=205
config.measurePsf.psfDeterminer['psfex'].kernelSizeMin=31
config.measurePsf.psfDeterminer['psfex'].kernelSizeMax=51
config.measurePsf.psfDeterminer['psfex'].sizeCellX=64
config.measurePsf.psfDeterminer['psfex'].sizeCellY=64
config.measurePsf.psfDeterminer['psfex'].samplingSize=0.2
config.measurePsf.psfDeterminer['psfex'].psfexBasis='PIXEL'
config.measurePsf.psfDeterminer['psfex'].recentroid=True
config.measurePsf.psfDeterminer['psfex'].lam=0.05
config.measurePsf.psfDeterminer['psfex'].spatialOrder=2
config.measurePsf.psfDeterminer['psfex'].spatialReject=3.0
config.measurePsf.psfDeterminer['psfex'].tolerance=0.01
config.measurePsf.psfDeterminer['psfex'].reducedChi2ForPsfCandidates=2.0

config.measurePsf.starSelector.name = 'objectSize'
config.measurePsf.starSelector['objectSize'].doFluxLimit=False
config.measurePsf.starSelector['objectSize'].fluxMin=1000.
config.measurePsf.starSelector['objectSize'].fluxMax=5000000.
config.measurePsf.starSelector['objectSize'].doSignalToNoiseLimit=True
config.measurePsf.starSelector['objectSize'].signalToNoiseMin=50.
config.measurePsf.starSelector['objectSize'].signalToNoiseMax=1000.
config.measurePsf.starSelector['objectSize'].widthMin = 0.5
config.measurePsf.starSelector['objectSize'].widthMax = 5.0
config.measurePsf.starSelector['objectSize'].widthStdAllowed = 0.3
config.measurePsf.starSelector['objectSize'].nSigmaClip = 2.0
config.measurePsf.starSelector['objectSize'].sourceFluxField = 'base_PsfFlux_instFlux'

Please let me know, if you have any idea on the possible cause or what can be helpful...

I have talked with Bob Amstrong. He immediately recognized the strange feature discussed above. He says that the feature is caused by the fact that the code only does the real analysis in the inner part of the PSF, while outside it does some sort of simplified averaging. The parameter that controls how large the region is in which ``real analysis'' is conducted is given by the parameter BASIS_NUMBER which is not accessible via python, but is accessible via the default configuration file. On GitHub, this file can be found at

On Monday I plan to play to see if the changing the BASIS_NUMBER changes the results 

I did conduct the change that Bob suggested and increased the BASIS_NUMBER from 20 to 40. The code ran 10 times longer (4517 seconds, compared to 467 when BASIS_NUMBER was 20), but I do get a similar result:

Thank you very much for this test ncaplar. I do appreciate if you get a further chance to talk to Bob or other folks. 

Kiyoto Yabe Yes, I am trying to get in contact with Bob to discuss further.

I have talked with Bob Amstrong about our problem when using PSFex. To repeat what I mentioned in the ticket, he says that the feature is caused by the fact that the code only does the real analysis in the inner part of the PSF, while outside it does some sort of simplified averaging. He first recommended increasing the BASIS_NUMBER parameter, but that did not change the result (only made code run longer).
He also shared with the Figure showing the HSC results when using third order lanczos and having different parameters of interpolation (`INTERPW` and `INTERPFAC`) then what we are using now. We see the same effect. Based on these we seem to have concluded that changing the interpolation properties via parameters `INTERPW` or `INTERPFAC` does not change the size of the interpolation box.

We discussed this during DRP telecom on March 27, 2020. We have reached the conclusion that at this point it is not worth investigating this further. Changing the size of the box would probably include considerable amounts of time and possibly working on the underlying C-code. 
If it turns out that this code is needed (in the case that my 2d PSF modeling is unsatisfactory), we plan to either try using Piff or return to the work in this ticket.