Short update - I have semi-implemented the algorithm described here (https://arxiv.org/abs/astro-ph/0606388) and used by A. Roodman with Dark Eenrgy Camera (http://spie.org/Publications/Proceedings/Paper/10.1117/12.857680). In the plot MoreZernike.png I am showing 1. at the left, starting proposal going up to zernike 22, 2. in the middle, adding 56 more Zernike components, 3. on the right, how would perfect subtractions look like.

What has not yet been done:
1. The algorithm only varies higher order Zernike, i.e., keeps the component up to z22 fixed, and varies higher order. I would need to implement possibility to vary all the orders.
2. The code tries to minimize the difference between the model and the science image in flux, not in chi**2.
3. The code is not integrated with the rest of the fitting routine.
This code to determine wavefront is also very fast, so the whole fitting routine would probably stand to benefit from it.

After discussion with Robert during the Monday meeting and after the plan is:
1. Use this newly developed method to fit all the residuals. Involves some work in extending the code so that it can run in parallel on the cluster.
2. Do PCA analysis on the residuals. This is to be done on the vectors containing Zernike coefficients which best describe the improvement in the residual images.
3. Use this prediction to predict high-order corrections when moving into focus

I have implemented a version of this code. With the full understanding that is not particularly transparent, the current code is at https://github.com/nevencaplar/PFS_Work_In_Progress/blob/master/Direct_and_Interpolation_Analysis_Extra_Zernike_Mar2019.py
The second part, basically outlined in the March 11 comment above has not been yet implemented, as we started to worry more about wrong characterization of the lower order Zernike (PIPE2D-393).

hassan I recommend that we close this ticket on this level (shows that it improves final results and we likely have to considered it in the future) and resurrect it in the future as one of the improvements to the PSF solution code.

ncaplar ok, but can you create a new ticket to track the extra work you proposed in your 11 March comment please?

Closed following ncaplar recent comment - additional tickets will be created to track further work.