https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-778 DRP 2-D Pipeline <p>This ticket is to investigate Problem nr. 2 described in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-777" title="Difference in centering between pipeline and 2d algorithm - centering and normalization" class="issue-link" data-issue-key="PIPE2D-777"><del>PIPE2D-777</del></a> i.e., that the 2d model generated by the pipeline based on Zernike modeling is different than the results directly generated my modeling algorithm. This is even after recentering them to have the same centers of light. I presume that this is due to differences in interpolation/downsampling details. See <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-777" title="Difference in centering between pipeline and 2d algorithm - centering and normalization" class="issue-link" data-issue-key="PIPE2D-777"><del>PIPE2D-777</del></a> for figures and more details. </p> PIPE2D-778

Difference between pipeline and 2d algorithm - interpolation/downsampling

Task Normal Done Done ncaplar ncaplar Fri, 12 Mar 2021 13:32:50 +0000 Wed, 28 Jul 2021 14:35:41 +0000 Sat, 26 Jun 2021 00:49:08 +0000 0 2 <p>This is somewhat tricky as Lancszos interpolation used in the pipeline is buried quite deep and within C layer of the code. On the other hand, I have not been able to find  readily available  Lancszos interpolation codes in python that would reliably reproduce the results of the pipeline. I have found an implementation in torch<img class="emoticon" src="https://pfspipe.ipmu.jp/jira/images/icons/emoticons/warning.png" height="16" width="16" align="absmiddle" alt="" border="0"/> which I could strip and modify, but it is not obvious it is worth the time and effort given the relatively small difference between currently used bilinear interpolation in my code and Lancszos code - and the fact that I will probably need to refactor my code to be LSST code based at some point in the future. I have to consider this a little bit further. </p> <p>It should be trivial to implement exactly the same Lanczos kernel in python (although I'm surprised that it's not in scipy).  I can do that if needs be.</p> <p>Indeed, I have been able to implement it on my own (see small example of shift with maaaany wiggles). I searched quite extensively and was not able to identify Lanszscos (capable to going to 5) implementation that was readily usable (for example, no mention in scipy <a href="https://docs.scipy.org/doc/scipy/reference/search.html?q=lanczos&amp;check_keywords=yes&amp;area=default" class="external-link" rel="nofollow">https://docs.scipy.org/doc/scipy/reference/search.html?q=lanczos&amp;check_keywords=yes&amp;area=default</a> , only Lanczos fitting algorithm / one could construct it with scipy.special.sinc). I am just expressing some worry rewriting everything with my implementation which might be prone to errors and the amount of time that might take given other priorities.</p> <p>After discussion with Robert on Slack, I have made slight modifications to the algorithm so that it can fully ran under LSST imports. There were some minor changes, which probably arise from slight differences in python version and other imports, but which do not change the final image. The only larger difference is that I was using <em>scikit-image</em> libarary for one resizing operation, which was trivially replaced.<br/> I have not yet tested if I could run the fitting algorithm also under LSST code (the algorithm which searches for best parameters of the fit), which may have other dependencies not in LSST at the moment (most likely there might be some problems in the way how I do multiprocesing?). I would have to check that.</p> <p>For the same test image in the focus, running on tiger-sumire, I am getting:</p> <p>without LSST stack: `840 ms ± 3.69 ms per loop(mean ± std. dev. of 7 runs, 1 loop each)`<br/> with LSST stack: `1.29 s ± 106 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)`</p> <p>I ran %timeit many times and the result are roughly the same each time. I am confused my much lower variance in `without LSST' result. This result suggests that the slow down is around 50% for in-focus images. I have to investigate where the slowdown is coming from (i.e., is it from FFT calculation and using/not using of MKL libraries)</p> <p>After some consideration, and talking with Josh Meyers, and looking how the pipeline code is set up, I think it is easiest and most convenient to use galsim.InterpolatedImage functionality. This makes it possible to use Lanczos5 as an interpolant and plays nicely with other functionality that I have + the pipeline as well. I have tested that the shift of 0.5 pixel with the pipeline function on an oversampled image in focus (with fuction lsst.afw.math import offsetImage, which is invoked in the pipeline) creates exactly the same change as when using galsim. This is shown in comparison_galsim_pipeline.png</p> <p>That sounds fine, but won't you still have to integrate with the "real" Rubin pipeline soon?</p> <p>Yes, indeed. I think we need to talk so that I can understand how this will be achieved, i.e., how will the psf object be constructed. But note, as you are aware, that galsim is LSST package. And I do need some functionality to do fitting of defocused and focused images which I dont think it would be easily run through ``real'' pipeline - but in this way it will be using the same algorithms as the ``real'' pipeline.</p> <p>I have incorporated galsim Lanczos5 procedures at all relevant places in my module, internal version 0.47a. Closing this ticket, and evaluating in the next release of the solutions. </p> Relates PIPE2D-874 PIPE2D-408 Development Rank 0|zzs4or:9 Sprint 2DDRP-2021 A 4 2DDRP-2021 A5 2DDRP-2021 A 6 Story Points 4.0