PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 [PIPE2D-355] Model the 2D PSF using PSFEx https://pfspipe.ipmu.jp/jira/browse/PIPE2D-355 DRP 2-D Pipeline <p>Using the external tool PSFEx, model the 2D PSF. </p> <p>This can subsequently be used by <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-333" title="Construct a 1D LSF from a 2D PSF" class="issue-link" data-issue-key="PIPE2D-333"><del>PIPE2D-333</del></a> to construct a 1D LSF.</p> <p>Success criteria will be provided in due course.</p> PIPE2D-355 Model the 2D PSF using PSFEx Task Normal Won't Fix Won't Fix Kiyoto Yabe hassan Wed, 13 Feb 2019 14:57:46 +0000 Mon, 4 Jan 2021 20:23:07 +0000 Fri, 27 Mar 2020 14:46:42 +0000 0 6 <p>Example PSFs from LAM data should be accessible in order to address this ticket.</p> <p>The example PSFs have been created in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-379" title="Create set of cutouts to be used to model PSF" class="issue-link" data-issue-key="PIPE2D-379"><del>PIPE2D-379</del></a> and they should be ready to use (I am using then now for my modeling).</p> <p>Clarification of my earlier comment: <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naka.keigo" class="user-hover" rel="naka.keigo">Keigo Nakamura</a> needs access to the example PSFs that have been created from LAM data as part of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-379" title="Create set of cutouts to be used to model PSF" class="issue-link" data-issue-key="PIPE2D-379"><del>PIPE2D-379</del></a>, which are situated on the Princeton cluster.</p> <p>I don't understand this.  The LSST stack that the PFS pipeline is based on has support for `PSFex` built in;  you can start with an arc image and run the PSF modelling code.</p> <p>When we come to see if piff works better we will have to think about interfaces – LSST is planning to integrate piff at some point, and we can either wait for this work or contribute it;  it may be reasonable to test piff on arc data before doing the integration work.</p> <p>Key LSST task to be used that makes use of PSFex is <tt>CharacterizeImageTask</tt>: <a href="https://pipelines.lsst.io/modules/lsst.pipe.tasks/tasks/lsst.pipe.tasks.characterizeImage.CharacterizeImageTask.html#lsst-task-lsst-pipe-tasks-characterizeimage-characterizeimagetask" class="external-link" rel="nofollow">https://pipelines.lsst.io/modules/lsst.pipe.tasks/tasks/lsst.pipe.tasks.characterizeImage.CharacterizeImageTask.html#lsst-task-lsst-pipe-tasks-characterizeimage-characterizeimagetask</a></p> <p>Documentation providing examples are missing. An LSST ticket has been raised to address this:<br/> <a href="https://jira.lsstcorp.org/browse/DM-18600" class="external-link" rel="nofollow">https://jira.lsstcorp.org/browse/DM-18600</a></p> <p>However a useful starting point for using the <tt>CharacterizeImageTask</tt> is available here: <a href="https://github.com/RobertLuptonTheGood/notebooks/blob/master/Demos/Image%20Processing.ipynb" class="external-link" rel="nofollow">https://github.com/RobertLuptonTheGood/notebooks/blob/master/Demos/Image%20Processing.ipynb</a></p> <p>I attached the output image of PSFEX. I tried simulated image of ARC in integration test (visit =31).<br/> At the top region, PSF has some fluctuation.</p> <p>The very initial results:</p> <p><span class="image-wrap" style=""><a id="12312_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12312/12312_arc_ne_17244_psfex_xstack.png" title="arc_ne_17244_psfex_xstack.png" file-preview-type="image" file-preview-id="12312" file-preview-title="arc_ne_17244_psfex_xstack.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12310/_thumb_12310.png" style="border: 0px solid black" /></a></span></p> <p>Modeled PSF with PSFEx at each position of the detector and spots used for the calculation (small circle; redder color means brighter). Ne arc PostISRCCD image (visit=17244) is used. `config.repair.doCosmicRay = False`  I'm not sure why very bright lines in the lower area of the image are not used by PSFEx, but this may be able to be controlled by tuning parameters. Current set of parameter is like this:</p> <p>lsst.meas.extensions.psfex.psfexPsfDeterminer.PsfexPsfDeterminerConfig(kernelSize=41.0, kernelSizeMin=31, kernelSizeMax=51, <em>PsfexPsfDeterminerConfig</em><em>nEigenComponents=4, spatialOrder=2, sizeCellX=256, sizeCellY=256, _PsfexPsfDeterminerConfig</em><em>nStarPerCell=3, samplingSize=1.0, badMaskBits=<span class="error">&#91;&#39;INTRP&#39;, &#39;SAT&#39;&#93;</span>, psfexBasis='PIXEL_AUTO', _PsfexPsfDeterminerConfig</em><em>borderWidth=0, _PsfexPsfDeterminerConfig</em><em>nStarPerCellSpatialFit=5, _PsfexPsfDeterminerConfig</em><em>constantWeight=True, _PsfexPsfDeterminerConfig</em>_nIterForPsf=3, tolerance=0.01, lam=0.05, reducedChi2ForPsfCandidates=2.0, spatialReject=3.0, recentroid=False)</p> <p>lsst.meas.algorithms.objectSizeStarSelector.ObjectSizeStarSelectorConfig(doFluxLimit=False, fluxMin=12500.0, fluxMax=0.0, doSignalToNoiseLimit=True, signalToNoiseMin=20.0, signalToNoiseMax=0.0, widthMin=0.0, widthMax=20.0, sourceFluxField='base_GaussianFlux_instFlux', widthStdAllowed=0.5, nSigmaClip=3.0, badFlags=<span class="error">&#91;&#39;base_PixelFlags_flag_edge&#39;, &#39;base_PixelFlags_flag_interpolatedCenter&#39;, &#39;base_PixelFlags_flag_saturatedCenter&#39;, &#39;base_PixelFlags_flag_crCenter&#39;, &#39;base_PixelFlags_flag_bad&#39;, &#39;base_PixelFlags_flag_interpolated&#39;&#93;</span>)</p> <p><span class="image-wrap" style=""><a id="12313_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12313/12313_arc_ne_17244_psfex_2d_residual_ex1.png" title="arc_ne_17244_psfex_2d_residual_ex1.png" file-preview-type="image" file-preview-id="12313" file-preview-title="arc_ne_17244_psfex_2d_residual_ex1.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12311/_thumb_12311.png" style="border: 0px solid black" /></a></span></p> <p>This is a 1D profile of each image along X direction in each location (+/-15 pix stack in Y direction).</p> <p><span class="image-wrap" style=""><a id="12400_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12400/12400_arc_ne_17244_psfex_2d_residual_1d_ex1.png" title="arc_ne_17244_psfex_2d_residual_1d_ex1.png" file-preview-type="image" file-preview-id="12400" file-preview-title="arc_ne_17244_psfex_2d_residual_1d_ex1.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12312/_thumb_12312.png" style="border: 0px solid black" /></a></span></p> <p>This is a 1D profile of each image along Y direction in each location (+/-15 pix stack in X direction).</p> <p>In each plot, the X location is shown at the top of each sub-panel, and color shows the Y location (bluer means lower Y).</p> <p> </p> <p>I'm perhaps doing stupid things, and I'm very happy to know a proper set of parameters.</p> <p>Anyway, I don't see a large change of the line profile in Y direction, but see slight change (1-2 percent level) in the wing of the profile.</p> <p> </p> <p><span class="image-wrap" style=""><a id="12401_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12401/12401_arc_ne_17244_psfex_1d_residual_map.png" title="arc_ne_17244_psfex_1d_residual_map.png" file-preview-type="image" file-preview-id="12401" file-preview-title="arc_ne_17244_psfex_1d_residual_map.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12313/_thumb_12313.png" style="border: 0px solid black" /></a></span></p> <p><span class="image-wrap" style=""><a id="12426_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12426/12426_oversampling_issue_psf_cutout.png" title="oversampling_issue_psf_cutout.png" file-preview-type="image" file-preview-id="12426" file-preview-title="oversampling_issue_psf_cutout.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12400/_thumb_12400.png" style="border: 0px solid black" /></a></span></p> <p>This is an example of 2D (top) and 1D (bottom) residual of a fairly bright spot at the center of the image. You see 1-2 percent level of residual (compared to the peak). Note that this is actually a good case, and there is worse case indeed. </p> <p> </p> <p><span class="image-wrap" style=""><a id="12401_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12401/12401_arc_ne_17244_psfex_1d_residual_map.png" title="arc_ne_17244_psfex_1d_residual_map.png" file-preview-type="image" file-preview-id="12401" file-preview-title="arc_ne_17244_psfex_1d_residual_map.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12401/_thumb_12401.png" style="border: 0px solid black" /></a></span></p> <p>This is a map of RMS of 1D residual divided by the peak of the original line. Most of the spots show 2-3% level of residual, but some spots show &gt;5% level residual. Bad spots in lower part of the image are fairly bright but the positional information of the detection result looks bad for some reason and this causes the large residual.</p> <p>In plots like arc_ne_17244_psfex_xstack.png what are the units? The y-axis is labelled "counts" &#8211; are they in fractions of the peak, or the total flux, or something else?</p> <p>Also, the PSF images are showing ringing. Did you turn on oversampling, and (if so) are you using the version of psfEx in the LSST that I fixed/hacked?</p> <p>Yes, it is a normalized value so that the total count is unity. </p> <p>samplingSize=1.0, so I think oversampling is OFF.</p> <p>I'm using ver. 18.1.0 for meas_extensions_psfex but I'm not sure if this is a version you fixed. I actually asked <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a> to install the package on a server at IPMU, but some steps described in README (such as checking out your subversion repo) may have been skipped.</p> <p>It has been installed with `eups distrib install meas_extensions_psfex v18_1_0` on top of installed LSST stack for pfs_pipe.</p> <p>The data is undersampled (as is normal for fibre spectrographs), so I think you need to turn on oversampling.</p> <p>As reported in previous telecon, turning on oversampling causes a strange result as shown below:</p> <p><span class="image-wrap" style=""><a id="12536_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12536/12536_image+%284%29.png" title="image (4).png" file-preview-type="image" file-preview-id="12536" file-preview-title="image (4).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12426/_thumb_12426.png" style="border: 0px solid black" /></a></span></p> <p>This is an image of the local PSF modeling near the FoV center for 5x oversampling. In the processing, 192 out of 237 PSF candidate objects (arc spots) are used for the modeling.</p> <p>A set of configuration is shown below:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">// code placeholder </span> config = CharacterizeImageTask.ConfigClass() config.psfIterations = 1 config.useSimplePsf = False config.repair.doCosmicRay = False config.doApCorr = False config.measurePsf.psfDeterminer.name = <span class="code-quote">'psfex'</span> config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].badMaskBits=[<span class="code-quote">'INTRP'</span>, <span class="code-quote">'SAT'</span>] config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].kernelSize=205 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].kernelSizeMin=31 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].kernelSizeMax=51 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].sizeCellX=64 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].sizeCellY=64 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].samplingSize=0.2 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].psfexBasis=<span class="code-quote">'PIXEL'</span> config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].recentroid=True config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].lam=0.05 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].spatialOrder=2 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].spatialReject=3.0 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].tolerance=0.01 config.measurePsf.psfDeterminer[<span class="code-quote">'psfex'</span>].reducedChi2ForPsfCandidates=2.0 config.measurePsf.starSelector.name = <span class="code-quote">'objectSize'</span> config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].doFluxLimit=False config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].fluxMin=1000. config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].fluxMax=5000000. config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].doSignalToNoiseLimit=True config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].signalToNoiseMin=50. config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].signalToNoiseMax=1000. config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].widthMin = 0.5 config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].widthMax = 5.0 config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].widthStdAllowed = 0.3 config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].nSigmaClip = 2.0 config.measurePsf.starSelector[<span class="code-quote">'objectSize'</span>].sourceFluxField = <span class="code-quote">'base_PsfFlux_instFlux'</span> </pre> </div></div> <p>Please let me know, if you have any idea on the possible cause or what can be helpful...</p> <p> </p> <p>I have talked with Bob Amstrong. He immediately recognized the strange feature discussed above. He says that the feature is caused by the fact that the code only does the real analysis in the inner part of the PSF, while outside it does some sort of simplified averaging. The parameter that controls how large the region is in which ``real analysis'' is conducted is given by the parameter <tt>BASIS_NUMBER</tt> which is not accessible via python, but is accessible via the default configuration file. On GitHub, this file can be found at</p> {[https://github.com/LSST-nonproject/meas_extensions_psfex/blob/master/config/default-lsst.psfex].} <p> </p> <p>On Monday I plan to play to see if the changing the <tt>BASIS_NUMBER</tt> changes the results </p> <p>I did conduct the change that Bob suggested and increased the <tt>BASIS_NUMBER</tt> from 20 to 40. The code ran 10 times longer (4517 seconds, compared to 467 when BASIS_NUMBER was 20), but I do get a similar result:</p> <p> </p> <p><span class="image-wrap" style=""><a id="12535_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12535/12535_Basis_number_40.png" title="Basis_number_40.png" file-preview-type="image" file-preview-id="12535" file-preview-title="Basis_number_40.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12535/_thumb_12535.png" style="border: 0px solid black" /></a></span></p> <p>Thank you very much for this test <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=ncaplar" class="user-hover" rel="ncaplar">ncaplar</a>. I do appreciate if you get a further chance to talk to Bob or other folks. </p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> Yes, I am trying to get in contact with Bob to discuss further.</p> <p> </p> <p>I have talked with Bob Amstrong about our problem when using PSFex. To repeat what I mentioned in the ticket, he says that the feature is caused by the fact that the code only does the real analysis in the inner part of the PSF, while outside it does some sort of simplified averaging. He first recommended increasing the BASIS_NUMBER parameter, but that did not change the result (only made code run longer). <br/> He also shared with the Figure showing the HSC results when using third order lanczos and having different parameters of interpolation (`INTERPW` and `INTERPFAC`) then what we are using now. We see the same effect. Based on these we seem to have concluded that changing the interpolation properties via parameters `INTERPW` or `INTERPFAC` does not change the size of the interpolation box.</p> <p> </p> <p><span class="image-wrap" style=""><a id="12536_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12536/12536_image+%284%29.png" title="image (4).png" file-preview-type="image" file-preview-id="12536" file-preview-title="image (4).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12536/_thumb_12536.png" style="border: 0px solid black" /></a></span></p> <p>We discussed this during DRP telecom on March 27, 2020. We have reached the conclusion that at this point it is not worth investigating this further. Changing the size of the box would probably include considerable amounts of time and possibly working on the underlying C-code. <br/> If it turns out that this code is needed (in the case that my 2d PSF modeling is unsatisfactory), we plan to either try using Piff or return to the work in this ticket.</p> Relates PIPE2D-379 Development Epic Link PIPE2D-351 Rank 0|02qprv:cc Sprint 2DDRP-2019 C 2DDRP-2019 D 2DDRP-2019 E 2DDRP-2019 F 2DDRP-2019 G 2DDRP-2019 H 2DDRP-2019 I 2DDRP-2019 J 2DDRP-2019 K 2DDRP-2020 A 2DDRP-2021 A Story Points 5.0