https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1154 DRP 2-D Pipeline <p>Here is the feedback from the redshift measurements on the faint galaxies of the November run, after comparison with the reference redshifts. As I have identified issues that look independent from each other to me I will create other tickets attached to this one. </p> <p>The horizontal lines in the "measured z" vs "reference z" plot denote a fixed feature (identified at 413.4 nm on the two extracts of spectra), that is not masked and is significantly detected. This feature is not systematically present, at least on spectra where the redshift is correct</p> PIPE2D-1154

November Run feedback

Bug Normal Done Done Takuji Yamashita vlebrun Wed, 1 Feb 2023 11:57:49 +0000 Tue, 14 Mar 2023 01:36:11 +0000 Tue, 14 Mar 2023 01:36:11 +0000 0 9 <p>Could you please provide the necessary information to reproduce this? In particular, we need to know what version was used, which visits were processed, and which fibers show the 413.4nm feature.</p> <p>the 'faint galaxies' dataset : pfsDesignId=0x608f5cd5e916bf12: visit=83244..83249</p> <p>I have attached a first list of objects affected, others to come</p> <p>What's the difference between the two lists?</p> <p>objects on both exhibit the feature, but it has been used differently to compute the redshift, our visualization tool can only select by redshift, hence the two lists</p> <p>I've been unable to identify any 413.4 nm feature in the <tt>pfsArm</tt> or <tt>pfsMerged</tt> spectra from <tt>83244b3</tt>. I am attempting to run the flux calibration code from NAOJ to see if it's coming from that. There is a mild absorption line in the stellar spectra around this wavelength, so it's possible that's involved.</p> <p>Indeed in the latest run ( Feb 13, PIPE2D w.2023.06) this feature is not present, but two other fainter ones are here at 566.0 and 579.5 nm. These are just faint residuals from sky lines which should not be corrected, but they are considered as real because of the noise underestimation (by a factor of about 2 from the warnings issued by PIPE1D on those spectra). </p> <p><span class="image-wrap" style=""><a id="15535_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15535/15535_NovRun_5795feature.png" title="NovRun_5795feature.png" file-preview-type="image" file-preview-id="15535" file-preview-title="NovRun_5795feature.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15534/_thumb_15534.png" style="border: 0px solid black" /></a></span><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15535/_thumb_15535.png" style="border: 0px solid black" /></span></p> <p>Those features create the two new horizontal lines in the z_PIPE1D vs z_ref plane, being mistaken as OIIIb <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15533/_thumb_15533.png" style="border: 0px solid black" /></span></p> <p>they represent about 25% of the full sample</p> <p>after further checks, it seems to be mainly present in faint objects spectra, while the noise estimate seems to be OK when there is som flux. I don't know how and if it relates to the discussion we had 3 weeks ago, about the noise estimate for bright star spectra (which is still the casein this run), but it seems that in these extreme case the noise estimate is not optimal</p> <p>There aren't sky lines at 566.0 or 579.5 nm. The closest are OI at 557 nm and the Na doublet at 589 nm. So I'm not sure what you are reporting.</p> <p>I will look into the noise model.</p> <p>Measuring the region 560-585 nm, the actual noise in a pfsObject is 10x larger than what the noise model reports it should be. I'm investigating the cause.</p> <p>The flux calibration vectors have some strong undulations as a function of wavelength, which I think is adversely affecting the quality of the coadded spectra. The undulations differ in strength, but have the same frequency.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15536/15536_pipe2d-1154-fluxCal.png" width="1000" style="border: 0px solid black" /></span></p> <p>The undulations look to be coming from the variance values of the flux calibration inputs, which undulate (due to resampling, as the original pixels have in turn larger and smaller contributions to the resampled pixel). One fiber in particular has a much lower variance level than the others (presumably a brighter target), so it's dominating the coadd (in <tt>ConstantFocalPlaneFunction.fitArrays</tt>), and because the inputs aren't all the same value (varying by an order of magnitude!), the coadd is getting differing contributions of different values, and the result is a mess.</p> <p>The fact that the flux calibration inputs have such a large range of values is a serious problem that needs to be fixed first. I'm handing this ticket over to <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=takuji.yamashita" class="user-hover" rel="takuji.yamashita">Takuji Yamashita</a>.</p> <p>This plot shows the large range of flux calibration vectors (left), with the resultant coadd (right):<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15538/15538_fluxCal-83249.png" width="1000" style="border: 0px solid black" /></span></p> <p>This plot shows the median of the flux calibration vectors (over 560-585 nm) as a function of position on the focal plane (and split by spectrograph):<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15537/15537_fluxCal-83249-spatial.png" width="1000" style="border: 0px solid black" /></span></p> <p>With regard to the undulations in fluxCal, I checked pfsFluxReference and found that this is probably due to a mismatch of LSFs. </p> <p>I see the data of visit=083248. This figure shows, from the bottom to top, fluxCal (blue), whitened pfsMerged spectra (thin lines), a median of whitened pfsMerged (thick black), and pfsFluxReference (thick lines). There are a lot of absorption lines in pfsFluxReference (the best fit model spectra of each standard). It looks like the positions and strengths of these lines coincide with each dips in fluxCal and whitened pfsMerged. The absorptions are diluted with actual LSFs and are observed as moderate undulations in pfsMerged (see a thick black line). On the other hand, the small undulations in fluxCal can be explained by smalll LSFs, which are applied to model spectra. </p> <p> </p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15545/15545_v083248_pipe2d-1154.png" height="352" width="842" style="border: 0px solid black" /></span></p> <p> </p> <p> </p> <p> </p> <p>I checked the photons that fell in a fiber and the large variance of flux calibration. I think that the large range of flux calibration comes from a variance of fiber positions. </p> <p>I calculated a fraction of observed fiber fluxes (fluxes in 600-620 nm of pfsMerged) to total fluxes (r-band psfFlux). And the fractional fluxes are compared with flux calibration, which is normalized by an average flux calibration across the focal plane (fluxCal). I use only standard stars. This figure shows that both are well correlated with each other. The fractional fluxes of fibers also vary by an order of magnitude.</p> <p>In Paul's plot of flux calibration distribution on the focal plane, we find the systematic variation of the median flux calibration value. This suggests that the fractional fluxes also correlate with fiber positions. Therefore, I think that the large range of flux calibration comes from fiber loss. </p> <p>The difference in flux calibration is in itself what we expect. I expect the difference will become small after the further fiber position adjustment. On the pipeline side, we need a better fit on the focal plane. </p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15546/15546_v083248_fractionalflux_pipe2d-1154.png" height="436" width="440" style="border: 0px solid black" /></span></p> <p> </p> <p>If this is the case, the fractions you measure should correlate with the fiber positioning errors. I don't think those are in the pfsConfig for this exposure, but they should be available from the opdb (<a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a>, could you help with this, please?).</p> <p>Also, the fractions should change with wavelength as the PSF changes.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> reports that the mapping from (ra, dec) to (x, y) is currently accurate to only about 0.5 arcsec, so that's probably responsible for the huge dispersion in flux calibration vectors, rather than the cobra positioning accuracy which is about 0.1 arcsec. I therefore think we have a couple of options.</p> <p>The first is to attempt to fit the flux calibration vector over the focal plane, assuming the alignment errors are a smooth function over the focal plane and that couples to the variation in throughput. The plot above shows that there's some focal plane structure that we can fit, but I'm concerned that it looks like it would need an order ~ 5, and we're not really sampling it very well. We will ultimately need to do a fit over the focal plane, but I wonder if this is the time to do it.</p> <p>The second option is to ignore the absolute scaling of the flux calibration and merely attempt to get the wavelength dependence right. The correct wavelength dependence allows us to remove telluric features and get the correct shapes of the spectra; the absolute calibration of individual objects could then be done by scaling the spectra to match broadband photometry (which we've always planned to be able to do, as a backup). One way to do this would be to simply average the flux calibration vectors (not a weighted average per-pixel with rejection iterations, as is being done currently, but a single weight per spectrum could be used). Another option would be to iteratively fit for the attenuation of each fiber (a low-order function of wavelength) relative to the best (or the average) throughput. I think this is what we should pursue for the sake of the coming engineering data release.</p> <p>Thank you for your suggestions! I have filed a ticket for the first option, PIPE2D-11183. The second option should be discussed. Do you have any thoughts about that? <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=msyktnk" class="user-hover" rel="msyktnk">Masayuki Tanaka</a>?</p> <p>I just share some plots here. <br/> For the large variation of flux calibration, I made a plot below is a wavelength dependence of the fractional fluxes. I calculated the fractions at a certain wavelength range in each band (g,r,i). The same color indicates the same fiber and the color changes with the fractions at g-band. This wavelength dependence is consistent with the expected PSF's wavelength dependence. <br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15555/15555_pipe2d-1154_fluxfractions_v83248.png" style="border: 0px solid black" /></span></p> <p>Apart from the flux calibration variation, I share plots about the undulations on fluxCal. The undulations are seen also in individual flux calibration of each standard's fiber and those look like a similar wavelength pattern and a similar strength to that of fluxCal. Some of them could have a small offset in the wavelength direction. We can find that there are these undulations in pfsArm and pfsMerged as well. That is not anything that a flat does. The wavelengths and strengths of the undulations appear to be consistent with those of pfsReference spectra. We cannot find undulations in fibers of science or sky. So, I conclude the undulations are stellar features of standard stars.<br/> These undulations should be not seen in flux calibration because those should be reproduced in pfsReference. Probably fitting could be not going well. I will look into this in more detail in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1148" title="Flux calibration performance check using Nov run data" class="issue-link" data-issue-key="PIPE2D-1148"><del>PIPE2D-1148</del></a>. </p> <p>Individual flux calibration (red) and fluxCal (black).<br/> <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/15556/15556_pipe2d-1154_fluxcals.pdf" title="pipe2d-1154_fluxcals.pdf attached to PIPE2D-1154">pipe2d-1154_fluxcals.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>pfsMerged, pfsArm, pfsReference of two FLUXSTD fibers with the highest signal-to-noise ratios.<br/> <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/15557/15557_pipe2d-1154_v083248_MergedArmRef_fiberId_253_464.pdf" title="pipe2d-1154_v083248_MergedArmRef_fiberId_253_464.pdf attached to PIPE2D-1154">pipe2d-1154_v083248_MergedArmRef_fiberId_253_464.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>pfsMerged of SCIENCE and SKY (red) and fluxCal (black). <br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15558/15558_pipe2d-1154_sci_sky_fluxcal.png" style="border: 0px solid black" /></span> </p> <p>In addition, when we scale the values of individual flux calibration to match with each other, the undulations become small. As Paul pointed out, the undulations in fluxCal are amplified by the large variation of flux calibration values. My initial thought that attributes the undulations to a mismatch of LSF was wrong. </p> <p>Averaged flux calibration vectors after scaling of values of individual flux calibrations.<br/> <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/15559/15559_averaged_fluxcalib_afterscaling_pipe2d-1154.pdf" title="averaged_fluxcalib_afterscaling_pipe2d-1154.pdf attached to PIPE2D-1154">averaged_fluxcalib_afterscaling_pipe2d-1154.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>My thinking here is the same as Paul's: we should ignore the absolute flux scaling and just average the individual flux calibration vectors (I might try scaling all the flux calibration vectors to the highest one, which likely has the star at the fiber center, and then taking the median). I am not really sure if we have time to do this relatively simple thing as there is the ASJ meeting next week, though. I do not mind leaving the flux calibration part as is and add a caveat to the release note.</p> <p>Maybe it is too late, but perhaps we could split this ticket into two, one for undulation and the other for underestimated variance? I am still not 100% sure if the undulations are due to stellar absorption; the pattern seems too periodic for absorption lines.</p> <p>This ticket deviated significantly from the original scope. There are three separate issues:</p> <ul> <li>absolute flux calibration: although the Nov2022 is not the right run to look into it, I've filed a new ticket so that we do not forget to check the absolute flux calibration in future runs.  <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1184" title="Absolute flux calibration using engineering data" class="issue-link" data-issue-key="PIPE2D-1184">PIPE2D-1184</a>.</li> <li>undulation in the flux calibration vector: this will be addressed further in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1148" title="Flux calibration performance check using Nov run data" class="issue-link" data-issue-key="PIPE2D-1148"><del>PIPE2D-1148</del></a>.</li> <li>under-estimated variance: it seems it was not the main driver of the low redshift success rate in the 1d pipeline, but it will certainly affect down-stream science analyses and we need to track it down. <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1185" title="Underestimated variance in Nov 2022 data" class="issue-link" data-issue-key="PIPE2D-1185">PIPE2D-1185</a>.</li> </ul> <p>With this summary, I am closing this ticket.</p> Relates PIPE2D-1155 PIPE2D-1185 PIPE2D-1184 Development Rank 0|02qph7:079 Sprint 2DDRP-2023 A