Screenshot 2018-04-20 09.59.53.png
[PIPE2D-278] Variance maps should accurately estimate noise in the images Created: 20/Apr/18 Updated: 11/Sep/18 Resolved: 11/Sep/18 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Normal |
| Reporter: | ncaplar | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
Screenshot 2018-04-20 09.59.53.png
|
| Reviewers: | ncaplar |
| Description |
|
The noise in the reduced images is currently not correctly captured by the variance map. For an example, looking at the image 8603 and taking random patch of the image without sources gives variance of around 20. On the other hand, mean value in the variance image is around 36, 37. These results seem to be consistent across the whole image.
Possible causes are 1. wrong gains, 2. given that there seems to be structure in the noise, wrong estimation of the readout, 3. other options.
|
| Comments |
| Comment by price [ 25/Apr/18 ] |
|
Work is being done on DM-14182. pprice@tiger-sumire:~/LSST/ip/isr (tickets/DM-14182=) $ detrend.py /tigress/HSC/PFS/LAM/ --calib /tigress/HSC/PFS/LAM/CALIB-PAP --rerun price/pipe2d-278-fix --id visit=8603 -c isr.doEmpiricalReadNoise=True ... detrend.isr INFO: Calculated empirical read noise for amp 0: 4.423752 detrend.isr INFO: Calculated empirical read noise for amp 1: 4.266983 detrend.isr INFO: Calculated empirical read noise for amp 2: 4.403465 detrend.isr INFO: Calculated empirical read noise for amp 3: 4.178715 detrend.isr INFO: Calculated empirical read noise for amp 4: 3.960904 detrend.isr INFO: Calculated empirical read noise for amp 5: 4.411193 detrend.isr INFO: Calculated empirical read noise for amp 6: 5.042430 detrend.isr INFO: Calculated empirical read noise for amp 7: 4.181415 Then, in python: >>> import lsst.afw.image >>> exposure = lsst.afw.image.ExposureF("/tigress/HSC/PFS/LAM/rerun/price/pipe2d-278-fix/calExp/2017-12-15/v0008603/PFLAr1.fits") >>> import numpy as np >>> np.var(exposure.image.array[760:2960,640:1140]) 21.075794 >>> np.mean(exposure.variance.array[760:2000,640:1140]) 19.159437 |
| Comment by price [ 25/Apr/18 ] |
|
Should I also update the read noise in the detector? They don't seem far off, but it might be good to update them with the hardware. >>> ff = pyfits.open("/home/lsst/pfs/obs_pfs/pfs/camera/r_1.fits") >>> ff[1].data["readnoise"] array([ 3.61, 3.78, 3.18, 2.95, 3.19, 3.8 , 4.51, 3.18]) |
| Comment by cloomis [ 25/Apr/18 ] |
|
Probably. With luck we will have time to figure out why the system is that much noisier in the real cryostat at LAM than in the lab dewar JHU. |
| Comment by price [ 26/Apr/18 ] |
|
I've updated the hard-wired read noise values. ncaplar, could you please verify this helps? The original values were: >>> import pyfits >>> ff = pyfits.open("/home/lsst/pfs/obs_pfs/pfs/camera/r_1.fits") >>> ff[1].data["readnoise"] array([ 3.61, 3.78, 3.18, 2.95, 3.19, 3.8 , 4.51, 3.18]) They are now:
array([ 4.423752, 4.266983, 4.403465, 4.178715, 3.960904, 4.411193,
5.04243 , 4.181415])
|
| Comment by ncaplar [ 11/Jul/18 ] |
|
Just to have it here for documentation purposes, this fix did help and variance is now consistent with what we observe in the data. |
| Comment by price [ 11/Sep/18 ] |
|
Merged to master. |