[PIPE2D-290] Fix normalisation of fibers in flats Created: 14/Sep/18  Updated: 19/Oct/18  Resolved: 17/Oct/18

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File after-5694.png     PNG File before-5694.png    
Issue Links:
Relates
relates to PIPE2D-298 Record throughput in FiberTrace Done
Story Points: 5
Reviewers: hassan

 Description   

We need to be able to calibrate all fibers on a CCD by calibrating just one, so the fiber flats need to preserve the fiber-to-fiber throughput variations, whereas currently they don't.

In the process, remove DetectorMap._throughput: throughput variations will be taken care of in the flat-field, and that also allows us to correct throughput as a function of wavelength rather than a single value per fiber.



 Comments   
Comment by price [ 03/Oct/18 ]

hassan, are you available to take a look at this, please?

price@neverland:~/pfs/drp_stella (tickets/PIPE2D-290=) $ git sub
commit 4415c44ee44de4e2ef7f8b53e6601c5be889187d
Author: Paul Price <price@astro.princeton.edu>
Date:   Thu Sep 13 14:32:15 2018 -0400

    remove SWIG bindings
    
    We're using pybind11 now. How have these managed to stick around so long?

 python/pfs/drp/stella/stellaLib.i | 152 --------------------------------------
 1 file changed, 152 deletions(-)

commit 80bdd284fd72c1c9e1e6337415e0bf9660cdd54f
Author: Paul Price <price@astro.princeton.edu>
Date:   Thu Sep 13 14:53:40 2018 -0400

    DetectorMap: strip throughput
    
    A single value per fiber isn't sufficient to characterise the throughput
    as a function of wavelength for each fiber. We'll do this in the flat-field
    instead of carrying it around here.

 include/pfs/drp/stella/DetectorMap.h             | 25 +---------
 python/pfs/drp/stella/constructFiberTraceTask.py | 62 ------------------------
 python/pfs/drp/stella/detectorMap.cc             | 27 +++--------
 python/pfs/drp/stella/detectorMapContinued.py    | 16 +-----
 python/pfs/drp/stella/extractSpectraTask.py      |  1 -
 src/DetectorMap.cc                               | 45 +----------------
 tests/test_DetectorMap.py                        | 32 ++++--------
 7 files changed, 20 insertions(+), 188 deletions(-)

commit 0b76eff8b900b509d7250d98e3c08414b1c383ee
Author: Paul Price <price@astro.princeton.edu>
Date:   Tue Oct 2 21:34:44 2018 +0000

    FiberTraceSet: add extractSpectra method
    
    That's why you have a FiberTrace, right? So let's add a method to do that.

 python/pfs/drp/stella/fiberTracesContinued.py | 29 +++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

commit 7ccc5150684ca409f821866d3f1625ecb8362c5a
Author: Paul Price <price@astro.princeton.edu>
Date:   Tue Oct 2 21:37:28 2018 +0000

    refactor flat field normalisation
    
    Previously, each fiber was independently normalised, which means
    that flux calibrating one doesn't allow you to flux calibrate
    anything else (well, we have a throughput measurement in the
    DetectorMap, but that's not a function of wavelength). Instead,
    normalise all fibers to the mean spectrum of the fibers.
    This keeps the fibers tied together.

 python/pfs/drp/stella/constructFiberFlatTask.py | 115 +++++++++++++-----------
 1 file changed, 61 insertions(+), 54 deletions(-)

commit 2801f730a15e1ab95cc21f169c174ad259f129de
Author: Paul Price <price@astro.princeton.edu>
Date:   Tue Oct 2 21:36:06 2018 -0400

    make some files flake8-clean

 python/pfs/drp/stella/constructFiberTraceTask.py | 2 --
 python/pfs/drp/stella/detectorMapContinued.py    | 3 +--
 python/pfs/drp/stella/spectraContinued.py        | 4 ++--
 3 files changed, 3 insertions(+), 6 deletions(-)
Comment by hassan [ 03/Oct/18 ]

I'm hoping to discuss this with cloomis later on today.

Comment by cloomis [ 03/Oct/18 ]

What is the advantage of calibrating all by calibrating one? Besides that I guess that for flats (fiber throughput) I'd think of them as 600 separate cameras. At least until we learn better.

Comment by price [ 16/Oct/18 ]

Thanks for the pushback, cloomis — it encouraged me to go and verify that what I'm doing actually makes sense. I tried flattening the flats, and seeing how they turned out. There wasn't much improvement, which forced me to get in and fix it up. Now it's a lot better, and I think the plots demonstrate why this is important. Here's the extracted spectra from a flat after flattening with the 4.0 release:

Notice that the spectra from the different fibers don't follow each other exactly. The standard deviation from summing the flux over rows 1000 to 3000 is 0.0084 mag (that's a raw sum; we could calculate this statistic as a function of wavelength, but we just want a value for comparison purposes).
Here's the same spectra after flattening with this submission:

The spectra are now much more closely aligned, and the standard deviation over the same aperture is 0.0013 mag. We have, in effect, calibrated all the fibers against a single spectrum (viz., a continuum fit to the mean flat-field). Now the flux calibration problem is reduced to calibrating that spectrum against some source with a known spectrum in physical units.

Comment by price [ 17/Oct/18 ]

Merged to master after receiving approval on GitHub.

Generated at Sat Feb 10 15:50:37 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.