[PIPE2D-935] Support HgCd and other multi and single element sources Created: 06/Nov/21  Updated: 15/Jul/22  Resolved: 29/Jan/22

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: hassan Assignee: hassan
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File log.reduceArc_67752_67757_hgcd_b    
Issue Links:
Duplicate
duplicates PIPE2D-756 Line list species filtering is ambiguous Done
Relates
relates to PIPE2D-974 Weekly build w.2022.05 fails Done
relates to PIPE2D-1062 Revise HgCd linelist Done
Story Points: 2
Sprint: 2DDRP-2021 A12, 2DDRP-2022 A
Reviewers: price

 Description   

[Updated title and description, as work reflects a bigger change than the original description implied]

The current reference lines construction code consists of a set of single-element linelists (Hg.txt, Ar.txt, Xe.txt, etc) as well as a linelist for the sky (skyLines.txt). For single-element lamps (Hg, Ar) the appropriate file is read, while for lamps that contain a mix of elements (eg HgAr) the Hg.txt and Ar.txt files are read and merged.

This is not entirely reflective of the real situation, where due to internal physics, the lamps that contain multiple elements may result in emission spectra where the lines for a given species may be shifted slightly compared to that of the single-element lamp case. So dedicate linelists for each lamp or light source is required.

Please add support for dedicated linelists for HgCd and HgAr lamps (the former is currently missing), while retaining support for single-element lamps.

Update reference line construction code to handle multiple lamps being illuminated (be they consisting of multiple elements or a single element).

For the HgCd lamp, update the ops_pfs code such that the lamp is correctly identified when the FITS keyword W_AITHGC is True in the raw exposure data.

 Comments   
Comment by arnaud.lefur [ 06/Nov/21 ]

Actually, I think that only Cd linelist is missing.

Comment by rhl [ 06/Nov/21 ]

I think we probably want a HgCd line list, as it isn't really a simple sum of Hg + Cd.

Comment by ncaplar [ 13/Nov/21 ]

I am just pasting, for possible future reference, the paragraph JEG wrote about air-vacuum conversions:
*************
The issue of the vacuum vs air wavelengths is complicated, not
by the physics particularly, but by convention. A simple Sellmeier
equation for the index is

n-1 = 0.057921/(238.0185 - 1/lam^2) + 0.001679/(57.362 - 1/lam^2) * (P/101325)*(288.15)/T

with the wavelength lam in microns, pressure in pascals, temeperature in
Kelvin; if you set the pressure and temperature terms to unity, you get the index for 15C and the standard sea level pressure, at which I think
the difference in vacuum and `air' wavelength is DEFINED, and this
is what you want to use, I think, not the ACTUAL air wavelength, on MK, which is at 0.6 bar pressure and 0C. A very useful reference is the SDSS
APOGEE one, which uses a different Sellmeier form with 5 constants
instead of 4 (there is a constant term of about 7e-5),
and with a nice discussion, is to be found at

https://www.as.utexas.edu/~hebe/apogee/docs/air_vacuum.pdf,

also attached.
*************

Comment by hassan [ 20/Nov/21 ]

Created new linelist HgCd.txt from a PFI linelist provided by Erin Kado-Fong, with wavelengths corrected to vacuum. It is assumed that the HgCd lamp consists of elements Hg, Cd and Ar (the latter element being the carrier; there is only one faint line at 433.945nm for this element), and updated the ops_pfs code to make this available.

Results from reduceArc.py look pretty OK:

reduceArc.fitDetectorMap INFO: Final result: chi2=119161.780643 dof=21362 xRMS=0.017879 yRMS=0.016282 xSoften=0.014306 ySoften=0.014033 from 11293 lines
Comment by hassan [ 20/Nov/21 ]

Full log of reduceArc run attached (log.reduceArc_67752_67757_hgcd_b)

Comment by price [ 21/Dec/21 ]

Please be aware of potential merge conflicts (PIPE2D-948), and please check that the following test cases produce reasonable results:

reduceArc.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-2021220 --rerun whatever --id visit=68351^68100 arm=r -c reduceExposure.isr.doFlat=False reduceExposure.isr.doApplyGains=True fitDetectorMap.doSlitOffsets=True reduceExposure.repair.doCosmicRay=False

reduceArc.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108 --rerun whatever --id visit=46671..46688 arm=r -C roomArc.py -c reduceExposure.isr.doFlat=False reduceExposure.isr.doApplyGains=True  fitDetectorMap.doSlitOffsets=True

where roomArc.py contains a suitably revised version of:

config.reduceExposure.readLineList.lampList = ["HgI", "KrI", "ArI"]  # otherwise we get sky
Comment by hassan [ 22/Jan/22 ] 
reduceArc.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108 --rerun hassans/pipe2d-935/iter3/test2 --id visit=46671..46688 arm=r -C roomArc.py -c reduceExposure.isr.doFlat=False reduceExposure.isr.doApplyGains=True fitDetectorMap.doSlitOffsets=True

Where roomArc.py contains

config.reduceExposure.readLineList.assumeSkyIfNoLamps = False  # otherwise we get sky
config.reduceExposure.readLineList.lightSources = ["HgAr", "Kr", "Ar"]

Yields the following result:

reduceArc.fitDetectorMap INFO: Final result: chi2=59086.066388 dof=14656 xRMS=0.031116 yRMS=0.115716 xSoften=0.000000 ySoften=0.112986 from 7651 lines
reduceArc.fitDetectorMap INFO: Stats for ArI: chi2=24129.617016 dof=9626 xRMS=0.022979 yRMS=0.080409 xSoften=0.000000 ySoften=0.058656 from 4813 lines
reduceArc.fitDetectorMap INFO: Stats for KrI: chi2=24788.003647 dof=1582 xRMS=0.040442 yRMS=0.243934 xSoften=0.003916 ySoften=0.445455 from 791 lines
reduceArc.fitDetectorMap INFO: Stats for HgI: chi2=10168.445725 dof=4094 xRMS=0.070457 yRMS=0.185180 xSoften=0.000000 ySoften=0.104983 from 2047 lines
reduceArc.fitDetectorMap INFO: Stats for fiberId=3: chi2=2855.063360 dof=70 xRMS=0.071605 yRMS=0.569879 xSoften=0.000000 ySoften=0.318329 from 35 lines (29 ArI, 6 KrI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=169: chi2=4.205491 dof=12 xRMS=0.066904 yRMS=0.048803 xSoften=0.000000 ySoften=0.000000 from 6 lines (6 HgI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=306: chi2=110.163186 dof=70 xRMS=0.017175 yRMS=0.068615 xSoften=0.000000 ySoften=0.039321 from 35 lines (12 HgI, 18 ArI, 5 KrI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=482: chi2=5.268089 dof=16 xRMS=0.027286 yRMS=0.059581 xSoften=0.000000 ySoften=0.000000 from 8 lines (7 HgI, 1 ArI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=649: chi2=63.746220 dof=60 xRMS=0.049800 yRMS=0.040935 xSoften=0.008278 ySoften=0.000000 from 30 lines (11 HgI, 18 ArI, 1 KrI)

which is comparable with that from the current weekly tag (w.2022.03):

reduceArc.fitDetectorMap INFO: Final result: chi2=51757.452781 dof=14922 xRMS=0.029859 yRMS=0.106771 xSoften=0.000000 ySoften=0.110645 from 7784 lines
reduceArc.fitDetectorMap INFO: Stats for HgI: chi2=11342.541964 dof=4152 xRMS=0.069827 yRMS=0.196695 xSoften=0.000000 ySoften=0.118273 from 2076 lines
reduceArc.fitDetectorMap INFO: Stats for KrI: chi2=19655.487699 dof=1720 xRMS=0.037106 yRMS=0.197794 xSoften=0.007738 ySoften=0.402704 from 860 lines
reduceArc.fitDetectorMap INFO: Stats for ArI: chi2=20759.423117 dof=9696 xRMS=0.021606 yRMS=0.074336 xSoften=0.000000 ySoften=0.057231 from 4848 lines
reduceArc.fitDetectorMap INFO: Stats for fiberId=3: chi2=59.019761 dof=72 xRMS=0.065215 yRMS=0.051048 xSoften=0.000000 ySoften=0.000000 from 36 lines (30 ArI, 6 KrI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=169: chi2=635.113404 dof=90 xRMS=0.028226 yRMS=0.148571 xSoften=0.000000 ySoften=0.095848 from 45 lines (33 ArI, 8 HgI, 4 KrI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=306: chi2=31.149488 dof=70 xRMS=0.024501 yRMS=0.029723 xSoften=0.000000 ySoften=0.000000 from 35 lines (17 ArI, 13 HgI, 5 KrI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=482: chi2=4.620672 dof=14 xRMS=0.032584 yRMS=0.068946 xSoften=0.000000 ySoften=0.000000 from 7 lines (7 HgI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=649: chi2=83.829390 dof=64 xRMS=0.047829 yRMS=0.050627 xSoften=0.005874 ySoften=0.000000 from 32 lines (20 ArI, 11 HgI, 1 KrI)
Comment by hassan [ 22/Jan/22 ]

Running 

reduceArc.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211220 --rerun hassans/pipe2d-935/iter3/test1 --id visit=68351^68100 arm=r -c reduceExposure.isr.doFlat=False reduceExposure.isr.doApplyGains=True fitDetectorMap.doSlitOffsets=True reduceExposure.repair.doCosmicRay=False

Yields

reduceArc.fitDetectorMap INFO: Final fit: chi2=13203932.913188 dof=2252213 xRMS=0.075809 yRMS=0.050710 (0.004378 nm) xSoften=0.075029 ySoften=0.000000 from 2229199/2246832 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=1635161.929196 xRMS=0.050164 yRMS=0.054724 (0.004725 nm) xSoften=0.080291 ySoften=0.000000 from 249648 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softened fit: chi2=2727141.743981 dof=2252213 xRMS=0.076250 yRMS=0.045583 (0.003936 nm) xSoften=0.075012 ySoften=0.000000 from 249648 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=368992.754672 xRMS=0.050122 yRMS=0.054923 (0.004742 nm) xSoften=0.080258 ySoften=0.000000 from 249648 lines
reduceArc.fitDetectorMap INFO: Slit offsets measurement: chi2=803656.550181 dof=2251109 xRMS=0.018615 yRMS=0.026924 xSoften=0.016077 ySoften=0.000000 from 2229141/2229199 lines
reduceArc.fitDetectorMap INFO: Unable to measure slit offsets for 22 fiberIds: [1, 3, 45, 92, 114, 137, 184, 204, 229, 273, 280, 309, 314, 336, 359, 370, 382, 426, 471, 515, 607, 651]
reduceArc.fitDetectorMap INFO: Final fit: chi2=791107.313052 dof=2252074 xRMS=0.018470 yRMS=0.026463 (0.002285 nm) xSoften=0.015888 ySoften=0.000000 from 2228995/2246652 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=191766.322777 xRMS=0.015909 yRMS=0.024306 (0.002099 nm) xSoften=0.023703 ySoften=0.000000 from 249628 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softened fit: chi2=2296324.289124 dof=2252074 xRMS=0.017464 yRMS=0.014524 (0.001254 nm) xSoften=0.015867 ySoften=0.000000 from 249628 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=503367.561480 xRMS=0.015864 yRMS=0.024285 (0.002097 nm) xSoften=0.023712 ySoften=0.000000 from 249628 lines
reduceArc.fitDetectorMap INFO: Slit offsets measurement: chi2=790558.408379 dof=2250881 xRMS=0.018461 yRMS=0.026872 xSoften=0.015839 ySoften=0.000000 from 2228847/2228995 lines
reduceArc.fitDetectorMap INFO: Unable to measure slit offsets for 22 fiberIds: [1, 3, 45, 92, 114, 137, 184, 204, 229, 273, 280, 309, 314, 336, 359, 370, 382, 426, 471, 515, 607, 651]
reduceArc.fitDetectorMap INFO: Final result: chi2=793459.472338 dof=2250871 xRMS=0.018495 yRMS=0.026891 xSoften=0.015868 ySoften=0.000000 from 2228980 lines
reduceArc.fitDetectorMap INFO: Stats for Trace: chi2=767501.018717 dof=2205757 xRMS=0.018354 yRMS=nan xSoften=0.015933 ySoften=0.000000 from 2205757 lines
reduceArc.fitDetectorMap INFO: Stats for NeI: chi2=25958.453621 dof=46446 xRMS=0.035183 yRMS=0.026891 xSoften=0.010827 ySoften=0.001561 from 23223 lines
reduceArc.fitDetectorMap INFO: Stats for fiberId=2: chi2=6793.950692 dof=3755 xRMS=0.041979 yRMS=0.035976 xSoften=0.041180 ySoften=0.000000 from 3718 lines (3681 Trace, 37 NeI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=159: chi2=991.126173 dof=3804 xRMS=0.015939 yRMS=0.023107 xSoften=0.011590 ySoften=0.000000 from 3768 lines (3732 Trace, 36 NeI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=338: chi2=4422.998858 dof=3763 xRMS=0.033767 yRMS=0.048579 xSoften=0.032699 ySoften=0.000000 from 3722 lines (3681 Trace, 41 NeI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=495: chi2=724.627922 dof=3797 xRMS=0.013638 yRMS=0.018094 xSoften=0.009688 ySoften=0.000000 from 3758 lines (3719 Trace, 39 NeI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=650: chi2=4821.861357 dof=3795 xRMS=0.035141 yRMS=0.026714 xSoften=0.034019 ySoften=0.000000 from 3755 lines (3715 Trace, 40 NeI)

which is comparable (=the same) as that from the current weekly tag (w.2022.03):

reduceArc.fitDetectorMap INFO: Final fit: chi2=13203932.913188 dof=2252213 xRMS=0.075809 yRMS=0.050710 (0.004378 nm) xSoften=0.075029 ySoften=0.000000 from 2229199/2246832 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=1635161.929196 xRMS=0.050164 yRMS=0.054724 (0.004725 nm) xSoften=0.080291 ySoften=0.000000 from 249648 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softened fit: chi2=2727141.743981 dof=2252213 xRMS=0.076250 yRMS=0.045583 (0.003936 nm) xSoften=0.075012 ySoften=0.000000 from 249648 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=368992.754672 xRMS=0.050122 yRMS=0.054923 (0.004742 nm) xSoften=0.080258 ySoften=0.000000 from 249648 lines
reduceArc.fitDetectorMap INFO: Slit offsets measurement: chi2=803656.550181 dof=2251109 xRMS=0.018615 yRMS=0.026924 xSoften=0.016077 ySoften=0.000000 from 2229141/2229199 lines
reduceArc.fitDetectorMap INFO: Unable to measure slit offsets for 22 fiberIds: [1, 3, 45, 92, 114, 137, 184, 204, 229, 273, 280, 309, 314, 336, 359, 370, 382, 426, 471, 515, 607, 651]
reduceArc.fitDetectorMap INFO: Final fit: chi2=791107.313052 dof=2252074 xRMS=0.018470 yRMS=0.026463 (0.002285 nm) xSoften=0.015888 ySoften=0.000000 from 2228995/2246652 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=191766.322777 xRMS=0.015909 yRMS=0.024306 (0.002099 nm) xSoften=0.023703 ySoften=0.000000 from 249628 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softened fit: chi2=2296324.289124 dof=2252074 xRMS=0.017464 yRMS=0.014524 (0.001254 nm) xSoften=0.015867 ySoften=0.000000 from 249628 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=503367.561480 xRMS=0.015864 yRMS=0.024285 (0.002097 nm) xSoften=0.023712 ySoften=0.000000 from 249628 lines
reduceArc.fitDetectorMap INFO: Slit offsets measurement: chi2=790558.408379 dof=2250881 xRMS=0.018461 yRMS=0.026872 xSoften=0.015839 ySoften=0.000000 from 2228847/2228995 lines
reduceArc.fitDetectorMap INFO: Unable to measure slit offsets for 22 fiberIds: [1, 3, 45, 92, 114, 137, 184, 204, 229, 273, 280, 309, 314, 336, 359, 370, 382, 426, 471, 515, 607, 651]
reduceArc.fitDetectorMap INFO: Final result: chi2=793459.472338 dof=2250871 xRMS=0.018495 yRMS=0.026891 xSoften=0.015868 ySoften=0.000000 from 2228980 lines
reduceArc.fitDetectorMap INFO: Stats for Trace: chi2=767501.018717 dof=2205757 xRMS=0.018354 yRMS=nan xSoften=0.015933 ySoften=0.000000 from 2205757 lines
reduceArc.fitDetectorMap INFO: Stats for NeI: chi2=25958.453621 dof=46446 xRMS=0.035183 yRMS=0.026891 xSoften=0.010827 ySoften=0.001561 from 23223 lines
reduceArc.fitDetectorMap INFO: Stats for fiberId=2: chi2=6793.950692 dof=3755 xRMS=0.041979 yRMS=0.035976 xSoften=0.041180 ySoften=0.000000 from 3718 lines (3681 Trace, 37 NeI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=159: chi2=991.126173 dof=3804 xRMS=0.015939 yRMS=0.023107 xSoften=0.011590 ySoften=0.000000 from 3768 lines (3732 Trace, 36 NeI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=338: chi2=4422.998858 dof=3763 xRMS=0.033767 yRMS=0.048579 xSoften=0.032699 ySoften=0.000000 from 3722 lines (3681 Trace, 41 NeI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=495: chi2=724.627922 dof=3797 xRMS=0.013638 yRMS=0.018094 xSoften=0.009688 ySoften=0.000000 from 3758 lines (3719 Trace, 39 NeI)
reduceArc.fitDetectorMap INFO: Stats for fiberId=650: chi2=4821.861357 dof=3795 xRMS=0.035141 yRMS=0.026714 xSoften=0.034019 ySoften=0.000000 from 3755 lines (3715 Trace, 40 NeI)
Comment by hassan [ 22/Jan/22 ]

price: please have another look prior to my merging.

Comment by hassan [ 29/Jan/22 ]

Merged to master

Generated at Sat Feb 10 16:00:02 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.