[PIPE2D-339] Model arc line amplitudes Created: 07/Feb/19  Updated: 29/Jan/22  Resolved: 18/Mar/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0

Type: Task Priority: Normal
Reporter: hassan Assignee: naoki.yasuda
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2019-04-27-21-19-25-398.png     PNG File image-2019-04-28-06-13-26-384.png     PNG File image-2019-04-30-13-13-29-311.png     PDF File PCA_of_Xe_arc.pdf     PNG File PC_fiberId_Xe.png     PNG File PCvector_Xe.png     PNG File PC_visit_Xe.png    
Issue Links:
Blocks
is blocked by PIPE2D-345 Provide a method for determining line... Done
is blocked by PIPE2D-404 Support for raw exposures with W_PFDS... Done
is blocked by PIPE2D-418 ConstructFiberTrace.py only identifie... Won't Fix
is blocked by PIPE2D-333 Construct a 1D LSF from a 2D PSF Done
Relates
relates to PIPE2D-332 Model HgAr lamp 1D continuum Done
relates to PIPE2D-580 Qualitatively estimate time dependenc... Done
relates to PIPE2D-329 Implement arc subtraction in 1D Open
Story Points: 5
Sprint: 2DDRP-2019 C, 2DDRP-2019 D, 2DDRP-2019 E, 2DDRP-2019 F, 2DDRP-2019 G, 2DDRP-2019 H, 2DDRP-2019 I, 2DDRP-2019 J, 2DDRP-2019 K, 2DDRP-2020 A, 2DDRP-2021 A

 Description   

Create model spectra by combining LAM arc data for different elements. Then for each combined spectrum, perform a PCA or similar analysis of the line intensities.

This analysis would be used in the future to estimate the intensities of lines from actual spectra taken at the summit. Line variability, for example that of the OH lines, can also then be estimated.

See also email 'Re: Homework to IPMU for PFS 2D DRP development' from RHL on 2019-01-08

 Comments   
Comment by naoki.yasuda [ 29/Mar/19 ]

Where can I get various arc data taken at LAM?

Comment by hassan [ 29/Mar/19 ]

You will in general need access to the Princeton cluster. We will try to arrange that for you. In the meantime, Keigo Nakamura does have access and should be able to provide you with some of that data.

Comment by naoki.yasuda [ 29/Mar/19 ]

I used to have an access to the Princeton cluster as "naoki.yasuda" but it seems to be deactivated now. Anyway, I will contact Keigo.

Comment by naoki.yasuda [ 12/Apr/19 ]

Recent arc data are taken with the same exposure times. Should I investigate the variability within these exposures? Or should I collect the data from different dates and compare them?

Comment by ncaplar [ 13/Apr/19 ]

naoki.yasuda Just in case if you are looking for this information, the comments in this ticket: https://pfspipe.ipmu.jp/jira/browse/PIPE2D-359 has the exposure numbers of the arc observations that we did so far. Soon there will be more. Perhaps we should organize a system to track this better....

Comment by hassan [ 17/Apr/19 ]

naoki.yasuda: in the case of LAM data variability does not need to be estimated. I apologize for the confusion. What you can do is to estimate the line intensity for a) a sample of individual arc exposures and b) combine arc exposures corresponding to different lamps, and then demonstrate that line intensities can be estimated for each element.

Once the method of estimated line intensities is demonstrated using arc data, then this method can be applied to 'real' spectra taken at the summit to measure line intensities, and the time-variability of the intensity of certain lines (such as the OH lines).

Comment by rhl [ 17/Apr/19 ]

Hassan and I just chatted.  I expect that the lines will vary as the lines warm up and the pressure changes, and probably the Ar to Hg ratio will change.

So we can model the arc lines from LAM, looking for variability (e.g. higher PCA components of the line intensities vector).  It's possible that the Ne lines will not change, but that means that we'll measure a variability that's consistent with zero — however, I fear that we'll find evidence that the line measurements aren't quite good enough.

Comment by naoki.yasuda [ 25/Apr/19 ]

When I have run reduceArc.py for visit=15539 arm=r, I will get a lot of warnings like the following.
Is this normal and can I ignore them?
Another thing I noticed is that resulting pfsArm-015539-r1.fits has size of 0 byte.
Do I have to run reduceExposure.py for the same data to see resulting spectra?

pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=214) - (GaussCoeffs[XC]=211)| = 3 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=233) - (GaussCoeffs[XC]=236)| = 3 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=389) - (GaussCoeffs[XC]=392)| = 3 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=466) - (GaussCoeffs[XC]=469)| = 3 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=971) - (GaussCoeffs[XC]=972.536)| = 1.53619 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=997) - (GaussCoeffs[XC]=995.46)| = 1.5401 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=1007) - (GaussCoeffs[XC]=1004.86)| = 2.13757 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=1489) - (GaussCoeffs[XC]=1490.59)| = 1.59155 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2033) - (GaussCoeffs[XC]=2034.66)| = 1.66064 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2400) - (GaussCoeffs[XC]=2397.87)| = 2.12793 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2423) - (GaussCoeffs[XC]=2420)| = 3 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2461) - (GaussCoeffs[XC]=2458.3)| = 2.7002 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2537) - (GaussCoeffs[XC]=2540)| = 3 >= 1.5 => Skipping line
pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2679) - (GaussCoeffs[XC]=2682)| = 3 >= 1.5 => Skipping line

 

Comment by ncaplar [ 26/Apr/19 ]

Hopefully this will be helpful for price, especially given he can easily access data on tigeress. This is done with pfs_pipe2d 5.0-10-gff9a74e and checkout of tickets/PIPE2D-404.

I noticed the same problem with e.g. reduceArc.py /tigress/ncaplar/ReducedData/NeonApr_2019 --calib /tigress/ncaplar/ReducedData/NeonApr_2019/CALIB --rerun Apr25_2019/arc --id visit=16292..16297 arm=r -j 20 -c reduceExposure.isr.doFlat=False

The first of the pfsArm files will have 0 bytes, e.g., see /tigress/ncaplar/ReducedData/NeonFeb_2019/rerun/Apr23_2019/arc/pfsArm/2019-02-16/v0012355.

But the others (e.g., /tigress/ncaplar/ReducedData/NeonFeb_2019/rerun/Apr23_2019/arc/pfsArm/2019-02-16/v0012356) are 2 mb.

Comment by ncaplar [ 26/Apr/19 ]

Same when running reduceArc.py /tigress/ncaplar/ReducedData/KrFeb_2019 --calib /tigress/ncaplar/ReducedData/KrFeb_2019/CALIB --rerun Apr25_2019/arc --id visit=13051..13056 -j 20.

This creates empty /tigress/ncaplar/ReducedData/KrFeb_2019/rerun/Apr25_2019/arc/pfsArm/2019-02-27/v0013051, while others seem to be full.

Comment by naoki.yasuda [ 26/Apr/19 ]

The following is the complete list of commands to reproduce the problem of 0 byte pfsArm file
on gfarm.ipmu.jp. I have used master branch for datamodel, drp_stella, and pfs_pipe2d and tickets/PIPE2D-404 branch for obs_pfs.

export REPO_DIR=/gpfs02/work/yasuda/PFS/repo

echo "lsst.obs.pfs.PfsMapper" > $REPO_DIR/_mapper
ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-04-04/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010
ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-04-05/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010

ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $OBS_PFS_DIR/pfs/camera/detectorMap-sim-1-r.fits --mode=copy --validity 1000

constructBias.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/bias --id visit=15390..15396:2^15397^15399^15402^15403^15406..15410:2^15411^15414^15415^15417 arm=r --batch-type=smp --cores=1
ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/bias/BIAS/*.fits --validity 1000

constructDark.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/dark --id visit=15419^15421^15424..15428:2^15429..15435:2^15438^15440^15441^15443^15446^15447 arm=r --batch-type=smp --cores=1
ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/dark/DARK/*.fits --validity 1000

constructFiberFlat.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/flat --id visit=15529..15538 arm=r --batch-type=smp --cores=1
ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/flat/FLAT/*.fits --validity 1000

constructFiberTrace.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/fiberTrace --id visit=15529..15538 arm=r slitOffset=0.0 --batch-type=smp --cores=1
ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/fiberTrace/FIBERTRACE/*.fits --validity 1000

reduceArc.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/arc --id visit=15539 arm=r
Comment by price [ 27/Apr/19 ]

Thanks. I was able to reproduce the problem, tracked it down and fixed it on PIPE2D-411. Please use the tickets/PIPE2D-411 branch of drp_stella to generate the pfsArm files.

Comment by naoki.yasuda [ 27/Apr/19 ]

I confirmed the psfArm file will be created. But the results does not look like promising.

This figure is made by pfs.datamodel.drp.PfsArm.plot().
I have processed one Hg-Ar arc file (visit=15749) with a command

reduceArc.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/arc --id visit=15749 arm=r -c reduceExposure.isr.doWrite=True

The output of the command says wavelength calibration is going well.

reduceArc.calibrateWavelengths INFO: FiberId  650, rms 0.013309 nm (0.156 pix) from 149 (0.102334 nm = 1.200 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 650: wavelength correction 0.000000 +/- 0.000000 nm  
reduceArc.calibrateWavelengths INFO: FiberId  588, rms 0.014110 nm (0.165 pix) from 142 (0.040758 nm = 0.477 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 588: wavelength correction 0.000000 +/- 0.000000 nm  
reduceArc.calibrateWavelengths INFO: FiberId  526, rms 0.014441 nm (0.168 pix) from 153 (0.099881 nm = 1.165 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 526: wavelength correction 0.000000 +/- 0.000000 nm  
reduceArc.calibrateWavelengths INFO: FiberId  465, rms 0.014325 nm (0.167 pix) from 146 (0.069775 nm = 0.815 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 465: wavelength correction 0.000000 +/- 0.000000 nm  
reduceArc.calibrateWavelengths INFO: FiberId  402, rms 0.014927 nm (0.174 pix) from 150 (0.113282 nm = 1.324 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 402: wavelength correction 0.000000 +/- 0.000000 nm  
reduceArc.calibrateWavelengths INFO: FiberId  340, rms 0.012951 nm (0.151 pix) from 158 (0.038495 nm = 0.450 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 340: wavelength correction 0.000000 +/- 0.000000 nm  
reduceArc.calibrateWavelengths INFO: FiberId  256, rms 0.013654 nm (0.160 pix) from 158 (0.043300 nm = 0.506 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 256: wavelength correction 0.000000 +/- 0.000000 nm  
reduceArc.calibrateWavelengths INFO: FiberId  193, rms 0.013723 nm (0.160 pix) from 144 (0.090711 nm = 1.059 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 193: wavelength correction 0.000000 +/- 0.000000 nm  
reduceArc.calibrateWavelengths INFO: FiberId   64, rms 0.013223 nm (0.155 pix) from 148 (0.103616 nm = 1.211 pix for 4 reserved points)                                                         
reduceArc.calibrateWavelengths INFO: Fiber 64: wavelength correction 0.000000 +/- 0.000000 nm   
reduceArc.calibrateWavelengths INFO: FiberId    3, rms 0.013612 nm (0.160 pix) from 140 (0.113453 nm = 1.331 pix for 4 reserved points)

 

I have updated flat using dithered flat data (visit=13882..14010). Previously I was using exposed flat data. In addition I have updated fiberTrace using one visit. Previously I was using 10 visits and this results in bad looking trace.

Comment by naoki.yasuda [ 28/Apr/19 ]

I have updated detectormap with the one created by reduceArc.py and rerun reduceArc.py to create pfsArm.

This is better than the previous one using the default detectormap ($OBS_PFS_DIR/pfs/camera/detectorMap-sim-1-r.fits) but I'm not sure this is good enough. It seems one fiber is offset by 2nm.

Comment by price [ 30/Apr/19 ]

naoki.yasuda: Please use the new detectorMaps in branch tickets/PIPE2D-411 of obs_pfs as the starting point, and let me know how that goes.

Comment by naoki.yasuda [ 30/Apr/19 ]

I have updated detectorMap but the overall accuracy seems not improved so much.

The double peaked profile may be come from mismatched fibertrace. So I have tried to make a fibertrace from visit=14000 which have reasonable overlap with visit=15749 which I'm now working on. But for visit=14000 only a half of fibers will be identified (left hand side).

Comment by naoki.yasuda [ 18/Nov/19 ]

How can I access the line lists of each elements from the pipeline?

Comment by price [ 19/Nov/19 ]

Here's an example: https://github.com/Subaru-PFS/drp_stella/blob/master/python/pfs/drp/stella/reduceArcTask.py#L159-L163

Comment by naoki.yasuda [ 25/Nov/19 ]

I have done very preliminary PCA analysis for 18 Xe arc exposures. I'm still not sure this is a required analysis for this ticket. 

PCA_of_Xe_arc.pdf

Comment by naoki.yasuda [ 09/Dec/19 ]

I have checked the dependency of each PC components on fiberId and visit number using the same dataset (Xe, 18 visits)  as in the previous comment. The variation of PC components depend more on fiberId.

Comment by hassan [ 18/Mar/21 ]

I think this specific ticket has gone as far as it can, and newer tickets will be filed to address the outstanding work.

Generated at Sat Feb 10 15:51:26 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.