[PIPE2D-1290] Define metrics to evaluate the extraction accuracy Created: 29/Aug/23  Updated: 14/Oct/23

Status: Open
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: Satoshi Hamano Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: EDR, QualityAssurance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File ____________________________2023-09-07_15.59.09_720.png     PDF File 97821 b1_calexp.pdf     PDF File 97821 b1_chi-2.pdf     PDF File 97821 b1_chi_coolwarm.pdf     PDF File 97821 b1_chistats-13.pdf     PDF File 97821 b1_chistats_v2.pdf     PDF File 97821 b1_chistats_v3.pdf     PDF File 98042 r1_mean.pdf     PDF File 98042 r1_median.pdf     PDF File 98042 r1_std.pdf     PDF File 98042 r1_subtracted-divided.pdf     PDF File 98042 r1_subtracted.pdf    
Issue Links:
Relates
relates to PIPE2D-1315 Define metrics for the detectorMap QA Open

 Description   

The metrics to evaluate the extraction accuracy should be defined to make a QA script.



 Comments   
Comment by Satoshi Hamano [ 30/Aug/23 ]

Yabe-san's notebook produces the extraction residual image (calexp - image simulated from traces and pfsArm) and the residual/calexp image.

 

98042 r1_subtracted.pdf

98042 r1_subtracted-divided.pdf

Comment by Satoshi Hamano [ 30/Aug/23 ]

Also, the stats (mean, median, and std dev) per 7x7 pix cell are plotted for the residual image.

 

98042 r1_mean.pdf

98042 r1_median.pdf

98042 r1_std.pdf

Comment by Satoshi Hamano [ 30/Aug/23 ]

I'd like to add a function to calculate the rms of the extraction residual for each fiber and for the whole detector.

Comment by Satoshi Hamano [ 07/Sep/23 ]

Residual/error image. np.sqrt(calexp.variance.array) is used as the error.

97821 b1_chi-2.pdf

Comment by Satoshi Hamano [ 07/Sep/23 ]

The chi^2 and the calexp average of each fiber are plotted. The fibers of bright objects and its adjacent fibers tend to have large chi^2. The example images of the residual/error, calexp, and mask for the fiber of the bright object is attached. The bumpy pattern is created by rearranging the distorted fiber trace into a straight line.

97821 b1_chistats-13.pdf

  

 

Comment by Masayuki Tanaka [ 07/Sep/23 ]

Thank you, Hamano-san! Can I make a few suggestions?

  • In 97821..chi-2.pdf, could you change the color scheme so that positive residual are red and negative residuals are blue?
  • In addition to chi^2, I would like to see chi because we are interested in the sign.
  • Could we show sigma(chi) for each fiber? We can get sigma from the interquartile range, |Q3-Q1| /1.354, for each fiber. If things behave, we expect sigma~1. Deviations from unity is a bad sign.
Comment by Satoshi Hamano [ 08/Sep/23 ]

Thank you for your comments. I updated the figures.

97821 b1_chi_coolwarm.pdf

97821 b1_chistats_v2.pdf

Comment by Satoshi Hamano [ 12/Sep/23 ]

The data points of different targetType are plotted with different colors.

97821 b1_chistats_v3.pdf

Generated at Sat Feb 10 16:05:33 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.