We have investigated further, and it happens that the 1D Pipeline actually uses the mask values provided by the conversion parser in the pipeline. We have actually discovered three issues from the file provided by Kiyoto Yabe in PIPE1D-51Bad redshift determination with version 0.22.0? :

• A mask value is provided in two extensions : the #2 and the 4th column of the table in extension #8 (which is the resampled table), but the two values are not compatible (see attached pictures).
• In the pipeline, the #2 extension is used, but in this one, the pixels around 1229 nm (where a fake emission line is used to give a redshift measure) are not masked 
• In the table of extension #8, the mask is >0 above 600 nm, which would indicate that only the blue arm spectrum is useful.

These problems explain the bad redshift measurement, and mask values have to be corrected

Thank you for the investigation. What about this one (pfsObject-00001-00000-0,0-0000000000000318-006-0x7ace412dc05971ff.fits)? This is a product of the latest pipeline. The mask values look very different from the previous one.

Indeed the values are different for the extension  #2 (#8 seems to be identical), did you try DRP-1D on this one? we'll try our version on our side

The mask seems to be correct with respect to strong features around 1220-1230 nm

and now we measure the correct redshift, which shows that everything seems OK. On the other side, the mask in the resampled data table are still weird. Should we raise a PIPE-2D ticket ?

Yes, the recent comparison (check_redshift_determination.pdf) shows better results. The redshifts are consistent for that object. I still see some discrepancy. I'll take a look into which objects are so.

Also here I invite price for discussion on mask values in pfsObject.

the value of the std deviation of 0.2 indicates that there are still a large number of catastrophic failures. Let's check that ...

Ali Allaoui

The latest weekly products (at IPMU) are here:

https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/

I also added reference redshift and [OII] luminosity in csv file:

https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.csv

Original input spectra are not in the weekly products but I can send you specific objects if you need them.

I have recently discovered and fixed some bugs in the generation of the mask. I hope these fixes will make it to a weekly release soon.

With the latest run (2D=w.2021.40 and 1D=0.24.0), the comparison between input and output redshifts is like this:

check_redshift_determination_20211006.pdf

The result looks obviously strange. Could you check this in your side? pfsObjects are here:

https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/INTEGRATION/rerun/integration/pipeline/pfsObject/00001/00000/0,0/

Input redshifts are here:

https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.csv

For the spectrum I looked, the noise level does not make sense; it is about 10^12 while the spectrum has a (normal) 10^16 average value, hence the noise weighted fit cannot work properly. On this plot Weekly_run_noise_level.png the good solution is found but only on the 3rd rank.

The weekly tests in this morning showed that the comparisons

https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.pdf

https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e_extended/check_redshift_determination.pdf

were fair enough. So I think we can close this ticket and post any further issues in different ticket (maybe PIPE1D-38?).