[INSTRM-484] Update FITS headers based on Subaru feedback of MCS example PFSC00005800 Created: 10/Sep/18  Updated: 29/May/23  Resolved: 29/May/23

Status: Done
Project: Instrument control development
Component/s: ics_actorkeys
Affects Version/s: None
Fix Version/s: None

Type: Epic Priority: Normal
Reporter: hassan Assignee: cloomis
Resolution: Done Votes: 0
Labels: FITS, ICS
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to INSTRM-130 Refactor CCD header generation Done
relates to INSTRM-354 On missing, unconverted, or invalid n... Done
relates to INSTRM-522 Move date/time and frame card creatio... Done
Epic Name: INSTRM-484

 Description   

In the email [PFS-Soft-Subaru:551] Hisanori Furusawa raised the following comments on the MCS FITS file example supplied by cloomis. These comments should be reviewed and implemented.

    • General comments to the whole headers

1. Could you provide a keyword dictionary that describes a range of values
and value format for each keyword?

2. We recommend to add WCS to the 2nd HDU (compressed image extension),
since it would benefit PFS commissioning as well as users, considering
this is the only data representing fiber positions on a 2-d plane without
dispersion.

    • Comments to HDU1

3. Missing mandatory keywords (labeled 'Common' in the current Subaru Basic
FITS header dictionary)

DET-TMP
GAIN (recommend a typical value here, and use W_XXX keywords for
different amplifiers)
LST

*1) except NAXIS1, NAXIS2; these two are always required in the current
rule,
but the new rule will not require this if NAXIS=0)
*2) missing keywords relevant to image extension are listed in the HDU2
section

4. Check value range for the below 4 keywords:

OBS-ALOC
OBS-MOD
FOC-POS
IMR-TYPE

We need to confirm that these keywords follow the rule and make sure
values are surely registered in the rule.

5. Inconsistent inline comment for UT and HST keywords

should be "HH:MM:SS.SSS ..." not "..SS.SS ..."
UT = '00:25:27.402' / HH:MM:SS.SS typical UTC at exposure
UT-STR = '00:25:27.402' / HH:MM:SS.SS UTC at exposure start
UT-END = '00:25:28.402' / HH:MM:SS.SS UTC at exposure end
HST = '14:25:27.402' / HH:MM:SS.SS typical HST at exposure
HST-STR = '14:25:27.402' / HH:MM:SS.SS HST at exposure start
HST-END = '14:25:28.402' / HH:MM:SS.SS HST at exposure end

6. Registration of the new keywords to the Basic FITS header dictionary
is required:

M2-POS3, M2-ANG3

7. Instrument-specific keywords (W_*) are not separated from a set of
basic keywords by appropriate comment lines

e..g, W_VISIT, W_M2OFF[1-3]

The new rule will allow similar/related basic & W_* keywords are
grouped in a place without separation.
Still appropriate comment lines are recommended.

8. If MCS camera uses any filter in the light path, we recommend to add
keyword 'FILTER01'

9. M2-TYPE should be 'IR'. Subaru plans to fix Gen2 for this.

10. Inline comment is required for these keywords:

EXTEND, DET-ID

11. Inline comments for FRAMEID and EXP-ID are the same.
More appropriate comments to describe the difference would be preferred.

12. General comment to Inline comment:
Inline comment wording should be derived from the Basic FITS header
dictionary for basic keywords whenever possible.
For archive users, the same keywords should have the same comments across
the instruments and data kinds.

13. EXP-ID should have 'E' in the 4th letter

14. DATA-TYP=ACQUISITION is recommended. To be discussed with PFS team

15. Make sure DETECTOR should be an identifier for a detector
If replaced, it will have a different value.

16. Wrong keyword names:

ADC-TYP --> ADC-TYPE
IMG-TYP --> IMG-TYPE

17. Inconsistent format of values

Please have them reviewed by PFS team.

ADC-STR (F20.3)
AZIMUTH, ALTITUDE (F20.5)
AIRMASS (F20.3)
DOM-PRS DOM-WND (F20.2)
EXPTIME (F20.2)
FOC-VAL (F20.3)
INST-PA (F20.3)
INR-STR, IMR-STR (F20.3)
M2-POS[1-3], M2-ANG[1-3] (F20.3)
MJD, MJD-STR, MJD-END (F20.8)
OUT-PRS OUT-WND (F20.2)
SEEING (F20.2)
TRANSP (F20.3)
ZD (F20.5)

18. Unnecessary keywords may be omitted
It might be worth considering to remove them to avoid potential confusion.

e.g., IMR* (if IMR is not used) , ADC* (if ADC is not used)

    • Comments to HDU2

19. Missing mandatory keywords (labeled 'Common' in the current Subaru
Basic FITS header dictionary)

BIN-FCT[1-2]
BLANK
BUNIT
WCS keywords - we recommend to include the WCS keywords for MCS data
CRPIX[1-2], CRVAL[1-2], CTYPE[1-2], CUNIT[1-2]
and (CD[1-2] or (PC[1-2][1-2], CDELT[1-2])) etc.

*) relevant to image extension only

20. Inline comment is required for these keywords:

BSCALE, BZERO

21. Subaru will update FITS rule including multi-extension FITS

22. There seem no problem with keywords for Tile Image Compression

23. Please make sure if ZPCOUNT, ZGCOUNT are necessary.

24. Please make sure keywords relevant to 'image data' as they are
in this HDU, in addition to the keywords for Tiled-image compression.
This would include:
BZERO, BSCALKE, WCS keywords etc.



 Comments   
Comment by hassan [ 17/Sep/18 ]

Child tasks for this epic and corresponding priorities will be created within the next few days.

Comment by cloomis [ 02/Oct/18 ]

1. INSTRM-493
2. INSTRM-388
3. Sure.
4. Do we get this from the NAOJ FITS table? From Subaru?
5. INSTRM-493
6. So we do not need to do anything?
7. So we do not need to do anything?
8. If there is no filter, what should we put in?
9. So we do not need to do anything?
10. Sure.
11. Sorry – I missed the distinction between the two cards. I did not realize there was a logical "E" filetype.
12. INSTRM-493
13. See 11
14. TBD
15. Good. Makes sense.
16. OK, INSTRM-493
17. INSTRM-493
18. TBD
19. INSTRM-493, INSTRM-388
20. INSTRM-493
21. So we do not need to do anything?
22. So we do not need to do anything?
23. INSTRM-388

Comment by hassan [ 03/Oct/18 ]

Hisanori Furusawa or a member of the Subaru/NAOJ teams: could you please respond to the questions Craig has added in the above section, for your comments 4, 6, 7, 8, 9, 21 and 22? Thanks.

Comment by Hisanori Furusawa [ 04/Oct/18 ]

1. INSTRM-493 (dictionary)
2. INSTRM-388 (WCS)
3. Sure. (missing keywords)
4. Do we get this from the NAOJ FITS table? From Subaru? (OBS-ALOC, OBS-MOD, FOC-POS)
--> needs to consult with TSC&Gen2(Eric maybe first?) for the current system.
and for the allowed value range, refer to the Subaru FITS rule (https://www.naoj.org/Obserinvg/fits and the new rule being prepared; Onodera-san).

5. INSTRM-493 (inconsistent comment)

6. So we do not need to do anything?
--> Basically no. But recommend to check with Onodera-san,
if you really need this keyword as 'Subaru's basic keywords'.

7. So we do not need to do anything?
--> Please check with Onodera-san to confirm the proposed order
of FITS cards is acceptable in the new rule.

8. If there is no filter, what should we put in?
--> FILTER01 is optional, so not required if the filter is never used.
FILTER01 = 'None' might be another option.

9. So we do not need to do anything?
--> Basically no. Onodera-san and Eric will respond.
Please make sure this is done in the engineering.

10. Sure. (Inline command for EXTEND, DET-ID)
11. Sorry ? I missed the distinction between the two cards. I did not realize there was a logical "E" filetype.
--> Likely I don't get this comment. Have you addressed the difference?
If not, could you elaborate your confusion so we could help?

12. INSTRM-493 (RE: inline comment)
13. See 11
--> Likewise.
14. TBD
--> let's check with the new rule.
15. Good. Makes sense.
16. OK, INSTRM-493
17. INSTRM-493
18. TBD
19. INSTRM-493, INSTRM-388
20. INSTRM-493
21. So we do not need to do anything? (multi-extension FITS)
--> Please communicated with Onodera-san to give him feedback and
establish the new rule.
22. So we do not need to do anything? (Tiled image)
--> Basically, no, but recommend to confirm with Onodera-san if
we decide to go for it.
23. INSTRM-388

Comment by Hisanori Furusawa [ 04/Oct/18 ]

The above is my quick response, but some of them apparently needs Onodera-san & Eric's attention. I will check with them, too.

Comment by hassan [ 09/Oct/18 ]

cloomis: can you look at the additional questions from Hisanori Furusawa?
Hisanori Furusawa: any news from Onodera-san and Eric?

Comment by monodera [ 09/Oct/18 ]

I'm looking into it.  Sorry for not very prompt response.

Comment by monodera [ 10/Oct/18 ]

FYI, the current table is available here: https://docs.google.com/spreadsheets/d/1chczRkX_Thif3FYD8ScgN7CgZQv1XNjcysYzEqPqfl4/edit?usp=sharing

I'm still working on the translation of the Subaru FITS rule https://www.naoj.org/staff/monodera/subaru-fits/ja/header/regulation/, but I believe Google translate works nicely. 
 

3. I'll add a description for NAXIS[1-2] to clarify that it's not mandatory when NAXIS=0.

4. The ranges for keywords listed here are the following.

OBS-ALOC: {Observation,Standby}
OBS-MOD: {IMAG,SPEC,IPOL,SPOL}_* (e.g., IMAG for HSC)
FOC-POS: {Cassegrain,COUDE,Nasmyth-IR,Nasmyth-Opt,PRIME}
IMR-TYPE: {BLUE,RED,IR,NONE}

6. I don't think there are any problems to include M2-POS3 and M2-ANG3 as optional keywords in the list. I'll work on it.

7. I don't see major problems which potentially cause confusion.

9. I will talk to Eric. As long as the value is taken from Gen2 as "FITS.SBR.M2-TYPE", it will be corrected automatically. We will let you know when we will make a change.
 
14.

DATA-TYP = {ACQUISITION,BIAS,COMPARISON,DARK,DOMEFLAT,DOMEFLAT_ON,DOMEFLAT_OFF,FLAT,FOCUSING,OBJECT,SKYFLAT,STANDARD,STANDARD_STAR,TEST}

16.
> IMG-TYP --> IMG-TYPE

Is it DATA-TYP or IMR-TYPE?

21. I still don't have clear description of the rule for MEF. What I have currently in mind is that keywords common to all extensions are put in the primary header, and other keywords relevant to each extension are present in the corresponding header.

22. So far I don't see major problems for keywords used for tiled-compression, but see items 23 and 24. 

Comment by cloomis [ 10/Oct/18 ]

Thanks for the spreadsheet! All the cards in one machine-parseable file, yay!

A couple of immediate questions:

4. OBS-MOD: PFS is a spectrograph, so the instrument and the science detectors are always SPEC. Should the MCS and AGCC images be IMAG? [Also, the names listed in the spreadsheet are the long form ("Imaging" vs. "IMAG")]
4. IMR-TYPE: what will be the right value for PFS?

For both of those, can I just use whatever value is in the keyword? Or should that be checked against the legal/expected values?

14. DATA-TYP: So for the science detectors, we should use "COMPARISON" for arc lamp exposures, and "OBJECT" for science exposures? And for the MCS, "OBJECT" for the exposures of the illuminated PFI? Or "ACQUISITION"?

And a general question:

17. There are various tables on the NAOJ FITS site showing per-instrument formats. Should I only use the ones in the spreadsheet you are developing?

Comment by eric [ 10/Oct/18 ]

LST

  • Gen2 status alias FITS.SBR.LST

OBS-ALOC

  • Gen2 status alias FITS.PFS.OBS-ALOC
  • will be "Observation" most of the time
  • is set when the instrument is allocated
  • Gen2 currently does not "unset" this value or ever set it to "Standby"

OBS-MOD

  • Gen2 status alias FITS.PFS.OBS-MOD
  • will be set according to the directory containing the observation scripts
  • check the rules promulgated by Onodera-san about acceptable values

FOC-POS

  • Gen2 status alias FITS.PFS.FOC-POS
  • This alias is currently missing from Gen2, although there are ones for
    other instruments
  • should be "Prime" most of the time?
  • check the rules promulgated by Onodera-san about acceptable values
  • Gen2 will add it

IMR-TYPE

  • Gen2 status alias FITS.SBR.IMR-TYPE
  • "NONE" if there is no image rotator being used

UT, UT-STR, UT-END

  • Gen2 status alias FITS.SBR.UT

HST, HST-STR, HST-END

  • Gen2 status alias FITS.SBR.HST

M2-POS3, M2-ANG3

  • Gen2 status alias FITS.SBR.M2-POS3 and FITS.SBR.M2-ANG3

FILTER01

  • Should be set to "None" if no filter used. Helpful for some displays
    like that used in integgui2

M2-TYPE

  • Gen2 status alias FITS.SBR.M2-TYPE
  • Which of the current Gen2 values are incorrect?

 

Comment by cloomis [ 12/Oct/18 ]

monodera eric
FITS.SBR.INSROT vs. FITS.SBR.IMGROT? Is IMGROT for the case that the detector is rotated w.r.t. the instrument or the instrument is rotated w.r.t. the rotator (whatever that might mean)?

Both descriptions contain "Typical angle of instrument rotator during the exposure (degree)." Both cards are optional. We care about one sensible value; which should PFS use and put in the header?

Comment by Hisanori Furusawa [ 12/Oct/18 ]

IMGROT is for the field rotation w.r.t. the sky at Nasmys foci, while INSROT is that for Cassegrain's, so only the INSROT should matter for MCS.

Comment by eric [ 12/Oct/18 ]

Concur with Furusawa-san's comment.

 

Comment by cloomis [ 16/Oct/18 ]

What is the correct way to determine if the system (telescope focus and rotator, at least) is configured correctly for PFS, and thus we can use the associate keywords to fill in the focus and rotator cards, generate valid WCS cards, and calculate PFI geometry? I'd like to avoid putting silly values in headers, and to be able to warn about unexpected configurations.

FITS.SBR.TELFOCUS == 'P_OPT2' looks like a plausible test, from looking at both the spreadsheet and the HSC headers. But is that sufficient to tell us that both the FITS.SBR.INSROT and FITS.SBR.FOC-VAL will be correct for PFS? 

If so, can we then also drop IMR-TYP?

Comment by eric [ 16/Oct/18 ]

I think that is a reasonable test.  If it is set so, then the other keywords can be assumed to be correct for the configuration (although this depends on the day and night crews setting the telescope up correctly from the TWS side).

As for dropping IMR-TYP, I think Onodera-san or Furusawa-san should comment on that.  I'd say if HSC doesn't include it then PFS shouldn't need to either.  But that's just an opinion.

Comment by Hisanori Furusawa [ 16/Oct/18 ]

I believe that we should simply drop IMR-TYP for non-Nasmyth instruments.

Comment by monodera [ 16/Oct/18 ]

I agree to drop IMR-TYPE and other keywords related to the image rotator.

For checks of TELFOCUS, you can test allowed values like INSROT > a && INSROT < b to be more conservative.

Comment by cloomis [ 17/Oct/18 ]

Please look at https://www.dropbox.com/s/frj8pd1m51b6oy1/PFSC00011400.fits. [I should have filled it with 0s instead of the camera noise. Sorry. ]

1

  • No keyword dictionary w/formats yet. Close.

2

  • MCS-to-PFI only, created by astropy.coordinates.
  • Linear only, for now.
  • Not sure what CTYPEn to use; the standard celestial projections are obviously wrong.

3

  • Done, but we do not populate DET-TMP yet. INSTRM-527

4

  • IMR-TYPE dropped.
  • OBS-MOD covered by INSTRM-528.
  • OBS-ALOC and FOC-POS taken directly from Gen2.

5

  • Changed comments to have either "[DMS]" or "[HMS]" prefixes.

8

  • MCS has no filter. Can add FILTER01='None' if desired.

10

  • DET-ID dropped
  • EXTEND fixed.

11 Fixed

12

  • I generally followed the NAOJ comments and rules. Except that units are always written as bracketed prefixes, following the FITS 4.0 reference. The WCS cards and most of the core FITS cards have the astropy-generated comments.

13 Fixed

14 Fixed

15

  • Fixed, except that we do not know the values yet.

16

  • ADC-TYPE fixed
  • IMG-TYP dropped.

17

  • See #1.

18

  • Have dropped some.

20 Fixed

24 Fixed

Comment by cloomis [ 24/Oct/18 ]

The MCS and P_OPT2 are mounted on the telescope, and PFS is set as the instrument.

  • M2_TYPE and TELFOCUS are both "P_OPT2". Makes sense.
  • FOC-POS and IMR-TYP are both "##NODATA##". Is this expected?
Comment by eric [ 24/Oct/18 ]

IMR-TYP is most likely undefined for this combination, since it refers to the NS_OPT image rotator.  From the comments above, I thought you had dropped it as suggested by Furusawa-san.

 

FOC-POS may need to be updated to handle this configuration.  We'll look into it.

Comment by eric [ 24/Oct/18 ]

Re: FOC-POS.  See my comment above: 

  • Gen2 status alias FITS.PFS.FOC-POS
  • This alias is currently missing from Gen2, although there are ones for
    other instruments

So, we'll add it.

 

Comment by eric [ 24/Oct/18 ]

Added FITS.PFS.FOC-POS

Comment by cloomis [ 24/Oct/18 ]

Oops. My mistake on IMR-TYP – somehow I did not drop it as promised.
Thanks on FITS.PFS.FOC-POS

Comment by Hisanori Furusawa [ 26/Oct/18 ]

congrats to your successful engineering! sorry for my delay, but here is a quick summary of my reviewing PFSC00011400.fits. I see big improvements have been done - thank you for your efforts, so the remaining are mostly something to be tuned with Gen2/telescope and/or new rule.
assumes IMR-TYP will be dropped.

1. mandatory keywords:
LST is missing

2. optional but suitable keywords:
Do we agree to remove FILTER01?

3. check valid value ranges (and default values)
OBJECT, OBS-MOD, FOC-POS; now all ##NODATA##

4. value format
DET-TMP has string 'NaN', should be F20.2.
plus all keywords listed in my original item 17 (Inconsistent format of values) still need to be treated.

5. validity of WCS keywords is TBD
but it is great that now we have them.

Comment by hassan [ 10/Dec/18 ]

1. cloomis to add LST
2. FILTER01 will be assigned 'NONE'
3. FOC-POS fixed already. OBS-MOD needs to be updated to one of the available directories.
4. See INSTRM-354
5. For MCS is ok, but separate tickets will be raised for science detectors (lambda versus slit hole ID and AGC (ra/dec)

Comment by yuki.moritani [ 29/May/23 ]

Comments on newer sample is discussed under INSTRM-1531 , so I close this ticket.

Generated at Sat Feb 10 16:25:30 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.