[PIPE2D-704] Header missing END card in pfsObject fits Created: 27/Jan/21 Updated: 28/Jan/21 Resolved: 27/Jan/21 |
|
| Status: | Won't Fix |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | Kiyoto Yabe | Assignee: | price |
| Resolution: | Won't Fix | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
Testing the new master branch `w.2021.04`, I got the following error when I read `pfsObject` by using `astropy`, which does not exist in the previous version. This seems to affect 1D DRP as well because they also use `astropy`. I could read the data themselves correctly, so we can ignore this message, but ...
--------------------------------------------------------------------------- OSError Traceback (most recent call last) <ipython-input-26-2f2f0d5421bb> in <module> 13 14 for pfsObject in pfsObjects: ---> 15 with fits.open(pfsObject) as hdul: 16 flx = hdul['FLUX'].data 17 var = hdul['COVAR'].data[0] /usr/local/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in fitsopen(name, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 149 150 return HDUList.fromfile(name, mode, memmap, save_backup, cache, --> 151 lazy_load_hdus, **kwargs) 152 153 /usr/local/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in fromfile(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 388 return cls._readfrom(fileobj=fileobj, mode=mode, memmap=memmap, 389 save_backup=save_backup, cache=cache, --> 390 lazy_load_hdus=lazy_load_hdus, **kwargs) 391 392 @classmethod /usr/local/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in _readfrom(cls, fileobj, data, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs) 1062 1063 # Make sure at least the PRIMARY HDU can be read -> 1064 read_one = hdulist._read_next_hdu() 1065 1066 # If we're trying to read only and no header units were found, /usr/local/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py in _read_next_hdu(self) 1137 fileobj.seek(offset, os.SEEK_SET) 1138 -> 1139 hdu = _BaseHDU.readfrom(fileobj, **kwargs) 1140 except EOFError: 1141 self._read_all = True /usr/local/lib/python3.7/site-packages/astropy/io/fits/hdu/base.py in readfrom(cls, fileobj, checksum, ignore_missing_end, **kwargs) 326 hdu = cls._readfrom_internal(fileobj, checksum=checksum, 327 ignore_missing_end=ignore_missing_end, --> 328 **kwargs) 329 330 # If the checksum had to be checked the data may have already been read /usr/local/lib/python3.7/site-packages/astropy/io/fits/hdu/base.py in _readfrom_internal(cls, data, header, checksum, ignore_missing_end, **kwargs) 391 if header is None: 392 header_offset = data.tell() --> 393 header = Header.fromfile(data, endcard=not ignore_missing_end) 394 hdu_fileobj = data 395 data_offset = data.tell() # *after* reading the header /usr/local/lib/python3.7/site-packages/astropy/io/fits/header.py in fromfile(cls, fileobj, sep, endcard, padding) 445 446 return cls._from_blocks(block_iter, is_binary, sep, endcard, --> 447 padding)[1] 448 finally: 449 if close_file: /usr/local/lib/python3.7/site-packages/astropy/io/fits/header.py in _from_blocks(cls, block_iter, is_binary, sep, endcard, padding) 514 # TODO: Pass this error to validation framework as an ERROR, 515 # rather than raising an exception --> 516 raise OSError('Header missing END card.') 517 518 header_str = ''.join(read_blocks) OSError: Header missing END card. |
| Comments |
| Comment by Kiyoto Yabe [ 27/Jan/21 ] |
|
Sorry, I was stupid, to say the least. I wrongly read pickle files in the same directory, so that was a natural behavior... Sorry again. |
| Comment by price [ 27/Jan/21 ] |
|
If you use the datamodel package to read the files, this mistake should never happen, and you're saved from having to know the file formats. |
| Comment by rhl [ 28/Jan/21 ] |
|
While I agree with Paul, I read this ticket as saying that astropy thinks that the file is invalid. That seems odd, but if it's true we need to fix it. Can Kiyoto Yabe clarify? |
| Comment by price [ 28/Jan/21 ] |
|
My reading is that astropy failed to read the file because it's actually a pickle file (containing an LSF). This is expected behaviour, and there's not a lot we can do about it until we upgrade the LSF datamodel. |
| Comment by rhl [ 28/Jan/21 ] |
|
Got it. As you were! |