[#PIPE1D-12] drp_1dpipe does not read pfsObject** file created by PFS spectrum simulator

[PIPE1D-12] drp_1dpipe does not read pfsObject** file created by PFS spectrum simulator Created: 29/Oct/19 Updated: 14/Aug/20 Resolved: 14/Aug/20
Status:	Done
Project:	DRP 1D pipeline
Component/s:	None
Affects Version/s:	None
Fix Version/s:	None

Type:

Task

Priority:

Normal

Reporter:

ishigaki

Assignee:

Pierre-Yves CHABAUD

Resolution:

Done

Votes:

Labels:

None

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Attachments:

myparams.json

myparams_linemeas.json

pfsObject-00000-0,0-000-00000001-01-0x395428ab.fits

pfsObject-001-00001-0,0-0000000000000000-001-0x8cf7641568bdb4ab.fits

Reviewers:

ishigaki

Description

I have run the drp_1dpipe on a pfsObject*** file, which is an output of the PFS spectrum simulator (Version 1.1). I've got following error messages:

---------------------------------------------------------------------------------

(base) miho@thomas:~/PFS/pipeline/GAtest$ drp_1dpipe --workdir /home/miho/PFS/pipeline/GAtest
Can't load spectrum : "Key 'wavelength' does not exist."
Can't load spectrum : "Key 'wavelength' does not exist."
Traceback (most recent call last):
File "/home/miho/anaconda3/bin/process_spectra", line 11, in <module>
load_entry_point('drp-1dpipe==0.12.1', 'console_scripts', 'process_spectra')()
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/process_spectra/process_spectra.py", line 96, in main
return run(args)
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/process_spectra/process_spectra.py", line 279, in run
amazed(args)
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/process_spectra/process_spectra.py", line 260, in amazed
args.lineflux in ['only', 'on'])
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/process_spectra/results.py", line 50, in _init_
self._read_lambda_ranges()
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/process_spectra/results.py", line 107, in _read_lambda_ranges
hdul = fits.open(path)
File "/home/miho/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 151, in fitsopen
lazy_load_hdus, **kwargs)
File "/home/miho/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 390, in fromfile
lazy_load_hdus=lazy_load_hdus, **kwargs)
File "/home/miho/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1039, in _readfrom
fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
File "/home/miho/anaconda3/lib/python3.7/site-packages/astropy/utils/decorators.py", line 503, in wrapper
return function(*args, **kwargs)
File "/home/miho/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 178, in _init_
self._open_filename(fileobj, mode, overwrite)
File "/home/miho/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 555, in _open_filename
self._file = fileobj_open(self.name, IO_FITS_MODES[mode])
File "/home/miho/anaconda3/lib/python3.7/site-packages/astropy/io/fits/util.py", line 388, in fileobj_open
return open(filename, mode, buffering=0)
FileNotFoundError: [Errno 2] No such file or directory: '/home/miho/PFS/pipeline/GAtest/spectra/pfsObject-1234-57011-0,0-000a083d-000002.fits'
— Logging error —
Traceback (most recent call last):
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/scheduler/scheduler.py", line 128, in run
'output-dir': 'output-'})
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/scheduler/local.py", line 82, in parallel
'{}'.format([f.result() for f in futures]))
Exception: A task returned non-zero value. [CompletedProcess(args=['process_spectra', '--spectra-listfile=/home/miho/PFS/pipeline/GAtest/spectralist_4fcgkdvu', '--workdir=/home/miho/PFS/pipeline/GAtest', '--logdir=/home/miho/PFS/pipeline/GAtest/logdir', '--loglevel=WARNING', '--lineflux=on', '--spectra-dir=spectra', '--output-dir=output-0'], returncode=1)]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/miho/anaconda3/lib/python3.7/logging/_init_.py", line 1034, in emit
msg = self.format(record)
File "/home/miho/anaconda3/lib/python3.7/logging/_init_.py", line 880, in format
return fmt.format(record)
File "/home/miho/anaconda3/lib/python3.7/logging/_init_.py", line 619, in format
record.message = record.getMessage()
File "/home/miho/anaconda3/lib/python3.7/logging/_init_.py", line 380, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/home/miho/anaconda3/bin/drp_1dpipe", line 11, in <module>
load_entry_point('drp-1dpipe==0.12.1', 'console_scripts', 'drp_1dpipe')()
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/scheduler/scheduler.py", line 60, in main
return run(args)
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/scheduler/scheduler.py", line 130, in run
logger.log(logging.ERROR, 'Error in process_spectra:', e)
Message: 'Error in process_spectra:'
Arguments: (Exception("A task returned non-zero value. [CompletedProcess(args=['process_spectra', '--spectra-listfile=/home/miho/PFS/pipeline/GAtest/spectralist_4fcgkdvu', '--workdir=/home/miho/PFS/pipeline/GAtest', '--logdir=/home/miho/PFS/pipeline/GAtest/logdir', '--loglevel=WARNING', '--lineflux=on', '--spectra-dir=spectra', '--output-dir=output-0'], returncode=1)]"),)
— Logging error —
Traceback (most recent call last):
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/scheduler/scheduler.py", line 128, in run
'output-dir': 'output-'})
File "/home/miho/anaconda3/lib/python3.7/site-packages/drp_1dpipe-0.12.1-py3.7.egg/drp_1dpipe/scheduler/local.py", line 82, in parallel
'{}'.format([f.result() for f in futures]))
Exception: A task returned non-zero value. [CompletedProcess(args=['process_spectra', '--spectra-listfile=/home/miho/PFS/pipeline/GAtest/spectralist_4fcgkdvu', '--workdir=/home/miho/PFS/pipeline/GAtest', '--logdir=/home/miho/PFS/pipeline/GAtest/logdir', '--loglevel=WARNING', '--lineflux=on', '--spectra-dir=spectra', '--output-dir=output-0'], returncode=1)]

During handling of the above exception, another exception occurred:

------------------------------------------------------------------------------

According to the first line of the messages, it seems that the input pfsObject*** file

does not have a fits header keyword 'wavelength'.

The drp_1dpipe code runs well with no errors for input spectra included in "pfs_datapack_sp8_v2.0".

I attach the input spectrum I'm trying to use (output from the PFS spectrum simulator Ver.1.1).

Comments

Comment by Pierre-Yves CHABAUD [ 29/Oct/19 ]

I propose to implement both in the 1D pipeline and in the spectrum simulator the data model tagged version 5.1.0

Comment by Kiyoto Yabe [ 08/Nov/19 ]

ETC (spectral simulator) will target tag `5.1.5`. Is it possible for you to use this release tag?

Comment by Pierre-Yves CHABAUD [ 13/Nov/19 ]

Data model 5.1.5 is now implement in branch tickets/~~PIPE1D-12~~.

Dear Kiyoto, can you attach here an example of pfsObject produced by ETC ? I would like to test ETC output before merging in develop branch.

Comment by Kiyoto Yabe [ 14/Nov/19 ]

Hi Pierre-Yves CHABAUD

I added an example from the simulator with new datamodel (pfsObject-001-00001-0,0-0000000000000000-001-0x8cf7641568bdb4ab.fits) as an attachment.

Comment by Kiyoto Yabe [ 14/Nov/19 ]

Hi ishigaki

Now I believe you can use the new ETC. At the directory of the repo, please do the following commands:

git pull
git submodule update --init
make
python setup.py install

Or it may be easier to remove the previous repository and get the new one like this:

git clone --recursive https://github.com/Subaru-PFS/spt_ExposureTimeCalculator.git 
cd spt_ExposureTimeCalculator 
make 
python setup.py install

If you get any difficulties, please let me know.

Comment by ishigaki [ 14/Nov/19 ]

Thank you for updating the ETC. The new version of the code was successfully installed and imported. I would like to use this version to obtain simulated spectra for testing.

Comment by Pierre-Yves CHABAUD [ 20/Nov/19 ]

Hi Kiyoto Yabe

Thank you for uploading file. I have a question.

What is the meaning of your MASK values ? (I guess 0 is ok).

Comment by Kiyoto Yabe [ 20/Nov/19 ]

Yes, "0" means OK, and I think there are no masked pixels in the output from the simulator.

Comment by Pierre-Yves CHABAUD [ 20/Nov/19 ]

Ok great!

I've detected "2" values (at the begining and at the end of the file). That's related to "NULL" values in variance (HDU#5 - COVAR[0][0:] ) and "0.0" values in flux (HDU#1 - FLUX extension).

Comment by Pierre-Yves CHABAUD [ 13/Dec/19 ]

Hi Kiyoto Yabe

To be compliant with datamodel, the floating data should be single precision floating value. I detect double precision floating value in your attached file. Can you check on your last development ?

Comment by Kiyoto Yabe [ 14/Dec/19 ]

I only rely on the datamodel package, but that may be a problem on my side. Anyway, I'll take a look and check.

Comment by Kiyoto Yabe [ 16/Dec/19 ]

Pierre-Yves CHABAUD This was my failure and I have pushed the fix on master. Check that again please.

Comment by Pierre-Yves CHABAUD [ 17/Dec/19 ]

Kiyoto Yabe Everything is ok after a last check. Thank you a lot!

Comment by ishigaki [ 21/Jan/20 ]

Thank you so much for your updating the code. I'm now trying to run drp_1dpipe ("develop" branch). For that, I'm trying to install the latest versions of both drp_1d and drp_1dpipe. When I first try to install drp_1d by "pip install .", following the instruction in the README file in its "develop" branch, I've got following messages attached here.

It would be great if you could check the messages and suggest on a possible solution.

Comment by ishigaki [ 21/Jan/20 ]

These are the messages:

(base) miho@thomas:~/PFS/pipeline/drp_1d$ pip install .
Processing /home/miho/PFS/pipeline/drp_1d
Requirement already satisfied: numpy>=1.16.0 in /home/miho/anaconda3/lib/python3.7/site-packages (from pylibamazed==0.7.dev0) (1.16.4)
Requirement already satisfied: astropy>=3.1.1 in /home/miho/anaconda3/lib/python3.7/site-packages (from pylibamazed==0.7.dev0) (3.1.1)
Building wheels for collected packages: pylibamazed
Building wheel for pylibamazed (setup.py) ... error
ERROR: Complete output from command /home/miho/anaconda3/bin/python -u -c 'import setuptools, tokenize;_file='"'"'/tmp/pip-req-build-2odqav55/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file_, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-j6bbrpie --python-tag cp37:
ERROR: running bdist_wheel
running build
running build_ext
building 'pyamazed/_redshift' extension
swigging pyamazed/redshift.i to pyamazed/redshift_wrap.cpp
swig -python -c++ -o pyamazed/redshift_wrap.cpp pyamazed/redshift.i
Unable to find file 'pyamazed/redshift.i'.
error: command 'swig' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for pylibamazed
Running setup.py clean for pylibamazed
Failed to build pylibamazed
Installing collected packages: pylibamazed
Running setup.py install for pylibamazed ... error
ERROR: Complete output from command /home/miho/anaconda3/bin/python -u -c 'import setuptools, tokenize;_file='"'"'/tmp/pip-req-build-2odqav55/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file_, '"'"'exec'"'"'))' install --record /tmp/pip-record-pbvf_au7/install-record.txt --single-version-externally-managed --compile:
ERROR: running install
running build
running build_ext
building 'pyamazed/_redshift' extension
swigging pyamazed/redshift.i to pyamazed/redshift_wrap.cpp
swig -python -c++ -o pyamazed/redshift_wrap.cpp pyamazed/redshift.i
Unable to find file 'pyamazed/redshift.i'.
error: command 'swig' failed with exit status 1
----------------------------------------
ERROR: Command "/home/miho/anaconda3/bin/python -u -c 'import setuptools, tokenize;_file='"'"'/tmp/pip-req-build-2odqav55/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file_, '"'"'exec'"'"'))' install --record /tmp/pip-record-pbvf_au7/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-req-build-2odqav55/

Comment by Pierre-Yves CHABAUD [ 21/Jan/20 ]

ishigaki The missing file "redshift.i" is no more in the "pyamazed" folder. Starting from a fresh install (from git clone) could solve your problem.

Comment by ishigaki [ 22/Jan/20 ]

Thanks for your prompt response! I understand there was an issue in installation and found that it was a problem of my environment, where the new installations of both drp_1d and drp_1dpipe were not reflected to the */site-packages/ directory. As you suggests, the older version was running. When I delete the older versions in that directly and freshly install the drp_1d and drp_1dpipe (after git checkout develop), the installations were both successful. Thanks again for your suggestion!

Comment by ishigaki [ 10/Feb/20 ]

Thanks again for your support on the drp_1dpipe. The code has been successful in running on a stellar spectrum, which has been created by the latest version of the PFS simulator provided by Yabe-san. I'm trying to modify the parameter files (*.json) to be suitable for the stellar velocity measurement. Could you provide me with an example *.json file to be used for a stellar spectrum or

with suggestions on the relevant parameters? Specifically, I was trying to switch to the "Templete" method from the "Linemodel" method and to search for stellar templates. The current parameter files are attached below:

Comment by ishigaki [ 10/Feb/20 ]

Attachement

myparams.json

^{myparams_linemeas.json}

Generated at Sat Feb 10 15:35:28 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.

[PIPE1D-12] drp_1dpipe does not read pfsObject** file created by PFS spectrum simulator Created: 29/Oct/19 Updated: 14/Aug/20 Resolved: 14/Aug/20