[PIPE1D-75] pfsZCandidates should always have 10 HDUs Created: 11/Oct/23  Updated: 25/Jan/24  Resolved: 25/Jan/24

Status: Done
Project: DRP 1D pipeline
Component/s: drp_1dpipe
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: Ali Allaoui Assignee: Ali Allaoui
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File pip-list.txt     Text File valid_report.txt    
Reviewers: Kiyoto Yabe

 Comments   
Comment by Kiyoto Yabe [ 06/Jan/24 ]

I'm now testing version 0.46.0. In the validation test, I got the following error:

Running drp_1dpipe 0.46.0
Traceback (most recent call last):
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 73, in json_schema_check
    validate(instance=self.accessor.parameters, schema=self.jsonSchema, resolver=resolver)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: False schema does not allow 3900Failed validating None in schema['properties']['lambdarange']:
    FalseOn instance['lambdarange'][0]:
    3900During handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/process_spectra", line 8, in <module>
    sys.exit(main())
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 269, in main
    return main_method(config)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 243, in main_method
    amazed(config)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 155, in amazed
    context, user_parameters = _setup_pass(config)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 136, in _setup_pass
    context = Context(vars(config), Parameters(params))
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/Parameters.py", line 54, in __init__
    Checker(accessor).check()
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 61, in check
    self.json_schema_check()
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 75, in json_schema_check
    raise APIException(
pylibamazed.Exception.APIException: INVALID_PARAMETER_FILE:False schema does not allow 3900
Traceback (most recent call last):
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 73, in json_schema_check
    validate(instance=self.accessor.parameters, schema=self.jsonSchema, resolver=resolver)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: False schema does not allow 3900Failed validating None in schema['properties']['lambdarange']:
    FalseOn instance['lambdarange'][0]:
    3900

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/process_spectra", line 8, in <module>
    sys.exit(main())
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 269, in main
    return main_method(config)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 243, in main_method
    amazed(config)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 155, in amazed
    context, user_parameters = _setup_pass(config)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 136, in _setup_pass
    context = Context(vars(config), Parameters(params))
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/Parameters.py", line 54, in __init__
    Checker(accessor).check()
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 61, in check
    self.json_schema_check()
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 75, in json_schema_check
    raise APIException(
pylibamazed.Exception.APIException: INVALID_PARAMETER_FILE:False schema does not allow 3900
Traceback (most recent call last):
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/merge_results", line 8, in <module>
    sys.exit(main())
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/merge_results/merge_results.py", line 98, in main
    return main_method(config)
  File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/merge_results/merge_results.py", line 71, in main_method
    raise FileNotFoundError("Bunch data directory not found : {}".format(bunch_data_dir))
FileNotFoundError: Bunch data directory not found : /simdata/proc-spt/drp/pipe_e2e/regular_processing_new/src/validation_1dpipe/datapack_0.46.0/test_output/B0/data
'test_output/data/pfsZcandidates-00005-00000-0,0-000000000000067d-002-0x2324f2caaa6c5d9d.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help.
'test_output/data/pfsZcandidates-00005-00000-0,0-0000000000000bc8-002-0x2324f2caaa6c5d9d.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help.
'test_output/data/pfsZcandidates-00006-00000-0,0-0000000000021368-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help.
'test_output/data/pfsZcandidates-00006-00000-0,0-000000000002368c-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help.
'test_output/data/pfsZcandidates-00006-00000-0,0-00000000001e8525-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help.
'test_output/data/pfsZcandidates-00006-00000-0,0-00000000001e8527-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help.
Validation successful
Comment by Ali Allaoui [ 10/Jan/24 ]

It seems to be a problem with jsonschema python library, can you send the result of

 pip list
Comment by Kiyoto Yabe [ 11/Jan/24 ]

Attached.

pip-list.txt

Comment by Kiyoto Yabe [ 19/Jan/24 ]

Validation test done with the new datapack_0.46.0.1. The test failed (see attached report) but it looks like the usual small numerical difference. Installation is successful and pipeline is working, so can we leave it for now or is the difference critical?

valid_report.txt

Comment by vlebrun [ 19/Jan/24 ]

I confirm the difference is not significant, good news

Comment by Ali Allaoui [ 25/Jan/24 ]

tickets/PIPE1D-75 is merged into master, with tag 0.46.0, we can consider 0.46.0  released

Generated at Sat Feb 10 15:36:16 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.