PFS-JIRA

PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us

8.3.4

803005

13-09-2019

[PIPE1D-75] pfsZCandidates should always have 10 HDUs https://pfspipe.ipmu.jp/jira/browse/PIPE1D-75 DRP 1D pipeline PIPE1D-75

pfsZCandidates should always have 10 HDUs

Bug Normal Done Done Ali Allaoui Ali Allaoui Wed, 11 Oct 2023 09:01:23 +0000 Wed, 24 Jan 2024 20:19:33 +0000 Wed, 24 Jan 2024 20:19:33 +0000 drp_1dpipe 0 3 I'm now testing version 0.46.0. In the validation test, I got the following error: <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-python"> Running drp_1dpipe 0.46.0 Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 73, in json_schema_check validate(instance=self.accessor.parameters, schema=self.jsonSchema, resolver=resolver) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/jsonschema/validators.py", line 934, in validate raise error jsonschema.exceptions.ValidationError: False schema does not allow 3900Failed validating None in schema['properties']['lambdarange']: FalseOn instance['lambdarange'][0]: 3900During handling of the above exception, another exception occurred:Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/process_spectra", line 8, in <module> sys.exit(main()) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 269, in main return main_method(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 243, in main_method amazed(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 155, in amazed context, user_parameters = _setup_pass(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 136, in _setup_pass context = Context(vars(config), Parameters(params)) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/Parameters.py", line 54, in __init__ Checker(accessor).check() File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 61, in check self.json_schema_check() File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 75, in json_schema_check raise APIException( pylibamazed.Exception.APIException: INVALID_PARAMETER_FILE:False schema does not allow 3900 Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 73, in json_schema_check validate(instance=self.accessor.parameters, schema=self.jsonSchema, resolver=resolver) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/jsonschema/validators.py", line 934, in validate raise error jsonschema.exceptions.ValidationError: False schema does not allow 3900Failed validating None in schema['properties']['lambdarange']: FalseOn instance['lambdarange'][0]: 3900 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/process_spectra", line 8, in <module> sys.exit(main()) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 269, in main return main_method(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 243, in main_method amazed(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 155, in amazed context, user_parameters = _setup_pass(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 136, in _setup_pass context = Context(vars(config), Parameters(params)) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/Parameters.py", line 54, in __init__ Checker(accessor).check() File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 61, in check self.json_schema_check() File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 75, in json_schema_check raise APIException( pylibamazed.Exception.APIException: INVALID_PARAMETER_FILE:False schema does not allow 3900 Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/merge_results", line 8, in <module> sys.exit(main()) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/merge_results/merge_results.py", line 98, in main return main_method(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/merge_results/merge_results.py", line 71, in main_method raise FileNotFoundError("Bunch data directory not found : {}".format(bunch_data_dir)) FileNotFoundError: Bunch data directory not found : /simdata/proc-spt/drp/pipe_e2e/regular_processing_new/src/validation_1dpipe/datapack_0.46.0/test_output/B0/data 'test_output/data/pfsZcandidates-00005-00000-0,0-000000000000067d-002-0x2324f2caaa6c5d9d.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00005-00000-0,0-0000000000000bc8-002-0x2324f2caaa6c5d9d.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00006-00000-0,0-0000000000021368-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00006-00000-0,0-000000000002368c-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00006-00000-0,0-00000000001e8525-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00006-00000-0,0-00000000001e8527-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. Validation successful</pre> </div></div> It seems to be a problem with jsonschema python library, can you send the result of <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> pip list</pre> </div></div> Attached. <a href="https://pfspipe.ipmu.jp/jira/secure/attachment/16998/16998_pip-list.txt" title="pip-list.txt attached to PIPE1D-75">pip-list.txt<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/></a> Validation test done with the new datapack_0.46.0.1. The test failed (see attached report) but it looks like the usual small numerical difference. Installation is successful and pipeline is working, so can we leave it for now or is the difference critical? <a href="https://pfspipe.ipmu.jp/jira/secure/attachment/17039/17039_valid_report.txt" title="valid_report.txt attached to PIPE1D-75">valid_report.txt<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/></a> I confirm the difference is not significant, good news tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE1D-75" title="pfsZCandidates should always have 10 HDUs" class="issue-link" data-issue-key="PIPE1D-75"><del>PIPE1D-75</del></a> is merged into master, with tag 0.46.0, we can consider 0.46.0 released Development Rank 0|zzszxr: Reviewers kiyoto.yabe