[PIPE1D-75] pfsZCandidates should always have 10 HDUs Created: 11/Oct/23 Updated: 25/Jan/24 Resolved: 25/Jan/24 |
|
| Status: | Done |
| Project: | DRP 1D pipeline |
| Component/s: | drp_1dpipe |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Normal |
| Reporter: | Ali Allaoui | Assignee: | Ali Allaoui |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Reviewers: | Kiyoto Yabe |
| Comments |
| Comment by Kiyoto Yabe [ 06/Jan/24 ] |
|
I'm now testing version 0.46.0. In the validation test, I got the following error: Running drp_1dpipe 0.46.0 Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 73, in json_schema_check validate(instance=self.accessor.parameters, schema=self.jsonSchema, resolver=resolver) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/jsonschema/validators.py", line 934, in validate raise error jsonschema.exceptions.ValidationError: False schema does not allow 3900Failed validating None in schema['properties']['lambdarange']: FalseOn instance['lambdarange'][0]: 3900During handling of the above exception, another exception occurred:Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/process_spectra", line 8, in <module> sys.exit(main()) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 269, in main return main_method(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 243, in main_method amazed(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 155, in amazed context, user_parameters = _setup_pass(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 136, in _setup_pass context = Context(vars(config), Parameters(params)) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/Parameters.py", line 54, in __init__ Checker(accessor).check() File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 61, in check self.json_schema_check() File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 75, in json_schema_check raise APIException( pylibamazed.Exception.APIException: INVALID_PARAMETER_FILE:False schema does not allow 3900 Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 73, in json_schema_check validate(instance=self.accessor.parameters, schema=self.jsonSchema, resolver=resolver) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/jsonschema/validators.py", line 934, in validate raise error jsonschema.exceptions.ValidationError: False schema does not allow 3900Failed validating None in schema['properties']['lambdarange']: FalseOn instance['lambdarange'][0]: 3900 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/process_spectra", line 8, in <module> sys.exit(main()) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 269, in main return main_method(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 243, in main_method amazed(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 155, in amazed context, user_parameters = _setup_pass(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/process_spectra/process_spectra.py", line 136, in _setup_pass context = Context(vars(config), Parameters(params)) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/Parameters.py", line 54, in __init__ Checker(accessor).check() File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 61, in check self.json_schema_check() File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/pylibamazed/ParametersChecker.py", line 75, in json_schema_check raise APIException( pylibamazed.Exception.APIException: INVALID_PARAMETER_FILE:False schema does not allow 3900 Traceback (most recent call last): File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/bin/merge_results", line 8, in <module> sys.exit(main()) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/merge_results/merge_results.py", line 98, in main return main_method(config) File "/opt/pfs/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/site-packages/drp_1dpipe/merge_results/merge_results.py", line 71, in main_method raise FileNotFoundError("Bunch data directory not found : {}".format(bunch_data_dir)) FileNotFoundError: Bunch data directory not found : /simdata/proc-spt/drp/pipe_e2e/regular_processing_new/src/validation_1dpipe/datapack_0.46.0/test_output/B0/data 'test_output/data/pfsZcandidates-00005-00000-0,0-000000000000067d-002-0x2324f2caaa6c5d9d.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00005-00000-0,0-0000000000000bc8-002-0x2324f2caaa6c5d9d.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00006-00000-0,0-0000000000021368-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00006-00000-0,0-000000000002368c-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00006-00000-0,0-00000000001e8525-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. 'test_output/data/pfsZcandidates-00006-00000-0,0-00000000001e8527-002-0x11fcdfd13c364618.fits' is not an existing file, directory, or wildcard pattern; see `fitsdiff --help` for more usage help. Validation successful |
| Comment by Ali Allaoui [ 10/Jan/24 ] |
|
It seems to be a problem with jsonschema python library, can you send the result of pip list |
| Comment by Kiyoto Yabe [ 11/Jan/24 ] |
|
Attached. |
| Comment by Kiyoto Yabe [ 19/Jan/24 ] |
|
Validation test done with the new datapack_0.46.0.1. The test failed (see attached report) but it looks like the usual small numerical difference. Installation is successful and pipeline is working, so can we leave it for now or is the difference critical? |
| Comment by vlebrun [ 19/Jan/24 ] |
|
I confirm the difference is not significant, good news |
| Comment by Ali Allaoui [ 25/Jan/24 ] |
|
tickets/ |