[PIPE2D-286] Create/update unit tests Created: 16/Aug/18 Updated: 11/Sep/18 Resolved: 11/Sep/18 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Reviewers: | hassan |
| Description |
|
Create/update unit tests for low-level components (Spectrum, FiberTrace, etc.) in drp_stella. |
| Comments |
| Comment by price [ 31/Aug/18 ] |
|
There are a lot of changes here, as the cleanup work has continued, new tests have been written and bugs have been identified and fixed. Not sure how we want to review this, so sending it to hassan to decide. There are changes in datamodel, but the bulk is in drp_stella.
price@pap-laptop:~/pfs/drp_stella (tickets/PIPE2D-286=) $ git sub
commit b1570f32aba0ac9e6f559c7f8aaa310f9f5f9be2
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 15 23:26:11 2018 -0400
FiberTrace: remove private calculation methods
The private calculation methods in FiberTrace existed to construct the
FiberTrace model, which is much simpler than the list of private members
implies. All we need for a FiberTrace is the trace profile and a fiberId.
Everything else was inserted to simplify the API of the various methods
called in the process of determining the model.
Now all of that has been moved out to findAndTraceApertures, which is
the only place it was used. Some vestigial parts (vectors of vectors of
points) have been stripped out.
The result is a much cleaner FiberTrace class, which could now be
constructed using a different algorithm or implementation than the
one that was hard-wired in. The state of the object is now much
clearer at all times, and simpler to represent.
include/pfs/drp/stella/Controls.h | 26 +-
include/pfs/drp/stella/FiberTraces.h | 101 +---
.../pfs/drp/stella/math/findAndTraceApertures.h | 26 -
python/pfs/drp/stella/fiberTraces.cc | 16 +-
src/FiberTraceFunction.cc | 42 +-
src/FiberTraces.cc | 639 +-------------------
src/math/findAndTraceApertures.cc | 672 +++++++++++++++++++--
7 files changed, 717 insertions(+), 805 deletions(-)
commit 6ee1cad0a983ff38606faea2c44384083fc9ecaf
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 16 16:08:21 2018 -0400
Export fiberMaskPlane constant to python
Especially when testing, we want to be able to mask pixels with the
proper mask plane.
include/pfs/drp/stella/FiberTraces.h | 2 +-
python/pfs/drp/stella/fiberTraces.cc | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
commit fd174cce83e83ec85cd778d60006a790e5b221eb
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 16 16:11:31 2018 -0400
FiberTrace: fix infinite loop in non-optimal extraction
The iterator being tested wasn't being incremented. The mask was being
iterated over but not being used.
src/FiberTraces.cc | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
commit 6979dca6cdcc9d547b35358677cf6d8cb25fa983
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 16 16:15:24 2018 -0400
FiberTraces: fix fixing of ends of spectrum
The last value in the array was being fixed even when it didn't need
to be. Be more careful about when we fix things up.
src/FiberTraces.cc | 22 +++++++++++++---------
1 file changed, 13 insertions(+), 9 deletions(-)
commit 2fd78f32f1707a31e8c09fc94a2e4c8e4d10d402
Author: Paul Price <price@astro.princeton.edu>
Date: Fri Aug 17 11:58:55 2018 -0400
fitProfile2d: mask bad results
We've long known we need to mask the spectral element when the fit
returns a negative chi^2; finally did it. Reordered the elements in
the tuple coming back from fitProfile2d to correspond to the classic
image,mask,variance order from lsst::afw::image::MaskedImage.
Bad spectral elements are masked as BAD; we probably want this to be
configurable, but this will do for now.
include/pfs/drp/stella/math/CurveFitting.h | 1 +
src/FiberTraces.cc | 12 +++++++++---
src/math/CurveFitting.cc | 11 ++++++-----
3 files changed, 16 insertions(+), 8 deletions(-)
commit 4434e04ff6c7c6bf03331b5befc5457be85ee891
Author: Paul Price <price@astro.princeton.edu>
Date: Fri Aug 17 15:08:08 2018 -0400
FiberTrace: be more careful about masking during extraction
We want the mask on the spectrum to be the OR of the mask values.
Refactored a bit to make this more efficient and do it correctly.
src/FiberTraces.cc | 39 +++++++++++++++++++--------------------
1 file changed, 19 insertions(+), 20 deletions(-)
commit fe164fb4216f14062eab72e89296efa973d26801
Author: Paul Price <price@astro.princeton.edu>
Date: Sat Aug 18 10:55:33 2018 -0400
FiberTrace: expect constructImage APIs
Want to be able to make an image of different size than the trace image,
and to add into an existing image.
include/pfs/drp/stella/FiberTraces.h | 28 ++++++++++++++++++--
python/pfs/drp/stella/fiberTraces.cc | 7 +++++
src/FiberTraces.cc | 50 ++++++++++++++++++++++++------------
3 files changed, 67 insertions(+), 18 deletions(-)
commit b8158faa431a9295d4b25c2f9d1c69feadfc481b
Author: Paul Price <price@astro.princeton.edu>
Date: Mon Aug 20 16:34:52 2018 -0400
FiberTraceSet: index with ptrdiff_t
This allows us to trap the use of negative indices that a python user
might try.
include/pfs/drp/stella/FiberTraces.h | 10 +++++-----
python/pfs/drp/stella/fiberTraces.cc | 4 ++--
2 files changed, 7 insertions(+), 7 deletions(-)
commit 596a0584e0b516abd3db7227257e90d077b8efe4
Author: Paul Price <price@astro.princeton.edu>
Date: Mon Aug 20 18:09:43 2018 -0400
FiberTraceSet: use caps for FITS header keywords
FITS header keywords are usually in caps; the FITS write/read cycle
will translate any lower-case keywords of 8 chars or less to caps.
python/pfs/drp/stella/fiberTracesContinued.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
commit 225c5250ec6dee4b353e04180a9215f3e15f9296
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 22 17:47:39 2018 -0400
DispersionCorrectionControl: remove order
It is unused.
include/pfs/drp/stella/Controls.h | 2 --
python/pfs/drp/stella/controls.cc | 1 -
2 files changed, 3 deletions(-)
commit f8f250ac552c20e7b8bb7eb041d26eb362638674
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 22 18:01:47 2018 -0400
ReferenceLine: convert floats to double
Python only uses doubles, so if we want everything to be identical,
they need to be double in C++ as well.
include/pfs/drp/stella/Spectra.h | 18 +++++++++---------
python/pfs/drp/stella/spectra.cc | 6 +++---
2 files changed, 12 insertions(+), 12 deletions(-)
commit fce3479382e8b8657533c7edb147208c16f97c79
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 10:56:00 2018 -0400
Spectrum: fix plotting with uneven rows
numpy.split refuses to split things unequally, so tell it where to
do the split.
python/pfs/drp/stella/spectraContinued.py | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
commit 7fb1a3262bfeeacf2bdf80b34efde8a0d40fc8ca
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 16:35:19 2018 -0400
Spectrum: have setMask make a deep copy
This matches what's done for the arrays (spectrum, background, etc.).
src/Spectra.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
commit 8abe77600de61461b79dfbefa5c92d7140b75a32
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 16:37:24 2018 -0400
SpectrumSet: attempt to preserve fiberId when converting to PfsArm
The fiberId values get persisted through the pfsConfig, which is saved
separately. If using the butler we can get them back, but if just using
writeFits/readFits, it's not possible.
python/pfs/drp/stella/spectraContinued.py | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
commit 7c2f459addcfe64932861077399b150df293bfd6
Author: Paul Price <price@astro.princeton.edu>
Date: Mon Aug 27 15:12:49 2018 -0400
remove DetectorMapIO
I/O for DetectorMap is now done within DetectorMapContinued.py.
include/pfs/drp/stella/DetectorMap.h | 1 -
include/pfs/drp/stella/DetectorMapIO.h | 90 -----------------------
python/pfs/drp/stella/calibrateWavelengthsTask.py | 6 +-
python/pfs/drp/stella/reduceArcTask.py | 6 +-
python/pfs/drp/stella/reduceExposure.py | 4 +-
5 files changed, 8 insertions(+), 99 deletions(-)
commit 97bc23c9ae7225e75d1ed5156c476427b0b05ba0
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Aug 28 11:22:33 2018 -0400
DetectorMap: fix pickle
The xCenter and wavelength values returned from DetectorMap are tweaked
for the slitOffsets, so can't be fed directly into the constructor.
Instead, construct using the spline vectors. But then there was a bug in
that because it reconstructed the splines.
python/pfs/drp/stella/detectorMap.cc | 24 +++++++++++++++++++-----
src/DetectorMap.cc | 1 -
2 files changed, 19 insertions(+), 6 deletions(-)
commit dd4052bfe4ca29c8149eef2d45fd9806cff11228
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Aug 28 11:23:41 2018 -0400
DetectorMap: cleanup bindings, add properties
python/pfs/drp/stella/detectorMap.cc | 26 ++++++++++++++++++++------
1 file changed, 20 insertions(+), 6 deletions(-)
commit 0ee5bcf3fcf4648b2141dc85ef8f1f6b4e3fc9db
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Aug 28 16:08:02 2018 -0400
DetectorMap: make metadata, VisitInfo pickleable
This requires modifying a Ctor to take these, and putting them into the
pickle. However, VisitInfo isn't pickleable by itself, so pull the snippet
from reduceArcTask that pickles them into its own file so everyone can
use it.
include/pfs/drp/stella/DetectorMap.h | 9 ++++++---
python/pfs/drp/stella/__init__.py | 1 +
python/pfs/drp/stella/detectorMap.cc | 14 ++++++++++----
python/pfs/drp/stella/pickleUtils.py | 25 +++++++++++++++++++++++++
python/pfs/drp/stella/reduceArcTask.py | 26 --------------------------
src/DetectorMap.cc | 8 +++++---
6 files changed, 47 insertions(+), 36 deletions(-)
commit fdaabae1e33050514eb6c8b87bd241e77f96d73f
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 29 10:09:19 2018 -0400
DetectorMap: persist metadata, visitInfo
If it's part of the object, we should persist it.
python/pfs/drp/stella/detectorMapContinued.py | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
commit 8c2d3eb3720885af2efb76f81366e03bc712d30d
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 17:07:22 2018 -0400
DetectorMap: add methods to get number of fibers
include/pfs/drp/stella/DetectorMap.h | 3 +++
python/pfs/drp/stella/detectorMap.cc | 3 +++
2 files changed, 6 insertions(+)
commit 3aae0c957a8cb82ba1b1dcdf2592b9286c864a8d
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 17:08:38 2018 -0400
add functions to generate synthetic data
python/pfs/drp/stella/synthetic.py | 179 +++++++++++++++++++++++++++++++++++++
1 file changed, 179 insertions(+)
commit 9c96f192f4648b6ef4ad20004584dc20e2e1e015
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 22:02:11 2018 -0400
DetectorMap::findFiberId: put a limit on the number of iterations
The "while" loop in this method could theoretically go on forever.
Put a practical limit on the number of iterations: twice the number
of fibers.
src/DetectorMap.cc | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
commit 54c639158214c9925d86b5198b7b6f7731fb4f10
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 22:03:54 2018 -0400
DetectorMap: include slitOffsets in find methods
Otherwise, we're not following the actual trace. The splines do not
include the slit offsets. The getXCenter() and getWavelength() results
do.
src/DetectorMap.cc | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
commit 3bb2dc76c598810b27d799a646cbf1ef35e1e93c
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 22:52:18 2018 -0400
DetectorMap: add more checks on Ctor
src/DetectorMap.cc | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
commit 5f65469664658c719911be74764270ae6beb80a3
Author: Paul Price <price@astro.princeton.edu>
Date: Fri Aug 17 15:56:11 2018 -0400
Remove unused tests
These aren't useful, or assume an environment we don't want
(drp_stella_data), or will be replaced by more thorough tests.
tests/Controls.py | 61 -----
tests/FiberTraces.py | 495 --------------------------------------
tests/Math.py | 94 --------
tests/Spectra.py | 660 ---------------------------------------------------
tests/example.py | 65 -----
tests/testDRP.py | 102 --------
ups/drp_stella.table | 2 -
7 files changed, 1479 deletions(-)
commit ec62e72a12b3099a9b84db824f8efdac2d5f5d8e
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 16:40:54 2018 -0400
Add tests for the C++-based building blocks
Spectrum, SpectrumSet, FiberTrace, FiberTraceSet, DetectorMap.
tests/SConscript | 12 +-
tests/test_DetectorMap.py | 498 +++++++++++++++++++++++++++++---------------
tests/test_FiberTrace.py | 329 +++++++++++++++++++++++++++++
tests/test_FiberTraceSet.py | 240 +++++++++++++++++++++
tests/test_Spectrum.py | 402 +++++++++++++++++++++++++++++++++++
5 files changed, 1298 insertions(+), 183 deletions(-)
commit 19fd73bead6e5e68b59b9a75cea86b68a9a347da (HEAD -> tickets/PIPE2D-286, origin/tickets/PIPE2D-286)
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 23:01:38 2018 -0400
git: ignore file generated by tests
.gitignore | 1 +
1 file changed, 1 insertion(+)
price@pap-laptop:~/pfs/datamodel (tickets/PIPE2D-286=) $ git sub
commit bc64dbdfd4375321cd22d1318aad761b353d3492
Author: Paul Price <price@astro.princeton.edu>
Date: Mon Aug 20 18:11:59 2018 -0400
PfsFiberTrace: strip FITS metadata
lsst.afw.fits.readMetadata does better at stripping than the MaskedImage
constructor.
Somehow, COMMENT doesn't get stripped.
python/pfs/datamodel/pfsFiberTrace.py | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
commit 0687d06e66d4bdcdbb3e1052dcf5ac5156b9638e (HEAD -> tickets/PIPE2D-286, origin/tickets/PIPE2D-286)
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 16:26:11 2018 -0400
pfsArm: fix ResourceWarning from not closing file
Got:
/home/lsst/pfs/drp_stella/python/pfs/drp/stella/spectraContinued.py:252: ResourceWarning: unclosed file <_io.FileIO name='tests/test_Spectrum/pfsArm-123456-r3.fits' mode='rb' closefd=True>
pfsArm.read(dirName=parsed.dirName, setPfsConfig=False)
This is fixed by using the file as a context manager, which ensures it's
closed when the context closes.
python/pfs/datamodel/pfsArm.py | 80 +++++++++++++++++++++---------------------
1 file changed, 40 insertions(+), 40 deletions(-)
|
| Comment by price [ 31/Aug/18 ] |
|
This is going to need a tweak, as Travis just failed: Traceback (most recent call last): File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-3-g388e494+18/python/lsst/ctrl/pool/pool.py", line 113, in wrapper return func(*args, **kwargs) File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-3-g388e494+18/python/lsst/ctrl/pool/pool.py", line 239, in wrapper return func(*args, **kwargs) File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-3-g388e494+18/python/lsst/ctrl/pool/pool.py", line 715, in reduce *args, **kwargs) File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-3-g388e494+18/python/lsst/ctrl/pool/pool.py", line 572, in _reduceQueue resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue] File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-3-g388e494+18/python/lsst/ctrl/pool/pool.py", line 572, in <listcomp> resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue] File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/drp_stella/19fd73bead6e5e68b59b9a75cea86b68a9a347da/python/pfs/drp/stella/constructFiberFlatTask.py", line 147, in combine traces = self.trace.run(exposure.maskedImage, detMap) File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/drp_stella/19fd73bead6e5e68b59b9a75cea86b68a9a347da/python/pfs/drp/stella/findAndTraceAperturesTask.py", line 39, in run return drpStella.findAndTraceApertures(maskedImage, detectorMap, finding, function, fitting) lsst.pex.exceptions.wrappers.RangeError: File "src/DetectorMap.cc", line 111, in std::size_t pfs::drp::stella::DetectorMap::getFiberIndex(int) const Unknown fiberId 0 {0} lsst::pex::exceptions::RangeError: 'Unknown fiberId 0' |
| Comment by price [ 05/Sep/18 ] |
|
Added a few commits to fix things. Here's where we're at now:
price@pap-laptop:~/pfs/datamodel (tickets/PIPE2D-286=) $ git sub
commit bc64dbdfd4375321cd22d1318aad761b353d3492
Author: Paul Price <price@astro.princeton.edu>
Date: Mon Aug 20 18:11:59 2018 -0400
PfsFiberTrace: strip FITS metadata
lsst.afw.fits.readMetadata does better at stripping than the MaskedImage
constructor.
Somehow, COMMENT doesn't get stripped.
python/pfs/datamodel/pfsFiberTrace.py | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
commit 0687d06e66d4bdcdbb3e1052dcf5ac5156b9638e (HEAD -> tickets/PIPE2D-286, origin/tickets/PIPE2D-286)
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 16:26:11 2018 -0400
pfsArm: fix ResourceWarning from not closing file
Got:
/home/lsst/pfs/drp_stella/python/pfs/drp/stella/spectraContinued.py:252: ResourceWarning: unclosed file <_io.FileIO name='tests/test_Spectrum/pfsArm-123456-r3.fits' mode='rb' closefd=True>
pfsArm.read(dirName=parsed.dirName, setPfsConfig=False)
This is fixed by using the file as a context manager, which ensures it's
closed when the context closes.
python/pfs/datamodel/pfsArm.py | 80 +++++++++++++++++++++---------------------
1 file changed, 40 insertions(+), 40 deletions(-)
price@pap-laptop:~/pfs/obs_pfs (tickets/PIPE2D-286=) $ git sub
commit f680d53a345e4657d5051bb73b924213bc556dff
Author: Paul Price <price@astro.princeton.edu>
Date: Fri Aug 31 23:04:04 2018 -0400
Fix reading a pfsArm
There were two problems:
1. The pfsConfigId wasn't set; default to 0. This is in the process of being
refactored anyway.
2. We were returning a SpectrumSet, which doesn't have a pfsConfig element.
Instead, return a PfsArm since that contains all the information. The
user can convert it if they want.
python/lsst/obs/pfs/pfsMapper.py | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
commit 4dfc79524772778e9141778c1a6ac42977db9901 (HEAD -> tickets/PIPE2D-286, origin/tickets/PIPE2D-286)
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Sep 4 15:45:13 2018 -0400
git: ignore directory generated by pytest
.gitignore | 1 +
1 file changed, 1 insertion(+)
price@pap-laptop:~/pfs/drp_stella (tickets/PIPE2D-286 %=) $ git sub
commit 8dbbe1217c536a16540128ef83d1619544d98c8a
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 15 23:26:11 2018 -0400
FiberTrace: remove private calculation methods
The private calculation methods in FiberTrace existed to construct the
FiberTrace model, which is much simpler than the list of private members
implies. All we need for a FiberTrace is the trace profile and a fiberId.
Everything else was inserted to simplify the API of the various methods
called in the process of determining the model.
Now all of that has been moved out to findAndTraceApertures, which is
the only place it was used. Some vestigial parts (vectors of vectors of
points) have been stripped out.
The result is a much cleaner FiberTrace class, which could now be
constructed using a different algorithm or implementation than the
one that was hard-wired in. The state of the object is now much
clearer at all times, and simpler to represent.
include/pfs/drp/stella/Controls.h | 26 +-
include/pfs/drp/stella/FiberTraces.h | 101 +---
.../pfs/drp/stella/math/findAndTraceApertures.h | 26 -
python/pfs/drp/stella/fiberTraces.cc | 16 +-
src/FiberTraceFunction.cc | 40 +-
src/FiberTraces.cc | 639 +-------------------
src/math/findAndTraceApertures.cc | 671 +++++++++++++++++++--
7 files changed, 714 insertions(+), 805 deletions(-)
commit a9b1ccdd8a47e8a96b67d01569b92d06bbf19dfb
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 16 16:08:21 2018 -0400
Export fiberMaskPlane constant to python
Especially when testing, we want to be able to mask pixels with the
proper mask plane.
include/pfs/drp/stella/FiberTraces.h | 2 +-
python/pfs/drp/stella/fiberTraces.cc | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
commit b659079f697d4e9bcf49642716057bb0c06c1e56
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 16 16:11:31 2018 -0400
FiberTrace: fix infinite loop in non-optimal extraction
The iterator being tested wasn't being incremented. The mask was being
iterated over but not being used.
src/FiberTraces.cc | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
commit 8f3040541e4d8714552aa0b7e4291b1b04516d7f
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 16 16:15:24 2018 -0400
FiberTraces: fix fixing of ends of spectrum
The last value in the array was being fixed even when it didn't need
to be. Be more careful about when we fix things up.
src/FiberTraces.cc | 22 +++++++++++++---------
1 file changed, 13 insertions(+), 9 deletions(-)
commit 59491f961ce472cc50c9fd1f0728b82c314c0880
Author: Paul Price <price@astro.princeton.edu>
Date: Fri Aug 17 11:58:55 2018 -0400
fitProfile2d: mask bad results
We've long known we need to mask the spectral element when the fit
returns a negative chi^2; finally did it. Reordered the elements in
the tuple coming back from fitProfile2d to correspond to the classic
image,mask,variance order from lsst::afw::image::MaskedImage.
Bad spectral elements are masked as BAD; we probably want this to be
configurable, but this will do for now.
include/pfs/drp/stella/math/CurveFitting.h | 1 +
src/FiberTraces.cc | 12 +++++++++---
src/math/CurveFitting.cc | 11 ++++++-----
3 files changed, 16 insertions(+), 8 deletions(-)
commit 3b711b40728c228ec31c5422b7c103e07a1630ff
Author: Paul Price <price@astro.princeton.edu>
Date: Fri Aug 17 15:08:08 2018 -0400
FiberTrace: be more careful about masking during extraction
We want the mask on the spectrum to be the OR of the mask values.
Refactored a bit to make this more efficient and do it correctly.
src/FiberTraces.cc | 39 +++++++++++++++++++--------------------
1 file changed, 19 insertions(+), 20 deletions(-)
commit bc5cf7cb54a9776bb745a93c0bd3d5ae0c638971
Author: Paul Price <price@astro.princeton.edu>
Date: Sat Aug 18 10:55:33 2018 -0400
FiberTrace: expect constructImage APIs
Want to be able to make an image of different size than the trace image,
and to add into an existing image.
include/pfs/drp/stella/FiberTraces.h | 28 +++++++++++++--
python/pfs/drp/stella/fiberTraces.cc | 7 ++++
src/FiberTraces.cc | 67 +++++++++++++++++++++++-------------
3 files changed, 76 insertions(+), 26 deletions(-)
commit 75700b08f015c8da56cced004aa53f8bf0a819d5
Author: Paul Price <price@astro.princeton.edu>
Date: Mon Aug 20 16:34:52 2018 -0400
FiberTraceSet: index with ptrdiff_t
This allows us to trap the use of negative indices that a python user
might try.
include/pfs/drp/stella/FiberTraces.h | 10 +++++-----
python/pfs/drp/stella/fiberTraces.cc | 4 ++--
2 files changed, 7 insertions(+), 7 deletions(-)
commit 41bbf7ab38d3a411b1ce92d54b5572f5e320a08a
Author: Paul Price <price@astro.princeton.edu>
Date: Mon Aug 20 18:09:43 2018 -0400
FiberTraceSet: use caps for FITS header keywords
FITS header keywords are usually in caps; the FITS write/read cycle
will translate any lower-case keywords of 8 chars or less to caps.
python/pfs/drp/stella/fiberTracesContinued.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
commit 1aee26e4c7c4c92072130ff587a466e6b2aff8e0
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 22 17:47:39 2018 -0400
DispersionCorrectionControl: remove order
It is unused.
include/pfs/drp/stella/Controls.h | 2 --
python/pfs/drp/stella/controls.cc | 1 -
2 files changed, 3 deletions(-)
commit 8fa8307b41b4037435e87992f6fb7585ba37391d
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 22 18:01:47 2018 -0400
ReferenceLine: convert floats to double
Python only uses doubles, so if we want everything to be identical,
they need to be double in C++ as well.
include/pfs/drp/stella/Spectra.h | 18 +++++++++---------
python/pfs/drp/stella/spectra.cc | 6 +++---
2 files changed, 12 insertions(+), 12 deletions(-)
commit ce27646c7a56c77de3fba61cf9481573111c56de
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 10:56:00 2018 -0400
Spectrum: fix plotting with uneven rows
numpy.split refuses to split things unequally, so tell it where to
do the split.
python/pfs/drp/stella/spectraContinued.py | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
commit cc8683cb5973a840e43c28e2763a5f05ce28c6be
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 16:35:19 2018 -0400
Spectrum: have setMask make a deep copy
This matches what's done for the arrays (spectrum, background, etc.).
src/Spectra.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
commit c5e574c1347891dab3579c5152588beafffe8532
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 16:37:24 2018 -0400
SpectrumSet: attempt to preserve fiberId when converting to PfsArm
The fiberId values get persisted through the pfsConfig, which is saved
separately. If using the butler we can get them back, but if just using
writeFits/readFits, it's not possible.
python/pfs/drp/stella/spectraContinued.py | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
commit ee29fdafcf0122e3d5329ed43e6db1c077771420
Author: Paul Price <price@astro.princeton.edu>
Date: Mon Aug 27 15:12:49 2018 -0400
remove DetectorMapIO
I/O for DetectorMap is now done within DetectorMapContinued.py.
include/pfs/drp/stella/DetectorMap.h | 1 -
include/pfs/drp/stella/DetectorMapIO.h | 90 -----------------------
python/pfs/drp/stella/calibrateWavelengthsTask.py | 6 +-
python/pfs/drp/stella/reduceArcTask.py | 6 +-
python/pfs/drp/stella/reduceExposure.py | 4 +-
5 files changed, 8 insertions(+), 99 deletions(-)
commit 4a99a7251ebb7264619392caabe6096c19030be5
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Aug 28 11:22:33 2018 -0400
DetectorMap: fix pickle
The xCenter and wavelength values returned from DetectorMap are tweaked
for the slitOffsets, so can't be fed directly into the constructor.
Instead, construct using the spline vectors. But then there was a bug in
that because it reconstructed the splines.
python/pfs/drp/stella/detectorMap.cc | 24 +++++++++++++++++++-----
src/DetectorMap.cc | 2 +-
2 files changed, 20 insertions(+), 6 deletions(-)
commit 081b6f6fde9488f82a70640f3930b43c5519a515
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Aug 28 11:23:41 2018 -0400
DetectorMap: cleanup bindings, add properties
python/pfs/drp/stella/detectorMap.cc | 25 +++++++++++++++++++------
1 file changed, 19 insertions(+), 6 deletions(-)
commit 3644c0552bb2ab54c7fa75bbfcba74f49586b91b
Author: Paul Price <price@astro.princeton.edu>
Date: Tue Aug 28 16:08:02 2018 -0400
DetectorMap: make metadata, VisitInfo pickleable
This requires modifying a Ctor to take these, and putting them into the
pickle. However, VisitInfo isn't pickleable by itself, so pull the snippet
from reduceArcTask that pickles them into its own file so everyone can
use it.
include/pfs/drp/stella/DetectorMap.h | 9 ++++++---
python/pfs/drp/stella/__init__.py | 1 +
python/pfs/drp/stella/detectorMap.cc | 14 ++++++++++----
python/pfs/drp/stella/pickleUtils.py | 25 +++++++++++++++++++++++++
python/pfs/drp/stella/reduceArcTask.py | 26 --------------------------
src/DetectorMap.cc | 8 +++++---
6 files changed, 47 insertions(+), 36 deletions(-)
commit 8af0398dde9833791391146632171953887b2fb5
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 29 10:09:19 2018 -0400
DetectorMap: persist metadata, visitInfo
If it's part of the object, we should persist it.
python/pfs/drp/stella/detectorMapContinued.py | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
commit 18e741ef79ab9d2ae6cd8cc2cf241f120198a300
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 17:07:22 2018 -0400
DetectorMap: add methods to get number of fibers
include/pfs/drp/stella/DetectorMap.h | 3 +++
python/pfs/drp/stella/detectorMap.cc | 3 +++
2 files changed, 6 insertions(+)
commit 1b3c390cd5266c9d2df5d301c689ecb40484699b
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 17:08:38 2018 -0400
add functions to generate synthetic data
python/pfs/drp/stella/synthetic.py | 179 +++++++++++++++++++++++++++++++++++++
1 file changed, 179 insertions(+)
commit fe1008ce0badd70f9d036169b7f1fa23e3b6e32d
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 22:02:11 2018 -0400
DetectorMap::findFiberId: put a limit on the number of iterations
The "while" loop in this method could theoretically go on forever.
Put a practical limit on the number of iterations: twice the number
of fibers.
src/DetectorMap.cc | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
commit f9c84c6a3db92ca5f9ba3310a7881359ae66c54e
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 22:03:54 2018 -0400
DetectorMap: include slitOffsets in find methods
Otherwise, we're not following the actual trace. The splines do not
include the slit offsets. The getXCenter() and getWavelength() results
do.
src/DetectorMap.cc | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
commit 135d552dc540866aa4cf741a451213914cc3517c
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 22:52:18 2018 -0400
DetectorMap: add more checks on Ctor
src/DetectorMap.cc | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
commit f78b0137828b1f35de26f61e419229598b14b211
Author: Paul Price <price@astro.princeton.edu>
Date: Fri Aug 17 15:56:11 2018 -0400
Remove unused tests
These aren't useful, or assume an environment we don't want
(drp_stella_data), or will be replaced by more thorough tests.
tests/Controls.py | 61 -----
tests/FiberTraces.py | 495 --------------------------------------
tests/Math.py | 94 --------
tests/Spectra.py | 660 ---------------------------------------------------
tests/example.py | 65 -----
tests/testDRP.py | 102 --------
ups/drp_stella.table | 2 -
7 files changed, 1479 deletions(-)
commit 3cea453adab972d06e5bea03ff601e727db83462
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 23 16:40:54 2018 -0400
Add tests for the C++-based building blocks
Spectrum, SpectrumSet, FiberTrace, FiberTraceSet, DetectorMap.
tests/SConscript | 12 +-
tests/test_DetectorMap.py | 498 +++++++++++++++++++++++++++++---------------
tests/test_FiberTrace.py | 326 +++++++++++++++++++++++++++++
tests/test_FiberTraceSet.py | 240 +++++++++++++++++++++
tests/test_Spectrum.py | 402 +++++++++++++++++++++++++++++++++++
5 files changed, 1295 insertions(+), 183 deletions(-)
commit a26dc4f323b103e19bc127a026fea592d95e7257
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Aug 30 23:01:38 2018 -0400
git: ignore file generated by tests
.gitignore | 1 +
1 file changed, 1 insertion(+)
commit 9fa02d1662b3e07eec256684e04b1858b71d71c7 (HEAD -> tickets/PIPE2D-286, origin/tickets/PIPE2D-286)
Author: Paul Price <price@astro.princeton.edu>
Date: Fri Aug 31 23:11:56 2018 -0400
constructFiberTrace: clear metadata before repopulating
To avoid conflicts with the existing metadata (e.g., CALIB_ID).
python/pfs/drp/stella/constructFiberTraceTask.py | 4 ++++
1 file changed, 4 insertions(+)
|
| Comment by price [ 11/Sep/18 ] |
|
Merged to master. |