PFS-JIRA

PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us

8.3.4

803005

13-09-2019

[PIPE2D-218] Discuss FiberTrace's architecture https://pfspipe.ipmu.jp/jira/browse/PIPE2D-218 DRP 2-D Pipeline <p>Having just been debugging some issues with <tt>FiberTrace</tt> I got the impression that it could be made significantly simpler and more robust.</p> <p>We should discuss this.</p> PIPE2D-218

Discuss FiberTrace's architecture

Task Normal Done Done rhl rhl Wed, 12 Jul 2017 17:18:33 +0000 Thu, 17 Aug 2017 20:00:45 +0000 Fri, 4 Aug 2017 19:26:18 +0000 0 2 <p>Some of the comments about unused code will be fixed on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-207" title="Run coverage tools and remove *all* unused code" class="issue-link" data-issue-key="PIPE2D-207"><del>PIPE2D-207</del></a> (or, in one case, <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-221" title="Remove obsolete makeNormFlatFiberTrace from FiberTraces.cc" class="issue-link" data-issue-key="PIPE2D-221"><del>PIPE2D-221</del></a>)</p> <p>Here's a first dump of my thoughts.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> The FiberTrace <span class="code-keyword">class </span>describes the profiles and positions of spectra on a detector. Creating a FiberTraceSet is done as follows: ftffc = drpStella.FiberTraceFunctionFindingControl() <span class="code-comment">// terrible name! </span> quartzImage = butler.get(<span class="code-quote">"calexp"</span>, dataId).getMaskedImage() fiberTraceSet = drpStella.findAndTraceApertures(quartzImage, ftffc) fiberTraceSet.sortTracesByXCenter() <span class="code-keyword">for</span> i in range(fiberTraceSet.size()): fiberTraceSet.getFiberTrace(i).setITrace(i) fiberTraceSet.setFiberTraceProfileFittingControl(fiberTraceProfileFittingControl) <span class="code-keyword">if</span> doAll: inFiberTraceSet.calcProfileAllTraces() <span class="code-keyword">else</span>: <span class="code-keyword">for</span> i in inTraceNumbers: inFiberTraceSet.getFiberTrace(i).calcProfile() Extracting a spectrum then consists of: # Read a model of the mapping from pixels to traces xCenters, wavelengths, traceIds = readWavelengthFile(wLenFile) # assign trace number to fiberTraceSet based on model drpStella.assignITrace(fiberTraceSet, traceIds, xCenters) # spectrumSet = drpStella.SpectrumSet() scienceImage = butler.get(<span class="code-quote">"calexp"</span>, dataId).getMaskedImage() <span class="code-keyword">for</span> i in traceIds: fiberTrace = fiberTraceSet.getFiberTrace(i) # Set pixels in FiberTrace from scienceImage fiberTrace.createTrace(scienceImage) # Extract spectrum from profile spectrum = fiberTrace.extractFromProfile() spectrumSet.addSpectrum(spectrum) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- I'd expect <span class="code-keyword">this</span> to look like: xCenters, wavelengths, fiberIds = readWavelengthFile(wLenFile) # we'd actually read the DetectorMap detectorMap = DetectorMap(xCenters, wavelengths, fiberIds, ...) quartzImage = butler.get(<span class="code-quote">"calexp"</span>, dataId).getMaskedImage() ftffc = drpStella.FiberTraceFunctionFindingControl() ftpfc = fiberTraceProfileFittingControl fiberTraceSet = drpStella.findAndTraceApertures(quartzImage, detectorMap, ftffc, ftpfc) which would now have profiles set. Extracting a spectrum would then consist of: scienceImage = butler.get(<span class="code-quote">"calexp"</span>, dataId).getMaskedImage() spectrumSet = drpStella.extractSpectrumSet(scienceImage, fiberTraceSet) Note in particular that we <span class="code-keyword">do</span> not set the <span class="code-quote">"trace"</span> images in the FiberTrace; in fact, I don't think that we need them at all. We <span class="code-keyword">do</span> need code to <span class="code-keyword">return</span> a MaskedImage of the Profile (in the CCD coordinate system, with XY0 set and mask bits to indicate the trace). The FiberTrace does, of course, need to know all about its profile but I think that the current code is more complex than is really necessary. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- A FiberTrace has a number of data members: std::vector<PTR(std::vector<<span class="code-object">float</span>>)> _overSampledProfileFitXPerSwath; std::vector<PTR(std::vector<<span class="code-object">float</span>>)> _overSampledProfileFitYPerSwath; std::vector<PTR(std::vector<<span class="code-object">float</span>>)> _profileFittingInputXPerSwath; std::vector<PTR(std::vector<<span class="code-object">float</span>>)> _profileFittingInputYPerSwath; std::vector<PTR(std::vector<<span class="code-object">float</span>>)> _profileFittingInputXMeanPerSwath; std::vector<PTR(std::vector<<span class="code-object">float</span>>)> _profileFittingInputYMeanPerSwath; What are these? 2-D arrays? why are they vectors of vectors, and what <span class="code-keyword">do</span> they all mean? PTR(lsst::afw::image::MaskedImage<ImageT, MaskT, VarianceT>) _trace; Why <span class="code-keyword">do</span> we need <span class="code-keyword">this</span> in the data structure? PTR(lsst::afw::image::Image<<span class="code-object">float</span>>) _profile; Why is <span class="code-keyword">this</span> an image, not a MaskedImage? ndarray::Array<<span class="code-object">float</span>, 2, 1> _xCentersMeas; What are these? Do they need to be in the class? ndarray::Array<<span class="code-object">float</span>, 1, 1> _xCenters; In what sense are these not measured? std::size_t _iTrace; Is <span class="code-keyword">this</span> a fiberId? bool _isTraceSet; bool _isProfileSet; bool _isFiberTraceProfileFittingControlSet; This will go once the profile extraction is part of the trace generation PTR(<span class="code-keyword">const</span> FiberTraceFunction) _fiberTraceFunction; This isn<span class="code-quote">'t a function, it'</span>s a set of parameters. What <span class="code-keyword">do</span> the coefficients mean -- there seems to be only one per trace. PTR(FiberTraceProfileFittingControl) _fiberTraceProfileFittingControl; There seems to be lots of old code in here. What<span class="code-quote">'s the telluric code doing -- we'</span>ll never turn it on. If it simplifies the code we should delete all references to Piskunov. And methods: <span class="code-comment">// ctors/dtors </span> explicit FiberTrace(std::size_t width = 0, std::size_t height = 0, std::size_t iTrace = 0); Do we need a <span class="code-keyword">default</span> ctor? explicit FiberTrace(PTR(<span class="code-keyword">const</span> MaskedImageT) <span class="code-keyword">const</span>& maskedImage, PTR(<span class="code-keyword">const</span> FiberTraceFunction) <span class="code-keyword">const</span>& fiberTraceFunction, std::size_t iTrace=0); FiberTrace(FiberTrace<ImageT, MaskT, VarianceT> <span class="code-keyword">const</span>& fiberTrace); FiberTrace(FiberTrace<ImageT, MaskT, VarianceT> & fiberTrace, bool <span class="code-keyword">const</span> deep); <span class="code-comment">// why two, not a <span class="code-keyword">default</span> argument (here or in the pybind11 bindings)? </span><span class="code-comment">// <span class="code-keyword">do</span> we ever need to copy FiberTraces? </span> virtual ~FiberTrace() {} -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- <span class="code-comment">// getters and setters </span> <span class="code-comment">// <span class="code-quote">"trace"</span> </span> PTR(MaskedImageT) getTrace(); <span class="code-keyword">const</span> PTR(<span class="code-keyword">const</span> MaskedImageT) getTrace() <span class="code-keyword">const</span>; void setTrace(PTR(MaskedImageT) & trace); void createTrace(PTR(<span class="code-keyword">const</span> MaskedImageT) <span class="code-keyword">const</span>& maskedImage); I think all of these can go. <span class="code-comment">// <span class="code-quote">"profile"</span> </span> PTR(lsst::afw::image::Image<<span class="code-object">float</span>>) getProfile() <span class="code-keyword">const</span>; This needs to be in CCD coordinates, not internal <span class="code-quote">"sliced"</span> images. void setProfile( PTR(lsst::afw::image::Image<<span class="code-object">float</span>>) <span class="code-keyword">const</span>& profile); When <span class="code-keyword">do</span> we <span class="code-keyword">do</span> <span class="code-keyword">this</span>? <span class="code-comment">// ID number of <span class="code-keyword">this</span> trace (_iTrace) to <span class="code-keyword">this</span> number </span> void setITrace(<span class="code-keyword">const</span> std::size_t iTrace); Shouldn't be needed std::size_t getITrace() <span class="code-keyword">const</span>; ITrace -> FiberId <span class="code-comment">// dimensions of <span class="code-keyword">this</span> FiberTrace </span> std::size_t getWidth() <span class="code-keyword">const</span>; std::size_t getHeight() <span class="code-keyword">const</span>; What are these used <span class="code-keyword">for</span>? <span class="code-comment">// Check <span class="code-keyword">if</span> things are set </span> bool isTraceSet() <span class="code-keyword">const</span>; bool isProfileSet() <span class="code-keyword">const</span>; bool isFiberTraceProfileFittingControlSet() <span class="code-keyword">const</span>; All of these must always be set in the construction step. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- <span class="code-comment">// Extract the spectrum of <span class="code-keyword">this</span> fiber trace using the _profile </span> PTR(SpectrumT) extractFromProfile(); No; <span class="code-keyword">this</span> is a free function that takes an image and a fiber trace. <span class="code-comment">// Simple Sum Extraction of <span class="code-keyword">this</span> fiber trace </span> PTR(SpectrumT) extractSum(); No; <span class="code-keyword">this</span> should be an option to the free function. <span class="code-comment">// FiberTraceFunction (<span class="code-keyword">const</span>) </span> <span class="code-keyword">const</span> PTR(<span class="code-keyword">const</span> FiberTraceFunction) getFiberTraceFunction() <span class="code-keyword">const</span>; void setFiberTraceFunction(PTR(FiberTraceFunction <span class="code-keyword">const</span>) fiberTraceFunction); Why <span class="code-keyword">do</span> you need these? Maybe the getter <span class="code-comment">// Return FiberTraceProfileFittingControl </span> PTR(FiberTraceProfileFittingControl) getFiberTraceProfileFittingControl() <span class="code-keyword">const</span>; I'm not sure you need the getter void setFiberTraceProfileFittingControl(PTR(FiberTraceProfileFittingControl) <span class="code-keyword">const</span>&); void setFiberTraceProfileFittingControl(PTR(FiberTraceProfileFittingControl <span class="code-keyword">const</span>) <span class="code-keyword">const</span>&); Why <span class="code-keyword">do</span> you need both of these? <span class="code-comment">// fitted x-centers of the fiber trace </span> <span class="code-keyword">const</span> ndarray::Array< <span class="code-object">float</span>, 1, 1 > getXCenters() <span class="code-keyword">const</span>; OK void setXCenters( ndarray::Array< <span class="code-object">float</span>, 1, 1 > <span class="code-keyword">const</span>& xCenters ); Why <span class="code-keyword">do</span> you need <span class="code-keyword">this</span>? <span class="code-comment">// Return the measured x-centers of the fiber trace </span> ndarray::Array< <span class="code-object">float</span>, 2, 1 > getXCentersMeas() <span class="code-keyword">const</span>; void setXCentersMeas( ndarray::Array< <span class="code-object">float</span>, 2, 1 > <span class="code-keyword">const</span>& ); How is <span class="code-keyword">this</span> different from getXCenters? <span class="code-comment">// Return an image containing the reconstructed 2D spectrum of the FiberTrace </span> <span class="code-comment">// @param spectrum : 1D spectrum to reconstruct the 2D image from </span> PTR(Image) getReconstructed2DSpectrum(SpectrumT <span class="code-keyword">const</span>& spectrum) <span class="code-keyword">const</span>; Should be a MaskedImage, I think. In CCD coords, of course. <span class="code-comment">// Return an image containing the reconstructed background of the FiberTrace </span> <span class="code-comment">// @param backgroundSpectrum : 1D spectrum to reconstruct the 2D image from </span> PTR(Image) getReconstructedBackground(SpectrumT <span class="code-keyword">const</span>& backgroundSpectrum) <span class="code-keyword">const</span>; Why <span class="code-keyword">do</span> you need <span class="code-keyword">this</span> too? <span class="code-comment">// Return an image containing the reconstructed 2D spectrum + background of the FiberTrace </span> <span class="code-comment">// @param spectrum : 1D spectrum to use <span class="code-keyword">for</span> the 2D reconstruction </span> <span class="code-comment">// @param background : 1D background spectrum to use <span class="code-keyword">for</span> the 2D reconstruction </span> PTR(Image) getReconstructed2DSpectrum(SpectrumT <span class="code-keyword">const</span>& spectrum, SpectrumT <span class="code-keyword">const</span>& background) <span class="code-keyword">const</span>; Why <span class="code-keyword">do</span> you need <span class="code-keyword">this</span> too? /** * @brief Calculate the spatial profile <span class="code-keyword">for</span> the FiberTrace * Normally <span class="code-keyword">this</span> would be a Flat FiberTrace, but in principle, <span class="code-keyword">if</span> the spectrum * shows some kind of continuum, the spatial profile can still be calculated */ void calcProfile(); What does <span class="code-keyword">this</span> <span class="code-keyword">do</span>? /** * @brief Helper function <span class="code-keyword">for</span> calcProfile, calculates profile <span class="code-keyword">for</span> a swath * A swath is approximately FiberTraceProfileFittingControl.swathWidth <span class="code-object">long</span> * Each swath is overlapping the previous swath <span class="code-keyword">for</span> half of the swath width * spectrum: * |----------------------------------------------------------------- * swaths: * |---------------|--------------|--------------|--------------|---- * |---------------|--------------|--------------|----------- * @param imageSwath : array containing the CCD image of the FiberTrace swath * @param maskSwath : array containing the mask of the FiberTrace swath * @param varianceSwath : array containing the variance of the FiberTrace swath * @param xCentersSwath : 1D array containing the x center positions <span class="code-keyword">for</span> the swath * @param iSwath : number of swath */ ndarray::Array<<span class="code-object">float</span>, 2, 1> calcProfileSwath(ndarray::Array<ImageT <span class="code-keyword">const</span>, 2, 1> <span class="code-keyword">const</span>& imageSwath, ndarray::Array<MaskT <span class="code-keyword">const</span>, 2, 1> <span class="code-keyword">const</span>& maskSwath, ndarray::Array<VarianceT <span class="code-keyword">const</span>, 2, 1> <span class="code-keyword">const</span>& varianceSwath, ndarray::Array<<span class="code-object">float</span> <span class="code-keyword">const</span>, 1, 1> <span class="code-keyword">const</span>& xCentersSwath, std::size_t <span class="code-keyword">const</span> iSwath); Why is <span class="code-keyword">this</span> part of the <span class="code-keyword">public</span> <span class="code-keyword">interface</span>? Those image swaths should be invisible to the user. /** * @brief Calculate boundaries <span class="code-keyword">for</span> the swaths used <span class="code-keyword">for</span> profile calculation * @param swathWidth_In : Approximate width <span class="code-keyword">for</span> the swaths, will be adjusted * to fill the length of the FiberTrace with equally sized swaths * @<span class="code-keyword">return</span> 2D array containing the pixel numbers <span class="code-keyword">for</span> the start and the end * of each swath */ ndarray::Array<std::size_t, 2, 1> calcSwathBoundY(<span class="code-keyword">const</span> std::size_t swathWidth_In) <span class="code-keyword">const</span>; Ditto <span class="code-comment">// Return the coefficients of the trace function of the xCenters </span> ndarray::Array<<span class="code-object">float</span>, 1, 1> getTraceCoefficients() <span class="code-keyword">const</span>; Why <span class="code-keyword">do</span> we need these? I can see a function that returns the model <span class="code-comment">// Return a smart pointer to <span class="code-keyword">this</span> FiberTrace </span> PTR(FiberTrace) getPointer(); Why <span class="code-keyword">do</span> we need <span class="code-keyword">this</span>? -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- template<typename ImageT, typename VarianceT=lsst::afw::image::VariancePixel> FindCenterPositionsOneTraceResult findCenterPositionsOneTrace( PTR(lsst::afw::image::Image<ImageT>) & ccdImage, PTR(lsst::afw::image::Image<VarianceT>) & ccdImageVariance, PTR(<span class="code-keyword">const</span> FiberTraceFunctionFindingControl) <span class="code-keyword">const</span>& fiberTraceFunctionFindingControl); should be in anon namespace ndarray::Array<<span class="code-object">float</span>, 1, 1> calculateXCenters(PTR(<span class="code-keyword">const</span> ::pfs::drp::stella::FiberTraceFunction) <span class="code-keyword">const</span>& fiberTraceFunctionIn, std::size_t <span class="code-keyword">const</span>& ccdHeightIn = 0, std::size_t <span class="code-keyword">const</span>& ccdWidthIn = 0); ndarray::Array<<span class="code-object">float</span>, 1, 1> calculateXCenters(PTR(<span class="code-keyword">const</span> ::pfs::drp::stella::FiberTraceFunction) <span class="code-keyword">const</span>& fiberTraceFunctionIn, ndarray::Array<<span class="code-object">float</span>, 1, 1> <span class="code-keyword">const</span>& yIn, std::size_t <span class="code-keyword">const</span>& ccdHeightIn = 0, std::size_t <span class="code-keyword">const</span>& ccdWidthIn = 0); Why <span class="code-keyword">do</span> we need both in the <span class="code-keyword">public</span> API? And why not a member function? template< typename ImageT, typename MaskT=lsst::afw::image::MaskPixel, typename VarianceT=lsst::afw::image::VariancePixel > PTR(FiberTrace< ImageT, MaskT, VarianceT >) makeNormFlatFiberTrace( PTR(<span class="code-keyword">const</span> lsst::afw::image::MaskedImage< ImageT, MaskT, VarianceT >) <span class="code-keyword">const</span>& maskedImage, PTR(<span class="code-keyword">const</span> ::pfs::drp::stella::FiberTraceFunction) <span class="code-keyword">const</span>& fiberTraceFunctionWide, PTR(<span class="code-keyword">const</span> ::pfs::drp::stella::FiberTraceFunctionControl) <span class="code-keyword">const</span>& fiberTraceFunctionControlNarrow, PTR(<span class="code-keyword">const</span> ::pfs::drp::stella::FiberTraceProfileFittingControl) <span class="code-keyword">const</span>& fiberTraceProfileFittingControl, ImageT minSNR = 100., std::size_t iTrace = 0); Unused template< typename ImageT, typename MaskT, typename VarianceT, typename T, typename U, <span class="code-object">int</span> I > void assignITrace( FiberTraceSet< ImageT, MaskT, VarianceT > & fiberTraceSet, ndarray::Array< T, 1, I > <span class="code-keyword">const</span>& traceIds, ndarray::Array< U, 1, I > <span class="code-keyword">const</span>& xCenters ); Part of the construction (and supported by DetectorMap) void addFiberTraceToCcdArray( FiberTrace< ImageT, MaskT, VarianceT > <span class="code-keyword">const</span>& fiberTrace, lsst::afw::image::Image< arrayT > <span class="code-keyword">const</span>& fiberTraceRepresentation, ndarray::Array< ccdImageT, 2, dim > & ccdArray ); void addFiberTraceToCcdImage( FiberTrace< ImageT, MaskT, VarianceT > <span class="code-keyword">const</span>& fiberTrace, lsst::afw::image::Image< arrayT > <span class="code-keyword">const</span>& fiberTraceRepresentation, lsst::afw::image::Image< ccdImageT > & ccdImage ); void addArrayIntoArray( ndarray::Array< smallT, 2, I > <span class="code-keyword">const</span>& smallArr, ndarray::Array< std::size_t, 2, 1 > <span class="code-keyword">const</span>& xMinMax, std::size_t <span class="code-keyword">const</span>& yMin, ndarray::Array< bigT, 2, J > & bigArr ); We don't need these once we can <span class="code-keyword">return</span> an image of the profile dataXY<CoordT> ccdToFiberTraceCoordinates( dataXY<CoordT> <span class="code-keyword">const</span>& ccdCoordinates, pfs::drp::stella::FiberTrace<ImageT, MaskT, VarianceT> <span class="code-keyword">const</span>& fiberTrace); Unused dataXY<CoordT> fiberTraceCoordinatesRelativeTo( dataXY<CoordT> <span class="code-keyword">const</span>& fiberTraceCoordinates, dataXY<CoordT> <span class="code-keyword">const</span>& ccdCoordinatesCenter, pfs::drp::stella::FiberTrace<ImageT, MaskT, VarianceT> <span class="code-keyword">const</span>& fiberTrace ); unused } template<typename T> <span class="code-keyword">const</span> T* getRawPointer(<span class="code-keyword">const</span> PTR(<span class="code-keyword">const</span> T) & ptr); Fortunately unused. This is a Very bad idea. /** * @brief mark FiberTrace pixels in Mask image * @param fiberTrace : FiberTrace to mark in maskedImage's Mask * @param mask : mask to mark the FiberTrace in * @param value : value to Or into the FiberTrace mask */ template< typename ImageT, typename MaskT, typename VarianceT > void markFiberTraceInMask( PTR( FiberTrace< ImageT, MaskT, VarianceT > ) <span class="code-keyword">const</span>& fiberTrace, PTR( lsst::afw::image::Mask< MaskT > ) <span class="code-keyword">const</span>& mask, MaskT value = 1); Superceded by getting the MaskedImage back. </pre> </div></div> <p>We discussed this 2017-07-28. Marking it as done.</p> Relates PIPE2D-219 PIPE2D-235 PIPE2D-236 PIPE2D-240 Development Rank 0|031vid:xi07fhk0i4