According to Jim Bosh for the butler to write/read these pfsArm files the write method should be renamed to writeFits and the read method needs to be a static method called readFits returning a new pfsArm object. Apparently this is by far the easiest way to make the butler call the appropriate methods. Shall I make these changes?

See tickets/PIPE2D-35 for drp_stella and datamodel.
The mystery of the empty file was caused by the butler wanting to write a temporary file and then rename that to the output file name. The PfsArm.write(...) method determined the file name independently of the butler what resulted in an empty file overwriting the output file.

Should be rebased on top of master after INFRA-36 has been merged, then put it back in review.

That change to the data model implementation is not acceptable – you're hardcoding details about your particular case. I understand why you need a method called readFits on some object, but the pfsArm object in the data model is not the right one. Please come up with some other solution.

    @staticmethod
    def readFits(fileName):
        dirName = os.path.dirname(fileName)
        visit = 4
        spectrograph = 2
        arm = 'r'
        pfsArm = PfsArm( visit, spectrograph, arm )
        
        pfsArm.read(dirName=dirName)
        return pfsArm

We do need a readFits(fileName) method for PfsArm but I added a getInfo(fileName) function that extracts the spectrograph, arm, and visit number from the fileName. drp_stella's reduceArc.py writes the correct PfsArm object now but when trying to read it back in with "myPfsArm = pfsArm.PfsArm.readFits(fileName)" I get the error

IOError: [Errno 2] No such file or directory: '/Users/azuri/spectra/pfs/PFS/rerun/azuri/tmp/pfsArm/2015-12-22/v0000004/pfsConfig-0x0bef64dc.fits'

Should the write method write the PfsConfig object as well?

You may not add a readFits method. Please use one of the other mechanisms available in the butler/mapper (using a bypass will work); I think you need

    def bypass_spArm(self, datasetType, pythonType, location, dataId):
        from pfs.datamodel.pfsArm import PfsArm

        for fn in location.locationList:
            dirName = os.path.dirname(fn)
            pfsa = PfsArm(spectrograph=dataId["spectrograph"], arm=dataId["arm"], visit=dataId["visit"])
            try:
                pfsa.read(dirName=dirName)
            except Exception as e:
                pass
            else:
                return pfsa

        raise RuntimeError("Unable to read pfsArm for %s: %s" % (dataId.items(), e))

(where the import needs to be at file scope). Note that that returns a pfsArm object, and you may need to unpack that into one of your internal objects (SpectrumSet) in your code after retrieval. Note that you'll have to change the name of this function when you fix the name from "spArm" to "pfsArm".

Did you read the data model? Yes, you do need that file, but it should be written by the simulator. There is code in datamodel to write it for you.

We discussed this at the meeting of 2016-09-19. We believe the code rhl added below should contain all we need here. This should be closed (or set to review) in time for the next meeting.

Actually, we believe rhl's code will fix reading, but we need to do something else for writing. rhl himself has volunteered to take a look at this.

This appears to be a weakness in the butler. While my fix for reading works beautifully, there's no equivalent for writing.

So we need to write a dummy class (pfsArmIO) that has readFits/writeFits methods and calls the pfsArm code.

I'll do it

Merged to master.

N.b. The work was done on branch tickets/PIPE2D-35-rhl. There is also a branch tickets/PIPE2D-35 that has a variety of changes, many unrelated to this ticket. I've filed PIPE2D-101 to salvage anything we need.