Is it okay to just list all bad pixels (width=height=1)? While sometimes there are 2 or more neighboring bad pixels, there is one irregular shape making up most of the bad pixels (36 out of 42 bad pixels). In other words, does the bad-pixel module reject neighboring bad pixels automatically?

Created obs_pfs/python/lsst/obs/pfs/genDefectList.py to create a list of defects for CCD 5 from 2 sets of flats without spectrograph with different exposure times, taken in the same exposure run with the same setup. All pixels which are non-linear are listed as bad pixels. While there were about 5 2s exposures, all longer exposures had only been taken twice. The 2s exposures are medianed, for the longer exposures I chose the minimum values from both exposures to get rid of cosmics. Apparently that worked pretty well. I have excluded the first and last 40 columns/rows as there is a systematic at the edges of the image FlatWithLongExpTime / FlatWithShortExpTime. Not quite sure why this is the case. There is also a few strange columns where the 2 physical chips meet. Seems to me that the last 3 columns of the 1st chip and the first 5 columns of the 2nd chip are bad...

Somehow the workflow has changed in Jira. I can't just move it from "In Progress" to "In Review" anymore...

The workflow is different in this project ("SCAM") because it's been configured that way. Since cloomis is listed as the administrator for the project, I don't just want to go in and change it unilaterally. Craig, is this part of some grand scheme, or is it just because nobody got around to configuring the project?

I suggest that you'll need input from somebody other than me (rhl?) as to how it's acceptable to list bad pixels.

Workflow issues should now be resolved (thanks Craig).

At our meeting of 2016-07-15, we agreed that Andreas would discuss the above issues with rhl and/or cloomis to understand better what's happening before we close this.

The procedure here is now well defined, but cloomis has provided some new data. aritter is going to regenerate the defect list based on that and then close this ticket. He assures us that process is very easy & quick.

Changes are in obs_pfs tickets/SCAM-3. Add genDefectsList.py and genDefectsFits.py to create defect list and fits files during compilation for CCD number 5 from the data taken 2015-12-04 by Craig Loomis. Pixels are assumed to be bad if they are not linear comparing 2s flats to 30s flats (taken without the spectrograph). 1 side of each amplifier appears to show a leftover of a structure seen in Biases which apparently doesn't get fully corrected for by bias subtraction.

Trying to create the dataset to test ticket on tiger. Will put back into review once done. At the moment I'm getting an error when running
ingestImages.py PFS raw/2015-12-04/*.fits --mode link -L warn
lsst.pex.exceptions.wrappers.InvalidParameterError:
File "src/table/AmpInfo.cc", line 128, in static std::shared_ptr<lsst::afw::table::AmpInfoTable> lsst::afw::table::AmpInfoTable::make(const lsst::afw::table::Schema&)
Schema for AmpInfo must contain at least the keys defined by makeMinimalSchema().

lsst::pex::exceptions::InvalidParameterError: 'Schema for AmpInfo must contain at least the keys defined by makeMinimalSchema().'

I remember having had this or a similar problem before but at least at the moment I can't remember/find the solution...

It's a bug in ip_isr that I fixed a couple of days ago. Use master ip_isr

See obs_pfs tickets/SCAM-3. You will need drp_stella tickets/PIPE2D-10 which contains the needed functions in the pfs.drp.stella(.math) namespace. On tiger obs_pfs, drp_stella_data, and drp_stella are in /tigress/HSC/PFS. To test run

python /tigress/HSC/PFS/obs_pfs/python/lsst/obs/pfs/genDefectList.py -homeDir='/tigress/HSC/PFS/PFS/rerun/azuri/tmp'

python /tigress/HSC/PFS/obs_pfs/python/lsst/obs/pfs/genDefectList.py -homeDir='/tigress/HSC/PFS/PFS/rerun/azuri/tmp'

Please explain that command. What does "-homeDir" do (and why only one -). How can I run this without using some data in an azuri tmp rerun?

I'm also confused as to why I have to run this. Are you re-calculating the defect maps? That's not what we want to do – we want to check them into jobs_pfs as they should be quite short and stable.

Moved back to "In Progress"

Do we want to create the defect list starting from raw images or from postISR images? I'm asking because the rerun/azuri/tmp refers to the place where the postISR images are currently stored...

Why are there print statements checked into genDefectFits.py? I would expect this sort of debugging information to be removed before you put the code up for review.

Using your proposed algorithm, I think you need to run post-ISR but with defect removal turned off.

For example, if there is fixed pattern noise in the bias or hot pixels then that will make your linearity approach unhappy. Another way to do this would be to use more than two thresholds and fit a straight line.

I suggest that we bootstrap from the imaging flats taken at JHU. We will have those for all detectors. The r1/b1 ones are vignetted, yes, but they should be a clean start for a defect list. I believe the CCD mask will be redone.

For now, in the tigress JHU CALIB/, there are r1 and b1 superflats. Not enough exposures to be good flats, but good enough for this.

I'm suspicious about this defect list. Almost all of the defects are single pixels, and they look just fine to me. Did you compare the list generated from two independent pairs of flats?

The exception is the cluster around (2303, 136). I don't see any problem in the flats that I looked at; can you attach postage stamps that illustrate the problem? (N.b. use ! attachment.png ! to inline the images when you describe them in your comment – no spaces between the !!)

Andreas stands by his defects! Please look at the 30s/2s flat pairs

Andreas: can you provide the visit numbers I should be using to check the defects?

I used visits 6301-6758 Biases and Flats with 2s and 30s. They are on tiger in /tigress/HSC/PFS/JHU/raw/2015-12-04. I should have mentioned that the visit numbers are set as defaults in genDefectList.py, parameters 'visitLow' and 'visitHigh'. The parameters 'expTimeLow' and 'expTimeHigh' specify the exposure times to use (defaulted to 2 and 30, respectively).

Probably postISRCCD, in which case you need to provide sufficient instructions in this ticket for generating the files you need.

Here is a postage stamp that illustrates the problem with the cluster of bad pixels around (2303, 136):

Re the individual pixels I found that the majority disappeared when I set the sigma-clipping factor to 5.5 or higher, possibly due the low SNR in the 2s exposure times or some excessive noise in the test data?

expTimeLow expTimeHigh sigma-clipping_factor nBadPixels
2 30 5.0 167
2 30 5.5 78
2 30 6.0 65
20 30 5.0 79
20 30 5.5 52
20 30 6.0 45

38 pixels out of the 45 found using Flats with 20s and 30s exposure (sigma-clipping factor 6.0) are in the cluster around (2303, 136).

Can I suggest trying the following data for r1, which are a bit newer (and so from my POV more trustworthy):

BIAS:
7251..7259

DARKs (1200s):
7291..7294

imaging FLATs (2s, ~1200 cnts):
7260..7264

I'd prefer not to have to answer for the early December data if I don't have to.

I have done a better inventory of the flats. Turns out that we do not have valid later sets of flats with two different levels (mainly because the monochrometer wavelength was not reset after QE tests, so while the flats were long the illumination was at ~1.1um...), but we do have considerable numbers of flats taken with non-final bias levels, etc. For the purpose of measuring non-linearity they should be fine, though you might well need to make super-biases for them, and only for them. Specifically the bias levels are ~3k instead of ~1k, and the relative levels will also be quite different from what they finally got set to.

See JHU/raw/2015-12-03/LOG.txt, specifically the exposures after:
2015-12-03T19:30:09 ... all flats (38 exposures)

If you can use the lower levels, you might be better off with the low flats listed in JHU/raw/2015-12-04/LOG.txt. Those should all have the same configuration and have close to the final electronic settings.

You will want to look at the raw images to choose the ones you want. Again, I don't vouch for the headers for these earlier files.

Craig, do you have an explanation why there are random pixels which show a higher noise than the others? Even setting the sigma clipping to 6.0 standard deviations does not exclude all pixels affected by apparently pure noise.

Can you give coordinates, please?

The lists of bad pixels with different sigma clippings are in /tigress/HSC/PFS/temp/INSTRM-3:
defects_10s_vs_30s_sig5.0.dat 83 bad pixels
defects_10s_vs_30s_sig5.5.dat 33 bad pixels
defects_10s_vs_30s_sig6.0.dat 22 bad pixels
defects_10s_vs_30s_sig6.5.dat 20 bad pixels

The code is in obs_pfs tickets/INSTRM-3. Data is on tiger in /tigress/HSC/PFS/temp/INSTRM-3.
The pipeline packages are in /tigress/HSC/PFS/PFS_DRP.
Command to execute genDefectList.py is:

genDefectList.py --root /tigress/HSC/PFS/temp/INSTRM-3 --expTimeLow 10 --expTimeHigh 30 --verbose

Defect list created is in obs_pfs/pfs/defects/2015-12-01/defects.dat. Currently there are 22 pixels in the list, 18 of which are in the cluster around (140,2307).
Documentation for genDefectList.py is in drp_stella/sphinx/user/genDefectList.rst.

cloomis — aritter and I have just been discussing his question above above. Do you have any thoughts about what's going on here?

The test data in /tigress/HSC/PFS/temp/INSTRM-3 are:

visits 6738-6749 Bias
visits 6341, 6342, 6427, 6428, 6519, 6520 10s Flat
visits 6365, 6366, 6451, 6452, 6543, 6544 20s Flat
visits 6713, 6714, 6735, 6736, 6757, 6758 30s Flat

aritter When I look at the postISRCCD images in /tigress/HSC/PFS/temp/INSTRM-3 (e.g. 6341, 6520, 6758) they seem to have BAD | INTRP set in the cores of the objects that you are labelling as defects (e.g. (132, 2303)). Can you confirm that they were not set in the data you used to determine the defects?

D'oh! I always had the isr.doDefect=False flag set but must have forgotten that in the INSTRM-3 directory. Having said that, there are even more bad pixels identified if the postISRCCD images are created without defect correction, so the structure around 132, 2303 is still non-linear.

Did you push the fix? Or update /tigress/HSC/PFS/temp/INSTRM-3?

I can't get the pipeline to run on tiger (Could not import lsstcppimport; ImportError: No module named _persistenceLib,...). Copying the data over from my machine...

It works for me.

My local setups are:

ctrl_pool             tag:rhl    	rhl setup
datamodel             tag:rhl    	rhl setup
drp_stella            LOCAL:/home/rhl/PFS/drp/stella 	setup
drp_stella_data       tag:rhl    	rhl setup
obs_pfs               tag:rhl    	rhl setup
pipe_drivers          tag:rhl    	rhl setup

Otherwise v12_1

You should not have to build your own stack on tiger. Please do not do so.

The major points that need to be resolved before we can call the defects work complete are:

1. You need to be working on raw files (and run ISR yourself as part of the script).
2. This algorithms isn't quite good enough. In particular, the defect that you have a nice PNG for isn't so much non-linear as black. I think we should be masking pixels that have a QE of less than (say) 50% of the median.

Please file tickets to address these points (and respond to the github comments) before merging this ticket to master.

According to RHL the CCD IDs should be 0-11