DRP end-to-end processing test

This ticket describes the end-to-end processing test of DRP at IPMU.

Scope of work in the DRP end-to-end processing test is as follows:
 
(1) Purpose
    (1.1) Confirm that the all pipelines (2D and 1D) can be processed end-to-end (we do this processing check step-by-step as follows) 
        (1.1.1) Confirm that 2D-DRP products can be passed to 1D-DRP
        (1.1.2) Confirm that the above process with Mineo-san’s system can be done
        (1.1.3) Confirm that any simulated spectra are realized with 2D-SIM and can be passed to the above process
        (1.1.4) Confirm that the processing including fiber assignment can be done as entire observing process
    (1.2) Confirm that all necessary information can be stored in (and extracted from) opDB
 
(2) Method
    (2.1) The following pipelines and processing software are tested:
        (2.1.1) 2D-SIM
        (2.1.2) 2D-DRP
        (2.1.3) 1D-DRP
        (2.1.4) Mineo-san’s processing scripts (if possible)
        (2.1.5) opDB
        (2.1.6) Yabe’s survey simulator including ETS
    (2.2) All these components are installed on a server at IPMU
    (2.3) The end-to-end processing is as follows:
        (2.3.1) Prepare target catalog with input spectra (Sean Johnson’s galaxy catalog and GA’s catalog if necessary)
        (2.3.2) Make fiber configuration and produce pfsDesign + pfsConfig
        (2.3.3) Create 2D simulated images (using 2D-SIM)
        (2.3.4) Process Mineo-san’s script to run 2D-DRP (if possible)
        (2.3.5) Process 1D-DRP with output pfsObject from 2D-DRP
        (2.3.6) Compare results to input spectra (and ETC results)
        (2.3.7) All info should be tracked with opDB
        (2.3.8) The process is repeated by several times
    (2.4) The processing is governed by a set of scripts, which should be a GitHub repository, and ideally we should have a mechanism to run the scripts automatically and regularly as continuous integration. The experience in `pfs_pipe2d` would be very useful.
    (2.5) Start these processing checks where we can start (see below)
 
(3) Tasks (probably should be filed as JIRA tickets)
    (3.1) Tasks for the purpose (1.1.1)
        (3.1.1) Install 2D-DRP & 1D-DRP on a server at IPMU (currently `gfarm` is assumed) (Note 2D-DRP has been already installed)
        (3.1.2) Run test scripts and confirm that all components work individually
        (3.1.3) Run 2D-DRP for the existing simulated data
        (3.1.4) Run 1D-DRP for the resulting products
        (3.1.5) Confirm that the results are consistent with the inputs (not caring about the performance)
    (3.2) Tasks for the purpose (1.1.2)
        (3.2.1) Install Mineo-san’s system on the same server
        (3.2.2) Run test scripts and confirm that the system works against the existing 2D-DRP system
        (3.2.3) Run the system for the above pre-defined data set
        (3.2.4) Run 1D-DRP for the resulting products
        (3.2.5) Confirm that the results are OK like the Task (3.1)
    (3.3) Tasks for the purpose (1.1.3)
        (3.3.1) Confirm that 2D-SIM works for any desired input dataset
        (3.3.2) Run the same process in (3.2) for this dataset
        (3.3.3) Check the products and confirm that the results are consistent with the inputs
        (3.3.4) Also compared to ETC results
    (3.4) Tasks for the purpose (1.1.4)
        (3.4.1) Implement the processing in the survey simulator
        (3.4.2) Prepare dedicated opDB on a server at IPMU
        (3.4.3) Run the survey simulator with fiber assignment software for real end-to-end simulations
        (3.4.4) Confirm that all processing can be completed and necessary information is stored in opDB
 
(4) Management Plan
    (4.1) Each components are manageable under GitHub and we keep them up to date on the server
    (4.2) Each task is filed as a JIRA ticket and relevant sub-tickets and we track the progress
    (4.3) Timeline (still floating)
        (4.3.1) File tickets for the tasks above: hopefully soon
        (4.3.2) Tasks (3.1): 2-3 wks? (after 1D-DRP is ready)
        (4.3.3) Tasks (3.2): 2-3 wks? (after Mineo-san’s system is ready)
        (4.3.4) Tasks (3.3): 2-3 wks?
        (4.3.5) Tasks (3.4): 2-3 wks?
    (4.4) JIRA ticketing
        (4.4.0) A root ticket for this end-to-end test => PIPE2D-605
        (4.4.1) A summary ticket for (3.1) => PIPE2D-608
            (4.4.1.1) a ticket for (3.1.1) => INFRA-278
            (4.4.1.2) a ticket for (3.1.2) => PIPE2D-606
            (4.4.1.3) a ticket for (3.1.3) => PIPE2D-607
            (4.4.1.4) a ticket for (3.1.4) & (3.1.5) => PIPE1D-14
        (4.4.2 and after) TBC (will be filed seeing the progress of task (3.1) and other components such as Mineo-san's script)