[PIPE2D-682] Move CI to Tiger Created: 11/Dec/20  Updated: 29/Jan/22  Resolved: 30/Jan/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Reviewers: hassan

 Description   

We have been using Travis for CI, but they are now setting limits that we have exceeded.

Rubin moved their Travis operations to GitHub Actions, but they only use it for linting, which is fast. We've been using Travis for integration testing, which takes much more compute power (we only have a handful of packages, so we could get away with it), so we're in a different regime.

I propose to move our CI to Jenkins running on Tiger. We already run weekly production runs and builds through this mechanism, and it wouldn't take much to extend it to run smaller-scale integration tests on demand.

  • Write a back-end script to run the integration test under Jenkins, and report the results via e-mail and/or Slack.
  • Add Jenkins job.
  • Add a front-end script to trigger the Jenkins job.
  • Remove Travis.
  • Circulate instructions.

 Comments   
Comment by price [ 23/Jan/21 ]

Instructions for use, which I'll circulate once this procedure is approved and the changes have merged:

To start an integration test automatically via Jenkins, run on the tiger cluster:

    setup pfs_pipe2d
    $PFS_PIPE2D_DIR/jenkins/trigger_integrationTest.py <branch>

This is a fire-and-forget mechanism: you can start the run and then forget all about it until it notifies you of the result via Slack (in #drp-2d-travis with an @ notification) and e-mail. Those notifications will contain the build number, a brief description of the result (success or failure) and the directory where you can poke around the run. The directory will contain a log file (integration_<build number>.log), a shallow clone of each of the software products used in the test, and the integration test data repo (INTEGRATION), so you can reproduce and debug the failures if you need.

Comment by price [ 23/Jan/21 ]

In addition to the main feature in pfs_pipe2d, there's a single change in each of the DRP2D packages (removing the .travis.yml file) that I didn't bother creating PRs for.

Comment by price [ 30/Jan/21 ]

Merged to master.

Comment by price [ 30/Jan/21 ]

Sent instructions on using this system to reduction@pfspipe.ipmu.jp.

Generated at Sat Feb 10 15:56:25 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.