[PIPE2D-678] Add visit and date restrictions for generateReductionSpec Created: 10/Dec/20  Updated: 14/Jan/21  Resolved: 14/Jan/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: sogo.mineo
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PIPE2D-423 Regular DRP processing of LAM and Sub... In Progress
Sprint: 2DDRP-2021 A
Reviewers: price

 Description   

generateReductionSpec.py currently extracts reduction specs for the entire opdb. This will not scale, so we should add options to restrict the list of visits by a visit and/or date range.



 Comments   
Comment by sogo.mineo [ 12/Jan/21 ]

I wonder how to express a time span in command lines.

First, I think that the end of a time span should be exclusive. `Sps_exposure.time_exp_start` has microsecond accuracy. If the end of a time span is inclusive, it is difficult to iterate over dates without losing unfortunate records in the gaps at midnights. I mean, if a user wants to do this:

    somecommand.py --daterange=2021-01-01T00:00:00..2021-01-01T23:59:59
    somecommand.py --daterange=2021-01-02T00:00:00..2021-01-02T23:59:59
    somecommand.py --daterange=2021-01-03T00:00:00..2021-01-03T23:59:59
    :

Then, a record at 2021-01-01T23:59:59.999999 will be lost. I don't know how well SQLite3 works, but I know some poor programs that occasionally output 2021-01-01T23:59:60.000000 by rounding 2021-01-01T23:59:59.9999996 (This is not related to leap seconds). So, it may be useless for the user to write

    somecommand.py --daterange=2021-01-01T00:00:00..2021-01-01T23:59:59.999999
    somecommand.py --daterange=2021-01-02T00:00:00..2021-01-02T23:59:59.999999
    somecommand.py --daterange=2021-01-03T00:00:00..2021-01-03T23:59:59.999999
    :

because these commands still cannot catch 2021-01-01T23:59:60.000000. The user cannot simply say 2021-01-01T23:59:60 because python's datetime.datetime does not permit 60 for the second.

Second, if I make the end of a time span exclusive, then it is confusing to keep using ... Should I use '/' as in https://en.wikipedia.org/wiki/ISO_8601 ?

Comment by price [ 13/Jan/21 ]

What about having separate --begin and --end arguments? Both could default to unset, specifying an open range.

Comment by sogo.mineo [ 13/Jan/21 ]

I added -visit-start, visit-end, date-start, and -date-end. Could you review the change?

Comment by price [ 14/Jan/21 ]

Very nice work, thanks!

Comment by sogo.mineo [ 14/Jan/21 ]

Merged to master. Thank you.

Generated at Sat Feb 10 15:56:20 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.