# add hsc driver script to validate_drp

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
8
• Team:
SQuaRE

#### Description

Add an equivalent of examples/runChftTest.sh to validate_drp to process validation_data_hsc.

#### Activity

Joshua Hoblitt created issue -
Field Original Value New Value
Epic Link DM-6196 [ 24712 ]
 Description Add an equivalent of {{examples/runChftTest.sh}} to {{validate_drp}} to process {{ validation_data_hsc}}. Add an equivalent of {{examples/runChftTest.sh}} to {{validate_drp}} to process {{ validation\_data\_hsc}}.
 Description Add an equivalent of {{examples/runChftTest.sh}} to {{validate_drp}} to process {{ validation\_data\_hsc}}. Add an equivalent of {{examples/runChftTest.sh}} to {{validate_drp}} to process {{validation_data_hsc}}.
Hide
Michael Wood-Vasey added a comment -

The full version of this would unpack the scons scripts in ci_hsc to get each of the single-visit, coadd, and photometry steps.

I would suggest that if you start with just getting the single-visit processing to run, it should be pretty straight forward.

Show
Michael Wood-Vasey added a comment - The full version of this would unpack the scons scripts in ci_hsc to get each of the single-visit, coadd, and photometry steps. I would suggest that if you start with just getting the single-visit processing to run, it should be pretty straight forward.
 Link This issue relates to DM-7606 [ DM-7606 ]
 Epic Link DM-6196 [ 24712 ] DM-5504 [ 23337 ]
Hide
Michael Wood-Vasey added a comment - - edited

Running the quick test takes less than an hour (in parallel with 16 cores) on lsst-dev01. Running the full test takes about 4 hours (in parallel with 16 cores) on lsst-dev01.

For a full run of the validation_data_hsc data using run, see on lsst-dev01.ncsa.illinois.edu:

 /home/wmwv/tmp/data_hsc 

Show
Michael Wood-Vasey added a comment - - edited Running the quick test takes less than an hour (in parallel with 16 cores) on lsst-dev01 . Running the full test takes about 4 hours (in parallel with 16 cores) on lsst-dev01 . For a full run of the validation_data_hsc data using run, see on lsst-dev01.ncsa.illinois.edu : /home/wmwv/tmp/data_hsc
 Status To Do [ 10001 ] In Progress [ 3 ]
Hide
Michael Wood-Vasey added a comment -

Basic version created with

 runHscTest.sh runHscQuickTest.sh 

examples. These do not use the runExample.sh framework because the call to singleFrameDriver.py used in the HSC processing is a bit different than the call to processCcd.py.

The runHscQuickTest.sh uses just the data available in ci_hsc, while runHscTest.sh uses the full set of data in validation_data_hsc.

Show
Michael Wood-Vasey added a comment - Basic version created with runHscTest.sh runHscQuickTest.sh examples. These do not use the runExample.sh framework because the call to singleFrameDriver.py used in the HSC processing is a bit different than the call to processCcd.py. The runHscQuickTest.sh uses just the data available in ci_hsc , while runHscTest.sh uses the full set of data in validation_data_hsc .
 Reviewers Joshua Hoblitt [ jhoblitt ] Status In Progress [ 3 ] In Review [ 10004 ]
 Comment [ Basic version created with runHscTest.sh runHscQuickTest.sh examples. These do not use the {{runExample.sh}} framework because the call to {{singleFrameDriver.py}} used in the HSC processing is a bit different than the call to {{processCcd.py}}. The {{runHscQuickTest.sh}} uses just the data available in {{ci_hsc}}, while {{runHscTest.sh}} uses the full set of data in {{validation_data_hsc}}. ]
 Story Points 4
Hide
Joshua Hoblitt added a comment -

Michael Wood-Vasey It looks like ci_hsc doesn't install anything. I've confirmed this from a clean lsstsw build in a container.

 [vagrant@f73cee0856e0 Linux64]$tree ci_hsc/ ci_hsc/ -- master-ga3affa3386  |-- _build.log  -- ups  |-- ci_hsc.table  -- pkginfo   2 directories, 3 files  Show Joshua Hoblitt added a comment - Michael Wood-Vasey It looks like ci_hsc doesn't install anything. I've confirmed this from a clean lsstsw build in a container. [vagrant @f73cee0856e0 Linux64]$ tree ci_hsc/ ci_hsc/ -- master-ga3affa3386 |-- _build.log -- ups |-- ci_hsc.table -- pkginfo   2 directories, 3 files
Hide
Michael Wood-Vasey added a comment -

Oh. I didn't realize this. I understand now. You're saying ci_hsc doesn't install in the stack install, it just runs the tests.

I've been directly running setup on a separate directory copy of the build directory; when doing that then CI_HSC_DIR gets defined. That's how I've been using it.

I'll have to think about how to refer to these data if they don't get installed.

Show
Michael Wood-Vasey added a comment - Oh. I didn't realize this. I understand now. You're saying ci_hsc doesn't install in the stack install, it just runs the tests. I've been directly running setup on a separate directory copy of the build directory; when doing that then CI_HSC_DIR gets defined. That's how I've been using it. I'll have to think about how to refer to these data if they don't get installed.
Hide
Joshua Hoblitt added a comment -

My preference would be for ci_hsc to install like other products but we can clone it by hand if necessary.

Show
Joshua Hoblitt added a comment - My preference would be for ci_hsc to install like other products but we can clone it by hand if necessary.
Hide
Michael Wood-Vasey added a comment - - edited

There's not a particular use case for runHscQuickTest.sh that's important. It's really only as a minor time savings to catch reasons runHscTest.sh might fail without having to download the 700 GB of validation_data_hsc.

I will add a descriptive note in the README about having to explicitly set up the cloned repo of ci_hsc if one wants to use runHscQuickTest.sh and a reminder that ci_hsc doesn't install anything into the stack dir.

The current ticket was meant to produce something that could be run, e.g., by SQuaSH. That's the full runHscTest.sh script.

Show
Michael Wood-Vasey added a comment - - edited There's not a particular use case for runHscQuickTest.sh that's important. It's really only as a minor time savings to catch reasons runHscTest.sh might fail without having to download the 700 GB of validation_data_hsc . I will add a descriptive note in the README about having to explicitly set up the cloned repo of ci_hsc if one wants to use runHscQuickTest.sh and a reminder that ci_hsc doesn't install anything into the stack dir. The current ticket was meant to produce something that could be run, e.g., by SQuaSH. That's the full runHscTest.sh script.
 Story Points 4 5
Hide
Joshua Hoblitt added a comment -

St. Michael Wood-Vasey has been very patient while we iterated through the ops model for running the hsc dataset. In the end, we determined that validation_data_hsc in too large and cumbersome to allow EUPS to manage it as a product dependency. It will be the responsibility of the end-user (or software agent) to manually clone/setup this the test dataset.

Show
Joshua Hoblitt added a comment - St. Michael Wood-Vasey has been very patient while we iterated through the ops model for running the hsc dataset. In the end, we determined that validation_data_hsc in too large and cumbersome to allow EUPS to manage it as a product dependency. It will be the responsibility of the end-user (or software agent) to manually clone/ setup this the test dataset.
 Status In Review [ 10004 ] Reviewed [ 10101 ]
 Link This issue is triggering DM-9259 [ DM-9259 ]
 Story Points 5 8
Hide
Michael Wood-Vasey added a comment -

Thanks very much to Joshua Hoblitt for the in-depth review that helped ensure that this work is ready for integration into regular processing.

Show
Michael Wood-Vasey added a comment - Thanks very much to Joshua Hoblitt for the in-depth review that helped ensure that this work is ready for integration into regular processing.
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]
 Link This issue is triggering DM-9290 [ DM-9290 ]
 Link This issue is triggering DM-9293 [ DM-9293 ]
 Epic Link DM-5504 [ 23337 ] DM-7605 [ 26790 ]

#### People

Assignee:
Michael Wood-Vasey
Reporter:
Joshua Hoblitt
Reviewers:
Joshua Hoblitt
Watchers:
Angelo Fausti, Frossie Economou, John Parejko, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey