Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6328

add hsc driver script to validate_drp

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: QA
    • Labels:

      Description

      Add an equivalent of examples/runChftTest.sh to validate_drp to process validation_data_hsc.

        Attachments

          Issue Links

            Activity

            jhoblitt Joshua Hoblitt created issue -
            jhoblitt Joshua Hoblitt made changes -
            Field Original Value New Value
            Epic Link DM-6196 [ 24712 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Description Add an equivalent of {{examples/runChftTest.sh}} to {{validate_drp}} to process {{ validation_data_hsc}}. Add an equivalent of {{examples/runChftTest.sh}} to {{validate_drp}} to process {{ validation\_data\_hsc}}.
            wmwood-vasey Michael Wood-Vasey made changes -
            Description Add an equivalent of {{examples/runChftTest.sh}} to {{validate_drp}} to process {{ validation\_data\_hsc}}. Add an equivalent of {{examples/runChftTest.sh}} to {{validate_drp}} to process {{validation_data_hsc}}.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            The full version of this would unpack the scons scripts in ci_hsc to get each of the single-visit, coadd, and photometry steps.

            I would suggest that if you start with just getting the single-visit processing to run, it should be pretty straight forward.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - The full version of this would unpack the scons scripts in ci_hsc to get each of the single-visit, coadd, and photometry steps. I would suggest that if you start with just getting the single-visit processing to run, it should be pretty straight forward.
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-7606 [ DM-7606 ]
            frossie Frossie Economou made changes -
            Epic Link DM-6196 [ 24712 ] DM-5504 [ 23337 ]
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment - - edited

            Running the quick test takes less than an hour (in parallel with 16 cores) on lsst-dev01. Running the full test takes about 4 hours (in parallel with 16 cores) on lsst-dev01.

            For a full run of the validation_data_hsc data using `run`, see on lsst-dev01.ncsa.illinois.edu:

            /home/wmwv/tmp/data_hsc
            

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - - edited Running the quick test takes less than an hour (in parallel with 16 cores) on lsst-dev01 . Running the full test takes about 4 hours (in parallel with 16 cores) on lsst-dev01 . For a full run of the validation_data_hsc data using `run`, see on lsst-dev01.ncsa.illinois.edu : /home/wmwv/tmp/data_hsc
            wmwood-vasey Michael Wood-Vasey made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Basic version created with

            runHscTest.sh
            runHscQuickTest.sh
            

            examples. These do not use the runExample.sh framework because the call to singleFrameDriver.py used in the HSC processing is a bit different than the call to processCcd.py.

            The runHscQuickTest.sh uses just the data available in ci_hsc, while runHscTest.sh uses the full set of data in validation_data_hsc.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Basic version created with runHscTest.sh runHscQuickTest.sh examples. These do not use the runExample.sh framework because the call to singleFrameDriver.py used in the HSC processing is a bit different than the call to processCcd.py. The runHscQuickTest.sh uses just the data available in ci_hsc , while runHscTest.sh uses the full set of data in validation_data_hsc .
            wmwood-vasey Michael Wood-Vasey made changes -
            Reviewers Joshua Hoblitt [ jhoblitt ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Comment [ Basic version created with

            runHscTest.sh
            runHscQuickTest.sh

            examples. These do not use the {{runExample.sh}} framework because the call to {{singleFrameDriver.py}} used in the HSC processing is a bit different than the call to {{processCcd.py}}.

            The {{runHscQuickTest.sh}} uses just the data available in {{ci_hsc}}, while {{runHscTest.sh}} uses the full set of data in {{validation_data_hsc}}. ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Story Points 4
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Michael Wood-Vasey It looks like ci_hsc doesn't install anything. I've confirmed this from a clean lsstsw build in a container.

            [vagrant@f73cee0856e0 Linux64]$ tree ci_hsc/
            ci_hsc/
            `-- master-ga3affa3386
                |-- _build.log
                `-- ups
                    |-- ci_hsc.table
                    `-- pkginfo
             
            2 directories, 3 files
            

            Show
            jhoblitt Joshua Hoblitt added a comment - Michael Wood-Vasey It looks like ci_hsc doesn't install anything. I've confirmed this from a clean lsstsw build in a container. [vagrant @f73cee0856e0 Linux64]$ tree ci_hsc/ ci_hsc/ `-- master-ga3affa3386 |-- _build.log `-- ups |-- ci_hsc.table `-- pkginfo   2 directories, 3 files
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Oh. I didn't realize this. I understand now. You're saying ci_hsc doesn't install in the stack install, it just runs the tests.

            I've been directly running setup on a separate directory copy of the build directory; when doing that then CI_HSC_DIR gets defined. That's how I've been using it.

            I'll have to think about how to refer to these data if they don't get installed.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Oh. I didn't realize this. I understand now. You're saying ci_hsc doesn't install in the stack install, it just runs the tests. I've been directly running setup on a separate directory copy of the build directory; when doing that then CI_HSC_DIR gets defined. That's how I've been using it. I'll have to think about how to refer to these data if they don't get installed.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            My preference would be for ci_hsc to install like other products but we can clone it by hand if necessary.

            Show
            jhoblitt Joshua Hoblitt added a comment - My preference would be for ci_hsc to install like other products but we can clone it by hand if necessary.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment - - edited

            There's not a particular use case for runHscQuickTest.sh that's important. It's really only as a minor time savings to catch reasons runHscTest.sh might fail without having to download the 700 GB of validation_data_hsc.

            I will add a descriptive note in the README about having to explicitly set up the cloned repo of ci_hsc if one wants to use runHscQuickTest.sh and a reminder that ci_hsc doesn't install anything into the stack dir.

            The current ticket was meant to produce something that could be run, e.g., by SQuaSH. That's the full runHscTest.sh script.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - - edited There's not a particular use case for runHscQuickTest.sh that's important. It's really only as a minor time savings to catch reasons runHscTest.sh might fail without having to download the 700 GB of validation_data_hsc . I will add a descriptive note in the README about having to explicitly set up the cloned repo of ci_hsc if one wants to use runHscQuickTest.sh and a reminder that ci_hsc doesn't install anything into the stack dir. The current ticket was meant to produce something that could be run, e.g., by SQuaSH. That's the full runHscTest.sh script.
            wmwood-vasey Michael Wood-Vasey made changes -
            Story Points 4 5
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            St. Michael Wood-Vasey has been very patient while we iterated through the ops model for running the `hsc` dataset. In the end, we determined that validation_data_hsc in too large and cumbersome to allow EUPS to manage it as a product dependency. It will be the responsibility of the end-user (or software agent) to manually clone/setup this the test dataset.

            Show
            jhoblitt Joshua Hoblitt added a comment - St. Michael Wood-Vasey has been very patient while we iterated through the ops model for running the `hsc` dataset. In the end, we determined that validation_data_hsc in too large and cumbersome to allow EUPS to manage it as a product dependency. It will be the responsibility of the end-user (or software agent) to manually clone/ setup this the test dataset.
            jhoblitt Joshua Hoblitt made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jhoblitt Joshua Hoblitt made changes -
            Link This issue is triggering DM-9259 [ DM-9259 ]
            wmwood-vasey Michael Wood-Vasey made changes -
            Story Points 5 8
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Thanks very much to Joshua Hoblitt for the in-depth review that helped ensure that this work is ready for integration into regular processing.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Thanks very much to Joshua Hoblitt for the in-depth review that helped ensure that this work is ready for integration into regular processing.
            wmwood-vasey Michael Wood-Vasey made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            jhoblitt Joshua Hoblitt made changes -
            Link This issue is triggering DM-9290 [ DM-9290 ]
            jhoblitt Joshua Hoblitt made changes -
            Link This issue is triggering DM-9293 [ DM-9293 ]
            frossie Frossie Economou made changes -
            Epic Link DM-5504 [ 23337 ] DM-7605 [ 26790 ]

              People

              Assignee:
              wmwood-vasey Michael Wood-Vasey
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Joshua Hoblitt
              Watchers:
              Angelo Fausti, Frossie Economou, John Parejko, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: