Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4706

Rerun and create a repository for CFHT astrometry test.

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_cfht, Test Data
    • Labels:
      None

      Description

      Understand, re-rerun, and recreate clean version of Dominique Boutigny 's CFHT astrometry test for the astrometry RMS for two sample CFHT observations. This test is on the NCSA machines in
      /lsst8/boutigny/valid_cfht

      Create a repository for this test with an eye toward it becoming integrated in a validation suite for the stack.

        Attachments

          Issue Links

            Activity

            No builds found.
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) created issue -
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) made changes -
            Field Original Value New Value
            Epic Link DM-3864 [ 20140 ]
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) made changes -
            Link This issue blocks DM-2518 [ DM-2518 ]
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) made changes -
            Link This issue blocks DM-3864 [ DM-3864 ]
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            I have tested and re-ran the CFHT validation test that Dominique Boutigny provided on lsst-dev.ncsa.illinois.edu in /lsst8/boutigny/valid_cfht.

            I created a clean directory for this tests to make sure the file dependencies were understood and started a repo `validate_drp_cfht` as

            https://github.com/wmwv/validate_drp_cfht

            Two important caveats/statements of currently functionality. Running this test (through the `run_test.sh` script) requires

            1. a set of input data that are not yet specifically provided in a package
            2. a custom `astrometry_net_data` bundle with the appropriate SDSS catalog for the input data.

            Proper placement and configuration of these input and reference catalog data is DM-4709

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - I have tested and re-ran the CFHT validation test that Dominique Boutigny provided on lsst-dev.ncsa.illinois.edu in /lsst8/boutigny/valid_cfht. I created a clean directory for this tests to make sure the file dependencies were understood and started a repo `validate_drp_cfht` as https://github.com/wmwv/validate_drp_cfht Two important caveats/statements of currently functionality. Running this test (through the `run_test.sh` script) requires 1. a set of input data that are not yet specifically provided in a package 2. a custom `astrometry_net_data` bundle with the appropriate SDSS catalog for the input data. Proper placement and configuration of these input and reference catalog data is DM-4709
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Close to done.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Close to done.
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Frossie Economou Can you create an `lsst/validate_drp_cfht` repo as a fork of `wmwv/validate_drp_cfht`? Future development will then be done in the `lsst/validate_drp_cfht` repo.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Frossie Economou Can you create an `lsst/validate_drp_cfht` repo as a fork of `wmwv/validate_drp_cfht`? Future development will then be done in the `lsst/validate_drp_cfht` repo.
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - - edited

            Created `validate_drp_cfht` and `validation_data_cfht`.

            These each need to be moved from github.com/wmwv to github.com/lsst. I don't have permissions to create these new repos under github.com/lsst.

            Run with

            ```
            setup lsst_apps
            git clone https://github.com/lsst/obs_cfht
            git clone https://github.com/wmwv/validation_data_cfht
            git clone https://github.com/wmwv/validate_drp_cfht
            cd obs_cfht; setup -r .; scons opt=3; cd -
            setup -r validation_data_cfht
            cd validate_drp_cfht
            sh run_test.sh
            ```

            should end with output like

            ```
            1069 sources in ccd : 12
            1083 sources in ccd : 13
            1047 sources in ccd : 14
            1102 sources in ccd : 21
            1110 sources in ccd : 22
            1119 sources in ccd : 23
            6530 Sources in reference visit : 849375
            1601 sources in ccd : 12
            3276 sources in ccd : 13
            5033 sources in ccd : 14
            6748 sources in ccd : 21
            8558 sources in ccd : 22
            10361 sources in ccd : 23
            Visit : 850587 5648 matches found
            Median value of the astrometric scatter - all magnitudes: 76.0983925574 mas
            Astrometric scatter (median) - mag < 21 : 22.7943173652 mas
            ```

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - - edited Created `validate_drp_cfht` and `validation_data_cfht`. These each need to be moved from github.com/wmwv to github.com/lsst. I don't have permissions to create these new repos under github.com/lsst. Run with ``` setup lsst_apps git clone https://github.com/lsst/obs_cfht git clone https://github.com/wmwv/validation_data_cfht git clone https://github.com/wmwv/validate_drp_cfht cd obs_cfht; setup -r .; scons opt=3; cd - setup -r validation_data_cfht cd validate_drp_cfht sh run_test.sh ``` should end with output like ``` 1069 sources in ccd : 12 1083 sources in ccd : 13 1047 sources in ccd : 14 1102 sources in ccd : 21 1110 sources in ccd : 22 1119 sources in ccd : 23 6530 Sources in reference visit : 849375 1601 sources in ccd : 12 3276 sources in ccd : 13 5033 sources in ccd : 14 6748 sources in ccd : 21 8558 sources in ccd : 22 10361 sources in ccd : 23 Visit : 850587 5648 matches found Median value of the astrometric scatter - all magnitudes: 76.0983925574 mas Astrometric scatter (median) - mag < 21 : 22.7943173652 mas ```
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            setup lsst_apps
            git clone https://github.com/lsst/obs_cfht
            git clone https://github.com/wmwv/validation_data_cfht
            git clone https://github.com/wmwv/validate_drp_cfht
            cd obs_cfht; setup -r .; scons opt=3; cd -
            setup -r validation_data_cfht
            cd validate_drp_cfht
            sh run_test.sh

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - setup lsst_apps git clone https://github.com/lsst/obs_cfht git clone https://github.com/wmwv/validation_data_cfht git clone https://github.com/wmwv/validate_drp_cfht cd obs_cfht; setup -r .; scons opt=3; cd - setup -r validation_data_cfht cd validate_drp_cfht sh run_test.sh
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) made changes -
            Reviewers David Nidever, Frossie Economou [ nidever, frossie ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - - edited

            I've moved the repos all over to github.com/lsst
            I also changed the name of `validate_drp_cfht` to the more general `validate_drp` because the validation calculations will be largely the same base code across different data sets.

            setup lsst_apps
            git clone https://github.com/lsst/obs_cfht
            git clone https://github.com/lsst/validation_data_cfht
            git clone https://github.com/lsst/validate_drp
            cd obs_cfht; git checkout 59e204f; setup -r .; scons opt=3; cd -
            setup -r validation_data_cfht
            cd validate_drp
            sh run_test.sh

            Note that something broke between 59e204f and 0351c12 in `obs_cfht`. Thus the explicit checkout of 59e204f.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - - edited I've moved the repos all over to github.com/lsst I also changed the name of `validate_drp_cfht` to the more general `validate_drp` because the validation calculations will be largely the same base code across different data sets. setup lsst_apps git clone https://github.com/lsst/obs_cfht git clone https://github.com/lsst/validation_data_cfht git clone https://github.com/lsst/validate_drp cd obs_cfht; git checkout 59e204f; setup -r .; scons opt=3; cd - setup -r validation_data_cfht cd validate_drp sh run_test.sh Note that something broke between 59e204f and 0351c12 in `obs_cfht`. Thus the explicit checkout of 59e204f.
            Hide
            swinbank John Swinbank added a comment -

            Renaming the repository to validate_drp caught my eye – it propels this from something I was only vaguely aware of to something that sounds like a key piece of infrastructure.

            With that in mind, can you fill me in on the longer term plans here, particularly as it relates to the other validation packages like lsst_dm_stack_demo and ci_hsc? From a quick glance, it looks like there's already quite a lot of functionailty duplication – is there a plan to merge them?

            Show
            swinbank John Swinbank added a comment - Renaming the repository to validate_drp caught my eye – it propels this from something I was only vaguely aware of to something that sounds like a key piece of infrastructure. With that in mind, can you fill me in on the longer term plans here, particularly as it relates to the other validation packages like lsst_dm_stack_demo and ci_hsc ? From a quick glance, it looks like there's already quite a lot of functionailty duplication – is there a plan to merge them?
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Yes, this is leading toward a key piece of the long term validation infrastructure. This Story is part of the Epic DM-3864: "Integration Dataset for metrics and regression tests - Part I":

            This present DM-4706 was covering specifically adding Dominique Boutigny 's astrometric performance check.

            You're very right to call out the connection of this epic and `ci_hsc`. I've just added DM-4817 as a Story under Epic DM-3864 to call this out explicitly. My intention is to develop a few more tests here and then look through the `ci_hsc` in the next month.

            I'm honestly not familiar with the state of how much of `ci_hsc` is runnable in the current DM stack (e.g., building `obs_subaru` with `lsstsw` `a4d9de0` and `lsst_build` `015c01a` current fails on the `psycopg2` dependency).

            The idea of "validation" vs. "testing" is 2-fold. One: it's a test of the integration rather than of specific modules. But two, and the distinction from continuous integration, is that the validation effort will be runnable at a variety of scales:
            1. Quick 1-minute turn around for continuous integration.
            2. Perhaps nightly 30-minute scale processing to alert if there's been any more subtle regression in performance.
            3. Weekly hours-long processing scale to calculate more detailed performance metrics that are part of the goals of the DM stack.

            The `bin/check_astrometry.py` in `lsst_dm_stack_demo` is exactly the same code that's being further developed in `validate_drp`. The duplication here is exactly intentional. `lsst_dm_stack_demo` has been a very loosely "validation" that the stack kind of actually works after an install. There's no strong current intent to develop `lsst_dm_stack_demo` that much further as part of Epic DM-3864. We can decide later whether `lsst_dm_stack_demo` should explicitly import from `validate_drp`, but my preference would be to actually not create any additional dependencies for `lsst_dm_stack_demo` until we have a clearer idea of the purpose of the `lsst_dm_stack_demo`.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Yes, this is leading toward a key piece of the long term validation infrastructure. This Story is part of the Epic DM-3864 : "Integration Dataset for metrics and regression tests - Part I": This present DM-4706 was covering specifically adding Dominique Boutigny 's astrometric performance check. You're very right to call out the connection of this epic and `ci_hsc`. I've just added DM-4817 as a Story under Epic DM-3864 to call this out explicitly. My intention is to develop a few more tests here and then look through the `ci_hsc` in the next month. I'm honestly not familiar with the state of how much of `ci_hsc` is runnable in the current DM stack (e.g., building `obs_subaru` with `lsstsw` `a4d9de0` and `lsst_build` `015c01a` current fails on the `psycopg2` dependency). The idea of "validation" vs. "testing" is 2-fold. One: it's a test of the integration rather than of specific modules. But two, and the distinction from continuous integration, is that the validation effort will be runnable at a variety of scales: 1. Quick 1-minute turn around for continuous integration. 2. Perhaps nightly 30-minute scale processing to alert if there's been any more subtle regression in performance. 3. Weekly hours-long processing scale to calculate more detailed performance metrics that are part of the goals of the DM stack. The `bin/check_astrometry.py` in `lsst_dm_stack_demo` is exactly the same code that's being further developed in `validate_drp`. The duplication here is exactly intentional. `lsst_dm_stack_demo` has been a very loosely "validation" that the stack kind of actually works after an install. There's no strong current intent to develop `lsst_dm_stack_demo` that much further as part of Epic DM-3864 . We can decide later whether `lsst_dm_stack_demo` should explicitly import from `validate_drp`, but my preference would be to actually not create any additional dependencies for `lsst_dm_stack_demo` until we have a clearer idea of the purpose of the `lsst_dm_stack_demo`.
            Hide
            swinbank John Swinbank added a comment -

            Thanks for that; sounds useful & reasonable.

            To clarify the status of obs_subaru: we're aware that it doesn't build with lsstsw at present. The fix is all ready to go (DM-3518), but unfortunately we are blocked from merging by DM-4733. As soon as that's resolved we should be up and running and will press ahead with getting ci_hsc integrated (that's DM-4307). This is essential for the HSC port, so it's top priority from our point of view.

            From a quick look at the ci_hsc and validate_drp packages (I claim expertise on neither), it seems like ci_hsc has a more thought-through infrastructure for adding and running tests (based on SCons, which seems like it would be just what you want for this kind of thing), but the individual tests are pretty simplistic compared to the astrometry check in validate_drp. It sounds like it would be a great idea to investigate whether they can adopt a common approach before they diverge further. In other words, on DM-4817!

            Show
            swinbank John Swinbank added a comment - Thanks for that; sounds useful & reasonable. To clarify the status of obs_subaru : we're aware that it doesn't build with lsstsw at present. The fix is all ready to go ( DM-3518 ), but unfortunately we are blocked from merging by DM-4733 . As soon as that's resolved we should be up and running and will press ahead with getting ci_hsc integrated (that's DM-4307 ). This is essential for the HSC port, so it's top priority from our point of view. From a quick look at the ci_hsc and validate_drp packages (I claim expertise on neither), it seems like ci_hsc has a more thought-through infrastructure for adding and running tests (based on SCons, which seems like it would be just what you want for this kind of thing), but the individual tests are pretty simplistic compared to the astrometry check in validate_drp . It sounds like it would be a great idea to investigate whether they can adopt a common approach before they diverge further. In other words, on DM-4817 !
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            David Nidever [X] Could you kindly take a moment to review this issue?

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - David Nidever [X] Could you kindly take a moment to review this issue?
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            David Nidever [X] Could you kindly take a look at this ticket?

            If you are unable to review it, please let me know and I will ask someone else.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - David Nidever [X] Could you kindly take a look at this ticket? If you are unable to review it, please let me know and I will ask someone else.
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            This issue ended up being completely covered by DM-4707, DM-4708, DM-4709, DM-4814, DM-4827. I am closing this issue as Done.

            By the time the prototype was reviewed (DM-4709), and the generalizations to SDSS (DM-4708), and DECam (DM-4707, DM-4814), and then further development to standardize structure of the code (DM-4827), this present issue, DM-4706, was covered and reviewed.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - This issue ended up being completely covered by DM-4707 , DM-4708 , DM-4709 , DM-4814 , DM-4827 . I am closing this issue as Done. By the time the prototype was reviewed ( DM-4709 ), and the generalizations to SDSS ( DM-4708 ), and DECam ( DM-4707 , DM-4814 ), and then further development to standardize structure of the code ( DM-4827 ), this present issue, DM-4706 , was covered and reviewed.
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status In Review [ 10004 ] Done [ 10002 ]
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) made changes -
            Reviewers David Nidever, Frossie Economou [ nidever, frossie ]

              People

              Assignee:
              wmwood-vasey Michael Wood-Vasey [X] (Inactive)
              Reporter:
              wmwood-vasey Michael Wood-Vasey [X] (Inactive)
              Watchers:
              David Nidever [X] (Inactive), Frossie Economou, John Swinbank, Michael Wood-Vasey [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Due:
                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.