Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5370

Create lsst_ci package as a continuous integration build target

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      Create an lsst_ci package to be built for the continuous integration testing.

      Plan:
      1. Create empty package that has dependencies on obs_cfht, obs_decam, obs_subaru, testdata_cfht, testdata_decam, testdata_subaru.
      2. Ensure above builds.
      3. Add obs_lsstSim and ensure that it builds.

      The following were moved to DM-5381:
      [ Tim Jenness : How can I get strikethrough to work in the following list?]
      3. Add dependencies on validation_data_cfht and validation_data_decam, and validate_drp.
      4. Run CFHT, DECam quick examples in validate_drp.
      5. Test for successful running of the above examples. Fail and trigger Jenkins FAILURE message if these examples fail.
      6. Check performance of CFHT, DECam runs against reference numbers. Fail if there is a significant regression.
      7. Decide how to include ci_hsc, which currently can take at least 30 minutes to process the image data.--

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            For the record, my understanding wasn't based on the discussion at DM-4251, but on a statement from SQuaRE about things they support.

            I'm not sure I understand concerns around the size of testdata_ packages and lsst_distrib: isn't the raison d'ĂȘtre of testdata_ to make it possible to include the corresponding obs_ in a top level product without imposing undue storage requirements? Maybe that's what Kian-Tat Lim is getting at in the summary above, but it's not completely clear.

            Show
            swinbank John Swinbank added a comment - For the record, my understanding wasn't based on the discussion at DM-4251 , but on a statement from SQuaRE about things they support. I'm not sure I understand concerns around the size of testdata_ packages and lsst_distrib : isn't the raison d'ĂȘtre of testdata_ to make it possible to include the corresponding obs_ in a top level product without imposing undue storage requirements? Maybe that's what Kian-Tat Lim is getting at in the summary above, but it's not completely clear.
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            I'm concerned that putting, e.g., obs_cfht in lsst_distrib immediately puts testdata_cfht in there as well due to the setupOptional dependency.

            I propose that the path forward for this present ticket, DM-5370, is to proceed with the plan outlined above, plus adding obs_lsstSim.

            The remaining work is now split off into a new ticket, DM-5381.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - I'm concerned that putting, e.g., obs_cfht in lsst_distrib immediately puts testdata_cfht in there as well due to the setupOptional dependency. I propose that the path forward for this present ticket, DM-5370 , is to proceed with the plan outlined above, plus adding obs_lsstSim . The remaining work is now split off into a new ticket, DM-5381 .
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Quick review of new lsst_ci package.
            https://github.com/lsst/lsst_ci/pull/1

            Depends on `obs_cfht`, `obs_decam`, `obs_subaru`, `obs_lsstSim`
            and their testdata
            `testdata_cfht`, `testdata_decam`, `testdata_subaru`

            Test of the utility and performance of the package as it currently stands:

            1. Successfully fails on current master of `obs_cfht`, `obs_decam`.
            https://ci.lsst.codes/job/stack-os-matrix/label=centos-6/8998/console

            2. Successfully passes on tickets/DM-5372 of `obs_cfht`, `obs_decam`.
            https://ci.lsst.codes/job/stack-os-matrix/9005/label=centos-6/console

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Quick review of new lsst_ci package. https://github.com/lsst/lsst_ci/pull/1 Depends on `obs_cfht`, `obs_decam`, `obs_subaru`, `obs_lsstSim` and their testdata `testdata_cfht`, `testdata_decam`, `testdata_subaru` Test of the utility and performance of the package as it currently stands: 1. Successfully fails on current master of `obs_cfht`, `obs_decam`. https://ci.lsst.codes/job/stack-os-matrix/label=centos-6/8998/console 2. Successfully passes on tickets/ DM-5372 of `obs_cfht`, `obs_decam`. https://ci.lsst.codes/job/stack-os-matrix/9005/label=centos-6/console
            Hide
            ktl Kian-Tat Lim added a comment -

            I was going to say that if this is a pure meta-package, it doesn't need an SConstruct, .build, or .cfg. But our other meta-packages like lsst_distrib and lsst_apps do have those, so this looks fine.

            Show
            ktl Kian-Tat Lim added a comment - I was going to say that if this is a pure meta-package, it doesn't need an SConstruct , .build , or .cfg . But our other meta-packages like lsst_distrib and lsst_apps do have those, so this looks fine.
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Merged to master.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Merged to master.

              People

              Assignee:
              wmwood-vasey Michael Wood-Vasey [X] (Inactive)
              Reporter:
              wmwood-vasey Michael Wood-Vasey [X] (Inactive)
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Frossie Economou, John Swinbank, Kian-Tat Lim, Michael Wood-Vasey [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.