Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20864

Attempt to build the stack with packaged third parties and pure conda-forge dependencies

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: conda, lsstsw, sconsUtils
    • Labels:
      None
    • Epic Link:
    • Sprint:
      Arch 2019-07-29, Arch 2019-08-05, Arch 2019-08-19, Arch 2019-08-26, Arch 2019-09-09, Arch 2019-09-16
    • Team:
      Architecture

      Description

      After the third parties are packaged and in conda-forge, make a new environment of our third party packages in conda-forge. Attempt to activate that environment and use the compilers meta package in conda-forge to build the stack, ideally with lsstsw, from scratch.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-20972 Update conda version used in lsstsw

          • Done
          is triggering

          Bug - A problem which impairs or prevents the functions of the product. DM-5299 data model for persisting long metadata to FITS

          • To Do

          Bug - A problem which impairs or prevents the functions of the product. DM-21003 pex_exceptions wrapper exceptions should only inherit from one kind of python exception

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-21000 obs_base has code that depends on pipe_base

          • Invalid
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21351 Move pipe_base tests to obs_test to normalize dependencies

          • Done
          links to

          Web Link AST GitHub Issue

            Activity

            Ascending order - Click to sort in descending order
            22 older comments
            Hide
            Permalink
            bvan Brian Van Klaveren added a comment -

            OSX gets to the same spot in Jointcal.

            So the only things not building are:
            jointcal, fgmcal, and lsst_distrib, where fgmcal and lsst_distrib rely on jointcal.

            https://gist.github.com/brianv0/42915d21efb9d364640a9969f447e4c5

            Show
            bvan Brian Van Klaveren added a comment - OSX gets to the same spot in Jointcal. So the only things not building are: jointcal, fgmcal, and lsst_distrib, where fgmcal and lsst_distrib rely on jointcal. https://gist.github.com/brianv0/42915d21efb9d364640a9969f447e4c5
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Is the eigen patch we are using locally something that upstream has? Is it a case of waiting for a new conda or does the conda-forge version need to be built in a special way?

            Show
            tjenness Tim Jenness added a comment - Is the eigen patch we are using locally something that upstream has? Is it a case of waiting for a new conda or does the conda-forge version need to be built in a special way?
            Hide
            Permalink
            price Paul Price added a comment -

            I believe upstream has it on the mainline, but not on the release branch.

            Show
            price Paul Price added a comment - I believe upstream has it on the mainline, but not on the release branch.
            Hide
            Permalink
            bvan Brian Van Klaveren added a comment -

            From some light research - I think this may be included in a 3.4 release but it was omitted from 3.3.7 release, which occurred after the changes were merged to master. I'm not sure what the rationale for that might be, but I'd probably feel slightly more comfortable leaving our eigen in eups until there's a 3.4 release. It sounds like that could be within a few months. The eigen mailing list mentioned a 3.4 beta was close to being put out earlier this year.

            A possible solution is to create an `eigen_cholmod`, which has this specific fix for jointcal. Jointcal could rely on that explicitly instead of the acquired one from conda (Right now it gets eigen transitively from afw it appears). Everything else uses conda eigen, until there's a release with the fix in it.

            Show
            bvan Brian Van Klaveren added a comment - From some light research - I think this may be included in a 3.4 release but it was omitted from 3.3.7 release, which occurred after the changes were merged to master. I'm not sure what the rationale for that might be, but I'd probably feel slightly more comfortable leaving our eigen in eups until there's a 3.4 release. It sounds like that could be within a few months. The eigen mailing list mentioned a 3.4 beta was close to being put out earlier this year. A possible solution is to create an `eigen_cholmod`, which has this specific fix for jointcal. Jointcal could rely on that explicitly instead of the acquired one from conda (Right now it gets eigen transitively from afw it appears). Everything else uses conda eigen, until there's a release with the fix in it.
            Hide
            Permalink
            bvan Brian Van Klaveren added a comment -

            I'm going to close this issue today because I think the intent is resolved for now.

            Some smaller lingering issues after this is closed
            1. Understand long keyword names with newer cfitsio libraries, some code may be needed to support this. Verify this fixes commented out afw test and modified pipe_task unit tests (DM-5299)
            2. Discussion around resolving eigen issues (should create a new issue for this)
            3. Push for a new release of esutil in conda-forge (I'll follow up with a PR)
            4. Possibly add peigen to conda-forge (though it's in pip now, probably low priority)

            Show
            bvan Brian Van Klaveren added a comment - I'm going to close this issue today because I think the intent is resolved for now. Some smaller lingering issues after this is closed 1. Understand long keyword names with newer cfitsio libraries, some code may be needed to support this. Verify this fixes commented out afw test and modified pipe_task unit tests ( DM-5299 ) 2. Discussion around resolving eigen issues (should create a new issue for this) 3. Push for a new release of esutil in conda-forge (I'll follow up with a PR) 4. Possibly add peigen to conda-forge (though it's in pip now, probably low priority)

              People

              Assignee:
              bvan Brian Van Klaveren
              Reporter:
              bvan Brian Van Klaveren
              Watchers:
              Brian Van Klaveren, Chris Walter, Gabriele Comoretto [X] (Inactive), Krzysztof Findeisen, Paul Price, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.