Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20864

Attempt to build the stack with packaged third parties and pure conda-forge dependencies

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: conda, lsstsw, sconsUtils
    • Labels:
      None
    • Sprint:
      Arch 2019-07-29, Arch 2019-08-05, Arch 2019-08-19, Arch 2019-08-26, Arch 2019-09-09, Arch 2019-09-16
    • Team:
      Architecture

      Description

      After the third parties are packaged and in conda-forge, make a new environment of our third party packages in conda-forge. Attempt to activate that environment and use the compilers meta package in conda-forge to build the stack, ideally with lsstsw, from scratch.

        Attachments

          Issue Links

            Activity

            Hide
            bvan Brian Van Klaveren added a comment -

            OSX gets to the same spot in Jointcal.

            So the only things not building are:
            jointcal, fgmcal, and lsst_distrib, where fgmcal and lsst_distrib rely on jointcal.

            https://gist.github.com/brianv0/42915d21efb9d364640a9969f447e4c5

            Show
            bvan Brian Van Klaveren added a comment - OSX gets to the same spot in Jointcal. So the only things not building are: jointcal, fgmcal, and lsst_distrib, where fgmcal and lsst_distrib rely on jointcal. https://gist.github.com/brianv0/42915d21efb9d364640a9969f447e4c5
            Hide
            tjenness Tim Jenness added a comment -

            Is the eigen patch we are using locally something that upstream has? Is it a case of waiting for a new conda or does the conda-forge version need to be built in a special way?

            Show
            tjenness Tim Jenness added a comment - Is the eigen patch we are using locally something that upstream has? Is it a case of waiting for a new conda or does the conda-forge version need to be built in a special way?
            Hide
            price Paul Price added a comment -

            I believe upstream has it on the mainline, but not on the release branch.

            Show
            price Paul Price added a comment - I believe upstream has it on the mainline, but not on the release branch.
            Hide
            bvan Brian Van Klaveren added a comment -

            From some light research - I think this may be included in a 3.4 release but it was omitted from 3.3.7 release, which occurred after the changes were merged to master. I'm not sure what the rationale for that might be, but I'd probably feel slightly more comfortable leaving our eigen in eups until there's a 3.4 release. It sounds like that could be within a few months. The eigen mailing list mentioned a 3.4 beta was close to being put out earlier this year.

            A possible solution is to create an `eigen_cholmod`, which has this specific fix for jointcal. Jointcal could rely on that explicitly instead of the acquired one from conda (Right now it gets eigen transitively from afw it appears). Everything else uses conda eigen, until there's a release with the fix in it.

            Show
            bvan Brian Van Klaveren added a comment - From some light research - I think this may be included in a 3.4 release but it was omitted from 3.3.7 release, which occurred after the changes were merged to master. I'm not sure what the rationale for that might be, but I'd probably feel slightly more comfortable leaving our eigen in eups until there's a 3.4 release. It sounds like that could be within a few months. The eigen mailing list mentioned a 3.4 beta was close to being put out earlier this year. A possible solution is to create an `eigen_cholmod`, which has this specific fix for jointcal. Jointcal could rely on that explicitly instead of the acquired one from conda (Right now it gets eigen transitively from afw it appears). Everything else uses conda eigen, until there's a release with the fix in it.
            Hide
            bvan Brian Van Klaveren added a comment -

            I'm going to close this issue today because I think the intent is resolved for now.

            Some smaller lingering issues after this is closed
            1. Understand long keyword names with newer cfitsio libraries, some code may be needed to support this. Verify this fixes commented out afw test and modified pipe_task unit tests (DM-5299)
            2. Discussion around resolving eigen issues (should create a new issue for this)
            3. Push for a new release of esutil in conda-forge (I'll follow up with a PR)
            4. Possibly add peigen to conda-forge (though it's in pip now, probably low priority)

            Show
            bvan Brian Van Klaveren added a comment - I'm going to close this issue today because I think the intent is resolved for now. Some smaller lingering issues after this is closed 1. Understand long keyword names with newer cfitsio libraries, some code may be needed to support this. Verify this fixes commented out afw test and modified pipe_task unit tests ( DM-5299 ) 2. Discussion around resolving eigen issues (should create a new issue for this) 3. Push for a new release of esutil in conda-forge (I'll follow up with a PR) 4. Possibly add peigen to conda-forge (though it's in pip now, probably low priority)

              People

              • Assignee:
                bvan Brian Van Klaveren
                Reporter:
                bvan Brian Van Klaveren
                Watchers:
                Brian Van Klaveren, Chris Walter, Gabriele Comoretto, Krzysztof Findeisen, Paul Price, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel