Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2948

Remove explicit buildbot dependency on datarel

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • buildbot, lsstDoxygen
    • None

    Description

      The buildbot scripts have an explicit dependency on the datarel package, which we'd like to remove from the stack. It uses datarel as the top-level product when building the cross-linked HTML documentation; lsstDoxygen's makeDocs script takes a single package, and generates the list of packages to include in the Doxygen build by finding all dependencies of that package.

      So, to remove the explicit dependency on datarel, we need to either:

      • find a new top-level product with a Doxygen build to pass to makeDocs (e.g. by adding a trivial Doxygen build to lsst_distrib)
      • modify the argument parsing in lsstDoxygen to take a list of multiple products (it looks like the limitation to one package is only in the argument parsing), and pass it a list of top-level products in the buildbot scripts.

      This is currently a blocker for DM-2928, which itself a blocker for DM-1766, which has now been lingering for a few weeks now. I'm going to look for other ways to remove the block on the latter, but I don't have a solution yet.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            Lowering the priority as I think I've found another way to unblock DM-1766.

            jbosch Jim Bosch added a comment - Lowering the priority as I think I've found another way to unblock DM-1766 .

            I'm going to do this in the near future, yell if you have any concerns not captured in the ticket.

            frossie Frossie Economou added a comment - I'm going to do this in the near future, yell if you have any concerns not captured in the ticket.
            tjenness Tim Jenness added a comment -

            Is this going to fix itself when jsick switches us away from makeDocs?

            tjenness Tim Jenness added a comment - Is this going to fix itself when jsick switches us away from makeDocs ?

            I've just linked this to the Pipelines doc epic so I can remember to close out this ticket when docs are being published that way. I don't have more specific tickets yet.

            jsick Jonathan Sick added a comment - I've just linked this to the Pipelines doc epic so I can remember to close out this ticket when docs are being published that way. I don't have more specific tickets yet.
            tjenness Tim Jenness added a comment -

            I don't think we can really mark this as invalid until we fix makeDocs or migrate the document building to the new system.

            tjenness Tim Jenness added a comment - I don't think we can really mark this as invalid until we fix makeDocs or migrate the document building to the new system.
            tjenness Tim Jenness added a comment -

            I have done some quick tests with makeDocs and it seems to me that we can switch from datarel to lsst_distrib without any need to change lsst_distrib. In my test I get a full set of documents if I ask for lsst_distrib (including ctrl_pool which is missing from datarel.

            tjenness Tim Jenness added a comment - I have done some quick tests with makeDocs and it seems to me that we can switch from datarel to lsst_distrib without any need to change lsst_distrib . In my test I get a full set of documents if I ask for lsst_distrib (including ctrl_pool which is missing from datarel .

            I'm still setting up a stack for this, so you're one step ahead of me tjenness.

            jsick Jonathan Sick added a comment - I'm still setting up a stack for this, so you're one step ahead of me tjenness .
            tjenness Tim Jenness added a comment -

            If it's a python 3 stack you'll need DM-9458.

            tjenness Tim Jenness added a comment - If it's a python 3 stack you'll need DM-9458 .
            tjenness Tim Jenness added a comment -

            Can I remove datarel from lsst_distrib now?

            tjenness Tim Jenness added a comment - Can I remove datarel from lsst_distrib now?

            I have a branch that removes datarel from the doc build but I don't think it has ever been tested. I'll rebase it and open a PR.

            jhoblitt Joshua Hoblitt added a comment - I have a branch that removes datarel from the doc build but I don't think it has ever been tested. I'll rebase it and open a PR.

            The branch replaced datarel with lsst_distrib https://github.com/lsst-sqre/ci-scripts/pull/44 but someone needs to test that makeDocs works with lsst_distrib (or any other product) before that's merged.

            jhoblitt Joshua Hoblitt added a comment - The branch replaced datarel with lsst_distrib https://github.com/lsst-sqre/ci-scripts/pull/44 but someone needs to test that makeDocs works with lsst_distrib (or any other product) before that's merged.
            tjenness Tim Jenness added a comment -

            See my comment from 15/Feb/17.

            tjenness Tim Jenness added a comment - See my comment from 15/Feb/17.
            tjenness Tim Jenness added a comment -

            One problem is that DM-13074 is indicating that we may see breakage regardless.

            tjenness Tim Jenness added a comment - One problem is that DM-13074 is indicating that we may see breakage regardless.

            Are you OK with disabling the doc build if that is the case? Otherwise, it will break the nightly/weekly.

            jhoblitt Joshua Hoblitt added a comment - Are you OK with disabling the doc build if that is the case? Otherwise, it will break the nightly/weekly.
            tjenness Tim Jenness added a comment -

            I'm not sure what you are asking. Doc builds are broken (DM-13074). In my tests a year ago everything worked fine when I switched datarel to lsst_distrib but I can't imagine changing this will fix DM-13074 (and I think on that ticket you said you had made the switch as part of debugging). Switching to lsst_distrib shouldn't make things worse though.

            tjenness Tim Jenness added a comment - I'm not sure what you are asking. Doc builds are broken ( DM-13074 ). In my tests a year ago everything worked fine when I switched datarel to lsst_distrib but I can't imagine changing this will fix DM-13074 (and I think on that ticket you said you had made the switch as part of debugging). Switching to lsst_distrib shouldn't make things worse though.
            jhoblitt Joshua Hoblitt added a comment - - edited

            Currently, the doxygen landing page is incorrect but the doc build is still running. I believe all the classes, etc. are still there if navigated to by the index. If the doc build starts failing, as in exiting non-zero, it will have to be disabled completely.

            jhoblitt Joshua Hoblitt added a comment - - edited Currently, the doxygen landing page is incorrect but the doc build is still running. I believe all the classes, etc. are still there if navigated to by the index. If the doc build starts failing, as in exiting non-zero, it will have to be disabled completely.
            tjenness Tim Jenness added a comment -

            Given my test from a year ago I believe switching to lsst_distrib will not be a problem.

            tjenness Tim Jenness added a comment - Given my test from a year ago I believe switching to lsst_distrib will not be a problem.

            OK - I have merged that PR. I'm currently re-run last nights release, so it will probably be several hours before this can be tested via release/run-rebuild.

            jhoblitt Joshua Hoblitt added a comment - OK - I have merged that PR. I'm currently re-run last nights release, so it will probably be several hours before this can be tested via release/run-rebuild .
            tjenness Tim Jenness added a comment -

            jhoblitt I have removed datarel from lsst_distrib and your change to makedocs has worked. Feel free to close this ticket (that was opened nearly 3 years ago)

            tjenness Tim Jenness added a comment - jhoblitt I have removed datarel from lsst_distrib and your change to makedocs has worked. Feel free to close this ticket (that was opened nearly 3 years ago)

            repos.yaml has been updated.

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - repos.yaml has been updated.

            gcomoretto +1

             

            tjenness I think DM-7298 can be invalidated now.

            jhoblitt Joshua Hoblitt added a comment - gcomoretto +1   tjenness I think DM-7298 can be invalidated now.
            tjenness Tim Jenness added a comment -

            Done. DM-7298 now closed.

            tjenness Tim Jenness added a comment - Done. DM-7298 now closed.

            People

              jhoblitt Joshua Hoblitt
              jbosch Jim Bosch
              Frossie Economou, Gabriele Comoretto [X] (Inactive), Jim Bosch, John Swinbank, Jonathan Sick, Joshua Hoblitt, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.