Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2948

Remove explicit buildbot dependency on datarel

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: buildbot, lsstDoxygen
    • Labels:
      None

      Description

      The buildbot scripts have an explicit dependency on the datarel package, which we'd like to remove from the stack. It uses datarel as the top-level product when building the cross-linked HTML documentation; lsstDoxygen's makeDocs script takes a single package, and generates the list of packages to include in the Doxygen build by finding all dependencies of that package.

      So, to remove the explicit dependency on datarel, we need to either:

      • find a new top-level product with a Doxygen build to pass to makeDocs (e.g. by adding a trivial Doxygen build to lsst_distrib)
      • modify the argument parsing in lsstDoxygen to take a list of multiple products (it looks like the limitation to one package is only in the argument parsing), and pass it a list of top-level products in the buildbot scripts.

      This is currently a blocker for DM-2928, which itself a blocker for DM-1766, which has now been lingering for a few weeks now. I'm going to look for other ways to remove the block on the latter, but I don't have a solution yet.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Lowering the priority as I think I've found another way to unblock DM-1766.

            Show
            jbosch Jim Bosch added a comment - Lowering the priority as I think I've found another way to unblock DM-1766 .
            Hide
            frossie Frossie Economou added a comment -

            I'm going to do this in the near future, yell if you have any concerns not captured in the ticket.

            Show
            frossie Frossie Economou added a comment - I'm going to do this in the near future, yell if you have any concerns not captured in the ticket.
            Hide
            tjenness Tim Jenness added a comment -

            Is this going to fix itself when Jonathan Sick switches us away from makeDocs?

            Show
            tjenness Tim Jenness added a comment - Is this going to fix itself when Jonathan Sick switches us away from makeDocs ?
            Hide
            jsick Jonathan Sick added a comment -

            I've just linked this to the Pipelines doc epic so I can remember to close out this ticket when docs are being published that way. I don't have more specific tickets yet.

            Show
            jsick Jonathan Sick added a comment - I've just linked this to the Pipelines doc epic so I can remember to close out this ticket when docs are being published that way. I don't have more specific tickets yet.
            Hide
            tjenness Tim Jenness added a comment -

            I don't think we can really mark this as invalid until we fix makeDocs or migrate the document building to the new system.

            Show
            tjenness Tim Jenness added a comment - I don't think we can really mark this as invalid until we fix makeDocs or migrate the document building to the new system.
            Hide
            tjenness Tim Jenness added a comment -

            I have done some quick tests with makeDocs and it seems to me that we can switch from datarel to lsst_distrib without any need to change lsst_distrib. In my test I get a full set of documents if I ask for lsst_distrib (including ctrl_pool which is missing from datarel.

            Show
            tjenness Tim Jenness added a comment - I have done some quick tests with makeDocs and it seems to me that we can switch from datarel to lsst_distrib without any need to change lsst_distrib . In my test I get a full set of documents if I ask for lsst_distrib (including ctrl_pool which is missing from datarel .
            Hide
            jsick Jonathan Sick added a comment -

            I'm still setting up a stack for this, so you're one step ahead of me Tim Jenness.

            Show
            jsick Jonathan Sick added a comment - I'm still setting up a stack for this, so you're one step ahead of me Tim Jenness .
            Hide
            tjenness Tim Jenness added a comment -

            If it's a python 3 stack you'll need DM-9458.

            Show
            tjenness Tim Jenness added a comment - If it's a python 3 stack you'll need DM-9458 .
            Hide
            tjenness Tim Jenness added a comment -

            Can I remove datarel from lsst_distrib now?

            Show
            tjenness Tim Jenness added a comment - Can I remove datarel from lsst_distrib now?
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I have a branch that removes datarel from the doc build but I don't think it has ever been tested. I'll rebase it and open a PR.

            Show
            jhoblitt Joshua Hoblitt added a comment - I have a branch that removes datarel from the doc build but I don't think it has ever been tested. I'll rebase it and open a PR.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The branch replaced datarel with lsst_distrib https://github.com/lsst-sqre/ci-scripts/pull/44 but someone needs to test that makeDocs works with lsst_distrib (or any other product) before that's merged.

            Show
            jhoblitt Joshua Hoblitt added a comment - The branch replaced datarel with lsst_distrib https://github.com/lsst-sqre/ci-scripts/pull/44 but someone needs to test that makeDocs works with lsst_distrib (or any other product) before that's merged.
            Hide
            tjenness Tim Jenness added a comment -

            See my comment from 15/Feb/17.

            Show
            tjenness Tim Jenness added a comment - See my comment from 15/Feb/17.
            Hide
            tjenness Tim Jenness added a comment -

            One problem is that DM-13074 is indicating that we may see breakage regardless.

            Show
            tjenness Tim Jenness added a comment - One problem is that DM-13074 is indicating that we may see breakage regardless.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Are you OK with disabling the doc build if that is the case? Otherwise, it will break the nightly/weekly.

            Show
            jhoblitt Joshua Hoblitt added a comment - Are you OK with disabling the doc build if that is the case? Otherwise, it will break the nightly/weekly.
            Hide
            tjenness Tim Jenness added a comment -

            I'm not sure what you are asking. Doc builds are broken (DM-13074). In my tests a year ago everything worked fine when I switched datarel to lsst_distrib but I can't imagine changing this will fix DM-13074 (and I think on that ticket you said you had made the switch as part of debugging). Switching to lsst_distrib shouldn't make things worse though.

            Show
            tjenness Tim Jenness added a comment - I'm not sure what you are asking. Doc builds are broken ( DM-13074 ). In my tests a year ago everything worked fine when I switched datarel to lsst_distrib but I can't imagine changing this will fix DM-13074 (and I think on that ticket you said you had made the switch as part of debugging). Switching to lsst_distrib shouldn't make things worse though.
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            Currently, the doxygen landing page is incorrect but the doc build is still running. I believe all the classes, etc. are still there if navigated to by the index. If the doc build starts failing, as in exiting non-zero, it will have to be disabled completely.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited Currently, the doxygen landing page is incorrect but the doc build is still running. I believe all the classes, etc. are still there if navigated to by the index. If the doc build starts failing, as in exiting non-zero, it will have to be disabled completely.
            Hide
            tjenness Tim Jenness added a comment -

            Given my test from a year ago I believe switching to lsst_distrib will not be a problem.

            Show
            tjenness Tim Jenness added a comment - Given my test from a year ago I believe switching to lsst_distrib will not be a problem.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            OK - I have merged that PR. I'm currently re-run last nights release, so it will probably be several hours before this can be tested via release/run-rebuild.

            Show
            jhoblitt Joshua Hoblitt added a comment - OK - I have merged that PR. I'm currently re-run last nights release, so it will probably be several hours before this can be tested via release/run-rebuild .
            Hide
            tjenness Tim Jenness added a comment -

            Joshua Hoblitt I have removed datarel from lsst_distrib and your change to makedocs has worked. Feel free to close this ticket (that was opened nearly 3 years ago)

            Show
            tjenness Tim Jenness added a comment - Joshua Hoblitt I have removed datarel from lsst_distrib and your change to makedocs has worked. Feel free to close this ticket (that was opened nearly 3 years ago)
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            repos.yaml has been updated.

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - repos.yaml has been updated.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Gabriele Comoretto [X] +1

             

            Tim Jenness I think DM-7298 can be invalidated now.

            Show
            jhoblitt Joshua Hoblitt added a comment - Gabriele Comoretto [X] +1   Tim Jenness I think DM-7298 can be invalidated now.
            Hide
            tjenness Tim Jenness added a comment -

            Done. DM-7298 now closed.

            Show
            tjenness Tim Jenness added a comment - Done. DM-7298 now closed.

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Frossie Economou, Gabriele Comoretto [X] (Inactive), Jim Bosch, John Swinbank, Jonathan Sick, Joshua Hoblitt, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.