Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1040

Doxygen doesn't include obs packages

    Details

    • Team:
      SQuaRE

      Description

      The doxygen pages (http://lsst-web.ncsa.illinois.edu/doxygen/x_masterDoxyDoc/annotated.html) currently don't include any of the obs packages (e.g., obs_lsstSim, obs_cfht, obs_subaru). These packages contain useful code, and their documentation should be merged in with that in Doxygen.

      This may be difficult, given that we want to keep the obs packages separate in some manner from the rest.

        Attachments

          Issue Links

            Activity

            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            I've looked over the BB code invoking the doxygen build. It builds the datarel product currently. I can easily make the changes to create the doxydoc for product lsst_distrib. The devenv/lsstDoxygen/bin/makedocs tool currently is restricted to building the documentation for a *single* product; it includes all of that product's dependencies.

            However, the Release aggregator: lsst_distrib, does not include obs_cfht nor obs_subaru. I believe that obs_cfht was removed as a Release artifact due to its decrepitude. I suspect that obs_subaru isn't a DM Release artifact for other reasons.

            So I will update the BB doxydoc generation to build the lsst_distrib product. I will also cleanup a handful of dependencies which should be excluded from consideration and are causing exceptions during doxydoc processing (i.e., the new sims externals: scisql, mysqlpython, ...).

            I might even look into moving the huge doxydoc generation log to the side since it is serving no purpose other than causing me to scroll beyond it to determine what happened during the final sanity check test.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - I've looked over the BB code invoking the doxygen build. It builds the datarel product currently. I can easily make the changes to create the doxydoc for product lsst_distrib. The devenv/lsstDoxygen/bin/makedocs tool currently is restricted to building the documentation for a * single * product; it includes all of that product's dependencies. However, the Release aggregator: lsst_distrib, does not include obs_cfht nor obs_subaru. I believe that obs_cfht was removed as a Release artifact due to its decrepitude. I suspect that obs_subaru isn't a DM Release artifact for other reasons. So I will update the BB doxydoc generation to build the lsst_distrib product. I will also cleanup a handful of dependencies which should be excluded from consideration and are causing exceptions during doxydoc processing (i.e., the new sims externals: scisql, mysqlpython, ...). I might even look into moving the huge doxydoc generation log to the side since it is serving no purpose other than causing me to scroll beyond it to determine what happened during the final sanity check test.
            Hide
            rhl Robert Lupton added a comment -

            It would be easy to make lsstDoxygen take a list of products, or you could add a new product with no directory and a table file listing e.g. lsst_apps + a set of obs files.

            Show
            rhl Robert Lupton added a comment - It would be easy to make lsstDoxygen take a list of products, or you could add a new product with no directory and a table file listing e.g. lsst_apps + a set of obs files.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Old Watchers and Newly Added Watchers: I'm agnostic on this issue. I'd like guidance on what is wanted wrt Project documentation support.

            • Will we be creating a doxydoc which embeds all of {lsst-core, sims, qserv}. - where lsst-core includes (future) MW doxydoc, too.
              * In the future when we have separate buildslaves for each of {lsst-core, sims, qserv}

              , will we generate separate doxydocs for each product set?

            • Should all viable camera specific packages be included in the lsst-core doxydoc?
              • then I'd like to know which are considered viable these days {obs_cfht obs_lsstSim obs_sdss obs_sst obs_subaru obs_wiyn}

            If there is no comment, then I will create a new package which includes lsst_distrib and all the obs_* that are viable.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Old Watchers and Newly Added Watchers: I'm agnostic on this issue. I'd like guidance on what is wanted wrt Project documentation support. Will we be creating a doxydoc which embeds all of {lsst-core, sims, qserv}. - where lsst-core includes (future) MW doxydoc, too. * In the future when we have separate buildslaves for each of {lsst-core, sims, qserv} , will we generate separate doxydocs for each product set? Should all viable camera specific packages be included in the lsst-core doxydoc? then I'd like to know which are considered viable these days {obs_cfht obs_lsstSim obs_sdss obs_sst obs_subaru obs_wiyn} If there is no comment, then I will create a new package which includes lsst_distrib and all the obs_* that are viable.
            Hide
            ktl Kian-Tat Lim added a comment -

            Here's what I'd like, but I don't know if you can do it with the tools available:

            • Each top-level product builds its own doxygen (and sphinx).
            • One or more separate products are created for plugin packages like the obs_* packages.
            • Shared packages are "owned" by lsst_distrib.
            • All the documentation is published to the same site, so that product pages can link to shared package pages. A (perhaps manual) directory page points to each top-level product.

            I believe that obs_cfht, obs_lsstSim, obs_sdss, and obs_subaru are "viable" in the sense that we support them and expect them to work as part of a release. If they're not currently in that state, they should be fixed. obs_sst and obs_wiyn seem to me to be contributed, and perhaps should be moved out of LSST/DMS.

            Show
            ktl Kian-Tat Lim added a comment - Here's what I'd like, but I don't know if you can do it with the tools available: Each top-level product builds its own doxygen (and sphinx). One or more separate products are created for plugin packages like the obs_* packages. Shared packages are "owned" by lsst_distrib . All the documentation is published to the same site, so that product pages can link to shared package pages. A (perhaps manual) directory page points to each top-level product. I believe that obs_cfht , obs_lsstSim , obs_sdss , and obs_subaru are "viable" in the sense that we support them and expect them to work as part of a release. If they're not currently in that state, they should be fixed. obs_sst and obs_wiyn seem to me to be contributed, and perhaps should be moved out of LSST/DMS .
            Hide
            rhl Robert Lupton added a comment -

            Re KT's comment:

            Each top-level product builds its own doxygen (and sphinx).

            I assume that you mean one documentation product, generated by some toolset (breathe?) that unifies doxygen + sphinx

            Is this proposal supposed to cover all products, or only obs packages?

            The reason why we use lsstDoxygen is to allow links to packages that you don't depend on (e.g. meas_algorithms can refer to pipe_tasks), and to allow us to generate documentation of peer packages (e.g. the LSST_task_documentation group). Why don't you just want to add the obs_* packages to this master build?

            Show
            rhl Robert Lupton added a comment - Re KT's comment: Each top-level product builds its own doxygen (and sphinx). I assume that you mean one documentation product, generated by some toolset (breathe?) that unifies doxygen + sphinx Is this proposal supposed to cover all products, or only obs packages? The reason why we use lsstDoxygen is to allow links to packages that you don't depend on (e.g. meas_algorithms can refer to pipe_tasks), and to allow us to generate documentation of peer packages (e.g. the LSST_task_documentation group). Why don't you just want to add the obs_* packages to this master build?
            Hide
            ktl Kian-Tat Lim added a comment -

            I was trying to separate out all plugins, which might be "official" or might be supported by others, from the main code base. But I could see drawing the dividing line between "official" and "unofficial" instead. I'd rather not fold obs_sst and obs_wiyn into the master build.

            My intent was to ensure that we have the tooling to handle an "ecosystem" model with multiple products, some official and some unofficial, relying on common core packages. I worry that the "Borg" model of assimilating all code into a single build (or, worse, repo or package) might be unsustainable.

            Show
            ktl Kian-Tat Lim added a comment - I was trying to separate out all plugins, which might be "official" or might be supported by others, from the main code base. But I could see drawing the dividing line between "official" and "unofficial" instead. I'd rather not fold obs_sst and obs_wiyn into the master build. My intent was to ensure that we have the tooling to handle an "ecosystem" model with multiple products, some official and some unofficial, relying on common core packages. I worry that the "Borg" model of assimilating all code into a single build (or, worse, repo or package) might be unsustainable.
            Hide
            frossie Frossie Economou added a comment -

            Hey folks,

            FYI I am entering three documentation-related epics in W15 relating to documentation infrastructure/tooling. Step one is to get some agreement on the tooling but I think we'd all be surprised at this point if sphinx wasn't involved

            I think this ticket is fleshing out a requirement for this work, and unless somebody corrects me I'll revisit this as part of that epic unless it can't wait for some reason?

            At this point I am agnostic about the disposition of the obs_ stuff, but I certainly agree that there should be a mechanism for easily dealing differently with "first party" versus "contributed" packages, whether we decide to use it at this point or not. That way we can change our minds later.

            Show
            frossie Frossie Economou added a comment - Hey folks, FYI I am entering three documentation-related epics in W15 relating to documentation infrastructure/tooling. Step one is to get some agreement on the tooling but I think we'd all be surprised at this point if sphinx wasn't involved I think this ticket is fleshing out a requirement for this work, and unless somebody corrects me I'll revisit this as part of that epic unless it can't wait for some reason? At this point I am agnostic about the disposition of the obs_ stuff, but I certainly agree that there should be a mechanism for easily dealing differently with "first party" versus "contributed" packages, whether we decide to use it at this point or not. That way we can change our minds later.
            Hide
            tjenness Tim Jenness added a comment -

            I am thinking that if DM-13074 switches from datarel to lsst_distrib as the anchor package that the obs packages would be documented.

            Show
            tjenness Tim Jenness added a comment - I am thinking that if DM-13074 switches from datarel to lsst_distrib as the anchor package that the obs packages would be documented.

              People

              • Assignee:
                frossie Frossie Economou
                Reporter:
                price Paul Price
                Watchers:
                Frossie Economou, Jim Bosch, Jonathan Sick, Kian-Tat Lim, Krzysztof Findeisen, Paul Price, Robert Lupton, Robyn Allsman [X] (Inactive), Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel