Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-252

LDM-493: DM Documentation Architecture v1

    Details

    • Type: RFC
    • Status: Withdrawn
    • Resolution: Done
    • Component/s: DM, TCT
    • Labels:
      None

      Description

      LDM-493: Data Management Documentation Architecture is a proposed description of the types of documentation that DM produces, and how we produce it. This RFC seeks to place LDM-493 under TCT change control.

      A draft of the document is available at https://ldm-493.lsst.io/v/v1.

        Attachments

          Issue Links

            Activity

            Hide
            womullan Wil O'Mullane added a comment -

            I would prefer TechNotes also in LatTex and in a single git repository. Existing "important" technotes should also be PDFed and lodged in Docushare. Robert McKercher would be willing to help with this perhaps.
            Userguides .. I can be convinced could be in rst/markdown .. in one of my other projects we still did the top level user guide in TeX but linked to the sphinx generated docs lower down.

            Tex Templates for some document types and a DM (now looking more like LSST documents) documentclass are here https://github.com/mjuric/lsst_texdoc

            I would say the assignment from Jacek absolutely stands - LDM-493 is completely avoiding the issues mentioned. Namely I do not see :
            Point 1. An inventory of all DM documentation including what lives where. Identify obsolete and out of date documentation
            Point 2. A document tree to Identify missing documentation

            Point 3. Come up with structure (what goes where). Consolidate (reduce) the number of technologies we use for documentation, and clearly define boundaries what should be documented using what technology.
            Rather LDM-493 seems to introduce a new technology and I do not understand the approach of having each doc in a single repo and with its own domain.

            Point 4. I do not see policies defined
            Without the above the other points are impossible to fulfil.

            I propose we tackle this in a more systematic way:
            A simple policy exists in LSST - Documents should be published as PDF to docushare - we should start to follow the established LSST policy (point 4) before adding our own. Then I have a strong preference for draft documents to be in Latex in a single git repo (I potentially can be convinced otherwise) - having such a repo or having followed the LSST policy would have already avoided the problem of trying to find docs (point 1).

            I will put a document tree (taxaonmy) together for DM documents and am happy to have some input to that - I do not have your (Jonathan Sick) work description but I would assume this could fall in your remit. LDM-294 as a top level DM management document should for me contain this, as well as the DM organisation (in dev and in OPS and a clue as to how we transition), Configuration Control, Document Control policies etc .. this will then help to identify which documents are missing (point 2)- then the inventory would be good to know what we have and where it is.

            But a couple of documents were given priority namely, risk process and test/validation plan, plus i have limited time since I do not yet work for LSST.

            Let me add - delivering DM is everyone's job and we are all responsible for documentation which is part of that delivery.

            Also my apologies if the inventory of all docs exists somewhere and also identifies what is missing.
            Now its well past my dinner time.

            Show
            womullan Wil O'Mullane added a comment - I would prefer TechNotes also in LatTex and in a single git repository. Existing "important" technotes should also be PDFed and lodged in Docushare. Robert McKercher would be willing to help with this perhaps. Userguides .. I can be convinced could be in rst/markdown .. in one of my other projects we still did the top level user guide in TeX but linked to the sphinx generated docs lower down. Tex Templates for some document types and a DM (now looking more like LSST documents) documentclass are here https://github.com/mjuric/lsst_texdoc I would say the assignment from Jacek absolutely stands - LDM-493 is completely avoiding the issues mentioned. Namely I do not see : Point 1. An inventory of all DM documentation including what lives where. Identify obsolete and out of date documentation Point 2. A document tree to Identify missing documentation Point 3. Come up with structure (what goes where). Consolidate (reduce) the number of technologies we use for documentation, and clearly define boundaries what should be documented using what technology. Rather LDM-493 seems to introduce a new technology and I do not understand the approach of having each doc in a single repo and with its own domain. Point 4. I do not see policies defined Without the above the other points are impossible to fulfil. I propose we tackle this in a more systematic way: A simple policy exists in LSST - Documents should be published as PDF to docushare - we should start to follow the established LSST policy (point 4) before adding our own. Then I have a strong preference for draft documents to be in Latex in a single git repo (I potentially can be convinced otherwise) - having such a repo or having followed the LSST policy would have already avoided the problem of trying to find docs (point 1). I will put a document tree (taxaonmy) together for DM documents and am happy to have some input to that - I do not have your ( Jonathan Sick ) work description but I would assume this could fall in your remit. LDM-294 as a top level DM management document should for me contain this, as well as the DM organisation (in dev and in OPS and a clue as to how we transition), Configuration Control, Document Control policies etc .. this will then help to identify which documents are missing (point 2)- then the inventory would be good to know what we have and where it is. But a couple of documents were given priority namely, risk process and test/validation plan, plus i have limited time since I do not yet work for LSST. Let me add - delivering DM is everyone's job and we are all responsible for documentation which is part of that delivery. Also my apologies if the inventory of all docs exists somewhere and also identifies what is missing. Now its well past my dinner time.
            Hide
            tjenness Tim Jenness added a comment -

            I don't think LDM-493 intended to be an index of current documentation.

            I really don't like a single repo for all documents because it doesn't allow an obvious way to branch when updating a new document. With the current scheme we can branch and the document is automatically built and available in a known location with its own domain. This approach has worked really well, even for the latex LDM-151.

            I'd be perfectly happy with your approach if there was a single git repo that had submodules for each LSST document and that umbrella repo was synced up every time a new version of a submodule document was updated on master. We could pretty easily automate the syncing to the umbrella repo.

            As for the language choice for tech notes, we explicitly came up with the tech note approach to deal with the untrackable content proliferating in random places on confluence. They've worked really well and I think one reason for that is that the cookiecutter plus sphinx approach is very easy to pick up and run with given that most of us are developing python documentation within the code using numpydoc and sphinx. I've been using latex for decades but I think the sphinx tech notes give nice results. Technotes do support bibtex in .bib files so bibtex should not be a reason for abandoning the current scheme.

            Show
            tjenness Tim Jenness added a comment - I don't think LDM-493 intended to be an index of current documentation. I really don't like a single repo for all documents because it doesn't allow an obvious way to branch when updating a new document. With the current scheme we can branch and the document is automatically built and available in a known location with its own domain. This approach has worked really well, even for the latex LDM-151. I'd be perfectly happy with your approach if there was a single git repo that had submodules for each LSST document and that umbrella repo was synced up every time a new version of a submodule document was updated on master. We could pretty easily automate the syncing to the umbrella repo. As for the language choice for tech notes, we explicitly came up with the tech note approach to deal with the untrackable content proliferating in random places on confluence. They've worked really well and I think one reason for that is that the cookiecutter plus sphinx approach is very easy to pick up and run with given that most of us are developing python documentation within the code using numpydoc and sphinx. I've been using latex for decades but I think the sphinx tech notes give nice results. Technotes do support bibtex in .bib files so bibtex should not be a reason for abandoning the current scheme.
            Hide
            frossie Frossie Economou added a comment -

            Alright folks,

            this isn't converging - and the process for an escalated RFC is that the TCT is responsible for getting it resolved, not the proposer (so Jonathan should stand down at this point).

            I believe we have three groups of issues here,

            1. There is the issue that Wil wants (reasonably) a map of DM project documentation. As TimJ said, it was not the intent to provide in this document but it can be the intent now.
            2. The implementation details, eg "should all the project docs be in one LaTeX monorepo or should they re-use the software docs toolchain"
            3. The apparently controversial "who writes the docs" section.

            As the T/CAM responsible for Jonathan's time (though I don't believe project docs, unlike software docs, to be in my scope at this point), here are my recommendations for each:

            1. TimJ as the DM system engineer (and amateur librarian) take the lead in producing a project document index/map (on a piece of paper for all I care - we can take care of presentation). We will include it in the next draft of LDM-493. I expect at that point Wil will say "but it SHOULD look like this instead" and we can come up with a plan DM-wide on how to produce the missing content.
            2. Jonathan's time is already planned out till March 1st. Ergo if his effort is involved, it won't kick in until Wil is here anyway and so we can discuss the pros and cons of implementation with Wil and Architecture in March without holding anything up (if his effort is not involved then it's moot anyway). I don't think we have had a proper chance of pitching to Wil why we (SQuaRE) think extending the software docs model may be good idea for some project-level docs, and it would be nice for our morale if we got that, even if the answer is still no.
            3. I propose we remove the offending section entirely, or just replace it with Wil's sentence ("we are all responsible") above as a guiding principle / call-to-arms.

            Tim Jenness as the TCT chair, would you please express a preference as to whether you want to leave the RFC as flagged until March, or whether you want us to withdraw it and resubmit it in a different form later.

            Show
            frossie Frossie Economou added a comment - Alright folks, this isn't converging - and the process for an escalated RFC is that the TCT is responsible for getting it resolved, not the proposer (so Jonathan should stand down at this point). I believe we have three groups of issues here, There is the issue that Wil wants (reasonably) a map of DM project documentation. As TimJ said, it was not the intent to provide in this document but it can be the intent now. The implementation details, eg "should all the project docs be in one LaTeX monorepo or should they re-use the software docs toolchain" The apparently controversial "who writes the docs" section. As the T/CAM responsible for Jonathan's time (though I don't believe project docs, unlike software docs, to be in my scope at this point), here are my recommendations for each: TimJ as the DM system engineer (and amateur librarian) take the lead in producing a project document index/map (on a piece of paper for all I care - we can take care of presentation). We will include it in the next draft of LDM-493. I expect at that point Wil will say "but it SHOULD look like this instead" and we can come up with a plan DM-wide on how to produce the missing content. Jonathan's time is already planned out till March 1st. Ergo if his effort is involved, it won't kick in until Wil is here anyway and so we can discuss the pros and cons of implementation with Wil and Architecture in March without holding anything up (if his effort is not involved then it's moot anyway). I don't think we have had a proper chance of pitching to Wil why we (SQuaRE) think extending the software docs model may be good idea for some project-level docs, and it would be nice for our morale if we got that, even if the answer is still no. I propose we remove the offending section entirely, or just replace it with Wil's sentence ("we are all responsible") above as a guiding principle / call-to-arms. Tim Jenness as the TCT chair, would you please express a preference as to whether you want to leave the RFC as flagged until March, or whether you want us to withdraw it and resubmit it in a different form later.
            Hide
            tjenness Tim Jenness added a comment -

            Frossie Economou it probably makes more sense to withdraw this RFC and to have a new discussion with Wil O'Mullane in late March.

            Show
            tjenness Tim Jenness added a comment - Frossie Economou it probably makes more sense to withdraw this RFC and to have a new discussion with Wil O'Mullane in late March.
            Hide
            tjenness Tim Jenness added a comment -

            Withdrawing in order to reassess the content of this document.

            Show
            tjenness Tim Jenness added a comment - Withdrawing in order to reassess the content of this document.

              People

              • Assignee:
                jsick Jonathan Sick
                Reporter:
                jsick Jonathan Sick
                Watchers:
                Donald Petravick, Frossie Economou, Joel Plutchak (Inactive), John Swinbank, Jonathan Sick, Kian-Tat Lim, Mandeep Gill [X] (Inactive), Mario Juric, Robert Lupton, Tim Jenness, Wil O'Mullane, Zeljko Ivezic
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel