Update of LDM-294

XMLWordPrintable

Details

• Type: RFC
• Status: Implemented
• Resolution: Done
• Component/s:
• Labels:
None

Description

For the JDR/JSR I woudl like to reissue LDM-294.

There are updates to Org charts, glossary, 3rd party software, risk management, milestones.

Document can be found at https://ldm-294.lsst.io/v/RFC-621/index.html

Activity

Wil O'Mullane created issue -
Field Original Value New Value
Summary New issue of LDM-294 Update of LDM-294
 Description For the JDR/JSR I woudl like to reissue LDM-294. Three are updates to Org charts, glossary, 3rd party software, risk management, milestones. For the JDR/JSR I woudl like to reissue LDM-294. There are updates to Org charts, glossary, 3rd party software, risk management, milestones.
 Status Proposed [ 10805 ] Flagged [ 10606 ]
 Description For the JDR/JSR I woudl like to reissue LDM-294. There are updates to Org charts, glossary, 3rd party software, risk management, milestones. For the JDR/JSR I woudl like to reissue LDM-294. There are updates to Org charts, glossary, 3rd party software, risk management, milestones. Document can be found at https://ldm-294.lsst.io/v/RFC-621/index.html
 Link This issue is triggered by DM-20304 [ DM-20304 ]
Hide
John Swinbank added a comment - - edited

There are substantial changes to the product tree (DM Product Properties.csv) in this version vs. the last release:

 $git diff --shortstat docushare-v11 DM\ Product\ Properties.csv  1 file changed, 112 insertions(+), 104 deletions(-)  (and that's on a 112-line-long file!) Unfortunately, the diff is very hard to read. I read this RFC as effectively baselining the updated product tree, so it would be very useful to have a short summary of exactly what the changes are. Show John Swinbank added a comment - - edited There are substantial changes to the product tree ( DM Product Properties.csv ) in this version vs. the last release:$ git diff --shortstat docushare-v11 DM\ Product\ Properties.csv 1 file changed, 112 insertions(+), 104 deletions(-) (and that's on a 112-line-long file!) Unfortunately, the diff is very hard to read. I read this RFC as effectively baselining the updated product tree, so it would be very useful to have a short summary of exactly what the changes are.
Hide
John Swinbank added a comment - - edited
• I see the mention of a glossary in the changelog, and I see the aglossary.tex file. However, the PDF at https://ldm-294.lsst.io/v/RFC-621/LDM-294.pdf does not actually seem to contain a glossary.
• Figure 11, “DM System Science Team organisation”, is blank.
• Several of the glossary entries chosen are questionable. For example, there are several instances of Data \gls{Release} Production, where \gls{Data Release Production} would be more appropriate.
• Not changed in this version, but I note that the description of the SUIT group and CalTech should be substantially revised. We should ticket this work before accepting this RFC.
• Also not changed in this version, but does the DM Systems Engineering Team actually exist? Apparently I am a member, but I am not aware of it ever having met. Were I a reviewer, I might well ask to see meeting notes or other evidence that it actually does something.
• aglossary.tex is unstable. That is, if I check out the source from git, I am provided with a copy of aglossary.tex, but if I run make that file is changed into something different. Which is correct?
• The caption for figure 1 appears in the middle of figure 1 (ie, superimposed on the figure itself).
Show
John Swinbank added a comment - - edited I see the mention of a glossary in the changelog, and I see the aglossary.tex file. However, the PDF at https://ldm-294.lsst.io/v/RFC-621/LDM-294.pdf does not actually seem to contain a glossary. Figure 11, “DM System Science Team organisation”, is blank. Several of the glossary entries chosen are questionable. For example, there are several instances of Data \gls{Release} Production , where \gls{Data Release Production } would be more appropriate. Not changed in this version, but I note that the description of the SUIT group and CalTech should be substantially revised. We should ticket this work before accepting this RFC. Also not changed in this version, but does the DM Systems Engineering Team actually exist? Apparently I am a member, but I am not aware of it ever having met. Were I a reviewer, I might well ask to see meeting notes or other evidence that it actually does something. aglossary.tex is unstable. That is, if I check out the source from git, I am provided with a copy of aglossary.tex , but if I run make that file is changed into something different. Which is correct? The caption for figure 1 appears in the middle of figure 1 (ie, superimposed on the figure itself).
Hide
John Swinbank added a comment -

PR at https://github.com/lsst/LDM-294/pull/78 address some of the above issues.

Show
John Swinbank added a comment - PR at https://github.com/lsst/LDM-294/pull/78 address some of the above issues.
Hide
Gabriele Comoretto added a comment -

This is what has changed in the product tree in respect to the tag v3.7

Services

• Prompt Services group:
• reintroduced "Prompt Proc. Ingest Service"
(from my understanding, this is required in order to distinguish between the prompt ingestion made at base enclave and the prompt processing made at NCSA. It was removed in one of the last changes.KT may be more precise)

Software Products

• LSP SW group
• LSP Web SW has been replaced by SUIT and SUIT Online Help
(after some discussion with Gregory the past months, my understanding is that these two software products will be used to implement the LSP portal)
•  Science Pipelines SW group
• all Calibration Softwares have been merged into Calibration SW
(the calibration software to use in each context, daily, DPR, etc, should be the same, but with different configurations and input data. This simplifies the approach, one SW product instead of 4, but provides a better granularity in respect to what we have now, all in lsst_distrib.)
• added Science Pipeline Distribution software (lsst_distrib)
(this is the only software product we are doing releases now, therefore, it has to be added to the product tree.)
(my understanding is that some packages in the Science Pipelines are plugins. The proposal here is to have one software product that includes all those packages)

Hardware and Cots

• added Third Party Libs group
(The distinction between COTS and Third Party Libs is: a COTS is a tool used to implement the infrastructure and a Third Party lib is a software on which our SW products depends on
at build time and run time. This list is not complete.)

Added new top-level group: Ref Data
(my understanding is that data product used in LSST processing are important, therefore they deserve their own section in the product tree. This list is not complete)

Show
Hide
John Swinbank added a comment -

Further PR at https://github.com/lsst/LDM-294/pull/79 addresses problems with the glossary (ensuring it gets built by Travis; ensuring that all entries get properly recursively expanded; fixing formatting issues).

Show
John Swinbank added a comment - Further PR at https://github.com/lsst/LDM-294/pull/79 addresses problems with the glossary (ensuring it gets built by Travis; ensuring that all entries get properly recursively expanded; fixing formatting issues).
Hide
John Swinbank added a comment -

Thank you Gabriele Comoretto — that summary is very helpful! A couple of comments:

• I believe that what you are calling “Calibration Software” is actually software for generating calibration products, not software for performing calibration. I think it would be more helpful if the name reflected that. (I notice, though, that this is also an issue in LDM-148 and apparently I didn't complain about it there, so maybe we should approve this for consistency and file a new ticket to fix it later.)
• Can we tighten up the definition of “plugin”? Do you mean this to refer to all meas_extensions_ packages, or everything which provides a meas_base-style measurement plugin, or something else? I'm not sure I understand why it's meaningful to distinguish these plugins from the rest of “Sci Pipelines Libs” — can you explain?

Thanks again!

Show
John Swinbank added a comment - Thank you Gabriele Comoretto — that summary is very helpful! A couple of comments: I believe that what you are calling “Calibration Software” is actually software for generating calibration products, not software for performing calibration. I think it would be more helpful if the name reflected that. (I notice, though, that this is also an issue in LDM-148 and apparently I didn't complain about it there, so maybe we should approve this for consistency and file a new ticket to fix it later.) Can we tighten up the definition of “plugin”? Do you mean this to refer to all meas_extensions_ packages, or everything which provides a meas_base-style measurement plugin, or something else? I'm not sure I understand why it's meaningful to distinguish these plugins from the rest of “Sci Pipelines Libs” — can you explain? Thanks again!
Hide
John Swinbank added a comment -

Adopting this so we can baseline the updated document for upcoming review: it's a clear step forward compared to what was there before.

I would like to revisit the design of the product tree; I'm a little bit worried that this RFC is effectively bouncing some immature thinking there into the baseline. I'll ticket that work to be done after we have the review out of the way.

Show
John Swinbank added a comment - Adopting this so we can baseline the updated document for upcoming review: it's a clear step forward compared to what was there before. I would like to revisit the design of the product tree; I'm a little bit worried that this RFC is effectively bouncing some immature thinking there into the baseline. I'll ticket that work to be done after we have the review out of the way.
 Status Flagged [ 10606 ] Adopted [ 10806 ]
 Resolution Done [ 10000 ] Status Adopted [ 10806 ] Implemented [ 11105 ]
Hide
Wil O'Mullane added a comment -
Show
Wil O'Mullane added a comment - v3.8   https://docushare.lsstcorp.org/docushare/dsweb/Get/Version-60624/LDM-294.pdf
 Link This issue relates to DM-20838 [ DM-20838 ]
Hide
Kian-Tat Lim added a comment -

Yes, "Calibration Software" is for generating calibration products. There was a general desire to keep names short, but adding "Products" to the name would be reasonable (while something like "Products Generation" might be less desirable).

The distinction between Science Pipelines Libraries and Plugins is that the former are part of every release and distribution; it was thought that various combinations of the latter may be released or distributed, and they might be released independently of the Libraries. Plugins, in my mind, include things like meas_ and obs_ packages that fulfill more-or-less-defined interfaces to general frameworks. In theory, all Tasks could be considered plugins, but most seem so fundamental that there's no reason to split them out.

Show
Kian-Tat Lim added a comment - Yes, "Calibration Software" is for generating calibration products. There was a general desire to keep names short, but adding "Products" to the name would be reasonable (while something like "Products Generation" might be less desirable). The distinction between Science Pipelines Libraries and Plugins is that the former are part of every release and distribution; it was thought that various combinations of the latter may be released or distributed, and they might be released independently of the Libraries. Plugins, in my mind, include things like meas_ and obs_ packages that fulfill more-or-less-defined interfaces to general frameworks. In theory, all Tasks could be considered plugins, but most seem so fundamental that there's no reason to split them out.

People

• Assignee:
Wil O'Mullane
Reporter:
Wil O'Mullane
Watchers:
Gabriele Comoretto, John Swinbank, Kian-Tat Lim, Wil O'Mullane