Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-356

Provide reference catalog fluxes in Maggies

    XMLWordPrintable

Details

    • RFC
    • Status: Retired
    • Resolution: Done
    • DM

    Description

      In order to ease the comparison of fluxes between our source catalogs and reference catalogs, I propose that the catalog in a sky circle or pixel box instantiated from a reference object loader provide fluxes in the same units as our source catalog fluxes (Maggies, per RFC-322). Also, our reference catalog standards need to be clearly documented in some location outside the source code (pipelines.lsst.io?).

      The on-disk representation can be in whatever units are most appropriate for that particular catalog. If those units are not Maggies, LoadReferenceObjectTask will convert before the catalog is provided to the user.

      This is a change from current reference catalog practice, where LoadReferenceObjectTask's docstring says that all fluxes are in units of jansky:

          - *referenceFilterName*_flux: brightness in the specified reference catalog filter (Jy)
              Note: the function lsst.afw.image.abMagFromFlux will convert flux in Jy to AB Magnitude.
          - *referenceFilterName*_fluxSigma (optional): brightness standard deviation (Jy);
              omitted if no data is available; possibly nan if data is available for some objects but not others
          - camFlux: brightness in default camera filter (Jy); omitted if defaultFilter not specified
          - camFluxSigma: brightness standard deviation for default camera filter;
              omitted if defaultFilter not specified or standard deviation not available that filter
          - *cameraFilterName*_camFlux: brightness in specified camera filter (Jy)
          - *cameraFilterName*_camFluxSigma (optional): brightness standard deviation
              in specified camera filter (Jy); omitted if no data is available;
              possibly nan if data is available for some objects but not others
      

      I believe the only impact of this RFC is direct users of reference catalogs: single frame calibration and meas_mosaic (and jointcal, which is what provoked this RFC). As such, this could be implemented before RFC-322, but is probably best implemented simultaneously with it.

      Attachments

        Issue Links

          Activity

            No builds found.
            Parejkoj John Parejko created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Link This issue relates to DM-9195 [ DM-9195 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to RFC-322 [ RFC-322 ]
            jsick Jonathan Sick made changes -
            Link This issue is triggering DM-11044 [ DM-11044 ]

            I agree we need reference documentation for the catalog schema outside of a task's docstring. I've created DM-11044 to implement that (regardless of this RFC's outcome).

            jsick Jonathan Sick added a comment - I agree we need reference documentation for the catalog schema outside of a task's docstring. I've created DM-11044 to implement that (regardless of this RFC's outcome).
            tjenness Tim Jenness made changes -
            Description In order to ease the comparison of fluxes between our source catalogs and reference catalogs, I propose that the catalog in a sky circle or pixel box instantiated from a reference object loader provide fluxes in the same units as our source catalog fluxes (Maggies, per RFC-322). Also, our reference catalog standards need to be clearly documented in some location outside the source code (pipelines.lsst.io?).

            The on-disk representation can be in whatever units are most appropriate for that particular catalog. If those units are not Maggies, {{LoadReferenceObjectTask}} will convert before the catalog is provided to the user.

            This is a change from current reference catalog practice, where {{LoadReferenceObjectTask}}'s docstring says that all fluxes are in units of Jansky:

            {code}
                - *referenceFilterName*_flux: brightness in the specified reference catalog filter (Jy)
                    Note: the function lsst.afw.image.abMagFromFlux will convert flux in Jy to AB Magnitude.
                - *referenceFilterName*_fluxSigma (optional): brightness standard deviation (Jy);
                    omitted if no data is available; possibly nan if data is available for some objects but not others
                - camFlux: brightness in default camera filter (Jy); omitted if defaultFilter not specified
                - camFluxSigma: brightness standard deviation for default camera filter;
                    omitted if defaultFilter not specified or standard deviation not available that filter
                - *cameraFilterName*_camFlux: brightness in specified camera filter (Jy)
                - *cameraFilterName*_camFluxSigma (optional): brightness standard deviation
                    in specified camera filter (Jy); omitted if no data is available;
                    possibly nan if data is available for some objects but not others
            {code}

            I believe the only impact of this RFC is direct users of reference catalogs: single frame calibration and meas_mosaic (and jointcal, which is what provoked this RFC). As such, this could be implemented before RFC-322, but is probably best implemented simultaneously with it.
            In order to ease the comparison of fluxes between our source catalogs and reference catalogs, I propose that the catalog in a sky circle or pixel box instantiated from a reference object loader provide fluxes in the same units as our source catalog fluxes (Maggies, per RFC-322). Also, our reference catalog standards need to be clearly documented in some location outside the source code (pipelines.lsst.io?).

            The on-disk representation can be in whatever units are most appropriate for that particular catalog. If those units are not Maggies, {{LoadReferenceObjectTask}} will convert before the catalog is provided to the user.

            This is a change from current reference catalog practice, where {{LoadReferenceObjectTask}}'s docstring says that all fluxes are in units of jansky:

            {code}
                - *referenceFilterName*_flux: brightness in the specified reference catalog filter (Jy)
                    Note: the function lsst.afw.image.abMagFromFlux will convert flux in Jy to AB Magnitude.
                - *referenceFilterName*_fluxSigma (optional): brightness standard deviation (Jy);
                    omitted if no data is available; possibly nan if data is available for some objects but not others
                - camFlux: brightness in default camera filter (Jy); omitted if defaultFilter not specified
                - camFluxSigma: brightness standard deviation for default camera filter;
                    omitted if defaultFilter not specified or standard deviation not available that filter
                - *cameraFilterName*_camFlux: brightness in specified camera filter (Jy)
                - *cameraFilterName*_camFluxSigma (optional): brightness standard deviation
                    in specified camera filter (Jy); omitted if no data is available;
                    possibly nan if data is available for some objects but not others
            {code}

            I believe the only impact of this RFC is direct users of reference catalogs: single frame calibration and meas_mosaic (and jointcal, which is what provoked this RFC). As such, this could be implemented before RFC-322, but is probably best implemented simultaneously with it.
            tjenness Tim Jenness added a comment -

            I thought we were moving away from maggies to janskys. I thought zivezic was in favor of Jy.

            tjenness Tim Jenness added a comment - I thought we were moving away from maggies to janskys. I thought zivezic was in favor of Jy.

            Yes, I am working on a memo to support an upcoming proposal to adopt Jy and
            hope to have it done within a few weeks. The main argument in favor of Jy is that
            rarely used maggies don't bring many benefits, while Jy is a de facto standard
            in astronomy. Finer points will be addressed in the promised memo.

            zivezic Zeljko Ivezic added a comment - Yes, I am working on a memo to support an upcoming proposal to adopt Jy and hope to have it done within a few weeks. The main argument in favor of Jy is that rarely used maggies don't bring many benefits, while Jy is a de facto standard in astronomy. Finer points will be addressed in the promised memo.
            jbosch Jim Bosch added a comment -

            As long as we use either Jy or Maggies consistently everywhere, I'm content. It sounds like we should defer the resolution of this RFC until zivezic's proposal is out.

            jbosch Jim Bosch added a comment - As long as we use either Jy or Maggies consistently everywhere, I'm content. It sounds like we should defer the resolution of this RFC until zivezic 's proposal is out.
            Parejkoj John Parejko added a comment -

            I was not aware of that upcoming proposal.

            What date should I defer this RFC to?

            Parejkoj John Parejko added a comment - I was not aware of that upcoming proposal. What date should I defer this RFC to?

            How about you set it to August first? That gives Zeljko a "few weeks" but isn't so far in the future to cause us to forget about it.

            krughoff Simon Krughoff (Inactive) added a comment - How about you set it to August first? That gives Zeljko a "few weeks" but isn't so far in the future to cause us to forget about it.
            Parejkoj John Parejko made changes -
            Planned End 01/Jul/17 12:22 AM 02/Aug/17 12:22 AM
            tjenness Tim Jenness added a comment -

            I have not heard anything on this for the past month. zivezic what's the status of your report? When do you envisage us being able to continue to discuss this RFC?

            tjenness Tim Jenness added a comment - I have not heard anything on this for the past month. zivezic what's the status of your report? When do you envisage us being able to continue to discuss this RFC?

            I almost finish my report in Aspen. I will be MIA this week, and hope to finish it by
            the end of the following week (i.e. by the AHM). After collecting all the technical
            materials and opinions from numerous people, I think an inevitable conclusion is
            that LSST should report fluxes in Jansky (and not in maggies). If the resolution
            of this issue is blocking work, please continue with Jansky. I will make sure that
            proper discussion is held at the PST and DMLT levels.

            zivezic Zeljko Ivezic added a comment - I almost finish my report in Aspen. I will be MIA this week, and hope to finish it by the end of the following week (i.e. by the AHM). After collecting all the technical materials and opinions from numerous people, I think an inevitable conclusion is that LSST should report fluxes in Jansky (and not in maggies). If the resolution of this issue is blocking work, please continue with Jansky. I will make sure that proper discussion is held at the PST and DMLT levels.
            tjenness Tim Jenness made changes -
            Planned End 02/Aug/17 12:22 AM 26/Aug/17 12:00 AM
            tjenness Tim Jenness added a comment -

            Thanks. I've changed the planned end of this RFC to the week after AHM. I hope that's okay with Parejkoj.

            tjenness Tim Jenness added a comment - Thanks. I've changed the planned end of this RFC to the week after AHM. I hope that's okay with Parejkoj .
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 15304 ]
            womullan Wil O'Mullane made changes -
            Remote Link This issue links to "Page (Confluence)" [ 15317 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 15317 ]
            tjenness Tim Jenness added a comment -

            zivezic can you let me know the new timescale for this report?

            tjenness Tim Jenness added a comment - zivezic can you let me know the new timescale for this report?
            mjuric Mario Juric added a comment -

            zivezic is on vacation, but the draft has been circulated to the PST. The thinking so far is that the DPDD may not need to be changed (other than for reasons of clarity). As a reminder, Section 2.3 of the DPDD says we will offer both nmgy and Jy to the end-users. The last paragraph says:

            Nevertheless, we acknowledge that the large majority of users will want to work with magnitudes. For convenience, we plan to provide columns with (Pogson) magnitudes, where values with negative flux will evaluate to NULL. Similarly, we will provide columns with flux expressed in Jy (and its error estimate that includes the relative and absolute calibration error contributions).

            mjuric Mario Juric added a comment - zivezic is on vacation, but the draft has been circulated to the PST. The thinking so far is that the DPDD may not need to be changed (other than for reasons of clarity). As a reminder, Section 2.3 of the DPDD says we will offer both nmgy and Jy to the end-users. The last paragraph says: Nevertheless, we acknowledge that the large majority of users will want to work with magnitudes. For convenience, we plan to provide columns with (Pogson) magnitudes, where values with negative flux will evaluate to NULL. Similarly, we will provide columns with flux expressed in Jy (and its error estimate that includes the relative and absolute calibration error contributions).
            mjuric Mario Juric made changes -
            Labels dm-sst
            Parejkoj John Parejko added a comment -

            Zeljko's suggestion is that we do everything internally in Jansky as well (hence his objection to my request to use Maggies for reference fluxes, etc.) This would require a change to the DPDD.

            Parejkoj John Parejko added a comment - Zeljko's suggestion is that we do everything internally in Jansky as well (hence his objection to my request to use Maggies for reference fluxes, etc.) This would require a change to the DPDD.
            mjuric Mario Juric added a comment -

            Still hashing this out with zivezic. I think part of the confusion may be that the definition of a maggie is incorrect in the DPDD; the way I wrote it there, it's just another unit for flux which I readily admit would be silly. The intent was to be similar to http://www.sdss3.org/dr8/algorithms/magnitudes.php#nmgy; especially note the last paragraph.

            We should be able to resolve this next week one way or another; is this RFC blocking work or can it wait for a few more days?

            mjuric Mario Juric added a comment - Still hashing this out with zivezic . I think part of the confusion may be that the definition of a maggie is incorrect in the DPDD; the way I wrote it there, it's just another unit for flux which I readily admit would be silly. The intent was to be similar to http://www.sdss3.org/dr8/algorithms/magnitudes.php#nmgy ; especially note the last paragraph. We should be able to resolve this next week one way or another; is this RFC blocking work or can it wait for a few more days?

            This is not currently blocking any work, but we should probably sort it out before we attempt to implement RFC-368 (since it has been requested that we make all necessary changes to our refcats at one time).

            Parejkoj John Parejko added a comment - This is not currently blocking any work, but we should probably sort it out before we attempt to implement RFC-368 (since it has been requested that we make all necessary changes to our refcats at one time).
            tjenness Tim Jenness added a comment -

            Parejkoj can you adjust the planned end date for this RFC so it drops below my radar again? Do you want to link RFC-368 with this one?

            tjenness Tim Jenness added a comment - Parejkoj can you adjust the planned end date for this RFC so it drops below my radar again? Do you want to link RFC-368 with this one?
            tjenness Tim Jenness added a comment -

            Any progress? Change of planned end date?

            tjenness Tim Jenness added a comment - Any progress? Change of planned end date?
            tjenness Tim Jenness added a comment -

            zivezic reports:

            We had a PST telecon where we decided to go with Jansky as the standard LSST
            flux unit (implying that fluxes on a linear absolute scale will be reported as the main
            quantity and other quantities, such as magnitudes, will be derived quantities).

            tjenness Tim Jenness added a comment - zivezic reports: We had a PST telecon where we decided to go with Jansky as the standard LSST flux unit (implying that fluxes on a linear absolute scale will be reported as the main quantity and other quantities, such as magnitudes, will be derived quantities).
            mjuric Mario Juric added a comment -

            What we discussed at the PST concerns the final results we give to the users (i.e., what’s in the DRs). The conclusion was, as zivezic says, to clarify the DPDD text so it’s clear what comes out in the end will be in Jansky’s with proper abs/rel errors (it already says so, but gets lost in the maggy discussion).

            I think that this RFC is something different — it talks about components internal to the pipeline that are used to get to these results. I’m (very) worried that this Jansky vs. maggies discussion is not misinterpreted as a mandate from above to use Jansky everywhere. That was not the intent – our discussion speaks only of the final, post-calibration, products. Use what makes sense elsewhere.

            For this RFC, I'd ask whether all input catalogs are always convertable to Janskys? Do we know absolute zero points? Or are maggies — which are units of flux relative to a standard source in that input catalog (see http://www.sdss3.org/dr8/algorithms/magnitudes.php), a better way to go? I don't know enough about our particular implementation to opine (@rhl, any thoughts?).

            mjuric Mario Juric added a comment - What we discussed at the PST concerns the final results we give to the users (i.e., what’s in the DRs). The conclusion was, as zivezic says, to clarify the DPDD text so it’s clear what comes out in the end will be in Jansky’s with proper abs/rel errors (it already says so, but gets lost in the maggy discussion). I think that this RFC is something different — it talks about components internal to the pipeline that are used to get to these results. I’m (very) worried that this Jansky vs. maggies discussion is not misinterpreted as a mandate from above to use Jansky everywhere . That was not the intent – our discussion speaks only of the final, post-calibration, products. Use what makes sense elsewhere. For this RFC, I'd ask whether all input catalogs are always convertable to Janskys? Do we know absolute zero points? Or are maggies — which are units of flux relative to a standard source in that input catalog (see http://www.sdss3.org/dr8/algorithms/magnitudes.php ), a better way to go? I don't know enough about our particular implementation to opine (@rhl, any thoughts?).
            Parejkoj John Parejko made changes -
            Planned End 26/Aug/17 12:00 AM 30/Sep/17 12:00 AM

            Note that a further conversation on this topic is happening on Community: https://community.lsst.org/t/photometric-units-in-the-stack-and-photocalib/2259

            Parejkoj John Parejko added a comment - Note that a further conversation on this topic is happening on Community: https://community.lsst.org/t/photometric-units-in-the-stack-and-photocalib/2259
            tjenness Tim Jenness added a comment -

            Can Parejkoj please summarize the current status of this RFC?

            tjenness Tim Jenness added a comment - Can Parejkoj please summarize the current status of this RFC?
            Parejkoj John Parejko made changes -
            Planned End 30/Sep/17 12:00 AM 12/Oct/17 12:00 AM
            tjenness Tim Jenness added a comment -

            Parejkoj what would you like to do?

            tjenness Tim Jenness added a comment - Parejkoj what would you like to do?

            Conversation on the Community post above petered out without an obvious conclusion, and with a few conflicting recommendations. I'm not sure how to proceed. Should we have a short voice chat with some of the people involved in the discussion, to see if we can come to a conclusion?

            Parejkoj John Parejko added a comment - Conversation on the Community post above petered out without an obvious conclusion, and with a few conflicting recommendations. I'm not sure how to proceed. Should we have a short voice chat with some of the people involved in the discussion, to see if we can come to a conclusion?
            tjenness Tim Jenness added a comment -

            Parejkoj Please arrange something so that we can resolve this.

            tjenness Tim Jenness added a comment - Parejkoj Please arrange something so that we can resolve this.
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-12572 [ DM-12572 ]
            Parejkoj John Parejko added a comment -

            Adopting this as written: the discussion on the Community post suggested converting to a maggie scale during LoadReferenceObjectsTask, which is in the Description above. The final design of our flux provenance system and how we provide the absolute flux scale to end users is beyond the scope of this RFC (but we can provide methods to help bridge the gap), and I've made that clear in the implementation ticket: DM-12572.

            Parejkoj John Parejko added a comment - Adopting this as written: the discussion on the Community post suggested converting to a maggie scale during LoadReferenceObjectsTask , which is in the Description above. The final design of our flux provenance system and how we provide the absolute flux scale to end users is beyond the scope of this RFC (but we can provide methods to help bridge the gap), and I've made that clear in the implementation ticket: DM-12572 .
            Parejkoj John Parejko made changes -
            Status Proposed [ 10805 ] Adopted [ 10806 ]
            gpdf Gregory Dubois-Felsmann made changes -
            Link This issue is triggering DM-12129 [ DM-12129 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to RFC-549 [ RFC-549 ]
            Parejkoj John Parejko added a comment -

            Closing as rejected: this has been entirely superseded by RFC-549.

            Parejkoj John Parejko added a comment - Closing as rejected: this has been entirely superseded by RFC-549 .
            Parejkoj John Parejko made changes -
            Resolution Rejected [ 10301 ] Done [ 10000 ]
            Status Adopted [ 10806 ] Retired [ 10705 ]
            lguy Leanne Guy made changes -
            Labels dm-sst DM-SST dm-sst
            lguy Leanne Guy made changes -
            Labels DM-SST dm-sst DM-SST

            People

              Parejkoj John Parejko
              Parejkoj John Parejko
              Colin Slater, Jim Bosch, John Parejko, Jonathan Sick, Paul Price, Simon Krughoff (Inactive), Tim Jenness, Zeljko Ivezic
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                Jenkins

                  No builds found.