Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-322

Rename "*_flux" fields to "*_instFlux" in SourceCatalogs

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Currently, the "flux" fields in our source catalogs are actually "on-chip, post-ISR counts", instead of "calibrated fluxes". This causes two problems: users looking for calibrated counts don't reach the correct objects, and we don't have a good place to put calibrated fluxes. I propose that all of our *Flux_flux/*Flux_fluxSigma fields be renamed *Flux_instFlux/*Flux_instFluxSigma, which will allow us to add e.g. *Flux_flux and *Flux_mag fields to our source catalogs to hold post-calibrated fluxes (in Maggies) and magnitudes (Pogson). The DPDD says in section 3.3 "Fluxes and Magnitudes" that we will distribute "flux" in units of "maggies", so this new naming is consistent with that.

      This developed while I was working on the new PhotoCalib object and I realized I'd inadvertantly overloaded the term "flux" in the code (sometimes meaning counts, sometimes calibrated flux). Some of this came from borrowing from the old Calib (where the term is very overloaded), and some from an attempt to get it right in the PhotoCalib code but then tripping over the input/output catalog field names.

      We will also rename the InstFlux slot to GaussianFlux, and remove the InstMag slot.

      In terms of work required, searching the stack (*.h, *.cc, *.py) found 111 matches in 33 files to Flux_flux, plus about 80 total matches in about 20 files (didn't compare overlap with the former) to several variations on + "_flux.

      In terms of timing, implementing this would wait until Calib was removed from the stack and replaced by PhotoCalib (not yet scheduled, but likely within the next couple months).

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            The use of ADU or DN in flux fields dates back a long way, past https://dev.lsstcorp.org/trac/wiki/CommonVocabulary, but I'm fine with renaming if it matches scientific expectations.

            Show
            ktl Kian-Tat Lim added a comment - The use of ADU or DN in flux fields dates back a long way, past https://dev.lsstcorp.org/trac/wiki/CommonVocabulary , but I'm fine with renaming if it matches scientific expectations.
            Hide
            jbosch Jim Bosch added a comment -

            Interesting idea. I like it!

            Show
            jbosch Jim Bosch added a comment - Interesting idea. I like it!
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Yes, please.

            A great majority of astronomers say "flux" when they mean "counts".

            I agree that it will be clearer if they did not. And it will be clearer if we do not.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Yes, please. A great majority of astronomers say "flux" when they mean "counts". I agree that it will be clearer if they did not. And it will be clearer if we do not.
            Hide
            price Paul Price added a comment -

            I don't see any problem with the current usage, and in fact I think it should be preferred. We are measuring flux in counts. I don't think we should name a field for its units. That's what the units element of the Field is for.

            Show
            price Paul Price added a comment - I don't see any problem with the current usage, and in fact I think it should be preferred. We are measuring flux in counts. I don't think we should name a field for its units. That's what the units element of the Field is for.
            Hide
            jbosch Jim Bosch added a comment -

            FWIW, I like this not because it avoids confusion (it might, but It think the "units" field Paul Price mentioned ought to be sufficient for that), but because it lets us add _flux (along with, say, _mag) when we add calibrated quantities without replacing something already there.

            Show
            jbosch Jim Bosch added a comment - FWIW, I like this not because it avoids confusion (it might, but It think the "units" field Paul Price mentioned ought to be sufficient for that), but because it lets us add _flux (along with, say, _mag ) when we add calibrated quantities without replacing something already there.
            Hide
            tjenness Tim Jenness added a comment - - edited

            John Parejko looks like you can adopt this (Assuming you can convince Paul Price).

            Show
            tjenness Tim Jenness added a comment - - edited John Parejko looks like you can adopt this (Assuming you can convince Paul Price ).
            Hide
            Parejkoj John Parejko added a comment -

            Paul Price How would you propose we add fields for "calibrated flux" and "magnitude" to our catalogs, then? And why do you prefer flux for a field that actually contains uncalibrated counts?

            Show
            Parejkoj John Parejko added a comment - Paul Price How would you propose we add fields for "calibrated flux" and "magnitude" to our catalogs, then? And why do you prefer flux for a field that actually contains uncalibrated counts?
            Hide
            price Paul Price added a comment - - edited

            "Flux" is a quantity. "Counts" is a unit. I believe we should name the column after the quantity being measured, and use the "units" field to record the units. Otherwise you're getting into Hungarian notation.

            For calibrated flux, what about *_flux_calibrated (or similar)? Magnitudes would naturally be *_mag.

            Show
            price Paul Price added a comment - - edited "Flux" is a quantity. "Counts" is a unit. I believe we should name the column after the quantity being measured, and use the "units" field to record the units. Otherwise you're getting into Hungarian notation. For calibrated flux, what about *_flux_calibrated (or similar)? Magnitudes would naturally be *_mag .
            Hide
            jbosch Jim Bosch added a comment -

            I think I might agree with Paul Price in general (or, maybe, in principle?) but disagree in this particular case: I think there's real value to having a short names for both (...says the guy who imposed "base_PsfFlux_flux" on the world, trying to atone for his sins...) and to do that I'm willing to play a little fast and loose with quantities vs. units. But maybe there's a name that can make everyone happy.

            As a side note, if we do go with a two-word phrase like for the calibrated flux, I think we should combine them with camelCase, not underscores; in the new schema naming conventions we want to distinguish scoping (which we use underscores) from word separation (for which we use camelCase). Granted, the distinction is not always super clear, but I think it is here.

            Show
            jbosch Jim Bosch added a comment - I think I might agree with Paul Price in general (or, maybe, in principle?) but disagree in this particular case: I think there's real value to having a short names for both (...says the guy who imposed "base_PsfFlux_flux" on the world, trying to atone for his sins...) and to do that I'm willing to play a little fast and loose with quantities vs. units. But maybe there's a name that can make everyone happy. As a side note, if we do go with a two-word phrase like for the calibrated flux, I think we should combine them with camelCase, not underscores; in the new schema naming conventions we want to distinguish scoping (which we use underscores) from word separation (for which we use camelCase). Granted, the distinction is not always super clear, but I think it is here.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            The Quantity of "flux" as measured by counting photo-electrons in a given region on a CCD is a different Quantity than the astrophysical "flux" that we wish to know. The latter is the flux that the user cares about and we should thus reserve the name "flux" to refer to the quantity that is meant to be the astrophysical flux of the object.

            In astronomy the flux we care about is the flux of an object at a given location independent of the system used to observe it. E.g. the flux impinging on the top of the atmosphere in energy/time/area or photons/time/area.

            In the processing of the signal recorded by a CCD we may counting the number of photons in an aperture and one may argue that this Quantity is a "flux" of photo-electrons on the area of the CCD with the units of "counts/time/sq. pixel", but this is not the same Quantity as the astrophysical "flux" of the object at the top of the atmosphere. The two different Quantities are certainly deeply related, but they are not the same Quantity. We should thus assign them different names to avoid confusion.

            I respect that calling one of them "counts" is perhaps philosophically objectionable, but we actually lack a good name for this Quantity and using "counts" has the substantial practical advantage of clarity. It will be very clear to most astronomers what is meant by naming a quantity with that term. In brief, they'll know that it's not the thing they wanted to know. The instead wanted the thing that include the name "flux".

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - The Quantity of "flux" as measured by counting photo-electrons in a given region on a CCD is a different Quantity than the astrophysical "flux" that we wish to know. The latter is the flux that the user cares about and we should thus reserve the name "flux" to refer to the quantity that is meant to be the astrophysical flux of the object. In astronomy the flux we care about is the flux of an object at a given location independent of the system used to observe it. E.g. the flux impinging on the top of the atmosphere in energy/time/area or photons/time/area. In the processing of the signal recorded by a CCD we may counting the number of photons in an aperture and one may argue that this Quantity is a "flux" of photo-electrons on the area of the CCD with the units of "counts/time/sq. pixel", but this is not the same Quantity as the astrophysical "flux" of the object at the top of the atmosphere. The two different Quantities are certainly deeply related, but they are not the same Quantity. We should thus assign them different names to avoid confusion. I respect that calling one of them "counts" is perhaps philosophically objectionable, but we actually lack a good name for this Quantity and using "counts" has the substantial practical advantage of clarity. It will be very clear to most astronomers what is meant by naming a quantity with that term. In brief, they'll know that it's not the thing they wanted to know. The instead wanted the thing that include the name "flux".
            Hide
            Parejkoj John Parejko added a comment -

            Related to the units, aren't post-ISR counts really in units of "ADU"?

            Show
            Parejkoj John Parejko added a comment - Related to the units, aren't post-ISR counts really in units of "ADU"?
            Hide
            price Paul Price added a comment -

            I have always considered "counts" and "ADU" as synonyms.

            Show
            price Paul Price added a comment - I have always considered "counts" and "ADU" as synonyms.
            Hide
            krughoff Simon Krughoff added a comment -

            Here is my proposal and I think it may actually address all the issues (but I'm probably wrong )

            There are three different conceptual values we may measure no a grid of pixels:

            • counts – This is an integer. It doesn't really matter whether it's drops of water in buckets, electrons, or binned voltage. This we almost never make any measurements of directly (at least not in the source photometry sense).
            • postIsr – This is a floating point value per pixel that is proportional to the number of photons that were converted in each pixel. This is a fluxy quantity because it is proportional, in a potentially spatially varying and color dependent way, to a physical flux unit. We care very much about measuring photometry in this system because it is what seeds the single frame photometric calibration.
            • Calibrated flux – This is a floating point value per pixel that is in units of a physical flux above the atmosphere and this is what we use for science.

            There are, of course, many things in between these three milestone pixel values, but these are the three I think we deal with most. I think the main point of contention in this thread is the second kind of thing. It's fluxy, but not in physical units. This is actually a solved problem because I believe most systems will call that value an "instrumental flux". I.e. it is the flux the instrument measures, but not necessarily what is the true flux. So, I propose the following:

            • counts --> *Flux_counts This will be so rarely used we may want to do away with it.
            • postIsr --> *Flux_instFlux This is the instrumental flux (and could have an associated Flux_InstMag I suppose}}
            • Calibrated flux --> *Flux_flux I think some may argue for *Flux_calFlux or similar, but I think we should steer toward understanding the word flux to refer to a physically calibrated flux and use modifier for every other kind. It doesn't matter that much to me.

            Does that help break the log jam?

            Show
            krughoff Simon Krughoff added a comment - Here is my proposal and I think it may actually address all the issues (but I'm probably wrong ) There are three different conceptual values we may measure no a grid of pixels: counts – This is an integer. It doesn't really matter whether it's drops of water in buckets, electrons, or binned voltage. This we almost never make any measurements of directly (at least not in the source photometry sense). postIsr – This is a floating point value per pixel that is proportional to the number of photons that were converted in each pixel. This is a fluxy quantity because it is proportional, in a potentially spatially varying and color dependent way, to a physical flux unit. We care very much about measuring photometry in this system because it is what seeds the single frame photometric calibration. Calibrated flux – This is a floating point value per pixel that is in units of a physical flux above the atmosphere and this is what we use for science. There are, of course, many things in between these three milestone pixel values, but these are the three I think we deal with most. I think the main point of contention in this thread is the second kind of thing. It's fluxy, but not in physical units. This is actually a solved problem because I believe most systems will call that value an "instrumental flux". I.e. it is the flux the instrument measures, but not necessarily what is the true flux. So, I propose the following: counts --> *Flux_counts This will be so rarely used we may want to do away with it. postIsr --> *Flux_instFlux This is the instrumental flux (and could have an associated Flux_InstMag I suppose}} Calibrated flux --> *Flux_flux I think some may argue for *Flux_calFlux or similar, but I think we should steer toward understanding the word flux to refer to a physically calibrated flux and use modifier for every other kind. It doesn't matter that much to me. Does that help break the log jam?
            Hide
            ctslater Colin Slater added a comment -

            Calling our current measurement instFlux sounds like a good idea to me. I think that accurately captures what it is.

            Show
            ctslater Colin Slater added a comment - Calling our current measurement instFlux sounds like a good idea to me. I think that accurately captures what it is.
            Hide
            jbosch Jim Bosch added a comment -

            I'm okay with this proposal.

            As a corollary, I propose on this RFC that we drop the InstFlux slot that's currently in the measurement tasks and the source catalogs, since it doesn't make any sense there (as Simon Krughoff said, it's a flux system, not a flux algorithm), and otherwise we'll have even more confusion if we use instFlux (correctly) here.

            Show
            jbosch Jim Bosch added a comment - I'm okay with this proposal. As a corollary, I propose on this RFC that we drop the InstFlux slot that's currently in the measurement tasks and the source catalogs, since it doesn't make any sense there (as Simon Krughoff said, it's a flux system, not a flux algorithm), and otherwise we'll have even more confusion if we use instFlux (correctly) here.
            Hide
            Parejkoj John Parejko added a comment -

            Jim Bosch I see both InstFlux and InstMag: should we drop both of those? (and what are they, anyway?)

            Show
            Parejkoj John Parejko added a comment - Jim Bosch I see both InstFlux and InstMag : should we drop both of those? (and what are they, anyway?)
            Hide
            jbosch Jim Bosch added a comment -

            Unless InstMag is something generated automatically from InstFlux, I'm not familiar with it.

            I have no idea what InstFlux was originally intended for. We've been abusing it as the slot we put GaussianFlux in for years.

            Show
            jbosch Jim Bosch added a comment - Unless InstMag is something generated automatically from InstFlux, I'm not familiar with it. I have no idea what InstFlux was originally intended for. We've been abusing it as the slot we put GaussianFlux in for years.
            Hide
            Parejkoj John Parejko added a comment -

            Simon Krughoff Your proposal seems reasonable. I'm not sure what your Flux_InstMag would represent; does anyone ever measure a "magnitude of counts" in that way?

            Also, we should be careful because the _flux that I'm proposing here is in Maggies (as our DPDD suggests), which is not quite a physical flux (there's an absolute calibration factor, which might just involve multiplying by 3631 Jy, but might also involve another small scale factor.

            Show
            Parejkoj John Parejko added a comment - Simon Krughoff Your proposal seems reasonable. I'm not sure what your Flux_InstMag would represent; does anyone ever measure a "magnitude of counts" in that way? Also, we should be careful because the _flux that I'm proposing here is in Maggies (as our DPDD suggests), which is not quite a physical flux (there's an absolute calibration factor, which might just involve multiplying by 3631 Jy , but might also involve another small scale factor.
            Hide
            krughoff Simon Krughoff added a comment -

            I suggested Flux_instMag because people seem to always want a magnitude, but I agree the "right" quantity to use is the fluxy one.

            Also, we should be careful because the _flux that I'm proposing here is in Maggies (as our DPDD suggests), which is not quite a physical flux (there's an absolute calibration factor, which might just involve multiplying by 3631 Jy, but might also involve another small scale factor.

            I specifically didn't mention a unit for the physical flux. Maggies is a physical flux unit in the sense that it is intended to represent the above atmosphere flux, it's just not Janskies. I do agree we need to be careful to be explicit about our convention for flux units.

            Show
            krughoff Simon Krughoff added a comment - I suggested Flux_instMag because people seem to always want a magnitude, but I agree the "right" quantity to use is the fluxy one. Also, we should be careful because the _flux that I'm proposing here is in Maggies (as our DPDD suggests), which is not quite a physical flux (there's an absolute calibration factor, which might just involve multiplying by 3631 Jy, but might also involve another small scale factor. I specifically didn't mention a unit for the physical flux. Maggies is a physical flux unit in the sense that it is intended to represent the above atmosphere flux, it's just not Janskies. I do agree we need to be careful to be explicit about our convention for flux units.
            Hide
            price Paul Price added a comment -

            I support Simon's proposal (if you remove the mention of counts) with Jim's addition.

            Show
            price Paul Price added a comment - I support Simon's proposal (if you remove the mention of counts ) with Jim's addition.
            Hide
            Parejkoj John Parejko added a comment -

            Thanks all.

            I've updated the Description with the final proposal, and filed DM-10302 as the implementation ticket.

            Show
            Parejkoj John Parejko added a comment - Thanks all. I've updated the Description with the final proposal, and filed DM-10302 as the implementation ticket.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Colin Slater, Gregory Dubois-Felsmann, Jim Bosch, John Parejko, John Swinbank, Kian-Tat Lim, Michael Wood-Vasey, Paul Price, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.