Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6927

Butler should support flags for writing source tables

    XMLWordPrintable

    Details

    • Team:
      Architecture

      Description

      Simon Krughoff discovered an interesting bug: several tasks that save source catalogs using butler.put want to specify a value for the flags argument of the FITS writer. They do this by passing flags=... to the butler.put.... However, the butler assumes that the flag argument is another data ID key:value pair. (Normally this extra item is ignored, which is why we have only now noticed it, but in some cases it can cause an exception).

      We might be able to manage without passing the flag at all, but I worry that is not a good solution. I'd be much happier having some way to pass the flags argument through to the persistence code.

        Attachments

          Issue Links

            Activity

            Hide
            krughoff Simon Krughoff added a comment -

            Did this get resolved in any way? I just ran into this again with people reducing DECam data.

            Show
            krughoff Simon Krughoff added a comment - Did this get resolved in any way? I just ran into this again with people reducing DECam data.
            Hide
            Parejkoj John Parejko added a comment -

            I just ran into this with CFHT data, and there was a discussion about it on slack. This is the sort of command that fails:

            data = butler.get("src", dataId=dataId, flags=lsst.afw.table.SOURCE_IO_NO_FOOTPRINTS)
            

            Can we get this taken care of, please?

            Show
            Parejkoj John Parejko added a comment - I just ran into this with CFHT data, and there was a discussion about it on slack. This is the sort of command that fails: data = butler.get("src", dataId=dataId, flags=lsst.afw.table.SOURCE_IO_NO_FOOTPRINTS) Can we get this taken care of, please?
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Kian-Tat LimWill
            A) This ticket be resolved by Gen3?
            B) We be able to load a Source Catalog without footprints in Gen3?

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Kian-Tat Lim Will A) This ticket be resolved by Gen3? B) We be able to load a Source Catalog without footprints in Gen3?
            Hide
            tjenness Tim Jenness added a comment -

            Gen3 does not allow butler.put to have write flags given by the user. The user has to strip the footprints out before writing.

            Flags on read to control footprints are covered by DM-26761.

            Show
            tjenness Tim Jenness added a comment - Gen3 does not allow butler.put to have write flags given by the user. The user has to strip the footprints out before writing. Flags on read to control footprints are covered by DM-26761 .
            Hide
            tjenness Tim Jenness added a comment -

            This came up again today on community (https://community.lsst.org/t/do-write-heavy-footprints-in-sources-appears-not-to-have-any-effect/7024). The outcome seems to be that:

            • Deciding whether or not to include the footprints on write should be a configuration parameter for the task and not for the butler to decide.
            • Controlling whether to read the heavy footprints is still needed.
            Show
            tjenness Tim Jenness added a comment - This came up again today on community ( https://community.lsst.org/t/do-write-heavy-footprints-in-sources-appears-not-to-have-any-effect/7024 ). The outcome seems to be that: Deciding whether or not to include the footprints on write should be a configuration parameter for the task and not for the butler to decide. Controlling whether to read the heavy footprints is still needed.

              People

              Assignee:
              ktl Kian-Tat Lim
              Reporter:
              rowen Russell Owen
              Watchers:
              Colin Slater, Jim Bosch, John Parejko, Kian-Tat Lim, Michael Wood-Vasey [X] (Inactive), Nate Pease [X] (Inactive), Paul Price, Russell Owen, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins Builds

                  No builds found.