Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17689

Expose maximum number of bits used by IdFactory

    XMLWordPrintable

Details

    • 1
    • Data Release Production

    Description

      DM-15843 (pipe_tasks) uses the magic number "64" when construct afw.table.IdFactory instance as there is currently no programmatic way to go from the number of bits in a packed data ID to the amount that ID would need to be shifted to fit in the upper bits of a source ID.

      I'm deferring fixing that to another ticket to avoid a period in which the weekly can't be used with master of pipe_tasks (what that's not required by our workflow, it seems a good practice when it's easy to do, as it is here).

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            nlust, this is the spin-off from your comment about magic numbers on DM-15843.

            Changes in afw, obs_base, and pipe_tasks.

            I'm hoping to merge afw this week and the others next week to avoid any periods (at least due to these changes) in which downstream packages can't use an afw weekly.

            jbosch Jim Bosch added a comment - nlust , this is the spin-off from your comment about magic numbers on DM-15843 . Changes in afw , obs_base , and pipe_tasks. I'm hoping to merge afw this week and the others next week to avoid any periods (at least due to these changes) in which downstream packages can't use an afw weekly.

            Hey jbosch — this has been reviewed but not merged for most of this year. Can we close it out?

            swinbank John Swinbank added a comment - Hey jbosch — this has been reviewed but not merged for most of this year. Can we close it out?
            jbosch Jim Bosch added a comment -

            It breaks the SDSS demo because it changes some source IDs.  At this point I've been thinking I'll just wait until that's gone, now that it's been RFC'd, and then update the changes here to reflect the stuff that's changed underneath it.

            jbosch Jim Bosch added a comment - It breaks the SDSS demo because it changes some source IDs.  At this point I've been thinking I'll just wait until that's gone, now that it's been RFC'd, and then update the changes here to reflect the stuff that's changed underneath it.
            tjenness Tim Jenness added a comment -

            Now that stack demo has gone can this ticket be resurrected?

            tjenness Tim Jenness added a comment - Now that stack demo has gone can this ticket be resurrected?
            jbosch Jim Bosch added a comment -

            Yeah, I'll try to get to it soon, and read up on DM-28796 at the same time to see if this will make things worse or better. Still, probably not until next week.

            jbosch Jim Bosch added a comment - Yeah, I'll try to get to it soon, and read up on DM-28796 at the same time to see if this will make things worse or better. Still, probably not until next week.
            jbosch Jim Bosch added a comment -

            This is ready for review again; tjenness, would you mind taking it this time?

            Overall, I tried to make our handling of source ID generation a bit more uniform, but without massively increasing the scope of the ticket. There are some related problems that I didn't try to solve - not just the longstanding duplication between translators and DimensionPackers, but also the fact that there's no code for unpacking Source IDs, even though we can now unpack the image IDs that get mangled into them. Someday this all merits a top-down rethink and an RFC, and while I don't think it'll be that hard it's not something I have time for now.

            Anyhow, there are PRs for obs_base and pipe_tasks; there were previous branches before the tickets stalled out, but the were so out of date that everything was basically rewritten (it's still not much code). There was also a PR for afw merged long ago that I think you can ignore.

            jbosch Jim Bosch added a comment - This is ready for review again; tjenness , would you mind taking it this time? Overall, I tried to make our handling of source ID generation a bit more uniform, but without massively increasing the scope of the ticket. There are some related problems that I didn't try to solve - not just the longstanding duplication between translators and DimensionPackers, but also the fact that there's no code for unpacking Source IDs, even though we can now unpack the image IDs that get mangled into them. Someday this all merits a top-down rethink and an RFC, and while I don't think it'll be that hard it's not something I have time for now. Anyhow, there are PRs for obs_base and pipe_tasks ; there were previous branches before the tickets stalled out, but the were so out of date that everything was basically rewritten (it's still not much code). There was also a PR for afw merged long ago that I think you can ignore.
            tjenness Tim Jenness added a comment -

            Looks okay to me. I'm wondering whether this is a new afw entanglement but I'm sure that will sort itself out eventually.

            tjenness Tim Jenness added a comment - Looks okay to me. I'm wondering whether this is a new afw entanglement but I'm sure that will sort itself out eventually.

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Tim Jenness
              Jim Bosch, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.