Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22178

queryDatasets produces lots of duplicate outputs

    Details

    • Story Points:
      2
    • Team:
      Data Release Production

      Description

      Running a query like

      list(butler.registry.queryDatasets("calexp", collections=["shared/ci_hsc_output"],
                                         skymap="discrete/ci_hsc", tract=0, patch=70))
      

      produces many duplicate records for unclear reasons (they're not from having multiple collections, so deduplicate doesn't help).  Try to fix this, or at least document when duplicate results must be expected.

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Description Running a query like

            {{list(butler.registry.queryDatasets("calexp", collections=["shared/ci_hsc_output"],}}
            {{ skymap="discrete/ci_hsc", tract=0, patch=70))}}

            produces many duplicate records for unclear reasons (they're not from having multiple collections, so {{deduplicate}} doesn't help).  Try to fix this, or at least document when duplicate results must be expected.
            Running a query like

            {code}
            list(butler.registry.queryDatasets("calexp", collections=["shared/ci_hsc_output"],
                                                                 skymap="discrete/ci_hsc", tract=0, patch=70))
            {code}

            produces many duplicate records for unclear reasons (they're not from having multiple collections, so {{deduplicate}} doesn't help).  Try to fix this, or at least document when duplicate results must be expected.
            tjenness Tim Jenness made changes -
            Description Running a query like

            {code}
            list(butler.registry.queryDatasets("calexp", collections=["shared/ci_hsc_output"],
                                                                 skymap="discrete/ci_hsc", tract=0, patch=70))
            {code}

            produces many duplicate records for unclear reasons (they're not from having multiple collections, so {{deduplicate}} doesn't help).  Try to fix this, or at least document when duplicate results must be expected.
            Running a query like

            {code}
            list(butler.registry.queryDatasets("calexp", collections=["shared/ci_hsc_output"],
                                                 skymap="discrete/ci_hsc", tract=0, patch=70))
            {code}

            produces many duplicate records for unclear reasons (they're not from having multiple collections, so {{deduplicate}} doesn't help).  Try to fix this, or at least document when duplicate results must be expected.
            tjenness Tim Jenness made changes -
            Description Running a query like

            {code}
            list(butler.registry.queryDatasets("calexp", collections=["shared/ci_hsc_output"],
                                                 skymap="discrete/ci_hsc", tract=0, patch=70))
            {code}

            produces many duplicate records for unclear reasons (they're not from having multiple collections, so {{deduplicate}} doesn't help).  Try to fix this, or at least document when duplicate results must be expected.
            Running a query like

            {code}
            list(butler.registry.queryDatasets("calexp", collections=["shared/ci_hsc_output"],
                                               skymap="discrete/ci_hsc", tract=0, patch=70))
            {code}

            produces many duplicate records for unclear reasons (they're not from having multiple collections, so {{deduplicate}} doesn't help).  Try to fix this, or at least document when duplicate results must be expected.
            Hide
            tjenness Tim Jenness added a comment -

            Since we know that all the DatasetRefs being returned must be coming from the same butler repository, this means that the ref.id must be unique. A short term fix is therefore to deduplicate the results by checking for duplicated ref.id – that's a couple of lines of code using a dict.

            Is it worth doing this quickly as a separate ticket and reserving this ticket for understanding why the query itself is producing duplicates?

            Show
            tjenness Tim Jenness added a comment - Since we know that all the DatasetRefs being returned must be coming from the same butler repository, this means that the ref.id must be unique. A short term fix is therefore to deduplicate the results by checking for duplicated ref.id – that's a couple of lines of code using a dict . Is it worth doing this quickly as a separate ticket and reserving this ticket for understanding why the query itself is producing duplicates?
            Hide
            jbosch Jim Bosch added a comment -

            Is it worth doing this quickly as a separate ticket and reserving this ticket for understanding why the query itself is producing duplicates?

            That sounds reasonable, though I may still not get to it super quickly.  Others are welcome (Arun Kannawadi expressed in another daf_butler issue that turns out to be trickier than this one; maybe he'd be interested?)

            Show
            jbosch Jim Bosch added a comment - Is it worth doing this quickly as a separate ticket and reserving this ticket for understanding why the query itself is producing duplicates? That sounds reasonable, though I may still not get to it super quickly.  Others are welcome ( Arun Kannawadi expressed in another daf_butler issue that turns out to be trickier than this one; maybe he'd be interested?)
            Hide
            kannawad Arun Kannawadi added a comment -

            Yes, I can work on it to remove the duplicates for now. Should I create another ticket? And is this a Gen3 butler?

            Show
            kannawad Arun Kannawadi added a comment - Yes, I can work on it to remove the duplicates for now. Should I create another ticket? And is this a Gen3 butler?
            Hide
            tjenness Tim Jenness added a comment -

            This is gen3. Yes, I think a new ticket.

            Show
            tjenness Tim Jenness added a comment - This is gen3. Yes, I think a new ticket.
            tjenness Tim Jenness made changes -
            Link This issue is duplicated by DM-21448 [ DM-21448 ]
            tjenness Tim Jenness made changes -
            Link This issue is duplicated by DM-21448 [ DM-21448 ]
            tjenness Tim Jenness made changes -
            Link This issue duplicates DM-21448 [ DM-21448 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-21448 [ DM-21448 ]
            tjenness Tim Jenness made changes -
            Link This issue duplicates DM-21448 [ DM-21448 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-22286 [ DM-22286 ]
            Hide
            tjenness Tim Jenness added a comment -

            DM-22286 was not closed quickly. Is this ticket still an issue?

            Show
            tjenness Tim Jenness added a comment - DM-22286 was not closed quickly. Is this ticket still an issue?
            Hide
            jbosch Jim Bosch added a comment -

            Closing this along with DM-22286 as Won't Fix.  DM-24938 will provide a way to get unique results, but I'm not making it the default.

            Show
            jbosch Jim Bosch added a comment - Closing this along with DM-22286 as Won't Fix.  DM-24938 will provide a way to get unique results, but I'm not making it the default.
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Won't Fix [ 10405 ]

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Arun Kannawadi, Jim Bosch, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel