Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33621

Support reading dataIds from external file

    XMLWordPrintable

    Details

    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Basically wanting Gen2 @file syntax. Tim Jenness said this "was on the todo list", now there's a ticket for it and everything.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30438 Add support for uploading data IDs to temporary tables and vectorize data ID expansion

          • To Do
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31725 Rewrite butler registry.queries subpackage

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33764 Write technote describing middleware functionality for "campaign management"

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Hmm. Section 4 of https://dmtn-181.lsst.io/v/u-ktl-initial-draft/index.html has the requirement – Kian-Tat Lim what's stopping us merging this to main? Maybe we decided that it superseded by https://rtn-023.lsst.io ?

            Show
            tjenness Tim Jenness added a comment - Hmm. Section 4 of https://dmtn-181.lsst.io/v/u-ktl-initial-draft/index.html has the requirement – Kian-Tat Lim what's stopping us merging this to main? Maybe we decided that it superseded by https://rtn-023.lsst.io ?
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            And also I have no idea what @file syntax is.

            I was envisaging a CSV with dataId values (or even serialized JSON DatasetRef) in it or else a text file with UUIDs in.

            Show
            tjenness Tim Jenness added a comment - And also I have no idea what @file syntax is. I was envisaging a CSV with dataId values (or even serialized JSON DatasetRef) in it or else a text file with UUIDs in.
            Hide
            Permalink
            mfisherlevine Merlin Fisher-Levine added a comment -

            In Gen2 the way one specified a file with a list of dataIds on the command line was with the @ symbol and the path to the file, that's all.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - In Gen2 the way one specified a file with a list of dataIds on the command line was with the @ symbol and the path to the file, that's all.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            What was the format of that file? How did you tell it which datasetTypes were relevant?

            Show
            tjenness Tim Jenness added a comment - What was the format of that file? How did you tell it which datasetTypes were relevant?
            Hide
            Permalink
            mfisherlevine Merlin Fisher-Levine added a comment -

            Well, in Gen2 land it was just a matter of passing dataIds to your task, as that would do a runDataRef() calling get() with the types it needed, using the supplied ids, so an example for processCcd just looked like

            --id dayObs=2020-03-12 seqNum=375
            		 
            --id dayObs=2020-03-12 seqNum=376
            		 
            --id dayObs=2020-03-12 seqNum=377

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Well, in Gen2 land it was just a matter of passing dataIds to your task, as that would do a runDataRef() calling get() with the types it needed, using the supplied ids, so an example for processCcd just looked like --id dayObs=2020-03-12 seqNum=375 --id dayObs=2020-03-12 seqNum=376 --id dayObs=2020-03-12 seqNum=377
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Linking DM-31725, where I had planned to do this (among many other things), and will at least do some prototyping around the problems involved with implementing it.

            Show
            jbosch Jim Bosch added a comment - Linking DM-31725 , where I had planned to do this (among many other things), and will at least do some prototyping around the problems involved with implementing it.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Jim Bosch The linked ticket is closed but I don't think we've implemented the ability to read in a file of dataIds/UUIDs in pipetask qgraph.

            Show
            tjenness Tim Jenness added a comment - Jim Bosch The linked ticket is closed but I don't think we've implemented the ability to read in a file of dataIds/UUIDs in pipetask qgraph.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Yes, this can stick around (as can DM-30438).  DM-31725 took a big step in the right direction but didn't get us all the way there.

            Show
            jbosch Jim Bosch added a comment - Yes, this can stick around (as can DM-30438 ).  DM-31725 took a big step in the right direction but didn't get us all the way there.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              mfisherlevine Merlin Fisher-Levine
              Watchers:
              Jim Bosch, Merlin Fisher-Levine, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.