Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4167

create a singleFileMapper and make it work with a CmdLineTask

    XMLWordPrintable

Details

    • Story
    • Status: Won't Fix
    • Resolution: Done
    • None
    • None
    • Data Access and Database

    Description

      Be able to write a command line task that has a single image as an input (via the dataId most likely), do something to that image, and then have 2 or more outputs. e.g. produce an output image and an afw table. This should execute from the command line with 2 variations:
      1. provide only the name of the input image (and have defaults for where the outputs would go)
      2. provide path for each output to go to
      The task needs to support running with a posix/sqlite registry as well as the above mechanism. (Will probably need to create a new mapper something like "singleFileMapper" and need a way to specify this. having a _mapper file is not ideal, would like a rule e.g. "if no _mapper is found, try the singleFileMapper, or maybe create an command line argument to the pipe_base argument parser to indicate the singleFileMapper).

      Attachments

        Issue Links

          Activity

            KT says: there's prior work for this in a thing obs_file, but it didn't work that well.
            This is an example of an approach that was taken. Dev (NP, me) should go look at it, parse it for whats good, what's bad, and figure out an approach from there.

            processFile is a CmdLineTasks that works only in single-file mode.
            https://github.com/lsst-dm/processFile

            npease Nate Pease [X] (Inactive) added a comment - KT says: there's prior work for this in a thing obs_file, but it didn't work that well. This is an example of an approach that was taken. Dev (NP, me) should go look at it, parse it for whats good, what's bad, and figure out an approach from there. processFile is a CmdLineTasks that works only in single-file mode. https://github.com/lsst-dm/processFile
            tjenness Tim Jenness added a comment -

            jbosch do we have a plan for "throw random file at task" in gen3 land? It's clearly a useful concept in the "I have a processed image with a WCS from random telescope, please run source extraction" use case but are we going to require ingest into butler before anything works?

            tjenness Tim Jenness added a comment - jbosch do we have a plan for "throw random file at task" in gen3 land? It's clearly a useful concept in the "I have a processed image with a WCS from random telescope, please run source extraction" use case but are we going to require ingest into butler before anything works?
            jbosch Jim Bosch added a comment -

            I would like the answer to not involve middleware beyond Task and Config at all - I don't think we gain anything from trying to use Butler or PipelineTask in this use case.

            jbosch Jim Bosch added a comment - I would like the answer to not involve middleware beyond Task and Config at all - I don't think we gain anything from trying to use Butler or PipelineTask in this use case.
            tjenness Tim Jenness added a comment -

            I think we could get something to work where you have to specify the Formatter for every input file and the datasetType in a config (maybe using file extensions to do that mapping in the simplest cases).

            tjenness Tim Jenness added a comment - I think we could get something to work where you have to specify the Formatter for every input file and the datasetType in a config (maybe using file extensions to do that mapping in the simplest cases).
            jbosch Jim Bosch added a comment -

            We could, but I don't see what you gain from butler/PipelineTask relative to a 20-line custom argparse script.

            I think historically the push for this has come from people who don't have a detailed idea of what such a script should do, and fail to notice that e.g.:

             - if you want to do ISR you need calibration lookup

             - if you want to do astrometric solving or forced photometry you need to load a reference catalog

             - if you want to use any of our tasks without starting from ISR you probably need to pre-process your image so it has the expected mask planes, a variance plane, maybe one of our PSF models...

            In other words, there is maybe a role for a generic deriver script that does essentially nothing more than SExtractor using DM code (and hence is in no way representative of what DM code can do), and that's the 20 or so lines or argparse I mentioned above.  Beyond that, I am totally happy telling people to either buy into our middleware wholesale and write an obs _* package, or write a custom Python script that uses our Tasks (but not through the PipelineTask interface).

             

            jbosch Jim Bosch added a comment - We could, but I don't see what you gain from butler/PipelineTask relative to a 20-line custom argparse script. I think historically the push for this has come from people who don't have a detailed idea of what such a script should do, and fail to notice that e.g.:  - if you want to do ISR you need calibration lookup  - if you want to do astrometric solving or forced photometry you need to load a reference catalog  - if you want to use any of our tasks  without starting from ISR you probably need to pre-process your image so it has the expected mask planes, a variance plane, maybe one of our PSF models... In other words, there is maybe a role for a generic deriver script that does essentially nothing more than SExtractor using DM code (and hence is in no way representative of what DM code can do), and that's the 20 or so lines or argparse I mentioned above.  Beyond that, I am totally happy telling people to either buy into our middleware wholesale and write an obs _* package, or write a custom Python script that uses our Tasks (but not through the PipelineTask interface).  
            tjenness Tim Jenness added a comment -

            I think it's clear from jbosch's comment that we are not sure how this fits into gen3 and for now I am closing this as won't fix. It can be re-opened (or a new ticket created) if we have a better idea on how to handle this and motivation.

            tjenness Tim Jenness added a comment - I think it's clear from jbosch 's comment that we are not sure how this fits into gen3 and for now I am closing this as won't fix. It can be re-opened (or a new ticket created) if we have a better idea on how to handle this and motivation.

            People

              Unassigned Unassigned
              npease Nate Pease [X] (Inactive)
              Andy Salnikov, Jim Bosch, Nate Pease [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.