Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-83

Rename pipe_tasks' ingest.py

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Retired
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None
    • Location:
      Jira

      Description

      The ingestImages.py bin script provides a camera-agnostic manner of creating a data repository (including a registry). The back-end code resides in pipe_tasks under the name ingest.py, and the IngestTask._DefaultName = "ingest", which means that configuration files in obs packages are also named ingest.py. This choice of name was unfortunate, as it may be confused with ingest of sources into the database. DM-3439 proposes changing ingest.py to ingestImages.py to reduce confusion. Suggestions for better names are also welcome.

      This will require coordinated changes to pipe_tasks, obs_subaru, obs_cfht and obs_decam.

        Attachments

          Issue Links

            Activity

            Hide
            rhl Robert Lupton added a comment -

            I'd avoid the term ingest entirely — registerImages?

            Show
            rhl Robert Lupton added a comment - I'd avoid the term ingest entirely — registerImages?
            Hide
            rowen Russell Owen added a comment -

            Your suggestion sounds like a big improvement. Nonetheless, I'll ask for even more: I think "ingestImages.py" is too ambiguous: it sounds as if images are being ingested to a database, or at least added to an existing collection of some kind. So I will request a clearer and less ambiguous name.

            I suggest "makeRepoFromImages". It is rather long, but quite clear and unambiguous. If the operation is sufficiently rare then the long name will not hurt, and auto-completion will work once one types "makeRepo".

            "repoFromImages" might do, but I miss the verb. I do not like "imagesToRepo" because it sounds as if it might be adding images to an existing repo.

            Show
            rowen Russell Owen added a comment - Your suggestion sounds like a big improvement. Nonetheless, I'll ask for even more: I think "ingestImages.py" is too ambiguous: it sounds as if images are being ingested to a database, or at least added to an existing collection of some kind. So I will request a clearer and less ambiguous name. I suggest "makeRepoFromImages". It is rather long, but quite clear and unambiguous. If the operation is sufficiently rare then the long name will not hurt, and auto-completion will work once one types "makeRepo". "repoFromImages" might do, but I miss the verb. I do not like "imagesToRepo" because it sounds as if it might be adding images to an existing repo.
            Hide
            price Paul Price added a comment -

            I would like to avoid the word "ingest", except that's what it does. The IngestTask has multiple functions: parsing an image, putting it in the data repo, and registering it in the registry. Registering is only a part of what it does, so it doesn't seem appropriate to call the whole "registerImages". I was therefore thinking to qualify the kind of ingestion.

            Show
            price Paul Price added a comment - I would like to avoid the word "ingest", except that's what it does. The IngestTask has multiple functions: parsing an image, putting it in the data repo, and registering it in the registry. Registering is only a part of what it does, so it doesn't seem appropriate to call the whole "registerImages". I was therefore thinking to qualify the kind of ingestion.
            Hide
            rowen Russell Owen added a comment -

            I saw Robert's suggestion after I posted mine. I agree that "registerImages" is better than "ingestImages" but I worry that it could easily be mistaken for warping.

            Show
            rowen Russell Owen added a comment - I saw Robert's suggestion after I posted mine. I agree that "registerImages" is better than "ingestImages" but I worry that it could easily be mistaken for warping.
            Hide
            jbosch Jim Bosch added a comment -

            How about ingestToRegistry.py and RegistryIngestTask?

            Show
            jbosch Jim Bosch added a comment - How about ingestToRegistry.py and RegistryIngestTask ?
            Hide
            rowen Russell Owen added a comment -

            If the bin script creates a new registry then I strongly feel the name should reflect that. ingestToRegistry implies adding images to an existing registry.

            If the binary code is more flexible, then name it accordingly. It sounds to me as if potentially 3 things could happen in the library code:

            • Create a new repository (which if done in place presumably just means adding a database and a bit of metadata to an existing dir)
            • Copy images or create symlinks to images. If creating a new repo, presumably that is done "in place" and so this step is not done, but it would be wanted if adding images to an existing repo.
            • Ingest data about the images to the repo db
            Show
            rowen Russell Owen added a comment - If the bin script creates a new registry then I strongly feel the name should reflect that. ingestToRegistry implies adding images to an existing registry. If the binary code is more flexible, then name it accordingly. It sounds to me as if potentially 3 things could happen in the library code: Create a new repository (which if done in place presumably just means adding a database and a bit of metadata to an existing dir) Copy images or create symlinks to images. If creating a new repo, presumably that is done "in place" and so this step is not done, but it would be wanted if adding images to an existing repo. Ingest data about the images to the repo db
            Hide
            price Paul Price added a comment -

            Thank you all for your input.

            The script creates a new registry if it is not present, but it does more. The registry really is an implementation detail of what I've been calling "data repositories", and what the script does is ingest images into the data repository. I therefore propose ingestToDataRepository.py. I think this captures the intent of all your suggestions.

            I will move the end date slightly to allow for objections to this solid proposal.

            Show
            price Paul Price added a comment - Thank you all for your input. The script creates a new registry if it is not present, but it does more. The registry really is an implementation detail of what I've been calling "data repositories", and what the script does is ingest images into the data repository. I therefore propose ingestToDataRepository.py . I think this captures the intent of all your suggestions. I will move the end date slightly to allow for objections to this solid proposal.
            Hide
            price Paul Price added a comment -

            Sorry, forgot to mention what was being ingested. New proposal: ingestImagesToDataRepository.py. Yeah, it's a bit long, but it's descriptive (and shell command completion is your friend).

            Show
            price Paul Price added a comment - Sorry, forgot to mention what was being ingested. New proposal: ingestImagesToDataRepository.py . Yeah, it's a bit long, but it's descriptive (and shell command completion is your friend).
            Hide
            price Paul Price added a comment -

            It shall be ingestImagesToDataRepository.py.

            Show
            price Paul Price added a comment - It shall be ingestImagesToDataRepository.py .
            Hide
            tjenness Tim Jenness added a comment -

            We decided that with gen3 now existing and with the triggered ticket marked invalid, we will not be implementing this RFC.

            Show
            tjenness Tim Jenness added a comment - We decided that with gen3 now existing and with the triggered ticket marked invalid, we will not be implementing this RFC.

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Watchers:
              Hsin-Fang Chiang, Jim Bosch, Paul Price, Robert Lupton, Russell Owen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.