Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-275

Add daf_ingest to lsst_distrib

    Details

    • Type: RFC
    • Status: Withdrawn
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None
    • Location:
      This ticket.

      Description

      As part of RFC-57 (DM-1674) the ingestion code was removed from datarel and placed in daf_ingest. daf_ingest was never added back into lsst_distrib. This RFC proposes that we put it back. I am assuming this is not controversial as it's nominally a rebranding of something that was already in lsst_distrib.

      The one wrinkle is that daf_ingest does not build with Python 3 (see DM-8268).

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment -

            I think https://community.lsst.org/t/unified-exposure-metadata/1214 is also relevant, as far as I know daf_ingest is not ingesting the exposure metadata and a unified metadata as implemented in DM-5503 was a requirement for that.

            Show
            afausti Angelo Fausti added a comment - I think https://community.lsst.org/t/unified-exposure-metadata/1214 is also relevant, as far as I know daf_ingest is not ingesting the exposure metadata and a unified metadata as implemented in DM-5503 was a requirement for that.
            Hide
            price Paul Price added a comment -

            I hope to get to DM-5766 soon. It would be good to get it done while Yusra AlSayyad is also playing with the warping.

            Show
            price Paul Price added a comment - I hope to get to DM-5766 soon. It would be good to get it done while Yusra AlSayyad is also playing with the warping.
            Hide
            jbosch Jim Bosch added a comment - - edited

            DM-5766 adds very useful functionality I'd love to have in the stack soon. I imagine implementing that will require adding daf_ingest as a dependency of pipe_tasks, which would bring it in as a dependency of lsst_apps, not just datarel. It looks like that will bring in sphgeom as well, which I don't anticipate adding any problems.

            The actual ingest-processing-to-SQL-databases code has probably bitrotted quite a bit and I don't think it has ever been fully integrated with the rest of the stack (at least not for modern pipeline outputs; it's possible it was used to ingest the old Stripe 82 processing into the PDAC database). I don't think we need that code right now (at least in Science Pipelines), but it's a very important piece of the puzzle in integrating Science Pipelines with DAX/SUIT in future incarnations of the PDAC.

            I don't see a clear gain or harm from adding daf_ingest to datarel earlier than DM-5766 (it's just unnecessary work vs. letting part of RFC-57 linger).

            Show
            jbosch Jim Bosch added a comment - - edited DM-5766 adds very useful functionality I'd love to have in the stack soon. I imagine implementing that will require adding daf_ingest as a dependency of pipe_tasks , which would bring it in as a dependency of lsst_apps , not just datarel . It looks like that will bring in sphgeom as well, which I don't anticipate adding any problems. The actual ingest-processing-to-SQL-databases code has probably bitrotted quite a bit and I don't think it has ever been fully integrated with the rest of the stack (at least not for modern pipeline outputs; it's possible it was used to ingest the old Stripe 82 processing into the PDAC database). I don't think we need that code right now (at least in Science Pipelines), but it's a very important piece of the puzzle in integrating Science Pipelines with DAX/SUIT in future incarnations of the PDAC. I don't see a clear gain or harm from adding daf_ingest to datarel earlier than DM-5766 (it's just unnecessary work vs. letting part of RFC-57 linger).
            Hide
            swinbank John Swinbank added a comment -

            Making pipe_tasks depend on daf_ingest because the former is actively using functionality provided by the latter certainly sits better with me than adding the dependency to _distrib on the basis that it might be useful someday.

            I'd prefer not to be adding dependencies which pull in the likely-bitrotted table-ingest code, but I don't think it's worth either trying to veto the dependency or to split daf_ingest into multiple packages on that basis.

            Show
            swinbank John Swinbank added a comment - Making pipe_tasks depend on daf_ingest because the former is actively using functionality provided by the latter certainly sits better with me than adding the dependency to _distrib on the basis that it might be useful someday. I'd prefer not to be adding dependencies which pull in the likely-bitrotted table-ingest code, but I don't think it's worth either trying to veto the dependency or to split daf_ingest into multiple packages on that basis.
            Hide
            tjenness Tim Jenness added a comment -

            I will withdraw this RFC on the basis that DM-5766 will pull daf_ingest into lsst_apps when the time is right and there is no need in the short term to add it to lsst_distrib just because the code from datarel used to be in lsst_distrib.

            I do not believe that DM-5766 will require an RFC to bring in daf_ingest based on the discussion in this RFC. If you feel that it does require one we can reopen this RFC at the time.

            Show
            tjenness Tim Jenness added a comment - I will withdraw this RFC on the basis that DM-5766 will pull daf_ingest into lsst_apps when the time is right and there is no need in the short term to add it to lsst_distrib just because the code from datarel used to be in lsst_distrib . I do not believe that DM-5766 will require an RFC to bring in daf_ingest based on the discussion in this RFC. If you feel that it does require one we can reopen this RFC at the time.

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                tjenness Tim Jenness
                Watchers:
                Angelo Fausti, Fritz Mueller, Hsin-Fang Chiang, Jim Bosch, John Swinbank, Kian-Tat Lim, Paul Price, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel