Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-275

Add daf_ingest to lsst_distrib

    XMLWordPrintable

Details

    • RFC
    • Status: Withdrawn
    • Resolution: Done
    • DM
    • None
    • This ticket.

    Description

      As part of RFC-57 (DM-1674) the ingestion code was removed from datarel and placed in daf_ingest. daf_ingest was never added back into lsst_distrib. This RFC proposes that we put it back. I am assuming this is not controversial as it's nominally a rebranding of something that was already in lsst_distrib.

      The one wrinkle is that daf_ingest does not build with Python 3 (see DM-8268).

      Attachments

        Issue Links

          Activity

            I think https://community.lsst.org/t/unified-exposure-metadata/1214 is also relevant, as far as I know daf_ingest is not ingesting the exposure metadata and a unified metadata as implemented in DM-5503 was a requirement for that.

            afausti Angelo Fausti added a comment - I think https://community.lsst.org/t/unified-exposure-metadata/1214 is also relevant, as far as I know daf_ingest is not ingesting the exposure metadata and a unified metadata as implemented in DM-5503 was a requirement for that.
            price Paul Price added a comment -

            I hope to get to DM-5766 soon. It would be good to get it done while yusra is also playing with the warping.

            price Paul Price added a comment - I hope to get to DM-5766 soon. It would be good to get it done while yusra is also playing with the warping.
            jbosch Jim Bosch added a comment - - edited

            DM-5766 adds very useful functionality I'd love to have in the stack soon. I imagine implementing that will require adding daf_ingest as a dependency of pipe_tasks, which would bring it in as a dependency of lsst_apps, not just datarel. It looks like that will bring in sphgeom as well, which I don't anticipate adding any problems.

            The actual ingest-processing-to-SQL-databases code has probably bitrotted quite a bit and I don't think it has ever been fully integrated with the rest of the stack (at least not for modern pipeline outputs; it's possible it was used to ingest the old Stripe 82 processing into the PDAC database). I don't think we need that code right now (at least in Science Pipelines), but it's a very important piece of the puzzle in integrating Science Pipelines with DAX/SUIT in future incarnations of the PDAC.

            I don't see a clear gain or harm from adding daf_ingest to datarel earlier than DM-5766 (it's just unnecessary work vs. letting part of RFC-57 linger).

            jbosch Jim Bosch added a comment - - edited DM-5766 adds very useful functionality I'd love to have in the stack soon. I imagine implementing that will require adding daf_ingest as a dependency of pipe_tasks , which would bring it in as a dependency of lsst_apps , not just datarel . It looks like that will bring in sphgeom as well, which I don't anticipate adding any problems. The actual ingest-processing-to-SQL-databases code has probably bitrotted quite a bit and I don't think it has ever been fully integrated with the rest of the stack (at least not for modern pipeline outputs; it's possible it was used to ingest the old Stripe 82 processing into the PDAC database). I don't think we need that code right now (at least in Science Pipelines), but it's a very important piece of the puzzle in integrating Science Pipelines with DAX/SUIT in future incarnations of the PDAC. I don't see a clear gain or harm from adding daf_ingest to datarel earlier than DM-5766 (it's just unnecessary work vs. letting part of RFC-57 linger).

            Making pipe_tasks depend on daf_ingest because the former is actively using functionality provided by the latter certainly sits better with me than adding the dependency to _distrib on the basis that it might be useful someday.

            I'd prefer not to be adding dependencies which pull in the likely-bitrotted table-ingest code, but I don't think it's worth either trying to veto the dependency or to split daf_ingest into multiple packages on that basis.

            swinbank John Swinbank added a comment - Making pipe_tasks depend on daf_ingest because the former is actively using functionality provided by the latter certainly sits better with me than adding the dependency to _distrib on the basis that it might be useful someday. I'd prefer not to be adding dependencies which pull in the likely-bitrotted table-ingest code, but I don't think it's worth either trying to veto the dependency or to split daf_ingest into multiple packages on that basis.
            tjenness Tim Jenness added a comment -

            I will withdraw this RFC on the basis that DM-5766 will pull daf_ingest into lsst_apps when the time is right and there is no need in the short term to add it to lsst_distrib just because the code from datarel used to be in lsst_distrib.

            I do not believe that DM-5766 will require an RFC to bring in daf_ingest based on the discussion in this RFC. If you feel that it does require one we can reopen this RFC at the time.

            tjenness Tim Jenness added a comment - I will withdraw this RFC on the basis that DM-5766 will pull daf_ingest into lsst_apps when the time is right and there is no need in the short term to add it to lsst_distrib just because the code from datarel used to be in lsst_distrib . I do not believe that DM-5766 will require an RFC to bring in daf_ingest based on the discussion in this RFC. If you feel that it does require one we can reopen this RFC at the time.

            People

              tjenness Tim Jenness
              tjenness Tim Jenness
              Angelo Fausti, Fritz Mueller, Hsin-Fang Chiang, Jim Bosch, John Swinbank, Kian-Tat Lim, Paul Price, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                Jenkins

                  No builds found.