As part of
RFC-57 ( DM-1674) the ingestion code was removed from datarel and placed in daf_ingest. daf_ingest was never added back into lsst_distrib. This RFC proposes that we put it back. I am assuming this is not controversial as it's nominally a rebranding of something that was already in lsst_distrib.
The one wrinkle is that daf_ingest does not build with Python 3 (see
I hope to get to
DM-5766 soon. It would be good to get it done while Yusra AlSayyad is also playing with the warping. DM-5766 adds very useful functionality I'd love to have in the stack soon. I imagine implementing that will require adding daf_ingest as a dependency of pipe_tasks, which would bring it in as a dependency of lsst_apps, not just datarel. It looks like that will bring in sphgeom as well, which I don't anticipate adding any problems.
The actual ingest-processing-to-SQL-databases code has probably bitrotted quite a bit and I don't think it has ever been fully integrated with the rest of the stack (at least not for modern pipeline outputs; it's possible it was used to ingest the old Stripe 82 processing into the PDAC database). I don't think we need that code right now (at least in Science Pipelines), but it's a very important piece of the puzzle in integrating Science Pipelines with DAX/SUIT in future incarnations of the PDAC.
I don't see a clear gain or harm from adding daf_ingest to datarel earlier than
DM-5766 (it's just unnecessary work vs. letting part of RFC-57 linger).
Making pipe_tasks depend on daf_ingest because the former is actively using functionality provided by the latter certainly sits better with me than adding the dependency to _distrib on the basis that it might be useful someday.
I'd prefer not to be adding dependencies which pull in the likely-bitrotted table-ingest code, but I don't think it's worth either trying to veto the dependency or to split daf_ingest into multiple packages on that basis.
I will withdraw this RFC on the basis that
DM-5766 will pull daf_ingest into lsst_apps when the time is right and there is no need in the short term to add it to lsst_distrib just because the code from datarel used to be in lsst_distrib.
I do not believe that
DM-5766 will require an RFC to bring in daf_ingest based on the discussion in this RFC. If you feel that it does require one we can reopen this RFC at the time.
I think https://community.lsst.org/t/unified-exposure-metadata/1214 is also relevant, as far as I know daf_ingest is not ingesting the exposure metadata and a unified metadata as implemented in
DM-5503was a requirement for that.