Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:6
-
Epic Link:
-
Sprint:DRP S21a (Dec Jan), DRP S21b, AP S22-1 (December)
-
Team:Alert Production
-
Urgent?:No
Description
Almost got this going on DM-24638, but couldn't get a gen3 DECam test repo set up in time.
end goal is to move
+subsets:
|
+ singleFrame:
|
+ subset:
|
+ - isr
|
+ - characterizeImage
|
+ - calibrate
|
+ - writeSourceTable
|
+ - transformSourceTable
|
+ - consolidateSourceTable
|
+ description: Set of tasks for complete single frame processing. Analogous to SingleFrameDriver.
|
+ sourceTable:
|
+ subset:
|
+ - writeSourceTable
|
+ - transformSourceTable
|
+ - consolidateSourceTable
|
+ description: Set of tasks to generate parquet Source Tables from output of processCcd subset.
|
Need a generic Source.yaml in obs_base or maybe Nate's snazzy new config field type.
Attachments
Issue Links
- blocks
-
- Done
-
- Done
- relates to
-
- To Do
-
- Done
-
- Done
-
- Adopted
Activity
I'm taking this over so we can get it merged. Most of the work should be done already, I just need to do some final testing.
Jenkins with ci_imsim ci_hsc ci_cpp_gen3: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35603/pipeline
Lee Kelvin: do you mind reviewing this? I think you were the last one to touch/create a source table yaml file (for DECam), which I've now deleted on this ticket. In particular, do the docstrings I've added-in a separate commit in pipe_tasks-make sense, and are they correct?
Do you have a DECam pipeline you could run with this ticket branch of obs_decam and pipe_tasks, to confirm that it's working? I think I checked that your new DECam files matched the ones Yusra created here from obs_subaru, but it'd be nice to have a run of something to confirm it.
PRs:
Thanks John (and Yusra), this is a great ticket, bringing all of the schema tables in under one roof. It should certainly help with catalogue consistency between surveys going forward.
I've ran through my test DECam data from the Merian survey using the pipe_tasks and obs_decam branches of this ticket (tickets/DM-28862), and all seems to work well. I attach the step 1 (single-frame), step 2 (post-single-frame), step 3 (coadd) and step 7 (post-run) processing logs to this ticket, for reference.
I haven't had the chance to test this on any HSC or ImSim data - I can't imagine anything would break, but it might be worth pushing through some (e.g.) HSC data for a few visits to double check that nothing there breaks either?
Other than that, I've left some comments on the GH PR for pipe_tasks regarding doc strings and a few typos. Some of these I likely propagated on my prior obs_decam ticket, so thanks for looking at this! I saw that the MacOS Jenkins run passed successfully, but not the CentOS run. Was this due to the unrelated Jenkins issues we were having on or around mid-December? If so, all seems well here, and this looks good to merge to me - thanks!
I ran ci_hsc and ci_imsim in Jenkins, which both use these files, so I think we're ok for those packages.
I didn't do any proofreading of the yaml files, I just copied them and checked that they matched the versions in each obs package, including DECam. So, your proofreading of them is useful!
Post review/rebase Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35624/pipeline
I now have a place to put the schema files drp_pipe/schemas and a gen3 decam repo to test with. Onward!