Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23074

Make the schema of the output Object parquet files input-independent

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_subaru, pipe_tasks
    • Labels:
      None
    • Templates:
    • Story Points:
      4
    • Team:
      DM Science

      Description

      Currently the schema of the Object table in the output parquet files depend on what filters are included in the input dataset. For example the Object table in a parquet file from the ci_hsc dataset has fewer columns than one from the HSC-RC2 dataset, because the former contains 2 filters and the latter contains 5. It is not ideal that the schema is not fixed. We would like the output schema to be deterministic and independent of the input dataset for the same camera (or obs package). Perhaps a config parameter in the obs package would be suitable.

      It's okay to only consider obs_subaru for now.

        Attachments

          Activity

            People

            • Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Colin Slater, Hsin-Fang Chiang, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel