Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15521

Run jointcal with HSC-RC2 and w_2018_34

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Run jointcal with the HSC-RC2 dataset, the w_2018_34 stack, and the default jointcal config.

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            With the purely default config for HSC data, I got errors in reference catalog: 

            lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'cal_ref_cat'}, tag=set())
            

            So, I'm adding config overrides as follows:

            config.astrometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
            config.photometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
            config.astrometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'}
            config.photometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'}
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - With the purely default config for HSC data, I got errors in reference catalog:  lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'cal_ref_cat'}, tag=set()) So, I'm adding config overrides as follows: config.astrometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110' config.photometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110' config.astrometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'} config.photometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'}
            Hide
            Parejkoj John Parejko added a comment -

            Oh yes, you'll definitely have to configure the reference object loaders. Using the same config as for single frame processing is probably correct, assuming that's what you did above.

            Show
            Parejkoj John Parejko added a comment - Oh yes, you'll definitely have to configure the reference object loaders. Using the same config as for single frame processing is probably correct, assuming that's what you did above.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I copied the configs from what you gave me before. For the filters in the HSC-RC2 data, they are the same as those in the single frame processing.

            Why don't we include these config overrides in obs_subaru or some other places?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I copied the configs from what you gave me before. For the filters in the HSC-RC2 data, they are the same as those in the single frame processing. Why don't we include these config overrides in obs_subaru or some other places?
            Hide
            Parejkoj John Parejko added a comment -

            That's a good question: I don't know what the policy on config overrides is, but it probably depends on what we assume the default reference catalogs to be: the dataset_name and filterMap are going to be different for PS1 vs. Gaia, for example.

            Show
            Parejkoj John Parejko added a comment - That's a good question: I don't know what the policy on config overrides is, but it probably depends on what we assume the default reference catalogs to be: the dataset_name and filterMap are going to be different for PS1 vs. Gaia, for example.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            As I'm just starting to include jointcal this week, I think you and the Science Pipelines team can decide the best configs and change the default in the Stack in the short term without problems from my side. I don't know if there are policies internal to the Science Pipelines team; maybe Yusra AlSayyad would know?

            Anyway, w_2018_34's jointcal results is now available at this separate repo:
            /datasets/hsc/repo/rerun/RC/w_2018_34/DM-15517-jointcal/
            and the log files are at /datasets/hsc/repo/rerun/RC/w_2018_34/DM-15517/logs/jointcal.
            I'll include jointcal for the future HSC-RC2 reprocessing too.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - As I'm just starting to include jointcal this week, I think you and the Science Pipelines team can decide the best configs and change the default in the Stack in the short term without problems from my side. I don't know if there are policies internal to the Science Pipelines team; maybe Yusra AlSayyad would know? Anyway, w_2018_34 's jointcal results is now available at this separate repo: /datasets/hsc/repo/rerun/RC/w_2018_34/ DM-15517 -jointcal/ and the log files are at /datasets/hsc/repo/rerun/RC/w_2018_34/ DM-15517 /logs/jointcal . I'll include jointcal for the future HSC-RC2 reprocessing too.
            Hide
            yusra Yusra AlSayyad added a comment -

            Hsin-Fang Chiang, our policy is to minimize the config overrides you need in order to run a DRP (and be internally consistent) so we will add these these to at least obs_subaru before w_2018_36.

            Show
            yusra Yusra AlSayyad added a comment - Hsin-Fang Chiang , our policy is to minimize the config overrides you need in order to run a DRP (and be internally consistent) so we will add these these to at least obs_subaru before w_2018_36.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Sounds great to me and +1 for adding these to obs_subaru. Thanks!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Sounds great to me and +1 for adding these to obs_subaru. Thanks!

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, John Parejko, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.