Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15521

Run jointcal with HSC-RC2 and w_2018_34

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      Run jointcal with the HSC-RC2 dataset, the w_2018_34 stack, and the default jointcal config.

      Attachments

        Issue Links

          Activity

            With the purely default config for HSC data, I got errors in reference catalog: 

            lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'cal_ref_cat'}, tag=set())
            

            So, I'm adding config overrides as follows:

            config.astrometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
            config.photometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
            config.astrometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'}
            config.photometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'}
            

            hchiang2 Hsin-Fang Chiang added a comment - With the purely default config for HSC data, I got errors in reference catalog:  lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'cal_ref_cat'}, tag=set()) So, I'm adding config overrides as follows: config.astrometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110' config.photometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110' config.astrometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'} config.photometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'}
            Parejkoj John Parejko added a comment -

            Oh yes, you'll definitely have to configure the reference object loaders. Using the same config as for single frame processing is probably correct, assuming that's what you did above.

            Parejkoj John Parejko added a comment - Oh yes, you'll definitely have to configure the reference object loaders. Using the same config as for single frame processing is probably correct, assuming that's what you did above.

            I copied the configs from what you gave me before. For the filters in the HSC-RC2 data, they are the same as those in the single frame processing.

            Why don't we include these config overrides in obs_subaru or some other places?

            hchiang2 Hsin-Fang Chiang added a comment - I copied the configs from what you gave me before. For the filters in the HSC-RC2 data, they are the same as those in the single frame processing. Why don't we include these config overrides in obs_subaru or some other places?
            Parejkoj John Parejko added a comment -

            That's a good question: I don't know what the policy on config overrides is, but it probably depends on what we assume the default reference catalogs to be: the dataset_name and filterMap are going to be different for PS1 vs. Gaia, for example.

            Parejkoj John Parejko added a comment - That's a good question: I don't know what the policy on config overrides is, but it probably depends on what we assume the default reference catalogs to be: the dataset_name and filterMap are going to be different for PS1 vs. Gaia, for example.

            As I'm just starting to include jointcal this week, I think you and the Science Pipelines team can decide the best configs and change the default in the Stack in the short term without problems from my side. I don't know if there are policies internal to the Science Pipelines team; maybe yusra would know?

            Anyway, w_2018_34's jointcal results is now available at this separate repo:
            /datasets/hsc/repo/rerun/RC/w_2018_34/DM-15517-jointcal/
            and the log files are at /datasets/hsc/repo/rerun/RC/w_2018_34/DM-15517/logs/jointcal.
            I'll include jointcal for the future HSC-RC2 reprocessing too.

            hchiang2 Hsin-Fang Chiang added a comment - As I'm just starting to include jointcal this week, I think you and the Science Pipelines team can decide the best configs and change the default in the Stack in the short term without problems from my side. I don't know if there are policies internal to the Science Pipelines team; maybe yusra would know? Anyway, w_2018_34 's jointcal results is now available at this separate repo: /datasets/hsc/repo/rerun/RC/w_2018_34/ DM-15517 -jointcal/ and the log files are at /datasets/hsc/repo/rerun/RC/w_2018_34/ DM-15517 /logs/jointcal . I'll include jointcal for the future HSC-RC2 reprocessing too.

            hchiang2, our policy is to minimize the config overrides you need in order to run a DRP (and be internally consistent) so we will add these these to at least obs_subaru before w_2018_36.

            yusra Yusra AlSayyad added a comment - hchiang2 , our policy is to minimize the config overrides you need in order to run a DRP (and be internally consistent) so we will add these these to at least obs_subaru before w_2018_36.

            Sounds great to me and +1 for adding these to obs_subaru. Thanks!

            hchiang2 Hsin-Fang Chiang added a comment - Sounds great to me and +1 for adding these to obs_subaru. Thanks!

            People

              hchiang2 Hsin-Fang Chiang
              hchiang2 Hsin-Fang Chiang
              Hsin-Fang Chiang, John Parejko, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.