Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22676

Create gaia dr2 refcat for ap_verify_hits2015

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ap_verify
    • None
    • 2
    • AP S20-1 (December)
    • Alert Production

    Description

      I'm using ap_verify_ci_hits2015 to test my gen2->gen3 repo converter code, and I ran into a problem due to the use of the gaia dr1 refcat, which uses htm depth=8. Gen3 repos all have to use depth=7 for the moment. The easiest solution is to just extract the necessary pixels from the new gaia dr2 refcat and replace the old one in ap_verify_hits. That also gains us an upgraded reference catalog.

      This should be an easy change to scripts/gaia_HiTS_2015.py and then a bit of processing on lsst-dev.

      Attachments

        Issue Links

          Activity

            Parejkoj John Parejko added a comment -

            mrawls: I think you're probably in the best place to review this. I suspect it may change some of the measured metrics. Probably the best way to test it is to run through each dataset's pipeline and see if things are still happy?

            Parejkoj John Parejko added a comment - mrawls : I think you're probably in the best place to review this. I suspect it may change some of the measured metrics. Probably the best way to test it is to run through each dataset's pipeline and see if things are still happy?
            Parejkoj John Parejko added a comment -

            Question: the gen3 butler doesn't allow `-` in a dataset name, so the `pan-starrs` refcat in these test data have to be renamed. I would probably recommend calling it either `ps1` (which is the data release it came from) or just `panstarrs` (even though the real name has a hyphen). Should I make that change on this ticket, or should we do it on another one? Would we need to change any configs other than the two in both ap_verify_hits `config/` directories?

            Parejkoj John Parejko added a comment - Question: the gen3 butler doesn't allow `-` in a dataset name, so the `pan-starrs` refcat in these test data have to be renamed. I would probably recommend calling it either `ps1` (which is the data release it came from) or just `panstarrs` (even though the real name has a hyphen). Should I make that change on this ticket, or should we do it on another one? Would we need to change any configs other than the two in both ap_verify_hits `config/` directories?
            ebellm Eric Bellm added a comment -

            Note that there are two PS1 data releases (DR1 and DR2).

            ebellm Eric Bellm added a comment - Note that there are two PS1 data releases (DR1 and DR2).

            A few things.

            The changes in the datasets themselves look fine, and thanks (John) for explaining HTM trixels and "depth" to me via DM.

            I ran ap_verify with the CI dataset on this ticket branch, and it failed with

            apPipe.ccdProcessor.calibrate.astromRefObjLoader INFO: Loading reference objects using center (154.967854, -5.784114) and radius 0.19703074992397315 deg
            apPipe FATAL: Failed on dataId={'date': '2015-02-18', 'filter': 'g', 'visit': 411420, 'hdu': 2, 'ccdnum': 5, 'object': 'Blind15A_40'}: RuntimeError: Unknown reference filter phot_g_mean_mag_flux 

            I think this failure is related to the new refcat and needs to be addressed before this ticket is done.

            It then subsequently failed with

            AttributeError: lsst.pipe.base.config.Connections has no attribute taskName 

            Which, incidentally, I am beginning to think is the new "no locations for get" (in its pervasiveness and opaqueness, not what it actually means). I believe this is the ap_pipe bug krzys caught and fixed earlier this week, so for now, don't worry about it.

            Regarding Pan-STARRS as the photometric refcat. Please specify in both datasets' READMEs that it includes Gaia DR2 and Pan-STARRS PS1 DRX (I don't know if it's DR1 or DR2?), per Eric's comment above. I am quite annoyed that gen3 does not permit hyphens in refcat names, as this is the kind of thing that will create significant overhead for folks who are just trying to do a processCcd rerun and suddenly it doesn't work anymore. That said, assuming it truly is necessary to banish hyphens from our refcats forever, I think it is fine to update the `pan-starrs` name to be hyphen free on this ticket. I think I would vote for pan_starrs or panstarrs (and not, e.g., PS1).

            mrawls Meredith Rawls added a comment - A few things. The changes in the datasets themselves look fine, and thanks (John) for explaining HTM trixels and "depth" to me via DM. I ran ap_verify with the CI dataset on this ticket branch, and it failed with apPipe.ccdProcessor.calibrate.astromRefObjLoader INFO: Loading reference objects using center ( 154.967854 , - 5.784114 ) and radius 0.19703074992397315 deg apPipe FATAL: Failed on dataId={ 'date' : '2015-02-18' , 'filter' : 'g' , 'visit' : 411420 , 'hdu' : 2 , 'ccdnum' : 5 , 'object' : 'Blind15A_40' }: RuntimeError: Unknown reference filter phot_g_mean_mag_flux I think this failure is related to the new refcat and needs to be addressed before this ticket is done. It then subsequently failed with AttributeError: lsst.pipe.base.config.Connections has no attribute taskName Which, incidentally, I am beginning to think is the new "no locations for get" (in its pervasiveness and opaqueness, not what it actually means). I believe this is the ap_pipe bug krzys caught and fixed earlier this week, so for now, don't worry about it. Regarding Pan-STARRS as the photometric refcat. Please specify in both datasets' READMEs that it includes Gaia DR2 and Pan-STARRS PS1 DRX (I don't know if it's DR1 or DR2?), per Eric's comment above. I am quite annoyed that gen3 does not permit hyphens in refcat names, as this is the kind of thing that will create significant overhead for folks who are just trying to do a processCcd rerun and suddenly it doesn't work anymore. That said, assuming it truly is necessary to banish hyphens from our refcats forever, I think it is fine to update the `pan-starrs` name to be hyphen free on this ticket. I think I would vote for pan_starrs or panstarrs (and not, e.g., PS1).

            Yes, I fixed it on DM-22641. (For the record, that error is not from the Butler at all: I was assigning to a config field that didn't exist. I think the AttributeError might have been raised by Python itself.)

            krzys Krzysztof Findeisen added a comment - Yes, I fixed it on DM-22641 . (For the record, that error is not from the Butler at all: I was assigning to a config field that didn't exist. I think the  AttributeError might have been raised by Python itself.)
            Parejkoj John Parejko added a comment -

            Thank you, mrawls. I've made some changes and got a successful run of ci_hits2015 on my desktop (I'm running the full hits2015 test overnight now). Could you please try it again and see if it works for you?

            Parejkoj John Parejko added a comment - Thank you, mrawls . I've made some changes and got a successful run of ci_hits2015 on my desktop (I'm running the full hits2015 test overnight now). Could you please try it again and see if it works for you?

            Thanks! I also ran the CI dataset and made a quick plot of the good DIA Sources on the sky on this ticket branch vs. the current master (plot is attached to this ticket). It looks essentially the same; the new version has 337 good sources vs. 362 good sources in master. Assuming nothing falls over when you run the full hits2015, I think this is all set.

            mrawls Meredith Rawls added a comment - Thanks! I also ran the CI dataset and made a quick plot of the good DIA Sources on the sky on this ticket branch vs. the current master (plot is attached to this ticket). It looks essentially the same; the new version has 337 good sources vs. 362 good sources in master. Assuming nothing falls over when you run the full hits2015, I think this is all set.
            Parejkoj John Parejko added a comment -

            Thanks mrawls; the full run didn't produce errors, so we should be good.

            Merged and done.

            Parejkoj John Parejko added a comment - Thanks mrawls ; the full run didn't produce errors, so we should be good. Merged and done.

            People

              Parejkoj John Parejko
              Parejkoj John Parejko
              Meredith Rawls
              Eric Bellm, Ian Sullivan, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.