Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4814

Create validation_data set for DECam validation test

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      Create a `validation_data_decam` to provide a few images for DECam validation tests.

      Use the COSMOS field data as currently available on NCSA being processed by nidever.

      Select just a few images for now.

      Attachments

        Issue Links

          Activity

            quick review.
            Not really that much to test here, this is the data repository.
            Do at least make sure you can check it out and the files seem good.

            Contains 2 DECam images of COSMOS field
            Contains SDSS DR9 fink-v5b astrometry.net-formated catalogs for astrometric and photometric calibration.

            In the future we should talk about expanding this validation set to some images in the Stripe82 field.

            wmwood-vasey Michael Wood-Vasey added a comment - quick review. Not really that much to test here, this is the data repository. Do at least make sure you can check it out and the files seem good. Contains 2 DECam images of COSMOS field Contains SDSS DR9 fink-v5b astrometry.net-formated catalogs for astrometric and photometric calibration. In the future we should talk about expanding this validation set to some images in the Stripe82 field.

            I have problems retrieving the branch, and it seems related to git-lfs authentication. I'm not sure if it's my local problem, but I am able to retrieve other git-lfs repo such as validation_data_cfht successfully.

            fatal: astrometry_net_data/sdss-dr9-fink-v5b_and_398_0.fits: smudge filter lfs failed
            

            Does it work just normally on your machine?

            hchiang2 Hsin-Fang Chiang added a comment - I have problems retrieving the branch, and it seems related to git-lfs authentication. I'm not sure if it's my local problem, but I am able to retrieve other git-lfs repo such as validation_data_cfht successfully. fatal: astrometry_net_data/sdss-dr9-fink-v5b_and_398_0.fits: smudge filter lfs failed Does it work just normally on your machine?

            Another note, the general policy is to rebase and squash, instead of the real-life improvement history
            https://confluence.lsstcorp.org/display/LDMDG/Git+Commit+Best+Practices
            http://docs.lsst.codes/en/latest/processes/workflow.html#merging
            Not sure if you were aware of that.

            hchiang2 Hsin-Fang Chiang added a comment - Another note, the general policy is to rebase and squash, instead of the real-life improvement history https://confluence.lsstcorp.org/display/LDMDG/Git+Commit+Best+Practices http://docs.lsst.codes/en/latest/processes/workflow.html#merging Not sure if you were aware of that.

            About the git-lfs problem, it seems to be very complicated and something about a shared build with multiple users and such. Same errors on lsst-dev with git-lfs/1.0.2. More discussions in HipChat Square room.

            Meanwhile, I can continue on nebula, where the checkout works.

            hchiang2 Hsin-Fang Chiang added a comment - About the git-lfs problem, it seems to be very complicated and something about a shared build with multiple users and such. Same errors on lsst-dev with git-lfs/1.0.2. More discussions in HipChat Square room. Meanwhile, I can continue on nebula, where the checkout works.

            Yes, thank you. I'm aware of the git best practices and rebase+squash to rewrite the history in a branch flow logically and clearly.

            I'll squash the little fix-up commits that come up from the review. (e.g., I'll squash the README URL update back in a006ba0 where I did most of the updates to the README).

            But I'm otherwise preserving the goal of each commit representing a discrete logical change.

            wmwood-vasey Michael Wood-Vasey added a comment - Yes, thank you. I'm aware of the git best practices and rebase+squash to rewrite the history in a branch flow logically and clearly. I'll squash the little fix-up commits that come up from the review. (e.g., I'll squash the README URL update back in a006ba0 where I did most of the updates to the README). But I'm otherwise preserving the goal of each commit representing a discrete logical change.

            Sorry you had troubles with the `git-lfs`. I read the discussion, and that does seem a bit subtle. `git-lfs` seems like a good kind of solution, but it's triggering a bunch of subtle things that don't quite work right.

            wmwood-vasey Michael Wood-Vasey added a comment - Sorry you had troubles with the `git-lfs`. I read the discussion, and that does seem a bit subtle. `git-lfs` seems like a good kind of solution, but it's triggering a bunch of subtle things that don't quite work right.

            The git-lfs permissions issue has been fixed. I am able to checkout the data in the ticket branch, and the files seem fine.

            hchiang2 Hsin-Fang Chiang added a comment - The git-lfs permissions issue has been fixed. I am able to checkout the data in the ticket branch, and the files seem fine.

            Merged into master.

            wmwood-vasey Michael Wood-Vasey added a comment - Merged into master.

            People

              wmwood-vasey Michael Wood-Vasey
              wmwood-vasey Michael Wood-Vasey
              Hsin-Fang Chiang
              Hsin-Fang Chiang, Michael Wood-Vasey
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.