Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4814

Create validation_data set for DECam validation test

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Create a `validation_data_decam` to provide a few images for DECam validation tests.

      Use the COSMOS field data as currently available on NCSA being processed by David Nidever [X].

      Select just a few images for now.

        Attachments

          Issue Links

            Activity

            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            quick review.
            Not really that much to test here, this is the data repository.
            Do at least make sure you can check it out and the files seem good.

            Contains 2 DECam images of COSMOS field
            Contains SDSS DR9 fink-v5b astrometry.net-formated catalogs for astrometric and photometric calibration.

            In the future we should talk about expanding this validation set to some images in the Stripe82 field.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - quick review. Not really that much to test here, this is the data repository. Do at least make sure you can check it out and the files seem good. Contains 2 DECam images of COSMOS field Contains SDSS DR9 fink-v5b astrometry.net-formated catalogs for astrometric and photometric calibration. In the future we should talk about expanding this validation set to some images in the Stripe82 field.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I have problems retrieving the branch, and it seems related to git-lfs authentication. I'm not sure if it's my local problem, but I am able to retrieve other git-lfs repo such as validation_data_cfht successfully.

            fatal: astrometry_net_data/sdss-dr9-fink-v5b_and_398_0.fits: smudge filter lfs failed
            

            Does it work just normally on your machine?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I have problems retrieving the branch, and it seems related to git-lfs authentication. I'm not sure if it's my local problem, but I am able to retrieve other git-lfs repo such as validation_data_cfht successfully. fatal: astrometry_net_data/sdss-dr9-fink-v5b_and_398_0.fits: smudge filter lfs failed Does it work just normally on your machine?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Another note, the general policy is to rebase and squash, instead of the real-life improvement history
            https://confluence.lsstcorp.org/display/LDMDG/Git+Commit+Best+Practices
            http://docs.lsst.codes/en/latest/processes/workflow.html#merging
            Not sure if you were aware of that.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Another note, the general policy is to rebase and squash, instead of the real-life improvement history https://confluence.lsstcorp.org/display/LDMDG/Git+Commit+Best+Practices http://docs.lsst.codes/en/latest/processes/workflow.html#merging Not sure if you were aware of that.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            About the git-lfs problem, it seems to be very complicated and something about a shared build with multiple users and such. Same errors on lsst-dev with git-lfs/1.0.2. More discussions in HipChat Square room.

            Meanwhile, I can continue on nebula, where the checkout works.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - About the git-lfs problem, it seems to be very complicated and something about a shared build with multiple users and such. Same errors on lsst-dev with git-lfs/1.0.2. More discussions in HipChat Square room. Meanwhile, I can continue on nebula, where the checkout works.
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Yes, thank you. I'm aware of the git best practices and rebase+squash to rewrite the history in a branch flow logically and clearly.

            I'll squash the little fix-up commits that come up from the review. (e.g., I'll squash the README URL update back in a006ba0 where I did most of the updates to the README).

            But I'm otherwise preserving the goal of each commit representing a discrete logical change.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Yes, thank you. I'm aware of the git best practices and rebase+squash to rewrite the history in a branch flow logically and clearly. I'll squash the little fix-up commits that come up from the review. (e.g., I'll squash the README URL update back in a006ba0 where I did most of the updates to the README). But I'm otherwise preserving the goal of each commit representing a discrete logical change.
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Sorry you had troubles with the `git-lfs`. I read the discussion, and that does seem a bit subtle. `git-lfs` seems like a good kind of solution, but it's triggering a bunch of subtle things that don't quite work right.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Sorry you had troubles with the `git-lfs`. I read the discussion, and that does seem a bit subtle. `git-lfs` seems like a good kind of solution, but it's triggering a bunch of subtle things that don't quite work right.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The git-lfs permissions issue has been fixed. I am able to checkout the data in the ticket branch, and the files seem fine.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The git-lfs permissions issue has been fixed. I am able to checkout the data in the ticket branch, and the files seem fine.
            Hide
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment -

            Merged into master.

            Show
            wmwood-vasey Michael Wood-Vasey [X] (Inactive) added a comment - Merged into master.

              People

              Assignee:
              wmwood-vasey Michael Wood-Vasey [X] (Inactive)
              Reporter:
              wmwood-vasey Michael Wood-Vasey [X] (Inactive)
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Michael Wood-Vasey [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Due:
                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.