Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18882

Add cron job to pull validation data on a regular basis

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Validation
    • Labels:
      None

      Description

      Add a cron job that will download the validation data to a known location. This location will be /project/shared/data/[testdata|validation_data] for now. I am using the lsstsw user to do the downloading so that we can make sure people are not inadvertently changing the branch etc.

      Datasets will be:
      validation_data_hsc
      validation_data_decam
      validation_data_cfht
      testdata_jointcal
      testdata_cfht
      testdata_subaru
      testdata_decam
      ap_verify_testdata
      ap_pipe_testdata
      ci_hsc
      afwdata

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          4 older comments
          Hide
          Permalink
          krughoff Simon Krughoff added a comment -

          John Swinbank I added words to the dev guide and updated the README in /project/shared/data. Will you have a look at the PR and let me know if that was what you were thinking.

          I'm installing the crontab entries now.

          I'm not sure what to do about the error when a normal user cd's to one of those directories. I think, if anything, it gets across that those directories are not intended to be mutable, maybe?

          Show
          krughoff Simon Krughoff added a comment - John Swinbank I added words to the dev guide and updated the README in /project/shared/data . Will you have a look at the PR and let me know if that was what you were thinking. I'm installing the crontab entries now. I'm not sure what to do about the error when a normal user cd's to one of those directories. I think, if anything, it gets across that those directories are not intended to be mutable, maybe?
          Hide
          Permalink
          krughoff Simon Krughoff added a comment -

          I confirm that the crontab I installed ran last night and executed as expected.

          Show
          krughoff Simon Krughoff added a comment - I confirm that the crontab I installed ran last night and executed as expected.
          Hide
          Permalink
          swinbank John Swinbank added a comment -

          Thanks Simon! I picked some nits on your Dev Guide update, but nothing of any great significance. Thank you very much for setting this up!

          Show
          swinbank John Swinbank added a comment - Thanks Simon! I picked some nits on your Dev Guide update, but nothing of any great significance. Thank you very much for setting this up!
          Hide
          Permalink
          krughoff Simon Krughoff added a comment -

          FYI, I added testdata_lsst.

          Show
          krughoff Simon Krughoff added a comment - FYI, I added testdata_lsst .
          Hide
          Permalink
          krughoff Simon Krughoff added a comment -

          Dev guide updated and all repositories pulled.

          Show
          krughoff Simon Krughoff added a comment - Dev guide updated and all repositories pulled.

            People

            Assignee:
            krughoff Simon Krughoff
            Reporter:
            krughoff Simon Krughoff
            Reviewers:
            John Swinbank
            Watchers:
            John Swinbank, Simon Krughoff
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.