Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19595

Make new ap_pipe rerun with latest weekly

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP S19-6
    • Team:
      Alert Production

      Description

      We should do new ap_pipe reruns on the HiTS2015 dataset approximately monthly with the latest weekly stack. In this case, we will not regenerate the coadd templates and will just use the CompareWarp coadds which are a few months old.

        Attachments

          Activity

          Hide
          mrawls Meredith Rawls added a comment - - edited

          Gabor Kovacs, will you please review this?

          The latest rerun with the latest weekly stack is on lsst-dev in /project/mrawls/hits2015/rerun/cw_processed5.

          There is no code to review; please just verify you can access the data products you expect at that location and feel free to switch to using this rerun for your diffim troubleshooting.

          Show
          mrawls Meredith Rawls added a comment - - edited Gabor Kovacs , will you please review this? The latest rerun with the latest weekly stack is on lsst-dev in /project/mrawls/hits2015/rerun/cw_processed5 . There is no code to review; please just verify you can access the data products you expect at that location and feel free to switch to using this rerun for your diffim troubleshooting.
          Hide
          gkovacs Gabor Kovacs added a comment - - edited

          Thanks for doing the re-runs.

           

          I'd recommend recording here the config file if any was given on the command line for ap_pipe and the version tag of the stack. (I know that this can be dug out from the repository, but perhaps it's easier to have it here as well.) If we have a log output of the run, we may want to have the filename here, too.

          Show
          gkovacs Gabor Kovacs added a comment - - edited Thanks for doing the re-runs.   I'd recommend recording here the config file if any was given on the command line for ap_pipe and the version tag of the stack. (I know that this can be dug out from the repository, but perhaps it's easier to have it here as well.) If we have a log output of the run, we may want to have the filename here, too.
          Hide
          mrawls Meredith Rawls added a comment -

          The only custom-set configs were the following:

          ppdb.db_url=sqlite:////project/mrawls/hits2015/rerun/cw_processed5/association.db
          ppdb.isolation_level=READ_UNCOMMITTED -c ppdb.connection_timeout=240
          ppdb.isolation_level=READ_UNCOMMITTED -c ppdb.connection_timeout=240
          ccdProcessor.calibrate.photoCal.match.referenceSelection.magLimit.fluxField="i_flux"
          ccdProcessor.calibrate.photoCal.match.referenceSelection.magLimit.maximum=22.0

          The way I have it set up to parallelize in slum results in the logs being overwritten as each new visit is processed. Nevertheless, the slurm output for the full run is in /project/mrawls/hits2015/slurm-154434.out and the logs for each ccd from the final visit (421606) are in /project/mrawls/hits2015/slurm/ap_pipe154434-*.out (where * is replaced by each ccdnum).

          The stack version used was the weekly tagged w_2019_21.

          Show
          mrawls Meredith Rawls added a comment - The only custom-set configs were the following: ppdb.db_url=sqlite:////project/mrawls/hits2015/rerun/cw_processed5/association.db ppdb.isolation_level=READ_UNCOMMITTED -c ppdb.connection_timeout=240 ppdb.isolation_level=READ_UNCOMMITTED -c ppdb.connection_timeout=240 ccdProcessor.calibrate.photoCal.match.referenceSelection.magLimit.fluxField="i_flux" ccdProcessor.calibrate.photoCal.match.referenceSelection.magLimit.maximum=22.0 The way I have it set up to parallelize in slum results in the logs being overwritten as each new visit is processed. Nevertheless, the slurm output for the full run is in /project/mrawls/hits2015/slurm-154434.out and the logs for each ccd from the final visit (421606) are in /project/mrawls/hits2015/slurm/ap_pipe154434-*.out (where * is replaced by each ccdnum). The stack version used was the weekly tagged w_2019_21.

            People

            • Assignee:
              mrawls Meredith Rawls
              Reporter:
              mrawls Meredith Rawls
              Reviewers:
              Gabor Kovacs
              Watchers:
              Eric Bellm, Gabor Kovacs, John Swinbank, Meredith Rawls
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: