Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14260

Re-run HiTS data processing with current ap_pipe

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      3
    • Epic Link:
    • Sprint:
      AP S18-6, AP F18-1
    • Team:
      Alert Production

      Description

      Re-run the data that Eric Bellm processed for the late-2017 LDM-503-3 milestone using a current (May 2018) version of ap_pipe.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            If possible, using SLURM per DM-14259; if that turns out to be difficult, we shouldn't let it block this but rather should run on a single node.

            Show
            swinbank John Swinbank added a comment - If possible, using SLURM per DM-14259 ; if that turns out to be difficult, we shouldn't let it block this but rather should run on a single node.
            Hide
            mrawls Meredith Rawls added a comment -

            I manually ingested the ap_verify_hits2015 dataset with no major issues. I'm not using slurm this time, so ap_pipe.py will be chugging along for a bit on lsst-dev.

            Show
            mrawls Meredith Rawls added a comment - I manually ingested the ap_verify_hits2015 dataset with no major issues. I'm not using slurm this time, so ap_pipe.py will be chugging along for a bit on lsst-dev.
            Hide
            mrawls Meredith Rawls added a comment - - edited

            The initial stab at this is on lsst-dev in /project/mrawls/hits2015/rerun/take1. It looks like about a third of the visits didn't process due to a problem with the calib validity ranges. The error was "No registry for lookup" for a bunch of cpBiases. I thought –validity 999 was adequate during calib ingestion, but it doesn't seem to be. I manually compared the calibRegistry with the one Eric Bellm made previously and his cpBias validity ranges are adequate to encompass all the data while mine are not.

            Show
            mrawls Meredith Rawls added a comment - - edited The initial stab at this is on lsst-dev in /project/mrawls/hits2015/rerun/take1 . It looks like about a third of the visits didn't process due to a problem with the calib validity ranges. The error was "No registry for lookup" for a bunch of cpBiases. I thought –validity 999 was adequate during calib ingestion, but it doesn't seem to be. I manually compared the calibRegistry with the one Eric Bellm made previously and his cpBias validity ranges are adequate to encompass all the data while mine are not.
            Hide
            mrawls Meredith Rawls added a comment -

            As Eric Bellm, John Swinbank, and I discussed in person, this ticket was successfully completed by doing a partial reprocessing on lsst-dev in /project/mrawls/hits2015. A new ticket to troubleshoot why ~1/3 of the data didn't process and fix the problem has been created as DM-14761.

            Show
            mrawls Meredith Rawls added a comment - As Eric Bellm , John Swinbank , and I discussed in person, this ticket was successfully completed by doing a partial reprocessing on lsst-dev in /project/mrawls/hits2015 . A new ticket to troubleshoot why ~1/3 of the data didn't process and fix the problem has been created as DM-14761 .
            Hide
            ebellm Eric Bellm added a comment - - edited

            Hi Meredith Rawls, could you just run your notebook from DM-13687 on this output and drop it in the ap_pipe/u/mrawls/notebooks-etc branch? No need to update the code, but it would be nice to have at least a small tangible record of the status of this reprocessing while we wait for DM-14762.

            Show
            ebellm Eric Bellm added a comment - - edited Hi Meredith Rawls , could you just run your notebook from DM-13687 on this output and drop it in the ap_pipe/u/mrawls/notebooks-etc branch? No need to update the code, but it would be nice to have at least a small tangible record of the status of this reprocessing while we wait for DM-14762 .
            Hide
            mrawls Meredith Rawls added a comment -

            As we discussed, I actually did have to update my notebook analysis code since the database schema changed significantly (in a good way!). I figure I may as well include this latest notebook with a few preliminary light curves in a new notebooks directory in ap_pipe. The PR I just made does this. If you disagree I am happy to move it somewhere else.

            Show
            mrawls Meredith Rawls added a comment - As we discussed, I actually did have to update my notebook analysis code since the database schema changed significantly (in a good way!). I figure I may as well include this latest notebook with a few preliminary light curves in a new notebooks directory in ap_pipe . The PR I just made does this. If you disagree I am happy to move it somewhere else.
            Hide
            ebellm Eric Bellm added a comment -

            Hurrah! Great work. Onwards to SLURM-ing (which will hopefully clean up the templates)...

            Show
            ebellm Eric Bellm added a comment - Hurrah! Great work. Onwards to SLURM-ing (which will hopefully clean up the templates)...

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              swinbank John Swinbank
              Reviewers:
              Eric Bellm
              Watchers:
              Eric Bellm, John Swinbank, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: