Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32870

Use MatchProbabilistic PipelineTask for DC2 plots

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      2
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Update the notebook from DM-31781 to use outputs from the PipelineTask created in DM-32034 (currently, the parquet tables are manually written to/read from disk).

        Attachments

          Activity

          Hide
          dtaranu Dan Taranu added a comment -

          Fred Moolekamp, can you see if this works for you without having to setup/install gauss2d/(meas_extensions_)multiprofit? I tested by including only modelling_research in my PYTHONPATH and it seemed to work.

          If you need to compare, I have w44 and w40 notebooks (the latter in an old subdirectory since it won't work as-is anymore).

          Show
          dtaranu Dan Taranu added a comment - Fred Moolekamp , can you see if this works for you without having to setup/install gauss2d/(meas_extensions_)multiprofit? I tested by including only modelling_research in my PYTHONPATH and it seemed to work. If you need to compare, I have w44 and w40 notebooks (the latter in an old subdirectory since it won't work as-is anymore).
          Hide
          fred3m Fred Moolekamp added a comment -

          Sure. Sorry, I forgot about this today but I'll test it out tomorrow.

          Show
          fred3m Fred Moolekamp added a comment - Sure. Sorry, I forgot about this today but I'll test it out tomorrow.
          Hide
          dtaranu Dan Taranu added a comment - - edited

          For reference, the command to run the matcher:

          pipetask --long-log run -b /repo/dc2 --input "2.2i/runs/test-med-1/w_2021_44/DM-32391,2.2i/truth_summary" --output "u/dtaranu/DM-32034/w_2021_44_match_0.5asec" -d "instrument = 'LSSTCam-imSim' and tract=3829 and skymap='DC2'" -p /project/dtaranu/dc2/match_catalog_pipe/match_tract_catalog_probabilistic.yaml --register-dataset-types

          Show
          dtaranu Dan Taranu added a comment - - edited For reference, the command to run the matcher: pipetask --long-log run -b /repo/dc2 --input "2.2i/runs/test-med-1/w_2021_44/ DM-32391 ,2.2i/truth_summary" --output "u/dtaranu/ DM-32034 /w_2021_44_match_0.5asec" -d "instrument = 'LSSTCam-imSim' and tract=3829 and skymap='DC2'" -p /project/dtaranu/dc2/match_catalog_pipe/match_tract_catalog_probabilistic.yaml --register-dataset-types
          Hide
          fred3m Fred Moolekamp added a comment -

          Something isn't right. When I run the notebook before using this task I get

          for the 2nd and 3rd plots in the notebook, but if I use the notebook contained in this ticket I get

          when run on the same data.

          The task command that I used to merge the catalogs was

          pipetask --long-log run -b /repo/dc2 --output-run "u/fred3m/DM-32436/DC2_44/20211210T214832Z" --extend-run -d "instrument = 'LSSTCam-imSim' and tract in (3828,3829) and skymap='DC2'" -p /project/dtaranu/dc2/match_catalog_pipe/match_tract_catalog_probabilistic.yaml --input 2.2i/runs/test-med-1/w_2021_44/DM-32391,2.2i/truth_summary --no-versions

          Show
          fred3m Fred Moolekamp added a comment - Something isn't right. When I run the notebook before using this task I get for the 2nd and 3rd plots in the notebook, but if I use the notebook contained in this ticket I get when run on the same data. The task command that I used to merge the catalogs was pipetask --long-log run -b /repo/dc2 --output-run "u/fred3m/ DM-32436 /DC2_44/20211210T214832Z" --extend-run -d "instrument = 'LSSTCam-imSim' and tract in (3828,3829) and skymap='DC2'" -p /project/dtaranu/dc2/match_catalog_pipe/match_tract_catalog_probabilistic.yaml --input 2.2i/runs/test-med-1/w_2021_44/ DM-32391 ,2.2i/truth_summary --no-versions
          Hide
          fred3m Fred Moolekamp added a comment -

          Ok, thanks to assistance from Dan Taranu in slack we were able to get the notebook working.

          Show
          fred3m Fred Moolekamp added a comment - Ok, thanks to assistance from Dan Taranu in slack we were able to get the notebook working.
          Hide
          fred3m Fred Moolekamp added a comment -

          The changes work to recreate the plots from the previous version of the notebook.

          Show
          fred3m Fred Moolekamp added a comment - The changes work to recreate the plots from the previous version of the notebook.

            People

            Assignee:
            dtaranu Dan Taranu
            Reporter:
            dtaranu Dan Taranu
            Reviewers:
            Fred Moolekamp
            Watchers:
            Dan Taranu, Fred Moolekamp
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.