Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13961

Run KPM30 tests at IN2P3

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Run KPM30 tests on LSST30 data on the lower half of IN2P3.

        Attachments

          Activity

          Hide
          vaikunth Vaikunth Thukral added a comment -

          Multiple regression tests with downgrading mariadb and qserv versions were performed and evaluated at IN2P3. Final results will be documented in DMTR-71 (DM-14294).

          Show
          vaikunth Vaikunth Thukral added a comment - Multiple regression tests with downgrading mariadb and qserv versions were performed and evaluated at IN2P3. Final results will be documented in DMTR-71 ( DM-14294 ).
          Hide
          vaikunth Vaikunth Thukral added a comment -

          Fritz Mueller associated code has now been pushed to the relevant ticket branch if you'd like to review that, or leave it in there as a record of changes made for KPM30's without merging to master.

          Show
          vaikunth Vaikunth Thukral added a comment - Fritz Mueller associated code has now been pushed to the relevant ticket branch if you'd like to review that, or leave it in there as a record of changes made for KPM30's without merging to master.
          Hide
          vaikunth Vaikunth Thukral added a comment -

          Just in case, all my KPM testing stuff at IN2P3 is on /sps/lsst/Qserv/vaikunth/kpm/ 

          Show
          vaikunth Vaikunth Thukral added a comment - Just in case, all my KPM testing stuff at IN2P3 is on /sps/lsst/Qserv/vaikunth/kpm/  
          Hide
          vaikunth Vaikunth Thukral added a comment -

          To get timings, sizes and counts out of KPM logs:

           

          cat runQueries.log | grep "QTYPE_LV FINISHED" | wc -l
          cat runQueries.log | grep "QTYPE_FTSObj FINISHED" | wc -l
          cat runQueries.log | grep "QTYPE_FTSSrc FINISHED" | wc -l
          cat runQueries.log | grep "QTYPE_FTSFSrc FINISHED" | wc -l
          cat runQueries.log | grep "QTYPE_joinObjSrc FINISHED" | wc -l
          cat runQueries.log | grep "QTYPE_joinObjFSrc FINISHED" | wc -l
          cat runQueries.log | grep "QTYPE_nearN FINISHED" | wc -l
          cat runQueries.log | grep "QTYPE_LV FINISHED" | awk '{sum +=$12; n++} END {print sum/n}'
          cat runQueries.log | grep "QTYPE_FTSObj FINISHED" | awk '{sum +=$12; n++} END {print sum/n}'
          cat runQueries.log | grep "QTYPE_FTSSrc FINISHED" | awk '{sum +=$12; n++} END {print sum/n}'
          cat runQueries.log | grep "QTYPE_FTSFSrc FINISHED" | awk '{sum +=$12; n++} END {print sum/n}'
          cat runQueries.log | grep "QTYPE_joinObjSrc FINISHED" | awk '{sum +=$12; n++} END {print sum/n}'
          cat runQueries.log | grep "QTYPE_joinObjFSrc FINISHED" | awk '{sum +=$12; n++} END {print sum/n}'
          cat runQueries.log | grep "QTYPE_nearN FINISHED" | awk '{sum +=$12; n++} END {print sum/n}'
          du -hc LV* | tail -1
          du -hc FTS* join* near*
          

           

          Show
          vaikunth Vaikunth Thukral added a comment - To get timings, sizes and counts out of KPM logs:   cat runQueries.log | grep "QTYPE_LV FINISHED" | wc -l cat runQueries.log | grep "QTYPE_FTSObj FINISHED" | wc -l cat runQueries.log | grep "QTYPE_FTSSrc FINISHED" | wc -l cat runQueries.log | grep "QTYPE_FTSFSrc FINISHED" | wc -l cat runQueries.log | grep "QTYPE_joinObjSrc FINISHED" | wc -l cat runQueries.log | grep "QTYPE_joinObjFSrc FINISHED" | wc -l cat runQueries.log | grep "QTYPE_nearN FINISHED" | wc -l cat runQueries.log | grep "QTYPE_LV FINISHED" | awk '{sum +=$12; n++} END {print sum/n}' cat runQueries.log | grep "QTYPE_FTSObj FINISHED" | awk '{sum +=$12; n++} END {print sum/n}' cat runQueries.log | grep "QTYPE_FTSSrc FINISHED" | awk '{sum +=$12; n++} END {print sum/n}' cat runQueries.log | grep "QTYPE_FTSFSrc FINISHED" | awk '{sum +=$12; n++} END {print sum/n}' cat runQueries.log | grep "QTYPE_joinObjSrc FINISHED" | awk '{sum +=$12; n++} END {print sum/n}' cat runQueries.log | grep "QTYPE_joinObjFSrc FINISHED" | awk '{sum +=$12; n++} END {print sum/n}' cat runQueries.log | grep "QTYPE_nearN FINISHED" | awk '{sum +=$12; n++} END {print sum/n}' du -hc LV* | tail - 1 du -hc FTS* join* near*  

            People

            Assignee:
            fritzm Fritz Mueller
            Reporter:
            vaikunth Vaikunth Thukral
            Reviewers:
            Fritz Mueller
            Watchers:
            Fritz Mueller, John Gates, Vaikunth Thukral
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: