Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26207

Investigate and fix unusually large memory footprint in Qserv czar

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      DB_F20_09, DB_S21_12, DB_F21_06
    • Team:
      Data Access and Database

      Description

      The problem

      When Qserv is processing many parallel shared scan queries changes in the memory footprint of the mysql-proxy may indicate a possible memory leak in the application. Here an example of the snapshots taken by UNIX tool top after launching one batch of 80 shared scan queries at a time:

      #queries |     VM |     PM | total CPU [m:s]
      ---------+--------+--------+-----------------
            80 |  89.9g |  81.5g |  932:55
           160 | 103.7g |  93.9g | 1891:04
           240 | 126.5g | 118.1g | 2844:07
           320 | 118.4g | 110.4g | 3843:00
           400 | 123.6g | 116.1g | 4832:27
      

      One more snapshot was taken while processing another batch of 80 queries at a level of 50% done (on average):

      #queries |     VM |     PM | total CPU [m:s]
      ---------+--------+--------+-----------------
           480 | 134.7g | 126.9g | 5303:35
      

      It looks like there is a big pile of memory allocated and held by the process. This memory is not released after finishing each batch (of 80 queries).

      The goal of this ticket is to investigate a source of the memory consumption, understand why the memory is not released after completing the queries, and find a fix for the problem.

      Documentation on tuning JEMALLOC arenas

        Attachments

          Activity

          Hide
          gapon Igor Gaponenko added a comment - - edited

          The NAIVE method to run JEMALLOC

          NOTE: this method uses jemalloc from the Conda environment.

          Installing JEMALLOC inside the container's Stack

          Log into the running container:

          docker exec -it qserv bash
          

          Then install jemalloc:

          source /qserv/stack/loadLSST.bash 
          setup -t qserv-dev qserv_distrib
          conda install jemalloc
          

          Locate a location of the library in conda:

          % conda env list
          # conda environments:
          #
          base                     /qserv/stack/conda/miniconda3-py37_4.8.2
          lsst-scipipe-ceb6bb6  *  /qserv/stack/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-ceb6bb6
           
          % ls -al /qserv/stack/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-ceb6bb6/lib/ | grep jemalloc
          lrwxrwxrwx   1 qserv qserv        16 Aug  3 00:37 libjemalloc.so -> libjemalloc.so.2
          -rwxrwxr-x   2 qserv qserv   4414136 Jul 21 11:48 libjemalloc.so.2
          

          Try using it to profile the application

          Log into the running container:

          docker exec -it qserv bash
          

          Configure mysql-proxy (by modifying the service startup script) to run with the following JEMALLOC environment:

          JEMU='LD_PRELOAD=/qserv/stack/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-ceb6bb6/lib/libjemalloc.so'
          JEMD='MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_prefix:/qserv/run/var/log/jeprof'
          

          The cold memory footprint of the service:

          % top | grep mysql-proxy
             6143 qserv     20   0 1401392  36484  15004 S   0.0  0.0   0:00.17 mysql-proxy
          

          Then start 20 queries:

          MYSQL='mysql --protocol=tcp -hlocalhost -P4040 -uqsmaster'
          for i in $(seq 1 20); do \
              $MYSQL -e "SELECT MAX(ra) FROM wise_00.allwise_p3as_mep WHERE ra < 1.0${i}1" &
          done
          

          Check for the result

          Unfortunately, no profile files were found where they're supposed to be:

          /qserv/run/var/log/jeprof.*
          

          An attempt to run another application with JEMALLOC shows the following problem:

          source /qserv/stack/loadLSST.bash 
          setup -t qserv-dev qserv_distrib
          LD_PRELOAD=/qserv/stack/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-ceb6bb6/lib/libjemalloc.so \
          MALLOC_CONF=prof:true,lg_prof_sample:1,prof_accum:false,prof_prefix:jeprof.out \
          qserv-replica-file-ingest
          <jemalloc>: Invalid conf pair: prof:true
          <jemalloc>: Invalid conf pair: lg_prof_sample:1
          <jemalloc>: Invalid conf pair: prof_accum:false
          <jemalloc>: Invalid conf pair: prof_prefix:jeprof.out
          ...
          

          Later it was found that this particular build of the library which comes from Conda was most likely built w/o the following option:

          ./configure --enable-prof
          

          Building a correct version of the library and injecting it into the container

          On the host machine qserv-master01 as usr gapon:

          cd /home/gapon/
          git clone https://github.com/jemalloc/jemalloc.git
          cd jemalloc
          mkdir ../jemalloc_install
          ./autogen.sh --prefix=/home/gapon/jemalloc_install --enable-prof
          make
          make install
           
          ls -al /home/gapon/jemalloc_install
          drwxr-xr-x  2 gapon grp_202   62 Aug  2 22:13 bin
          drwxr-xr-x  3 gapon grp_202   22 Aug  2 22:13 include
          drwxr-xr-x  3 gapon grp_202  115 Aug  2 22:13 lib
          drwxr-xr-x  4 gapon grp_202   28 Aug  2 22:13 share
           
          ls -al /home/gapon/jemalloc_install/lib/
          -rwxr-xr-x 1 gapon grp_202 33883948 Aug  2 22:13 libjemalloc.a
          -rwxr-xr-x 1 gapon grp_202 33752306 Aug  2 22:13 libjemalloc_pic.a
          lrwxrwxrwx 1 gapon grp_202       16 Aug  2 22:13 libjemalloc.so -> libjemalloc.so.2
          -rwxr-xr-x 1 gapon grp_202  6107728 Aug  2 22:13 libjemalloc.so.2
          drwxr-xr-x 2 gapon grp_202       25 Aug  2 22:13 pkgconfig
          

          Copy the results into a folder which is mounted into the container:

          \sudo -u qserv mkdir /qserv/qserv-prod/data/jemalloc
          \sudo -u qserv cp -rf ~/jemalloc_install/* /qserv/qserv-prod/data/jemalloc/
           
          ls -al /qserv/qserv-prod/data/jemalloc
          total 0
          drwxrwxr-x 6 qserv qserv  76 Aug  2 22:15 .
          drwxr-xr-x 7 qserv qserv  75 Aug  2 23:03 ..
          drwxr-xr-x 2 qserv qserv  62 Aug  2 22:15 bin
          drwxr-xr-x 3 qserv qserv  22 Aug  2 22:15 include
          drwxr-xr-x 3 qserv qserv 115 Aug  2 22:15 lib
          drwxr-xr-x 4 qserv qserv  40 Aug  2 22:15 share
           
          ls -al /qserv/qserv-prod/data/jemalloc/lib/
          -rwxr-xr-x 1 qserv qserv 33883948 Aug  2 22:15 libjemalloc.a
          -rwxr-xr-x 1 qserv qserv 33752306 Aug  2 22:15 libjemalloc_pic.a
          lrwxrwxrwx 1 qserv qserv       16 Aug  2 22:15 libjemalloc.so -> libjemalloc.so.2
          -rwxr-xr-x 1 qserv qserv  6107728 Aug  2 22:15 libjemalloc.so.2
          drwxr-xr-x 2 qserv qserv       25 Aug  2 22:15 pkgconfig
          

          Show
          gapon Igor Gaponenko added a comment - - edited The NAIVE method to run JEMALLOC NOTE : this method uses jemalloc from the Conda environment. Installing JEMALLOC inside the container's Stack Log into the running container: docker exec -it qserv bash Then install jemalloc : source /qserv/stack/loadLSST . bash setup -t qserv-dev qserv_distrib conda install jemalloc Locate a location of the library in conda: % conda env list # conda environments: # base /qserv/stack/conda/miniconda3-py37_4 .8.2 lsst-scipipe-ceb6bb6 * /qserv/stack/conda/miniconda3-py37_4 .8.2 /envs/lsst-scipipe-ceb6bb6   % ls -al /qserv/stack/conda/miniconda3-py37_4 .8.2 /envs/lsst-scipipe-ceb6bb6/lib/ | grep jemalloc lrwxrwxrwx 1 qserv qserv 16 Aug 3 00:37 libjemalloc.so -> libjemalloc.so.2 -rwxrwxr-x 2 qserv qserv 4414136 Jul 21 11:48 libjemalloc.so.2 Try using it to profile the application Log into the running container: docker exec -it qserv bash Configure mysql-proxy (by modifying the service startup script) to run with the following JEMALLOC environment: JEMU= 'LD_PRELOAD=/qserv/stack/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-ceb6bb6/lib/libjemalloc.so' JEMD= 'MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_prefix:/qserv/run/var/log/jeprof' The cold memory footprint of the service: % top | grep mysql-proxy 6143 qserv 20 0 1401392 36484 15004 S 0.0 0.0 0:00.17 mysql-proxy Then start 20 queries: MYSQL= 'mysql --protocol=tcp -hlocalhost -P4040 -uqsmaster' for i in $( seq 1 20); do \ $MYSQL -e "SELECT MAX(ra) FROM wise_00.allwise_p3as_mep WHERE ra < 1.0${i}1" & done Check for the result Unfortunately, no profile files were found where they're supposed to be: /qserv/run/var/log/jeprof.* An attempt to run another application with JEMALLOC shows the following problem: source /qserv/stack/loadLSST . bash setup -t qserv-dev qserv_distrib LD_PRELOAD= /qserv/stack/conda/miniconda3-py37_4 .8.2 /envs/lsst-scipipe-ceb6bb6/lib/libjemalloc .so \ MALLOC_CONF=prof: true ,lg_prof_sample:1,prof_accum: false ,prof_prefix:jeprof.out \ qserv-replica- file -ingest <jemalloc>: Invalid conf pair: prof: true <jemalloc>: Invalid conf pair: lg_prof_sample:1 <jemalloc>: Invalid conf pair: prof_accum: false <jemalloc>: Invalid conf pair: prof_prefix:jeprof.out ... Later it was found that this particular build of the library which comes from Conda was most likely built w/o the following option: ./configure --enable-prof Building a correct version of the library and injecting it into the container On the host machine qserv-master01 as usr gapon : cd /home/gapon/ git clone https: //github .com /jemalloc/jemalloc .git cd jemalloc mkdir .. /jemalloc_install . /autogen .sh --prefix= /home/gapon/jemalloc_install -- enable -prof make make install   ls -al /home/gapon/jemalloc_install drwxr-xr-x 2 gapon grp_202 62 Aug 2 22:13 bin drwxr-xr-x 3 gapon grp_202 22 Aug 2 22:13 include drwxr-xr-x 3 gapon grp_202 115 Aug 2 22:13 lib drwxr-xr-x 4 gapon grp_202 28 Aug 2 22:13 share   ls -al /home/gapon/jemalloc_install/lib/ -rwxr-xr-x 1 gapon grp_202 33883948 Aug 2 22:13 libjemalloc.a -rwxr-xr-x 1 gapon grp_202 33752306 Aug 2 22:13 libjemalloc_pic.a lrwxrwxrwx 1 gapon grp_202 16 Aug 2 22:13 libjemalloc.so -> libjemalloc.so.2 -rwxr-xr-x 1 gapon grp_202 6107728 Aug 2 22:13 libjemalloc.so.2 drwxr-xr-x 2 gapon grp_202 25 Aug 2 22:13 pkgconfig Copy the results into a folder which is mounted into the container: \ sudo -u qserv mkdir /qserv/qserv-prod/data/jemalloc \ sudo -u qserv cp -rf ~ /jemalloc_install/ * /qserv/qserv-prod/data/jemalloc/   ls -al /qserv/qserv-prod/data/jemalloc total 0 drwxrwxr-x 6 qserv qserv 76 Aug 2 22:15 . drwxr-xr-x 7 qserv qserv 75 Aug 2 23:03 .. drwxr-xr-x 2 qserv qserv 62 Aug 2 22:15 bin drwxr-xr-x 3 qserv qserv 22 Aug 2 22:15 include drwxr-xr-x 3 qserv qserv 115 Aug 2 22:15 lib drwxr-xr-x 4 qserv qserv 40 Aug 2 22:15 share   ls -al /qserv/qserv-prod/data/jemalloc/lib/ -rwxr-xr-x 1 qserv qserv 33883948 Aug 2 22:15 libjemalloc.a -rwxr-xr-x 1 qserv qserv 33752306 Aug 2 22:15 libjemalloc_pic.a lrwxrwxrwx 1 qserv qserv 16 Aug 2 22:15 libjemalloc.so -> libjemalloc.so.2 -rwxr-xr-x 1 qserv qserv 6107728 Aug 2 22:15 libjemalloc.so.2 drwxr-xr-x 2 qserv qserv 25 Aug 2 22:15 pkgconfig
          Hide
          gapon Igor Gaponenko added a comment - - edited

          Using the CORRECT version of library to profile mysql-proxy

          Log into the container and check if the library is available:

          docker exec -it qserv bash
          ldd /qserv/data/jemalloc/lib/libjemalloc.so.2
          	linux-vdso.so.1 =>  (0x00007ffd9caff000)
          	libm.so.6 => /lib64/libm.so.6 (0x00007fd4bf498000)
          	libdl.so.2 => /lib64/libdl.so.2 (0x00007fd4bf294000)
          	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd4bf07e000)
          	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd4bee62000)
          	libc.so.6 => /lib64/libc.so.6 (0x00007fd4bea94000)
          	/lib64/ld-linux-x86-64.so.2 (0x00007fd4bfc63000)
          

          Reconfigure and run mysql-proxy

          Create a folder where the memory profile snapshots will be stored:

          mkdir /qserv/run/var/log/mysql-proxy/
          

          Put this into the startup script:

          JEMU='LD_PRELOAD=/qserv/data/jemalloc/lib/libjemalloc.so.2'
          JEMD='MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_prefix:/qserv/run/var/log/mysql-proxy/jeprof'
          

          Restart the application:

          /qserv/run/etc/init.d/mysql-proxy stop
          /qserv/run/etc/init.d/mysql-proxy start
          

          Launch test queries and see results

          From the host machine:

          MYSQL='mysql --protocol=tcp -hlocalhost -P4040 -uqsmaster'
          for i in $(seq 1 20); do \
              $MYSQL -e "SELECT MAX(ra) FROM wise_00.allwise_p3as_mep WHERE ra < 1.0${i}1" &
          done
          

          Result files started to appear at:

          ls -al /qserv/qserv-prod/log/mysql-proxy
          total 12888
          drwxrwxr-x 2 qserv qserv   4096 Aug  2 23:13 .
          drwxr-xr-x 3 qserv qserv    328 Aug  2 23:12 ..
          -rw-r--r-- 1 qserv qserv 138756 Aug  2 23:13 jeprof.10483.0.i0.heap
          -rw-r--r-- 1 qserv qserv 235655 Aug  2 23:13 jeprof.10483.10.i10.heap
          -rw-r--r-- 1 qserv qserv 240210 Aug  2 23:13 jeprof.10483.11.i11.heap
          ...
          

          When all queries finished, mysql-proxy was reconfigured to run w/o JEMALLOC, and restarted in the normal regime.
          There are 2624 files in the output folder:

          ls -1 /qserv/qserv-prod/log/mysql-proxy | wc -l
          3624
          du -hs /qserv/qserv-prod/log/mysql-proxy
          1.1G	/qserv/qserv-prod/log/mysql-proxy
          

          What's next?

          • Analyze the memory profile snapshots.
          • Possibly, retune the configuration of the library and make more tests
          Show
          gapon Igor Gaponenko added a comment - - edited Using the CORRECT version of library to profile mysql-proxy Log into the container and check if the library is available: docker exec -it qserv bash ldd /qserv/data/jemalloc/lib/libjemalloc .so.2 linux-vdso.so.1 => (0x00007ffd9caff000) libm.so.6 => /lib64/libm .so.6 (0x00007fd4bf498000) libdl.so.2 => /lib64/libdl .so.2 (0x00007fd4bf294000) libgcc_s.so.1 => /lib64/libgcc_s .so.1 (0x00007fd4bf07e000) libpthread.so.0 => /lib64/libpthread .so.0 (0x00007fd4bee62000) libc.so.6 => /lib64/libc .so.6 (0x00007fd4bea94000) /lib64/ld-linux-x86-64 .so.2 (0x00007fd4bfc63000) Reconfigure and run mysql-proxy Create a folder where the memory profile snapshots will be stored: mkdir /qserv/run/var/log/mysql-proxy/ Put this into the startup script: JEMU= 'LD_PRELOAD=/qserv/data/jemalloc/lib/libjemalloc.so.2' JEMD= 'MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_prefix:/qserv/run/var/log/mysql-proxy/jeprof' Restart the application: /qserv/run/etc/init .d /mysql-proxy stop /qserv/run/etc/init .d /mysql-proxy start Launch test queries and see results From the host machine: MYSQL= 'mysql --protocol=tcp -hlocalhost -P4040 -uqsmaster' for i in $( seq 1 20); do \ $MYSQL -e "SELECT MAX(ra) FROM wise_00.allwise_p3as_mep WHERE ra < 1.0${i}1" & done Result files started to appear at: ls -al /qserv/qserv-prod/log/mysql-proxy total 12888 drwxrwxr-x 2 qserv qserv 4096 Aug 2 23:13 . drwxr-xr-x 3 qserv qserv 328 Aug 2 23:12 .. -rw-r--r-- 1 qserv qserv 138756 Aug 2 23:13 jeprof.10483.0.i0.heap -rw-r--r-- 1 qserv qserv 235655 Aug 2 23:13 jeprof.10483.10.i10.heap -rw-r--r-- 1 qserv qserv 240210 Aug 2 23:13 jeprof.10483.11.i11.heap ... When all queries finished, mysql-proxy was reconfigured to run w/o JEMALLOC, and restarted in the normal regime. There are 2624 files in the output folder: ls -1 /qserv/qserv-prod/log/mysql-proxy | wc -l 3624 du -hs /qserv/qserv-prod/log/mysql-proxy 1.1G /qserv/qserv-prod/log/mysql-proxy What's next? Analyze the memory profile snapshots. Possibly, retune the configuration of the library and make more tests
          Hide
          gapon Igor Gaponenko added a comment - - edited

          Analyzing memory usage

          From inside the container:

          /qserv/data/jemalloc/bin/jeprof --show_bytes --gif \
            /qserv/stack/stack/current/Linux64/mysqlproxy/lsst-dev-g9882a7ccab/bin/mysql-proxy  \
            /qserv/run/var/log/mysql-proxy/*.heap \
            > $PWD/report.gif&
          

          The operation failed with the following error message:

          Dropping nodes with <= 398046606257 B; edges with <= 79609321251 abs(B)
          Can't exec "dot": No such file or directory at /qserv/data/jemalloc/bin/jeprof line 2081.
          | dot -Tgif: No such file or directory
          If you want to investigate this profile further, you can do:
           
            jeprof \
              /qserv/stack/stack/current/Linux64/mysqlproxy/lsst-dev-g9882a7ccab/bin/mysql-proxy \
              /qserv/run/var/log/mysql-proxy/jeprof.10483.9.i9.heap \
              /qserv/run/var/log/mysql-proxy/jeprof.10483.99.i99.heap \
              ...
          

          Install the following package which provides dot:

          conda install graphviz
          

          Then make another try to generate the report.
          The operation has failed with:

           Dropping nodes with <= 398046606257 B; edges with <= 79609321251 abs(B)
          Format: "gif" not recognized. Use one of: canon cmap cmapx cmapx_np dot dot_json eps fig gv imap imap_np ismap json json0 mp pdf pic plain plain-ext png pov ps ps2 svg svgz tk vdx vml vmlz xdot xdot1.2 xdot1.4 xdot_json
           
          [1]+  Broken pipe             /qserv/data/jemalloc/bin/jeprof --show_bytes --gif /qserv/stack/stack/current/Linux64/mysqlproxy/lsst-dev-g9882a7ccab/bin/mysql-proxy /qserv/run/var/log/mysql-proxy/*.heap > $PWD/report.gif
          

          It looks like, there is inconsistency between jeprof and doc.
          Restarted with an option to generate the dot file:

          /qserv/data/jemalloc/bin/jeprof --show_bytes --dot /qserv/stack/stack/current/Linux64/mysqlproxy/lsst-dev-g9882a7ccab/bin/mysql-proxy    /qserv/run/var/log/mysql-proxy/*.heap   > $PWD/report.dot&
          

          The last attept was a success:

          ls -al report.*
          -rw-rw-r-- 1 qserv qserv  16111 Aug  4 00:07 report.gv
          

          Make the PNG file:

          % dot -Tpng report.dot -o report.png
          % ls -al report.*
          -rw-rw-r-- 1 qserv qserv  16111 Aug  4 00:07 report.dot
          -rw-rw-r-- 1 qserv qserv 266668 Aug  4 00:33 report.png
          

          Copy both files to the host's file system (at lsst-qserv-master01):

          % pwd
          /home/gapon/development/qserv
          % docker cp qserv:/qserv/report.dot .
          % docker cp qserv:/qserv/report.png .
          % ls -al report.*
          -rw-rw-r-- 1 gapon grp_202  16111 Aug  3 19:07 report.dot
          -rw-rw-r-- 1 gapon grp_202 266668 Aug  3 19:33 report.png
          

          The report is attached to this ticket is shown below:

          What's next?

          • analyze the report
          • look at Qserv code to see if there are any memory leaks or other memory management issues
          Show
          gapon Igor Gaponenko added a comment - - edited Analyzing memory usage From inside the container: /qserv/data/jemalloc/bin/jeprof --show_bytes --gif \ /qserv/stack/stack/current/Linux64/mysqlproxy/lsst-dev-g9882a7ccab/bin/mysql-proxy \ /qserv/run/var/log/mysql-proxy/ *.heap \ > $PWD /report .gif& The operation failed with the following error message: Dropping nodes with <= 398046606257 B; edges with <= 79609321251 abs(B) Can't exec "dot": No such file or directory at /qserv/data/jemalloc/bin/jeprof line 2081. | dot -Tgif: No such file or directory If you want to investigate this profile further, you can do:   jeprof \ /qserv/stack/stack/current/Linux64/mysqlproxy/lsst-dev-g9882a7ccab/bin/mysql-proxy \ /qserv/run/var/log/mysql-proxy/jeprof.10483.9.i9.heap \ /qserv/run/var/log/mysql-proxy/jeprof.10483.99.i99.heap \ ... Install the following package which provides dot : conda install graphviz Then make another try to generate the report. The operation has failed with: Dropping nodes with <= 398046606257 B; edges with <= 79609321251 abs(B) Format: "gif" not recognized. Use one of: canon cmap cmapx cmapx_np dot dot_json eps fig gv imap imap_np ismap json json0 mp pdf pic plain plain-ext png pov ps ps2 svg svgz tk vdx vml vmlz xdot xdot1.2 xdot1.4 xdot_json   [1]+ Broken pipe /qserv/data/jemalloc/bin/jeprof --show_bytes --gif /qserv/stack/stack/current/Linux64/mysqlproxy/lsst-dev-g9882a7ccab/bin/mysql-proxy /qserv/run/var/log/mysql-proxy/*.heap > $PWD/report.gif It looks like, there is inconsistency between jeprof and doc . Restarted with an option to generate the dot file: /qserv/data/jemalloc/bin/jeprof --show_bytes --dot /qserv/stack/stack/current/Linux64/mysqlproxy/lsst-dev-g9882a7ccab/bin/mysql-proxy /qserv/run/var/log/mysql-proxy/ *.heap > $PWD /report .dot& The last attept was a success: ls -al report.* -rw-rw-r-- 1 qserv qserv 16111 Aug 4 00:07 report.gv Make the PNG file: % dot -Tpng report.dot -o report.png % ls -al report.* -rw-rw-r-- 1 qserv qserv 16111 Aug 4 00:07 report.dot -rw-rw-r-- 1 qserv qserv 266668 Aug 4 00:33 report.png Copy both files to the host's file system (at lsst-qserv-master01 ): % pwd /home/gapon/development/qserv % docker cp qserv: /qserv/report .dot . % docker cp qserv: /qserv/report .png . % ls -al report.* -rw-rw-r-- 1 gapon grp_202 16111 Aug 3 19:07 report.dot -rw-rw-r-- 1 gapon grp_202 266668 Aug 3 19:33 report.png The report is attached to this ticket is shown below: What's next? analyze the report look at Qserv code to see if there are any memory leaks or other memory management issues

            People

            Assignee:
            gapon Igor Gaponenko
            Reporter:
            gapon Igor Gaponenko
            Watchers:
            Fritz Mueller, Igor Gaponenko, John Gates, Nate Pease
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: