Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-360

Add jemalloc to eups distrib, to enable use as a potential dependancy for qserv

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      jemalloc is a drop in mallc replacement for C/C++. The primary advantage here is that it provides application memory profiling with little cost. Once installed, enabling the profiling features is a matter of setting environent variables. jemalloc getting started

        Attachments

          Issue Links

            Activity

            Hide
            rhl Robert Lupton added a comment -

            While debugging mallocs are very useful they tend not to be as efficient as system mallocs. I don't know anything about jemalloc, is it only used via dynamic loading patching if enabled? And how will this work on OS/X platforms (where I think that the system malloc has much of this built into – see mallocinfo)?

            Show
            rhl Robert Lupton added a comment - While debugging mallocs are very useful they tend not to be as efficient as system mallocs. I don't know anything about jemalloc , is it only used via dynamic loading patching if enabled? And how will this work on OS/X platforms (where I think that the system malloc has much of this built into – see mallocinfo )?
            Hide
            bvan Brian Van Klaveren added a comment -

            jemalloc is actually the main malloc of FreeBSD, so it is a system malloc.

            Show
            bvan Brian Van Klaveren added a comment - jemalloc is actually the main malloc of FreeBSD, so it is a system malloc.
            Hide
            rhl Robert Lupton added a comment -

            So on FreeBSD we don't need it. I'm worried that on some hardware the vendor may have "value added" a custom malloc that knows about e.g. their memory backplane.

            Show
            rhl Robert Lupton added a comment - So on FreeBSD we don't need it. I'm worried that on some hardware the vendor may have "value added" a custom malloc that knows about e.g. their memory backplane.
            Hide
            jgates John Gates added a comment -

            This is a good explanation for why jemalloc is useful for memory profiling Debugging Java Native Memory Leaks

            The library is only used if you set the LD_PRELOAD environment variable and the profiling only occurs if you set the MALLOC_CONF environment variable appropriately. Otherwise, it uses the system malloc.

            So, if you want to use the memory profiling, do this before starting the program. It should generate a report every time a GB of memory is allocated.

            export LD_PRELOAD=/usr/local/lib/libjemalloc.so
            export MALLOC_CONF='prof:true,prof_prefix:/qserv/run/var/log/jeprof.out,lg_prof_interval:31,lg_prof_sample:22'
            

            Show
            jgates John Gates added a comment - This is a good explanation for why jemalloc is useful for memory profiling Debugging Java Native Memory Leaks The library is only used if you set the LD_PRELOAD environment variable and the profiling only occurs if you set the MALLOC_CONF environment variable appropriately. Otherwise, it uses the system malloc. So, if you want to use the memory profiling, do this before starting the program. It should generate a report every time a GB of memory is allocated. export LD_PRELOAD=/usr/local/lib/libjemalloc.so export MALLOC_CONF='prof:true,prof_prefix:/qserv/run/var/log/jeprof.out,lg_prof_interval:31,lg_prof_sample:22'
            Hide
            fritzm Fritz Mueller added a comment -

            Please note in title above, the proposed dependency add is only for Qserv

            Show
            fritzm Fritz Mueller added a comment - Please note in title above, the proposed dependency add is only for Qserv
            Hide
            tjenness Tim Jenness added a comment -

            I see no reason why this RFC can't be adopted. Is the implementation ticket DM-10971? (for switching qserv).

            Show
            tjenness Tim Jenness added a comment - I see no reason why this RFC can't be adopted. Is the implementation ticket DM-10971 ? (for switching qserv).
            Hide
            fritzm Fritz Mueller added a comment -

            jemalloc was packaged and published in ticket DM-11214. This has allowed us to use jemalloc for the intended debug purposes in branch builds. It remains to be seen whether we'll cut Qserv over to jemalloc in general, but that decision and work is not under the umbrella of this RFC.

            Show
            fritzm Fritz Mueller added a comment - jemalloc was packaged and published in ticket DM-11214 . This has allowed us to use jemalloc for the intended debug purposes in branch builds. It remains to be seen whether we'll cut Qserv over to jemalloc in general, but that decision and work is not under the umbrella of this RFC.

              People

              Assignee:
              jgates John Gates
              Reporter:
              jgates John Gates
              Watchers:
              Andy Hanushevsky, Brian Van Klaveren, Fritz Mueller, John Gates, Robert Lupton, Tim Jenness
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.