Hi,
Environment: MacBookPrp running Yosemite
I am currently trying LogMX and there are a lot of features I do like however it doesn't seem to work well with very large logfiles or has a Memory leak somewhere.
I had LogMX running for one or two days, monitoring a log from my dev system running JBoss. I've restarted the server a few times for updates while LogMX was still monitoring it.
Today I've noticed that my machine was running low on memory and I checked who the offender was: LogMX with 5.9 GB.
My dev system is generating a lot of log output (level is set to debug). It seems LogMX loads everything into memory.
Is there a way to tell LogMX to only have a 'window' of the last few hundret MB or e.g. 10 000 lines?
Using LogMX with large logfiles
Moderator: admin
Re: Using LogMX with large logfiles
Hello,
If LogMX uses more than 50 MB of memory while no file is opened and that you have already clicked on the "Garbage collection" icon () in the memory status panel, then it is indeed a memory leak (bug). If it's your case, please let us know.
If it's not your case (i.e. > 50 MB while files are opened or garbage collection not forced manually), you have two ways to limit the amount of memory LogMX can use :
Xavier
If LogMX uses more than 50 MB of memory while no file is opened and that you have already clicked on the "Garbage collection" icon () in the memory status panel, then it is indeed a memory leak (bug). If it's your case, please let us know.
If it's not your case (i.e. > 50 MB while files are opened or garbage collection not forced manually), you have two ways to limit the amount of memory LogMX can use :
- For each monitored file (i.e. in Auto Refresh mode), you can set the maximum number of log entries to keep in memory using these controls in the AutoRefresh toolbar:
Only the N most recent log entries will be kept in memory (1,000,000 in this screenshot), and oldest entries couldn't be displayed anymore. - A more global and brutal way is to change the maximum amount of memory that LogMX can use: in the file "startup.conf" in the LogMX directory, you can set it on the line "MAX_MEMORY=" (see comments just before this line). Note that if you set this parameter to 1 GB, it will mean 1 GB for all opened files in LogMX, plus LogMX internal data. By default, this setting is left empty, which means 600 MB max (or 400 MB max in LogMX v5.2.0 and older). So I guess someone has already modified this parameter on your LogMX install since you managed to use 5.9 GB.
Xavier
Re: Using LogMX with large logfiles
This is the memory usage reported by LogMX
These are some stats reported by LogMX
And this is what the OS X Activity Monitor reports:
So yes I think there is a big memory leak here.Re: Using LogMX with large logfiles
What bothers me is (for what I can see in your screenshot) that the JVM (Java) reports that LogMX currently uses only 70 MB, but 612 MB are still allocated to LogMX. It means that at some point, you had plenty of loaded log entries (so you used 600 MB at this point), then you flushed or closed some entries and/or files, to get to 70 MB used, but the JVM still keeps 600 MB. This situation can occur, but should not persist over time. So it's more a garbage collection issue than a memory leak in my opinion (but I agree it must be fixed anyway). This is problem number 1. I can see 3 answers to this problem:
Could you please tell me which version of Java and LogMX you are using ("Help" > "About" > blue "i" icon > "java.version") ? Also, could you please tell me if the items 1. and 2. in the above list are right so that we can fix this issue?
Xavier
- You didn't wait enough time before taking the screenshot (very likely)
- You didn't force a manual garbage collection before taking the screenshot (click on trash can icon, several clicks may also force the VM to release some more memory)
- The policy of your automatic garbage collection process of your VM doesn't want to release this memory yet. Maybe this policy should be enforced by LogMX (currently, LogMX uses the default garbage collection policy)
Could you please tell me which version of Java and LogMX you are using ("Help" > "About" > blue "i" icon > "java.version") ? Also, could you please tell me if the items 1. and 2. in the above list are right so that we can fix this issue?
Xavier
Re: Using LogMX with large logfiles
Hello,
After some further research, it seems that problem 1 is due to Java default behavior when using a 64 bits JDK/JRE on a multi-processors host (=> using a parallel GC that doesn't release memory back to the OS, see http://stackoverflow.com/questions/1191 ... at-runtime and http://stackoverflow.com/questions/6755 ... mory-to-os).
Indeed, now 64-bits JDKs/JREs are by default only supporting "server" VM (instead of 32-bits JDKs/JREs which provides "client" and "server" VMs, and using the "client" VM by default). And this "server" VM is tuned for high availability applications: using more RAM, for enhanced performances, whereas the "client" VM tries to use a smaller memory footprint instead. The good news is that the behavior of "server" VM can be tuned to request it to give back memory to the OS when possible: in the file "logmx.command" that you use to start LogMX, update the last line to use something like:
(add "-XX:+UseSerialGC -Xminf0.5 -Xmaxf0.5" just before "-Xmx$MAX_MEMORY")
Please note that:
As for problem 2 (OS X Activity Monitor report), I just don't get it. I found something that may be related: https://forum.xojo.com/16483-yosemite-u ... ry-fixed/0. Maybe you can look at this Activity Monitor just after LogMX starts, without any opened file in LogMX, just to see if it matches LogMX memory status. Or maybe applying the above workaround will fix this displayed value...
Please let me know.
Xavier
After some further research, it seems that problem 1 is due to Java default behavior when using a 64 bits JDK/JRE on a multi-processors host (=> using a parallel GC that doesn't release memory back to the OS, see http://stackoverflow.com/questions/1191 ... at-runtime and http://stackoverflow.com/questions/6755 ... mory-to-os).
Indeed, now 64-bits JDKs/JREs are by default only supporting "server" VM (instead of 32-bits JDKs/JREs which provides "client" and "server" VMs, and using the "client" VM by default). And this "server" VM is tuned for high availability applications: using more RAM, for enhanced performances, whereas the "client" VM tries to use a smaller memory footprint instead. The good news is that the behavior of "server" VM can be tuned to request it to give back memory to the OS when possible: in the file "logmx.command" that you use to start LogMX, update the last line to use something like:
Code: Select all
java -XX:+UseSerialGC -Xminf0.5 -Xmaxf0.5 -Xmx$MAX_MEMORY .........
Please note that:
- this works for Oracle and IBM Java VMs, please tell me it doesn't work, you may use another VM (see output of "java -version")
- you will still need to force (maybe several times) a garbage collection using the LogMX memory panel to release the memory (next LogMX version will automatically force a GC when closing a file containing a great number of log entries)
As for problem 2 (OS X Activity Monitor report), I just don't get it. I found something that may be related: https://forum.xojo.com/16483-yosemite-u ... ry-fixed/0. Maybe you can look at this Activity Monitor just after LogMX starts, without any opened file in LogMX, just to see if it matches LogMX memory status. Or maybe applying the above workaround will fix this displayed value...
Please let me know.
Xavier