QM 1.7.1 Froze Today

[itcanadian]

Hi,

My organization recently transitioned to an Asterisk/Queue Metrics solution for phones and call center services.  The systems have been in place for about two months now.  We've had repeated issues with our Asterisk servers, but our Queue Metrics server have had zero problems until today.

I got a call this morning that Agents couldn't log in, pause unpause, add or remove member etc.  When I attempted to browse to the webui, I found it non-responsive.  The server doesn't provide an error, it simply doesn't respond at all.  This happened again about twenty minutes ago.  The services are restored with a "service queuemetrics restart."  I'm attempting to review the logs in /usr/local/queuemetrics/tomcat/logs but it's challenging to sift through all of extra information tomcat is spitting into the logs.  I'm not thoroughly trained on Queue Metrics, having hired an integrator to install it and not having time to get up to speed as of yet.  I'm hoping somebody on the forum might be able to point me in the right direct.  Here are some details:

This is an HA cluster using heartbeat for resource management, and DRBD for distributed storage.

QM: 1.7.1
OS Platform: PBX In a Flash, Purple, with Queue Metrics installed over top
Heartbeat: 2.1.4
DRBD: 8.3.8

Can anybody recommend where I might start to determine why it's all of a sudden freezing?  No changes have been made to the queue metrics boxes recently with the exception of additional agents added etc.  Thanks in advance for any suggestions and help!

Mike.

[itcanadian]

I've some serious grep filtering, I did find the following occurring just prior to both outages:

2011-10-14 14:41:20 StandardContext[/queuemetrics]LowayTransactionController: [B969035C65C58DC6347643DFCDAA22E0] [ERR] -- Inner Exception --
Exception: java.lang.OutOfMemoryError
Error:
Java heap spaceStack trace:
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
        at java.lang.StringBuilder.append(StringBuilder.java:119)
        at java.io.UnixFileSystem.resolve(UnixFileSystem.java:93)
        at java.io.File.<init>(File.java:207)
        at java.io.File.listFiles(File.java:1056)
        at it.loway.app.queuemetrics.callListen.listeners.PluggableListener.listFiles(PluggableListener.java:81)
        at it.loway.app.queuemetrics.callListen.listeners.PluggableListener.listFiles(PluggableListener.java:99)
        at it.loway.app.queuemetrics.callListen.listeners.LocalFiles.search(LocalFiles.java:81)
        at it.loway.app.queuemetrics.callListen.listeners.LocalFiles.isCallPresent(LocalFiles.java:29)
        at it.loway.app.queuemetrics.callListen.searchAudioFiles.doRun(searchAudioFiles.java:107)
        at it.loway.tpf.transaction.servlets.LowayTransactionController.runVerb(LowayTransactionController.java:262)
        at it.loway.tpf.transaction.servlets.LowayTransactionController.serveRequest(LowayTransactionController.java:552)
        at it.loway.tpf.transaction.servlets.LowayTransactionController.serveRequestWrapper(LowayTransactionController.java:373)
        at it.loway.tpf.transaction.servlets.LowayTransactionController.doGet(LowayTransactionController.java:217)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
        at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
        at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
        at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
        at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
-- End Inner Exception --

I know the memory settings can be modified in tomcat, any suggestions where I can modify them and to what I should change them?  Thanks!

[itcanadian]

I've decided to take the following approach:

modify /etc/init.d/queuemetrics as follows:

Change:
JAVA_OPTS="-Xms128M -Xmx128M"

To:
JAVA_OPTS="-Xms2048M -Xmx2048M"

We have 18G of memory on the box.  Any thoughts against this?

[itcanadian]

For anybody who might find this post useful, here's the latest update.  I made the changes referenced above.  The first attempt used 2048, and I found that java would start, but queuemetrics itself wouldn't actually start up.  I then tweaked it down to 1024, and the service restarted and came up without problems.  I'm still open to any feedback regarding potential negative impact from this change.  Thanks again!

[QueueMetrics]

There should be no issues with that. Try and monitor the available RAM usage by using something like:  http://queuemetrics.com/manuals/QM_AdvancedConfig-chunked/ar01s07.html#_remote_monitoring_with_visualvm