Rotated Logs Per SIGXFSZ (exceeded files limit)

[wilsonchua]

I got this error message on my asterisk :
" Asterisk queue logger restarted".
"Rotated Logs Per SIGXFSZ (exceeded files limit)"

Then from that time on, I couldnt get any stats on the queuemetrics.
Has anyone come across this yet? Any ideas on how to resolve this?

[wilsonchua]

further INfo:
I went into my /var/log/asterisk and saw a lot of empty sized event_log.xxxxx
Could these be related?

What could be causing it? And how is it affecting queuemetrics?

[wilsonchua]

I read this:
http://www.mail-archive.com/asterisk-users@lists.digium.com/msg188549.html

Basically, the logrotate is also triggered when a large file exceed 2gb in size. This causes the the SIGXFSZ to be triggered, which in turn triggers the log rotation...

So i used this command to look for large files :
find / -type f -size +50000k -exec ls -lh {} \;| awk '{print $9": " $5}'

And found this:
/var/log/asterisk/cdr-csv/Master.csv: 2.0G

I will back this file up to Master.csv.bak and then rm the Master.csv and see if this resolves the situation.

[wilsonchua]

It did! Apparently, the Master.csv was never rotated, so its file size just kept on increasing.
AFter a while, it hit the 2gb limit and that triggered the SIGXFSZ (exceeded files limit), in turn , this triggered the log files to rotate.
This results in a LOT of event_log.* and Full.x.* files being made of zero size file.

I restarted qloader with /etc/init.d/qloader restart

I can now see data in the 'today' , real time monitoring.

I need your advise if i am to do anything about the past 2 days where no data is still showing in the queuemetrics.
Should i just wait for the qloader to process everything>? Or should i manually run some script?

[marcos]

Hi,

It depends.
If the problem was related to asterisk, it probably was not able to update the queue_log file.
If you're able to identify events for the past days, on that file, they probably were already update to your database by the qloader.
If you're able to identify events that are present in the queue_log file but not in the queue_log table (i.e. the qloader already updated more recent events) your qloader daemon will not be able to update past events automatically.
In order to fill the hole you would like to use the queuePartialUpdater tool as explained here: http://queuemetrics.com/manuals/QLoader-chunked/ar01s03.html

Thank you and regards,
Marco Signorini.

[wilsonchua]

Hi Marco,

thanks for looking into this. The past 2 days of activities were not in the mysql. The most recent calls are being added correctly into the mysql database.

I see a lot of queue_log.xxxx in the /var/log/asterisk directory, a partial list looks like this:
rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:38 queue_log.9964
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:38 queue_log.9965
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:43 queue_log.9966
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:43 queue_log.9967
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:44 queue_log.9968
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:44 queue_log.9969
-rw-rw-r--   1 asterisk asterisk        40 Jul 22 12:14 queue_log.997
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:45 queue_log.9970
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:45 queue_log.9971
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:45 queue_log.9972
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:45 queue_log.9973
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:45 queue_log.9974
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:45 queue_log.9975
-rw-rw-r--   1 asterisk asterisk        99 Jul 23 08:47 queue_log.9976
-rw-rw-r--   1 asterisk asterisk       113 Jul 23 08:47 queue_log.9977
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:50 queue_log.9978
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:50 queue_log.9979
-rw-rw-r--   1 asterisk asterisk        40 Jul 22 12:14 queue_log.998
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:51 queue_log.9980
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:51 queue_log.9981
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:51 queue_log.9982
-rw-rw-r--   1 asterisk asterisk       126 Jul 23 08:52 queue_log.9983
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:52 queue_log.9984
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:52 queue_log.9985
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:52 queue_log.9986
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:52 queue_log.9987
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:52 queue_log.9988
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:53 queue_log.9989
-rw-rw-r--   1 asterisk asterisk        40 Jul 22 12:14 queue_log.999
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:53 queue_log.9990
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:54 queue_log.9991
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:54 queue_log.9992
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:54 queue_log.9993
-rw-rw-r--   1 asterisk asterisk       155 Jul 23 08:55 queue_log.9994
-rw-rw-r--   1 asterisk asterisk        93 Jul 23 08:55 queue_log.9995
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:55 queue_log.9996
-rw-rw-r--   1 asterisk asterisk       155 Jul 23 08:56 queue_log.9997
-rw-rw-r--   1 asterisk asterisk       114 Jul 23 08:56 queue_log.9998
-rw-rw-r--   1 asterisk asterisk        40 Jul 23 08:57 queue_log.9999


Can  I just merge all the queue_log.xxx into the queue_log and restart qloaderd ?

If the above were possible, im planning to do this:

find . -name 'queue_log.*' | xargs cat >> queue_log

Will this approach work?

[wilsonchua]

When I ran the queuePartialUpdater tool (found in /usr/local/qloader) against the file, it did not restore items that were greater than the current timestamp (aug 2, 2011), the missing data were for July 23 and July 24th, 2011...

Is there a workaround for this?

[torontob]

Two days, I got exactly the same thing. Mine is a latest install with Espresso and using PBXinaFLASH. I have seen tons of /var/log/full.2343 and other full.xxxx files as well.

I am sure this is 100% caused by Queuemetrics install. It's probably that the yum repository doesn't configure the log rotate properly.

THIS IS A BUG. PLEASE FIX.

My issue went away with a restart for now.

[QueueMetrics]

I am sure this is 100% caused by Queuemetrics install. It's probably that the yum repository doesn't configure the log rotate properly.

THIS IS A BUG. PLEASE FIX.

The handling of the full log is completely outside the scope of QM and left to the distro.

[QueueMetrics]

When I ran the queuePartialUpdater tool (found in /usr/local/qloader) against the file, it did not restore items that were greater than the current timestamp (aug 2, 2011), the missing data were for July 23 and July 24th, 2011...

Is there a workaround for this?

The QueueLogPartialUpdater is meant to overcome the "problem" the qloader has, that is that it only inserts NEWER lines.

Are you sure that you have lines in your file but they are not inserted in the DB? given your previous history, I would not be surprised if Asterisk just stopped writing to all files.

[wilsonchua]

Hi Lenz,

Thanks for the response to this issue. We first merged all the queue_log.xxx into one big file.
Then ran the queuepartialupdater
But since it only inserts data that is NEWER, the old data in the merged file was not inserted into it.

Is there any way to reinsert the old data back into the database?

[QueueMetrics]

run the updater multiple times on each file, maybe the files were not merged in order.

[wilsonchua]

argh!!! there are 1000s of these files.
It means we have to run it 1000s of times as well?

Surely, there must be an easier way?

[QueueMetrics]

Yes: try if one works, and if it does, create a batch script based on the output of the ls command :-)