In the past few days we had three incidents where the qloaderd wrote timestamps into the database that were very far in the future. This lead to a complete standstill of all statistics and the inability of agents to make call classifications. We haven't found the cause yet (probably a broken line in queue_log).
To prevent this from happening again and to analyze the source of the problem, I made the following modifications to qloader.pl (posted as a diff):
--- old/qloader.pl 2009-08-19 16:28:56.000000000 +0200
+++ qloader.pl 2010-03-02 16:00:30.574029731 +0100
@@ -46,6 +46,7 @@
my $log_queries = 0; # set to 1 to log all queries
my $timezone_offset = 0 * 3600; # in seconds
+my $max_future_ts = 15; # in seconds, how many seconds a timestamp can lie in the future
my $heartbeat_delay = 15 * 60; # in seconds
my $use_subqueue = 0; # 0 no; 1 yes
my $split_subq_name = 0; # 0 no; 1 yes - turn a subqueue name from 'xxx/yyy" to "xxx"
@@ -116,7 +117,11 @@
$rowdata[3] = agentRewriteByDB( $rowdata[3] );
}
- if ( $tst < $highWaterMark ) {
+ if ( $tst > time() + $max_future_ts ) {
+ syslog("Timestamp $tst is too far in the future. Skipping line '$_'");
+ skipRow( @rowdata );
+ }
+ elsif ( $tst < $highWaterMark ) {
# salta le righe obsolete
skipRow( @rowdata );
}
If I get broken timestamps again I will post the results from the log output. Until then I'd love to hear your comments.
