Author Topic: Manually dump old log files into qloader database  (Read 4431 times)

jhfitness

  • Newbie
  • *
  • Posts: 26
  • Karma: 1
    • View Profile
    • Buy Treadmill Parts
Manually dump old log files into qloader database
« on: September 09, 2010, 20:40:52 »
I have data from the last couple of weeks going into the sql:P001 database. However I have some old logs such as queue_log.2, queue_log.4, etc, that I would like to manually dump into the database before they are rotated away.

I found this in the user manuals, is this what I want to do?

Quote
To test that the qloader script is working, try the following command:

./qloader.pl /var/log/asterisk/queue_log P02 /var/log/asterisk/qloader.log

This command will try uploading the contents of the file /var/log/asterisk/queue_log to partition P02 of the chosen database, writing its own log on /var/log/asterisk/qloader.log. When the program is running no output is written on stdout; if you want to know how the loading is going you have to consult its own log file.

but with /queue_log.X P001 ?

Thanks
« Last Edit: September 09, 2010, 20:43:43 by jahedlund »

jhfitness

  • Newbie
  • *
  • Posts: 26
  • Karma: 1
    • View Profile
    • Buy Treadmill Parts
Re: Manually dump old log files into qloader database
« Reply #1 on: September 09, 2010, 21:35:53 »
Actually, I got it to work once I passed the correct database connection parameters but the log says "Ignoring timestamps below XXXXXXXX" and everything is skipped.

I found another topic that says it won't accept data older than existing data and this seems to confirm that. Is there a quick temporary to override that or do I need to go the route of deleting the database records and importing my 5 logs one by one?

jhfitness

  • Newbie
  • *
  • Posts: 26
  • Karma: 1
    • View Profile
    • Buy Treadmill Parts
Re: Manually dump old log files into qloader database
« Reply #2 on: September 09, 2010, 21:50:41 »
OK, so here's what I did. P001 was my only partition so I loaded the missing logs into a new partition, P002, except I forgot to do it in reverse time order so I went on P003:

Code: [Select]
./qloader.pl -h localhost -d queuemetrics -u queuemetrics -p (password) /var/log/asterisk/queue_log.5 P003 /var/log/asterisk/qloader.log
./qloader.pl -h localhost -d queuemetrics -u queuemetrics -p (password) /var/log/asterisk/queue_log.4 P003 /var/log/asterisk/qloader.log
./qloader.pl -h localhost -d queuemetrics -u queuemetrics -p (password) /var/log/asterisk/queue_log.2 P003 /var/log/asterisk/qloader.log

Then I went into mysql and changed the partition name.

Code: [Select]
update queue_log set partition='P001' where partition='P003';
Also, of course, deleting where partition was P002. Now all of my data shows up in P001 like I wanted.

Hope this can help someone else... Feel free to let me know if I've messed something up and should change something. Everything looks good though.
« Last Edit: September 09, 2010, 21:55:08 by jahedlund »

marcos

  • Loway
  • Full Member
  • *
  • Posts: 138
  • Karma: 3
    • View Profile
Re: Manually dump old log files into qloader database
« Reply #3 on: September 10, 2010, 11:10:13 »
Hi,
What you did is correct.
Morehover, we provided a script that you can use for inserting timestamps in an already present queue log.
The script is called queuePartialUpdater.pl and you can use it in this way:

queuePartialUpdater.pl /my/queue_log/file partition_name /my/queue_log/logfile

The script takes the /my/queue_log/file as input and merge the events found with the information stored in the database for the specified partition.
You have to customize it a little bit, before to run it for the first time, in order to better adapt to your database configuration.

Thank you and regards,
Marco Signorini.