Author Topic: [major bug] unicode issues  (Read 3873 times)

qmax

  • Jr. Member
  • **
  • Posts: 51
  • Karma: 0
  • Asterisk-11.7.0-ubuntu, Queuemetrics-15.10.1-tgz
    • View Profile
    • Email
[major bug] unicode issues
« on: July 15, 2009, 11:03:06 »
Installation:

mysql server is configured to use utf8 by all defaults:
character-set-client-handshake    TRUE
character-set-filesystem          binary
character-set-server              utf8
default-character-set             utf8

all database tables were created or converted to:
CREATE TABLE ... ( ... ) DEFAULT CHARSET=utf8

mysql jdbcdriver is set up like following:
jdbc:mysql://localhost/queuemetrics?...&useUnicode=true&characterEncoding=utf8&characterSetResults=utf8


file agents.conf are like following:
agent=>001,,"Василий Пупкин
(here is utf-8 string after quote)

Content-Type of all pages is: text/html; charset=UTF-8

User selected Language is "Russian"

Issues:

1. Setup Wizard
when loading agents from file agents.conf, after specifying filenames,
after page "The specified sources have been succesfully read."
on stage "Source > Agents" it displays unicode strings incorrectly.
(a string of invalid chars 2 times longer then initial)

if to proceed further, on last stage it produces error:
Quote
Fail: Problema update DB: java.sql.SQLException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INVALID CHARS' at line 1
the "INVALID CHARS" is string invalid chars (4 times longer then initial string.)

when agent name is put into database tables directly, using mysql-tools, then quemetrics displays it correctly everywhere.

2. Track QA
when track QA window is opened, and there comment put, it saves in database incorrectly,
and then redisplayed in the wrong way.

when comment is put into database tables directly, using mysql-tools, then quemetrics displays it correctly

3. Exporting
Exporting just any table into XLS or CSV replaces all unicode characters into "?"
Only XML export works.

qmax

  • Jr. Member
  • **
  • Posts: 51
  • Karma: 0
  • Asterisk-11.7.0-ubuntu, Queuemetrics-15.10.1-tgz
    • View Profile
    • Email
Re: [major bug] unicode issues
« Reply #1 on: July 15, 2009, 11:38:44 »
Some details:

in the list of loaded agents to select for inmorting,
each byte of original utf8 string is displayed as U+FFDD "replacement char"

when comments are placed in QA form,
every byte of string is saved as U+00xx, where xx is value of that byte.
e.g. character U+0410 in utf presended as bytes D0 90 and thansaved as U+00D0 U+0090

P.S.
headers and questions in QA forms are all saved and displayed properly.

marcos

  • Loway
  • Full Member
  • *
  • Posts: 138
  • Karma: 3
    • View Profile
Re: [major bug] unicode issues
« Reply #2 on: July 16, 2009, 10:02:04 »
Hi,

Thank you for your detailed explanations.
I've submitted this as bug #775 in our internal bug tracker and it will be planned to be fixed on next coming release 1.5.4.

QueueMetrics

  • Loway
  • Hero Member
  • *
  • Posts: 2999
  • Karma: 39
    • View Profile
    • QueueMetrics
Re: [major bug] unicode issues
« Reply #3 on: August 17, 2009, 10:44:35 »
That's fixed now.

marcos

  • Loway
  • Full Member
  • *
  • Posts: 138
  • Karma: 3
    • View Profile
Re: [major bug] unicode issues
« Reply #4 on: September 03, 2009, 11:10:45 »
The bug is fixed since the version 1.5.4.
To have it working, please be sure to set the key:

default.charsetEncoding=UTF8

in the configuration.properties file.

Remember to restart the queuemetrics service and to log-off and log-on again to QM session.

qmax

  • Jr. Member
  • **
  • Posts: 51
  • Karma: 0
  • Asterisk-11.7.0-ubuntu, Queuemetrics-15.10.1-tgz
    • View Profile
    • Email
Re: [major bug] unicode issues
« Reply #5 on: November 12, 2009, 13:28:15 »
in 1.5.5
exporting tables in csv/xsl still produces "?" instead of nonlatin letters

default.charsetEncoding=UTF8

however, for excel exporting we'd like to specify another encoding.
is it possible ?

QueueMetrics

  • Loway
  • Hero Member
  • *
  • Posts: 2999
  • Karma: 39
    • View Profile
    • QueueMetrics
Re: [major bug] unicode issues
« Reply #6 on: November 13, 2009, 09:03:51 »
This does not work as a text file does miss the encoding.
The correct thing to do is:

Code: [Select]
default.useXmlExcel=true   
True: Generate the Excel file as an XML file (mandatory for UTF charsets); false: generate as an ISO-8859 CSV file

This will work on not-too-old version of Excel/OpenOffice.