Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gzuncompress() #1133

Closed
anonymous-matomo-user opened this issue Jan 30, 2010 · 37 comments
Closed

gzuncompress() #1133

anonymous-matomo-user opened this issue Jan 30, 2010 · 37 comments
Assignees
Labels
Bug For errors / faults / flaws / inconsistencies etc. Critical Indicates the severity of an issue is very critical and the issue has a very high priority.
Milestone

Comments

@anonymous-matomo-user
Copy link

First of all I already tried droping table and it doesn't work.

You still have problem with:
gzuncompress() href='function.gzuncompress'>function.gzuncompress</a>: data error in /htdocs/piwik/core/Archive/Single.php on line 260

This error is shown for all widgets on Dashboard beside "Last visit graph", I had few page visits. it's been failing from start

I'm using piwik 0.54, PHP Version 5.2.11-0.dotdeb.1, mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2 UTF-8.

I can help debug error, just let me know what do you need.
I'm attaching backtrace.

Cheers,
Lukasz

@anonymous-matomo-user
Copy link
Author

Attachment: backtrace
backtrace.txt

@robocoder
Copy link
Contributor

Make sure magic_quotes_runtime is off in php.ini.

Try changing the adapter in config/config.ini.php to PDO_MYSQL.

Drop the affected archive_blob table and see if any of the above makes a difference.

If not, I'll work on a script to collect more info.

@anonymous-matomo-user
Copy link
Author

magic_quotes_runtime = Off
magic_quotes_gpc = On
adapter = "PDO_MYSQL"
I've droped databases and it didn't solve problem.

I've also tried to delete from all log tables, drop archive and used logging for other site but this didn't help too.

I've also tried this with MYSQLI adapter with same result.

Let me know how I can help.

@robocoder
Copy link
Contributor

Either there's a bug in gzcompress/gzuncompress, or the blob isn't being stored/fetched properly from the database. We can't test this by configuration changes.

Can you use the mysql client to query the database directly?

SELECT value FROM piwik_archive_blob_2010_01
WHERE idSite='1' AND date1='2010-01-30' AND date2='2010-01-30' AND period='1' AND name='Referers_searchEngineByKeyword'; 

@anonymous-matomo-user
Copy link
Author

I have direct access to console.
result of your query:

+-----------------------------------+
| value                             |
+-----------------------------------+
| xK25  |
| xK25  |
+-----------------------------------+

@anonymous-matomo-user
Copy link
Author

If you want you can reach me on gtalk 'lkarolewski@gmail.com'

I've also runned unit tests:
27/28 test cases complete: 975 passes, 0 fails and 0 exceptions.

@robocoder
Copy link
Contributor

Let's try to narrow down the problem. In core/ArchiveProcessing/Record/Blob.php, change:


26    function __construct( $name, $value)
27    {
28      $value = gzcompress($value);
29      parent::__construct( $name, $value );
30    } 

to:

      function __construct( $name, $value)
      {
        try {
            $newValue = gzcompress($value);
            gzuncompress($newValue); // check
            $value = $newValue;
        } catch(Exception $e) {
            throw new Exception("gzcompress/gzuncompress data error: $value");
        }           
        parent::__construct( $name, $value );
      }

And drop the archive_blob table again.

@robocoder
Copy link
Contributor

In core/ArchiveProcessing.php, add the following after this code:

513     $query = "INSERT IGNORE INTO ".$table->getTableName()." (idarchive, idsite, date1, date2, period, ts_archived, name, value)
514           VALUES (?,?,?,?,?,?,?,?)";
515     Piwik_Query($query,
516               array(  $this->idArchive,
517                   $this->idsite,
518                   $this->strDateStart,
519                   $this->strDateEnd,
520                   $this->periodId,
521                   date("Y-m-d H:i:s"),
522                   $record->name,
523                   $record->value,
524               )
525           );

add:

       $db = Zend_Registry::get('db');
       $value = $db->fetchOne("SELECT value
                    FROM ".$table->getTableName()."
                    WHERE idarchive = ?
                      AND name = ?", 
                    array( $this->idArchive , $record->name)
                  );
    if($value != $record->value) {
       throw new Exception('store/fetch blob mismatch: '.$record->value." vs $value");
//       throw new Exception('store/fetch blob mismatch: '.$record->value." vs $value (".gzuncompress($record->value).')');
    }


@anonymous-matomo-user
Copy link
Author

Attachment: logger_error
logger_error

@anonymous-matomo-user
Copy link
Author

Attachment: logger_exception
logger_exception

@anonymous-matomo-user
Copy link
Author

I've attached logs, we have store/fetch blob mismatch.

I had to .base64_encode($value) like this :

if($value != $record->value) 
{
    throw new Exception('store/fetch blob mismatch: '.base64_encode($record->value)." vs ".base64_encode($value)." (".gzuncompress($record->value).')');
}

because without it in logger_excetion message was ending on

'store/fetch blob mismatch: '$record->value```
(there is \0 at the end ?)

I'm wondering if you properly escape value when passing to insert query...

@robocoder
Copy link
Contributor

The logger_error should be a quick fix.

The logger_exception -- that's what we store in a blob? But I digress... -- doesn't make sense. If it was an escaping issue, I would have expected some similarity (where most characters don't require escaping).

There may be a logic error in the debuging code. Because of the 'IGNORE' in the 'INSERT', we should check the number of affected rows before the fetch.

Otherwise I'll have to look in the Zend code...

@robocoder
Copy link
Contributor

[1796] - fix incompatible declaration messages with php 5.3 and E_STRICT

@robocoder
Copy link
Contributor

Small change to only check a successfully inserted blob:

    $db = Zend_Registry::get('db');
    $rowsAffected = mysqli_affected_rows($db->getConnection());
    if($rowsAffected === 1)
    {
        $value = $db->fetchOne("SELECT value
                        FROM ".$table->getTableName()."
                        WHERE idarchive = ?
                        AND name = ?",   
                    array( $this->idArchive , $record->name)
                );
        if($value != $record->value) 
        {
            throw new Exception('store/fetch blob mismatch: '.base64_encode($record->value)." vs ".base64_encode($value)." (".gzuncompress($record->value).')');
        }
    }

@robocoder
Copy link
Contributor

Stumbled upon http://bugs.mysql.com/bug.php?id=37514 ... which raises the question: did you get an installation warning about mismatched MySQL client/server character sets?

@robocoder
Copy link
Contributor

Scratch that. This is a more likely cause: http://bugs.mysql.com/bug.php?id=36851 -- since newer php versions use mysqlnd.

Are you running 0.5.4 or from our development trunk? (ie does it have the IGNORE keyword in the SELECT? (added in [1767])

@anonymous-matomo-user
Copy link
Author

Hi,
I've downloaded 0.5.4 version from website - lastet.zip ( and I don't have mentioned IGNORE )
I didn't have any warnings about client server mismath on install.

I'd like to tell you that when i've removed gzcompress from core/ArchiveProcessing/Record/Blob.php and gzuncompress from core/Archive/Single.php piwik works just fine so at least we have a workaround.

I'm just checking out source from svn , i'll do setup and upload logs.
Cheers,
Lukasz

@anonymous-matomo-user
Copy link
Author

oh after adding

 $rowsAffected = mysqli_affected_rows( ```
I still have store/fetch blob mismatch.

@robocoder
Copy link
Contributor

Looking at a hex dump of the base64_decode'd strings, the fetch'd string is an encoded version of the store'd string. This is leaning towards a MySQL issue.

Where you check rowsAffected, change the query to:

        $values = $db->fetchOne("SELECT value, hex(value)
                        FROM ".$table->getTableName()."
                        WHERE idarchive = ?
                        AND name = ?",   
                    array( $this->idArchive , $record->name)
                );

The hex should tell us when the encoding occurs.

And it would help if you can add a query before throwing the exception.

        var_dump(Piwik_FetchALL('SHOW VARIABLES'));

@anonymous-matomo-user
Copy link
Author

Attachment: show variables
show_variables.txt

@anonymous-matomo-user
Copy link
Author

Attachment: logger_exception_hex(value)
logger_exception.2

@anonymous-matomo-user
Copy link
Author

Current format of exception message:

'store/fetch blob mismatch: '.base64_encode($record->value)." vs ".base64_encode($value)." (".gzuncompress($record->value).') hex:'.$value_hex

@anonymous-matomo-user
Copy link
Author

Attachment: logger_exception_hex(value)_bin2hex(record->value)
logger_exception.3

@anonymous-matomo-user
Copy link
Author

in logger_exception.3 i've added bin2hex($record->value) at the end of message,

Please note that bin2hex conversion is done byte-wise with the high-nibble first, i don't know how sql hex() is done.

@robocoder
Copy link
Contributor

Thanks. The problem is that the blob is stored incorrectly. Binary blobs don't have a character set, but MySQL is mistakenly applying some character set conversion on the field.

See if this is a workaround: drop the archive tables and add 'charset = utf8' to your database settings in config.ini.php.

@robocoder
Copy link
Contributor

Were you able to test the config change in comment:19 ?

@anonymous-matomo-user
Copy link
Author

Currently I'm on business trip and i have very limited time. i'll be able to get back to you on Friday.

@mattab
Copy link
Member

mattab commented Feb 19, 2010

lkarolew, any update?

@robocoder
Copy link
Contributor

At this point, my guess is that the dotdeb build is bad (e.g., missing the force_libmysqlclient_r.patch, perhaps?).

My suggestion would be to try dotdeb's 5.2.12 package to see if the problem has been resolved, rollback to an official debian package (5.2.6), or test drive the squeeze distribution 5.2.12).

@anonymous-matomo-user
Copy link
Author

Hi, sorry for such delay but I've been extreamly busy lately, for now it should be better with my time.

I'm not admin on the server, and i found it impossible to update mysql configuration files.

What i did is I added Piwik_Query('SET NAMES utf8') before Piwik_Query($query,...) and it solved problem.

But probably it is not the best place it.
Can we put this query after each connection? somewhere in connection constructor? or something to that effect ?

@robocoder
Copy link
Contributor

Adding 'charset = utf8' to your database connection settings in config.ini.php should accomplish the same effect.

@mattab
Copy link
Member

mattab commented Mar 2, 2010

vipsoft, could this charset=utf8 be automatically added when the detection showed a mismatch between client and server? Currently I believe this is an information message, but instead can we just add it in the config? thx

@robocoder
Copy link
Contributor

I thought the idea of auto-adding it to the config was nixed back in #984.

@robocoder
Copy link
Contributor

Just to be clear: the proposed change is to add: charset=utf8 to the config for new installatons only (or where the tables are deleted during a re-install).

@anonymous-matomo-user
Copy link
Author

just to confirm adding 'charset = utf8' to database connection settings in config.ini.php resolved my issue.

@robocoder
Copy link
Contributor

(In [1881]) fixes #1133 - adds 'charset = utf8' to configuration file on new installs (or reinstalls where tables are deleted).

I also made some changes to address ZF-1743 (PHP 5.2.0) issues.

@robocoder
Copy link
Contributor

(In [2065]) refs #1133 - simplify logic re: setting the charset when installer (re)generates the config file

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. Critical Indicates the severity of an issue is very critical and the issue has a very high priority.
Projects
None yet
Development

No branches or pull requests

3 participants