Skip to content

Allow BLOB columns with utf8mb3 charset#144

Open
juan-ferrer-toribio wants to merge 1 commit intonevill:masterfrom
juan-ferrer-toribio:fix-143
Open

Allow BLOB columns with utf8mb3 charset#144
juan-ferrer-toribio wants to merge 1 commit intonevill:masterfrom
juan-ferrer-toribio:fix-143

Conversation

@juan-ferrer-toribio
Copy link
Copy Markdown

Fixes #143

@rhwinter
Copy link
Copy Markdown

I was running into this issue when using @rodrigogs/mysql-events. Since I don't have access to configure the MySQL server, which is likely the culprit of the issue (wrong charset encoding settings).

The following error would be thrown when something changed in certain databases/tables/columns:

Error: Encoding not recognized: 'utf8mb3' (searched as: 'utf8mb3')
      at Object.getCodec (_____/node_modules/iconv-lite/lib/index.js:106:23)
      at Object.getDecoder (_____/node_modules/iconv-lite/lib/index.js:127:23)
      at Object.decode (_____/node_modules/iconv-lite/lib/index.js:40:25)
      at exports.readMysqlValue (_____/node_modules/@rodrigogs/zongji/lib/common.js:461:24)
      at readRow (_____/node_modules/@rodrigogs/zongji/lib/rows_event.js:111:16)
      at RowsEvent._fetchOneRow (_____/node_modules/@rodrigogs/zongji/lib/rows_event.js:93:10)
      at DeleteRows.RowsEvent (_____/node_modules/@rodrigogs/zongji/lib/rows_event.js:62:27)
      at new DeleteRows (_____/node_modules/@rodrigogs/zongji/lib/rows_event.js:128:13)
      at BinlogHeader.parse (_____/node_modules/@rodrigogs/zongji/lib/packet/binlog_header.js:47:23)
      at Protocol._parsePacket (_____/node_modules/mysql/lib/protocol/Protocol.js:272:12)

which would be followed by a mysql-events connection error:

Error: Packets out of order. Got: 161 Expected: 42
      at Parser._tryReadPacketHeader (_____/node_modules/mysql/lib/protocol/Parser.js:470:15)
      at Parser.write (_____/node_modules/mysql/lib/protocol/Parser.js:33:29)
      at Protocol.write (_____/node_modules/mysql/lib/protocol/Protocol.js:38:16)
      at Socket.<anonymous> (_____/node_modules/mysql/lib/Connection.js:88:28)
      at Socket.<anonymous> (_____/node_modules/mysql/lib/Connection.js:526:10)
      at Socket.emit (node:events:511:28)
      at Socket.emit (node:domain:489:12)
      at Readable.read (node:internal/streams/readable:547:10)
      at Socket.read (node:net:765:39)
      at flow (node:internal/streams/readable:1031:34)
      --------------------
      at Protocol._enqueue (_____/node_modules/mysql/lib/protocol/Protocol.js:144:48)
      at Immediate._start (_____/node_modules/@rodrigogs/zongji/index.js:240:31)
      at process.processImmediate (node:internal/timers:478:21) {
    code: 'PROTOCOL_PACKETS_OUT_OF_ORDER',
    fatal: true
  }

I patched @rodrigogs/zongji's common.js as follows:

diff --git a/node_modules/@rodrigogs/zongji/lib/common.js b/node_modules/@rodrigogs/zongji/lib/common.js
index 89bf7ed..1cb9c6f 100644
--- a/node_modules/@rodrigogs/zongji/lib/common.js
+++ b/node_modules/@rodrigogs/zongji/lib/common.js
@@ -440,7 +440,7 @@ exports.readMysqlValue = function(
       } else {
         if (column.charset !== null) {
           // Javascript UTF8 always allows up to 4 bytes per character
-          column.charset = column.charset === 'utf8mb4' ? 'utf8' : column.charset;
+          column.charset = column.charset.startsWith('utf8mb') ? 'utf8' : column.charset;
           parser._encoding = column.charset;
         }
         result = parser.parseString(size);
@@ -457,7 +457,7 @@ exports.readMysqlValue = function(
       // e.g. TINYTEXT, MEDIUMTEXT, LONGTEXT, TEXT data types
       if (column.charset !== null) {
         // Javascript UTF8 always allows up to 4 bytes per character
-        column.charset = column.charset === 'utf8mb4' ? 'utf8' : column.charset;
+        column.charset = column.charset.startsWith('utf8mb') ? 'utf8' : column.charset;
         result = iconv.decode(result, column.charset);
       }
       break;

Maybe this can be of help to someone out there (I couldn't find any information about these errors and issue).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crash when parsing utf8mb3 BLOB columns (MariaDB 10.7) Tag: v0.5.1

2 participants