Troubleshooting Incremental Data Sync Failure in OceanBase OBLogProxy Binlog Mode

1 Background

A customer originally used the data analysis pipeline XXDB → Flink CDC → Doris . To replace the data source with OceanBase Community Edition, the pipeline was changed to OceanBase → OBLogProxy → Flink CDC → Doris . OceanBase serves as a distributed database, and its change logs are captured by OBLogProxy for downstream incremental synchronization.

OceanBase (data source) : native distributed database that overcomes single‑node capacity limits; its DDL/DML logs are captured by OBLogProxy.

OBLogProxy (log proxy service) : pulls incremental logs (transaction logs, binlog) from the OceanBase cluster and converts them into an ordered change data stream.

Flink CDC (real‑time data sync) : uses the Flink CDC Connector to connect to OBLogProxy, achieving full‑plus‑incremental data synchronization.

Doris (real‑time analytics database) : receives the processed data stream via the Flink Doris Connector.

Note on version compatibility : Ensure that the versions of OceanBase, OBLogProxy, and Flink CDC match.

OBLogProxy Binlog Mode

Binlog mode was created to be compatible with MySQL binlog, allowing existing MySQL binlog tools to synchronize OceanBase. It requires configuring obconfig_url and working together with OBProxy.

After the migration, downstream users could not read newly added data, suspecting an issue with OBLogProxy.

2 Troubleshooting Process

Check OBLogProxy status – it was abnormal due to insufficient VM resources.

MySQL [(none)]> show binlog instances\G
*************************** 1. row ***************************
    name: evan_cluster#tenant_evan
    ob_cluster: evan_cluster
    ob_tenant: tenant_evan
    ip: 10.186.64.164
    port: 2983
    running: No
    state: Running
    obcdc_running: No
    obcdc_state: Running
    service_mode: disabled
    convert_running: No
    ...

Start the OBLogProxy process – status returned to normal.

[root@oblogproxy-server oblogproxy]# ./run.sh start

MySQL [(none)]> show binlog instances\G
*************************** 1. row ***************************
    name: evan_cluster#tenant_evan
    ob_cluster: evan_cluster
    ob_tenant: tenant_evan
    ip: 10.186.64.164
    port: 2983
    running: Yes
    state: Running
    obcdc_running: Yes
    service_mode: enabled
    convert_running: Yes
    ...

Verify that new binlog files were generated in the binlog directory.

Developers still could not read the new data, so further steps were taken.

Log into the OceanBase MySQL tenant, create a test table, and insert several rows.

MySQL [evan_db]> CREATE TABLE `evan_test` (
    `id` int(11) NOT NULL,
    `name` varchar(10) DEFAULT NULL,
    `country` varchar(100) DEFAULT NULL,
    `age` int(10) DEFAULT NULL,
    PRIMARY KEY (`id`)
);
Query OK, 0 rows affected (0.05 sec)

MySQL [evan_db]> INSERT INTO evan_test (id, name, country, age) VALUES (1, 'test', 'China', 18);
Query OK, 1 row affected (0.01 sec)
... (additional inserts) ...

Run show master status – the current binlog file was mysql-bin.000227 .

Search mysql-bin.000227 for the test table – no entries were found.

Inspect the timestamp range of the binlog file and realize that the log covered an earlier period (250124‑250206) while the inserts occurred later (250219).

[root@oblogproxy-server data]# mysqlbinlog -vv mysql-bin.000227 | head | grep '#'
# at 4
#250124 15:39:07 server id 1147473732  end_log_pos 123 CRC32 0xc09acde6   Start: binlog v 4, server v 5.7.38 created 250124 15:39:07 at startup
...
# at 268435771
#250206  6:42:49 server id 1147473732  end_log_pos 268435818 CRC32 0xe056957f   Rotate to mysql-bin.000228  pos: 4
# End of log file

The binlog continuity was broken: mysql-bin.000110 dated Dec 3 2024 and mysql-bin.000111 dated Feb 19 2025, indicating a gap.

-rw-r--r--. 1 root root 524336507 Dec  3 13:02 mysql-bin.000109
-rw-r--r--. 1 root root 524288421 Dec  3 16:48 mysql-bin.000110
-rw-r--r--. 1 root root 287303474 Feb 19 13:59 mysql-bin.000111
-rw-r--r--. 1 root root 524308964 Feb 19 14:00 mysql-bin.000112

The older binlog ( mysql-bin.000111 ) started at 241203 16:48:57, meaning OBLogProxy had been parsing previously unprocessed redo logs, so the downstream could not see the latest changes until the backlog was flushed.

The next day, the latest binlog ( mysql-bin.000232 ) contained entries for the test table, confirming that the backlog had been processed.

[root@oblogproxy-server data]# mysqlbinlog -vv mysql-bin.000232 | grep -i "evan_test"
CREATE TABLE `evan_test`(...
#250219 18:10:14 ... INSERT INTO `evan_db`.`evan_test`
#250219 18:10:20 ... INSERT INTO `evan_db`.`evan_test`
... (additional inserts) ...

Business users confirmed that incremental data could now be read correctly.

3 Conclusion

The failure to read incremental data downstream was caused by an OBLogProxy exception that halted clog parsing. After restarting OBLogProxy, it needed time to pull and convert the previously unprocessed clog into binlog, during which the downstream could not read the latest changes.

4 Optimization Measures

Configure monitoring and alerts for OBLogProxy status to detect and resolve issues promptly.

References

[1] Binlog mode: https://www.oceanbase.com/docs/community-oblogproxy-doc-1000000000861455