Overview and Practical Guide to Debezium MongoDB Source Connector

Debezium’s MongoDB Source connector monitors document change events in MongoDB replica sets or sharded clusters by reading the oplog and persisting those events to Kafka topics, thereby offering a streaming data synchronization solution from MongoDB to Kafka.

The connector works only on replica sets or sharded clusters because it relies on the oplog; standalone MongoDB instances must be converted to a replica set to be compatible.

MongoDB clusters consist of an oplog for immutable change records, replica sets where the primary writes to the oplog and secondaries read from it, and sharded clusters composed of shards (each a replica set), configuration replica sets storing metadata, and mongos routing servers. When a new node joins, it performs an initial snapshot and then streams incremental changes.

Connector operation includes a logical name used as a topic prefix and offset identifier, a snapshot task that captures the current state when no offset is found, and a streaming task that converts oplog create/insert/delete operations into Debezium change events. Topics are named logic_name.database_name.collection_name , and partitioning is based on the message key to ensure ordering.

To deploy, download the Debezium MongoDB connector archive, place its JARs in the Kafka Connect plugin.path , and restart the Connect worker. The connector is started via the REST API with a JSON configuration that defines properties such as mongodb.hosts , mongodb.name , authentication credentials, whitelist/blacklist for databases and collections, snapshot mode, and task limits.

Configuration examples include the Connect worker properties file, the curl command to create the connector, and a table of key connector properties with descriptions. Monitoring is achieved through JMX metrics exposed to Prometheus and visualized in Grafana dashboards.

The FAQ addresses common issues: adjusting the topic name delimiter from "." to "-" in the source code, granting the necessary readAnyDatabase and read roles on the admin and local databases for oplog access, and ensuring the collection.whitelist follows the database.collection format.

References to the official Kafka Connector documentation, Debezium documentation, the connector archive, and a detailed guide on Kafka Connect, REST API, JMX, and Prometheus are provided for further reading.