Understanding Full-Link Tracing in OceanBase: Concepts, Mechanisms, and Usage

Full-Link Tracing in OceanBase

OceanBase is a distributed database whose request chains can be complex. When a timeout occurs, it is difficult to pinpoint whether the problem lies in an internal component or the network. The full‑link tracing feature introduced in OceanBase 4.x records detailed information for each request, helping operators quickly locate the problematic stage.

1. What is Full‑Link Tracing?

Full‑link tracing records the SQL request’s execution across all components and stages, visualising the path and enabling rapid diagnosis of component‑level issues.

2. Working Principle

2.1 Trace Logs

The system generates trace log entries that contain the total execution time of a SQL statement. Depending on the access method, logs are written to obproxy_trace.log (for ODP access) and/or trace.log on the OBServer.

2.2 Trace Process

A complete request trace (a “Trace”) consists of multiple spans. Each span represents a processing step and is linked by parent_id and id . Tags store additional details such as the executed SQL text, session ID, and plan‑cache hits.

2.2.1 Common Span Types

Client‑side spans: obclient , JDBC

OBProxy spans: ob_proxy , ob_proxy_partition_location_lookup , ob_proxy_server_process_req

OBServer spans: com_query_process , sql_compile , sql_execute , px_schedule , px_task , etc.

2.2.2 Common Tag Types

com_query_entry : log_trace_id , err_code

sql_compile : sess_id , sql_text , sql_id , hit_plan

px_schedule : dfo_id , used_worker_cnt , qc_id

px_task : task_id , dfo_id , sqc_id , qc_id , group_id

3. How to View Trace Information

3.1 Command‑Line (show trace)

Supported on OceanBase 4.2+ and ODP/JDBC/OBClient versions listed in the requirements. Enable the feature by setting ob_enable_show_trace='on' and the protocol flag enable_ob_protocol_v2_with_client=true . Example:

alter proxyconfig set enable_ob_protocol_v2_with_client=true;
alter proxyconfig set enable_ob_protocol_v2=true;
export ENABLE_PROTOCOL_OB20=1;
export ENABLE_TRACE=1;
set ob_enable_show_trace='on';
show trace;

The output shows each span with its start time and elapsed time; the span with the highest ElapseTime indicates the bottleneck.

3.2 OCP Visualisation

OCP provides a UI to query traces by time, Trace ID, or SQL ID. It requires an OpenSearch cluster for storage. The article includes Docker commands to deploy OpenSearch, generate root certificates, and configure OCP system parameters.

3.2.1 Deploy OpenSearch

docker pull oceanbase/opensearch:3.3.2
# generate root‑ca
openssl genrsa -out ./root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key ./root-ca-key.pem -subj "/C=CA/ST=ONTARIO/L=TORONTO/O=ORG/OU=UNIT/CN=ROOT" -out ./root-ca.pem
# run container
docker run -d --net host -v /opt/key/root-ca-key.pem:/opt/opensearch/config/root-ca-key.pem -v /opt/key/root-ca.pem:/opt/opensearch/config/root-ca.pem ... oceanbase/opensearch:3.3.2

After configuring the trace‑query parameters in OCP and restarting the container, users can enable tenant‑level tracing via DBMS_MONITOR procedures, query the configuration table GV$OB_FLT_TRACE_CONFIG , and view detailed span/tag information in the UI.

4. Frequently Asked Questions

Q1: Why does show trace not display the obclient span?

A: The ODP flag enable_ob_protocol_v2_with_client was not enabled. Enabling it makes the client span appear.

Q2: OpenSearch Docker fails with “Can’t open /opt/opensearch/config/root-ca.pem”.

A: The root certificate generation step was omitted. Generate the certificate as shown above and mount it into the container.

Conclusion

The article explains what full‑link tracing is, its internal mechanics, how to enable it in OceanBase 4.x, and how to view the collected data via command line or OCP. Using this feature dramatically reduces the time needed for DBAs to diagnose slow‑query problems.