Why Rust Powers oss_pipe: A High‑Performance Cloud File Migration Tool

Motivation for Building oss_pipe

As business grows, file count and size increase dramatically, making large‑scale migration challenging. oss_pipe is a Rust‑written tool to support massive file migration scenarios.

Common OSS Migration Tools

Existing tools include:

ossimport – Alibaba – Java – 16 MB executable

ossutil – Alibaba – Go – auxiliary management client

COS Migration – Tencent – Java – 31 MB

oss_pipe – JD – Rust – 12 MB

Problems with Existing Solutions

Languages with VMs (Java, Go) can cause OOM.

Feature‑rich tools like ossimport require a Java runtime, making deployment cumbersome.

Runtime tuning and JVM optimization increase operational complexity.

Technical Benefits of Rust

Memory safety via ownership, borrowing, and lifetimes, eliminating need for garbage collection.

Performance close to C/C++ with no runtime overhead.

Simplified concurrent programming through the ownership model.

High reliability and productivity thanks to strong typing, pattern matching, and rich tooling.

Cross‑platform support for various OS and hardware.

Modern language features combining functional and generic programming.

Active community and mature ecosystem.

Core Features of oss_pipe

Full‑volume, incremental, and breakpoint‑resume migrations; large‑file split upload; regex filtering; configurable thread count and bandwidth control.

Supports multiple object storage providers: JD Cloud, Alibaba Cloud, Tencent Cloud, Huawei Cloud, AWS, MinIO, etc.

Implementation Overview

Performance Tests

File upload test using JD Cloud enhanced SSD:

<code>task_id: '7171391438628982785'
name: transfer local to oss
task_desc:
  type: transfer
  source: /mnt/ext/t_upload
  target:
    provider: JD
    access_key_id: JDC_xxxxxxx
    secret_access_key: xxxxx61xxx
    endpoint: http://s3-internal.cn-north-1.jdcloud-oss.com
    region: cn-north-1
    bucket: jsw-bucket
  attributes:
    objects_per_batch: 1
    task_parallelism: 8
    max_errors: 1
    meta_dir: /tmp/meta_dir
    target_exists_skip: false
    start_from_checkpoint: false
    large_file_size: 500m
    multi_part_chunk_size: 100m
    multi_part_chunk_per_batch: 20
    multi_part_parallelism: 24
    transfer_type: stock
</code>

Network peak exceeded 1 Gbps, disk I/O near full load, achieving 1.3 TB transfer in 1 689 seconds.

OSS‑to‑OSS Transfer Test

Configuration:

<code>task_id: '7178591798162493441'
name: transfer oss to oss
task_desc:
  type: transfer
  source:
    provider: JD
    access_key_id: JDC_xxxxxx
    secret_access_key: 53A1D2xxxx
    endpoint: http://s3-internal.cn-north-1.jdcloud-oss.com
    region: cn-north-1
    bucket: jsw-bucket
  target:
    provider: JD
    access_key_id: JDC_xxxxx
    secret_access_key: 53Axxxxxxx
    endpoint: http://s3-internal.cn-north-1.jdcloud-oss.com
    region: cn-north-1
    bucket: jsw-bucket-1
  attributes:
    objects_per_batch: 1
    task_parallelism: 4
    max_errors: 1
    meta_dir: /tmp/meta_dir
    target_exists_skip: false
    start_from_checkpoint: false
    large_file_size: 500m
    multi_part_chunk_size: 100m
    multi_part_chunks_per_batch: 10
    multi_part_parallelism: 24
    transfer_type: stock
</code>

1.3 TB internal transfer completed in 2 110 seconds.

Conclusion

oss_pipe is still under development; the author invites collaborators. To address high CPU usage from OSS signature calculations, the Rust AWS S3 SDK was customized for migration scenarios.