Rolling Upgrade of HDFS from 2.7 to 3.2: Experience, Issues and Solutions

Hadoop 3.0 was released at the end of 2017 and the latest stable version is 3.2.1. Hadoop 3 introduces many useful new features such as Erasure Coding, multiple NameNodes, Router‑Based Federation, Intra‑datanode balancer and numerous performance optimizations and bug fixes.

The most attractive feature for the team is Erasure Coding, which can reduce the number of storage replicas while keeping data reliability, aligning with the company’s cost‑reduction goals. Additional bugs and optimizations that have already been fixed in the community also motivated the upgrade.

Two upgrade methods are available: Express (stop the service, start with the new version) and Rolling (upgrade components one by one without stopping the service). For large offline clusters, Rolling is preferred because it avoids user impact.

Industry vendors (Cloudera, Hortonworks) still recommend Express for a 2.x → 3.x migration, and the Apache Hadoop Wiki lists several unresolved issues for Rolling upgrades, the most critical being EditLog incompatibility.

Key issues encountered during testing:

EditLog incompatibility when the Active NameNode (3.2) writes EC‑related structures that a Standby NameNode (2.7) cannot read.

Fsimage incompatibility because EC information is written into the Fsimage by a 3.2 NameNode, which a 2.7 NameNode cannot load.

StringTable changes causing Fsimage incompatibility, solvable by rolling back the offending commit.

Authentication password mismatch (ISSUE HDFS‑14509) due to internal username/password mechanisms, requiring an intermediate upgrade to the latest 2.x version.

Solutions applied:

When generating EditLog or Fsimage, check whether the effective version supports EC; if not, omit EC information.

Use the fallback Fsimage generated before the upgrade to recover if needed.

Rollback the problematic commit for the StringTable issue, then perform a minor version upgrade after the rolling upgrade stabilises.

Upgrade to the latest 2.x release first to align authentication mechanisms before proceeding to 3.x.

Rolling upgrade procedure (summarised from the official Hadoop documentation):

1. JournalNode upgrade – restart each JournalNode with the new version

2. NameNode upgrade
   2.1 Prepare – generate fallback fsimage
   2.2 Restart Standby NameNode with new Hadoop, restart ZKFC
   2.3 Failover – make the upgraded NameNode Active
   2.4 Restart the other NameNode, restart ZKFC

3. DataNode upgrade – restart all DataNodes with the new Hadoop

4. Finalize – confirm the cluster is now at version 3.2

Testing and production rollout timeline:

1. ~May 2019 – multiple upgrade rehearsals on the full Hadoop, Hive, Spark stack; solution validated.
2. July 2019 – offline small cluster (≈100 nodes) upgraded to 3.2 with no user impact.
3. Oct 2019 – offline medium cluster (several hundred nodes) upgraded to 3.2 with no user impact.
4. End Nov 2019 – offline large cluster (thousands of nodes) upgraded to 3.2 with no user impact.

During the upgrade, DataNode deletion of blocks is performed by moving blocks to a Trash directory rather than immediate removal. A custom tool was developed to archive and purge Trash files older than one day, reducing storage pressure during the three‑week upgrade window.

Result: The team successfully completed a rolling upgrade from HDFS 2.7 to 3.2 on large‑scale clusters, achieving improved stability, performance and cost efficiency while staying at the forefront of industry practice.