Implementing a Webpack Plugin for Incremental CDN Upload with Caching

In modern web development, uploading static assets to a CDN (Content Delivery Network) is a common optimization that can significantly improve resource loading speed. However, as project size grows, the number and size of static assets increase, making it time‑consuming to upload all assets on every build. This article describes how to create a Webpack plugin that uploads build outputs to a CDN on the fly, replaces the original relative paths with CDN URLs, and uses a cache to avoid redundant uploads.

Basic Idea

Uploading every static asset to the CDN on each build wastes time and bandwidth, especially for large projects. By introducing a cache, the plugin checks whether a local cache entry already exists before uploading. If the cache is present, the upload is skipped and only the path replacement is performed, reducing build time and network usage.

Implementation Steps

1. Create a Webpack plugin with a custom asset‑lookup scheme

Create a plugin named webpack-upload-static-to-cdn . The plugin adds a custom resource‑lookup function to the generated Webpack code and replaces __webpack_require__.p with the new logic.

const ASSET_LOOKUP_DEF = `
;(function () {
  __webpack_require__.__webpack_asset_map__ = 1;
  __webpack_require__.__asset__ = function (path, wR) {
    return __webpack_require__.__webpack_asset_map__[path] || (wR.p + path);
  };
})();`;

compiler.hooks.compilation.tap(pluginName, (compilation) => {
  // 在编译时的钩子
  compilation.hooks.processAssets.tap(
    {
      name: pluginName,
      stage: Compilation.PROCESS_ASSETS_STAGE_OPTIMIZE, // 优化阶段
    },
    (assets) => {
      // 往js里面插东西，处理 `__webpack_require__.p`
      Object.entries(assets).forEach(([filename, source]) => {
        if (filename.endsWith('.js')) { // 如果是js文件
          try {
            const { source, map } = source.sourceAndMap(); // 获取源和映射
            // replacePublicPath很关键，处理webpack中的路径__webpack_require__.p，替换为读取一个对象中cdn地址
            const newSource = new SourceMapSource(replacePublicPath(source as string), filename, map); // 替换公共路径
            compilation.updateAsset(filename, newSource); // 更新资产
          } catch (e) {}
        }
      });
    }
  );
});

The replacePublicPath function rewrites occurrences of __webpack_require__.p and transforms expressions like (__webpack_require__.p + path) into __webpack_require__.__asset__(path, __webpack_require__) , which then resolves the CDN URL from a shared urlMap .

// In Webpack generated code, add and use a custom resource lookup function __webpack_require__.__asset__
const ASSET_LOOKUP_DEF = `
;(function () {
  __webpack_require__.__webpack_asset_map__ = 1;
  __webpack_require__.__asset__ = function (path, wR) {
    return __webpack_require__.__webpack_asset_map__[path] || (wR.p + path);
  };
})();
`;

/**
 * Replace public path in generated code with custom asset lookup
 * @param str 
 * @returns 
 */
export const replacePublicPath = function (str: string) {
  return (
    str
      // __webpack_require__.p is Webpack's public path variable. Insert custom lookup here.
      .replace(
        /__webpack_require__\.p\s*=\s*["']/g,
        (m) => `${ASSET_LOOKUP_DEF}\n${m}`
      )
      // Replace patterns like (__webpack_require__.p + path)
      .replace(
        /(?:\(__webpack_require__\.p\s*\+\s*)([^\n]+?)\)(;?)$/gm,
        (_, g1, g2) => `__webpack_require__.__asset__(${g1}, __webpack_require__)${g2}`
      )
      .replace(
        /(?:__webpack_require__\.p\s*\+\s*)([^\n]+?)(;?)$/gm,
        (_, g1, g2) => `__webpack_require__.__asset__(${g1}, __webpack_require__)${g2}`
      )
  );
};

2. Create a shared urlMap

A shared urlMap object stores the CDN URLs of all uploaded static assets. After each successful upload, the map is updated so that subsequent builds can quickly retrieve the CDN address.

3. Update urlMap with CDN URLs

When a static asset is uploaded successfully, its original relative path becomes the key and the returned CDN URL becomes the value in urlMap . This enables fast lookup in later builds.

compiler.hooks.afterEmit.tapPromise(pluginName, async (compilation) => {
  // ... omitted code
  // Create URL map, crucial for replacing paths after upload
  const urlMap = new Map
();
  const uploadFile = async (
    // 上传文件的异步函数
    name: string,
    content: string | Buffer,
    shouldOverwrite?: boolean // 是否覆盖的选项
  ) => {
    const fileLocation = stats.outputPath + '/' + name; // 文件位置
    const url = await uploadContent({
      file: name,
      fileLocation,
      content,
      extname: extname(name), // 获取文件扩展名
    });
    if (url && typeof url === 'string') { // 如果URL有效
      urlMap.set(name, url); // 将URL添加到映射中
    }
  };

  // ... omitted resource upload, replacement

  // Parallel upload of style files
  await Promise.all(
    Array.from(styleNames).map((name) =>
      uploadFile(
        name,
        replaceCSSUrls(name, assetMap.get(name) as string, urlMap), // 替换CSS URL并上传
        true // 设置为覆盖
      )
    )
  );

  // ... omitted html upload, replacement

  // ... omitted other upload, replacement code
});

4. Add a cache mechanism

To further optimize the upload process, a cache.json file is created. Before uploading each static asset, the plugin checks whether the hash stored in the cache matches the current file hash. If the hashes match, the upload is skipped; otherwise, the file is uploaded and the cache is updated.

/**
 * compatible API for cdn when enable cache
 * @param {Cdn} cdn
 * @param {object=} option
 * @param {object=} option.passToCdn passToCdn needs to be saved
 * @param {string=} option.cacheLocation where to put cache file
 * @returns {Cdn}
 */
const compatCache = (cdn, option = {}) => {
  // init to save option
  Cache.init(option)
  const upload = async (files) => {
    const { toUpload, pairFromCache, localHashMap } = files.reduce(
      (last, file) => {
        const fileContent = read(file)
        // using relative location so cache could be shared among developers
        const relativeLocation = path.relative(__dirname, file)
        const locationHash = Cache.getHash(relativeLocation)
        const hash = Cache.getHash(fileContent)
        if (Cache.shouldUpload(hash, locationHash)) {
          return Object.assign(last, {
            toUpload: last.toUpload.concat(file),
            localHashMap: Object.assign(last.localHashMap, {
              [file]: locationHash + hash,
            }),
          })
        }
        return Object.assign(last, {
          pairFromCache: Object.assign(last.pairFromCache, {
            [file]: Cache.getUrl(locationHash + hash),
          }),
        })
      },
      {
        localHashMap: {},
        toUpload: [],
        pairFromCache: {},
      }
    )
    const res = toUpload.length
      ? await cdn.upload(toUpload)
      : await Promise.resolve({})

    // new pair to cache
    const newPair = Object.entries(res).reduce((_, [localPath, cdnUrl]) => {
      const hash = localHashMap[localPath]
      return Cache.update(hash, cdnUrl)
    }, {})
    // update cache
    Cache.end(newPair)
    sourceCount.cacheTotal += Object.keys(pairFromCache).length
    sourceCount.filesTotal += files.length
    return Object.assign(res, pairFromCache)
  }
  return {
    upload,
    getSourceCount,
  }
}

Result Comparison

In a large project with 309 small files (≈5 KB each), the build time without caching was about 40 seconds. After adding the cache, the build time dropped to 17 seconds.

Without cache:

With cache:

Conclusion

Implementing a caching mechanism in Webpack can dramatically improve the efficiency of uploading static assets to a CDN. It reduces build time and saves network bandwidth, which becomes increasingly important as projects grow. The ideas and steps presented here aim to help developers apply caching in real projects to boost development efficiency.