Implementing iOS 16 Depth Effect Using Image Segmentation with CoreML and Vision

iOS 16 introduces a striking lock‑screen visual called Depth Effect , which creates a sense of depth by separating foreground and background elements of a single image. This article shows how to reproduce the effect inside an app using image‑segmentation techniques rather than a dedicated UIKit control.

Traditional segmentation methods such as the Watershed algorithm require manual seed points and are not suitable for fully automatic processing. Modern machine‑learning models, like Apple’s pre‑trained DeeplabV3 , can perform automatic segmentation of an image into multiple classes.

The DeeplabV3 model can be downloaded from Apple’s Machine Learning website. After adding the .mlmodel file to an Xcode project, Xcode generates a Swift class named DeepLabV3 . The model expects a 513 × 513 RGB image and outputs a 513 × 513 Int32 matrix where each pixel value indicates its class (0 = background, non‑zero = foreground).

lazy var model = try! DeepLabV3(configuration: {
    let config = MLModelConfiguration()
    config.allowLowPrecisionAccumulationOnGPU = true
    config.computeUnits = .cpuAndNeuralEngine
    return config
}())

Using the generated model, create a VNCoreMLRequest to run the segmentation on a given image. The request’s completion handler receives an array of VNCoreMLFeatureValueObservation objects, from which the multiArrayValue contains the segmentation mask.

lazy var request = VNCoreMLRequest(model: try! VNCoreMLModel(for: model.model)) { [unowned self] request, error in
    if let results = request.results as? [VNCoreMLFeatureValueObservation],
       let feature = results.first?.featureValue,
       let arrayValue = feature.multiArrayValue {
        // The mask is stored in arrayValue
        let width = arrayValue.shape[0].intValue
        let height = arrayValue.shape[1].intValue
        let stride = arrayValue.strides[0].intValue
        // ...process mask...
    }
}

To execute the request, instantiate a VNImageRequestHandler with the resized image (513 × 513) and call perform(_:) . The handler and request should run on a background thread to avoid blocking the UI.

private func segment() {
    if let image = self.imageView.image {
        imageSize = image.size
        DispatchQueue.global().async { [unowned self] in
            self.request.imageCropAndScaleOption = .scaleFill
            let handler = VNImageRequestHandler(cgImage: image.resize(to: .init(width: 513, height: 513)).cgImage!)
            try? handler.perform([self.request])
        }
    }
}

Note: The request callback runs on the same thread that invokes the handler, so it is advisable to dispatch the whole operation to a background queue. Set imageCropAndScaleOption to .scaleFill ; otherwise the handler will crop the central region of the image, producing unexpected results.

After obtaining the arrayValue , convert it to a binary mask (background = transparent, foreground = opaque) and overlay it on the original picture. By stacking the original image, UI components, and the mask view, the final depth‑effect UI is achieved.

The approach works well for portraits but may fail on complex scenes such as landscapes. The complete demo source code is available on GitHub.