In my last article I implemented a spoiler feature from Telegram. The team at Drawww was inspired by it and asked me to recreate a new feature — removing background from images. Luckily, it's very easy with a new iOS 17 API — VNGenerateForegroundInstanceMaskRequest. Fasten your seatbelts, we're ready to start!

Preparations

First, we create an image and a button:

struct ContentView: View {

    @State private var image = UIImage.cat

    var body: some View {
        VStack {
            Image(uiImage: image)
                .resizable()
                .scaledToFit()
            Button("Create a sticker") {
                createSticker()
            }
        }
        .padding()
    }

    // MARK: - Private

    private func createSticker() {
        // Main logic
    }
}

I added a photo of my lovely cat Trisha to the app bundle, and here is the result:

Handlers and requests

To remove the background from the image, we will generate a mask image and apply it to the original image. Let's create a CIImage, we will use it further:

private func createSticker() {
    guard let inputImage = CIImage(image: image) else {
        print("Failed to create CIImage")
        return
    }
}

Next, let's create a new subjectMaskImage(from:) function, that returns a mask image with foreground objects:

private func subjectMaskImage(from inputImage: CIImage) -> CIImage? {
}

To work with the inputImage, we will use the Vision framework and two main classes:

• VNImageRequestHandler allows to perform image analysis requests pertaining to a single image;
• VNGenerateForegroundInstanceMaskRequest is a special request that generates an instance mask of noticable objects.

The next step is to perform the request:

// At the top of ContentView
import Vision

// in the subjectMaskImage(from:) function
let handler = VNImageRequestHandler(ciImage: inputImage)
let request = VNGenerateForegroundInstanceMaskRequest()
do {
    try handler.perform([request])
} catch {
    print(error)
    return nil
}

The requests may fail. For example, handler throws an error if you try to run the code on an iOS Simulator:

Error Domain=com.apple.Vision Code=9 "Could not create inference context" UserInfo={NSLocalizedDescription=Could not create inference context}

The result of the requests is an array of VNInstanceMaskObservation objects. We need to check them, take the first one and generate the mask:

guard let result = request.results?.first else {
    print("No observations found")
    return nil
}
do {
    let maskPixelBuffer = try result.generateScaledMaskForImage(forInstances: result.allInstances, from: handler)
    return CIImage(cvPixelBuffer: maskPixelBuffer)
} catch {
    print(error)
    return nil
}

Now we're ready to call the function and get the mask image:

private func createSticker() {
    guard let inputImage = CIImage(image: image) else {
        print("Failed to create CIImage")
        return
    }
    guard let maskImage = subjectMaskImage(from: inputImage) else {
        print("Failed to create mask image")
        return
    }
}

This is what we got:

Applying mask

The mask is ready, therefore we're ready to apply it to the original image. With the help of CIFilter.blendWithMask(), we can generate a final image:

// At the top of ContentView
import CoreImage.CIFilterBuiltins

private func apply(mask: CIImage, to image: CIImage) -> CIImage {
    let filter = CIFilter.blendWithMask()
    filter.inputImage = image
    filter.maskImage = mask
    filter.backgroundImage = CIImage.empty()
    return filter.outputImage!
}

Finally, we render the result to a UIImage:

private func render(ciImage: CIImage) -> UIImage {
    guard let cgImage = CIContext(options: nil).createCGImage(ciImage, from: ciImage.extent) else {
        fatalError("Failed to render CGImage")
    }
    return UIImage(cgImage: cgImage)
}

Put it all together and check the result! Note that performing the requests is a time-consuming operation. So we'll add a separate queue to prevent main thread blocking:

// In ContentView
private var processingQueue = DispatchQueue(label: "ProcessingQueue")

private func createSticker() {
    guard let inputImage = CIImage(image: image) else {
        print("Failed to create CIImage")
        return
    }
    processingQueue.async {
        guard let maskImage = subjectMaskImage(from: inputImage) else {
            print("Failed to create mask image")
            DispatchQueue.main.async {
                isLoading = false
            }
            return
        }
        let outputImage = apply(maskImage: maskImage, to: inputImage)
        let image = render(ciImage: outputImage)
        DispatchQueue.main.async {
            self.image = image
        }
    }
}

That's it! But the transition between states looks a bit boring, let's add some animations and magic particles.

Animation and particles 💅

We create a StickerView with two bindings: an original image and a generated sticker.

import SwiftUI

struct StickerView: View {

    @Binding var image: UIImage
    @Binding var sticker: UIImage?

    var body: some View {
        ZStack {
            originalImage
            stickerImage
        }
    }

    // MARK: - Private

    @ViewBuilder
    private var originalImage: some View {
        // Layout for image
    }

    @ViewBuilder
    private var stickerImage: some View {
        // Layout for sticker
    }

When the sticker is set, we'll add some animations:

• Hide the image;
• Bounce the sticker via scale changes;
• Show the particles.

Starting with the original image, we add basic modifiers and an opacity animation based on the sticker's state:

Image(uiImage: image)
   .resizable()
   .scaledToFit()
   .opacity(sticker == nil ? 1 : 0)
   .animation(.easeOut(duration: 1), value: sticker)

Looks better! Now we add the sticker image layout and scale it for highlighting:

// In StickerView
@State private var stickerScale: Double = 1

@ViewBuilder
private var stickerImage: some View {
    if let sticker {
        Image(uiImage: sticker)
            .resizable()
            .scaledToFit()
            .scaleEffect(stickerScale)
            .onAppear {
                withAnimation(.easeOut(duration: 1)) {
                    stickerScale = 1.1
                } completion: {
                    withAnimation(.easeOut(duration: 1)) {
                        stickerScale = 1
                    }
                }
            }
    }
}

It remains only to add the particles. We'll reuse a SpoilerView from the previous article and add it as an overlay for originalImage:

// In StickerView
@State private var spoilerViewOpacity: Double = 0

// Add to originalImage
.overlay {
    SpoilerView(isOn: true)
        .opacity(spoilerViewOpacity)
}

// Update stickerImage
.onAppear {
    withAnimation(.easeOut(duration: 1)) {
        spoilerViewOpacity = 1
        stickerScale = 1.1
    } completion: {
        withAnimation(.linear) {
            spoilerViewOpacity = 0
        }
        withAnimation(.easeOut(duration: 1)) {
            stickerScale = 1
        }
    }                    
}

Now we can see the final result:

If you want to see the final code, check out StickerViewExample project on Github. I added small changes like a loading state. Feel free to share your thoughts, ask questions and suggest improvements on Twitter. Thank you for reading!

References

• Lift subjects from images in your app, WWDC 2023
• Applying visual effects to foreground subjects