SwiftUI multi-touch and a gesture arbitration story

Motecraft has a few experiences whose entire pitch is multi-finger. In Swarm Drones, one finger forms a ring, two fingers form a line, three a triangle, four-plus a grid. In Curtain each finger was meant to grab a different point on the cloth and drag it independently. The renderers ported across from the POC as single-touch only because the first cut of every experience used SwiftUI's Canvas plus a DragGesture — and DragGesture delivers exactly one touch, no matter how many fingers are down.

This post is the story of the fix, the bug the fix introduced, and the real solution that shipped — including which experiences I rolled back to single-touch on purpose because the multi-touch story wasn't worth the regression risk this close to launch.

The setup

The Motecraft renderer player is a SwiftUI Canvas inside a chrome wrapper. The wrapper owns the controls bar, the tab strip, the share button, the favorite heart. Each experience plugs its own renderer into the wrapper's content slot:

// Simplified — the real wrapper layers a TimelineView, an
// AmbientBackdrop, touch indicators, watermark, and recording capture.
RendererHost {
  experience.body                 // <- the renderer's SwiftUI body
}
.gesture(
  DragGesture(minimumDistance: 0) // chrome-level: tap-to-reveal chrome,
    .onChanged { ... }            // swipe-to-dismiss, etc.
)

That outer DragGesture(minimumDistance: 0) is non-negotiable — without it, taps in the canvas don't toggle the chrome, and swipes between experiences in the feed don't register. So every renderer has to coexist with it.

For single-touch experiences the easy pattern is to put DragGesture on the renderer's own Canvas and let SwiftUI's high-priority arbitration decide who wins (it gives priority to the deepest gesture). That's what Lotus Breath, Mandala Ember, and most of the catalog ship with.

But for the multi-finger renderers, DragGesture simply can't deliver the goods. Apple's docs are explicit: DragGesture is single-touch. To get N fingers you have to drop down to UIKit.

The first attempt — MultiTouchView

The fix was a UIViewRepresentable wrapping a UIView with isMultipleTouchEnabled = true, exposing every finger with a stable ID across frames:

struct MultiTouch: Identifiable, Equatable {
  let id: Int
  let position: CGPoint
}
 
struct MultiTouchView: UIViewRepresentable {
  let onChange: ([MultiTouch]) -> Void
 
  func makeUIView(context: Context) -> MultiTouchTrackingUIView {
    let v = MultiTouchTrackingUIView()
    v.isMultipleTouchEnabled = true
    v.isUserInteractionEnabled = true
    v.backgroundColor = .clear
    v.onChange = onChange
    return v
  }
}

Inside the tracking UIView, the four touches* callbacks maintain an active dictionary keyed by ObjectIdentifier(UITouch) — a stable identity that survives across touchesMoved frames — paired with a monotonic id the renderer can match per-finger state to.

Renderers dropped it into a ZStack alongside their Canvas, set .allowsHitTesting(false) on the Canvas so the UIView received all touches, and read [MultiTouch] callbacks.

This worked. Swarm Drones formed formations. Predator Prey tracked multiple agents. Curtain held multiple grip points.

It also broke, in two different ways.

Bug 1 — the UIView was 0×0

The first symptom: in the first port, Firecracker renderers stopped receiving taps entirely. No fingers, no formations, nothing. They worked perfectly in the test harness. Inside the real player hierarchy, they were dead.

The cause was iOS 16+ UIViewRepresentable sizing. If you don't implement sizeThatFits(_:uiView:context:), SwiftUI defaults the representable to a zero-sized intrinsic content size. When the representable is wrapped in a layout that respects intrinsic size — or in an .overlay() modifier without an explicit frame — the resulting UIView is 0pt × 0pt. UIKit then routes touches to whatever's behind it. From the user's perspective: the renderer doesn't see touches at all.

Fix:

func sizeThatFits(
  _ proposal: ProposedViewSize,
  uiView: MultiTouchTrackingUIView,
  context: Context
) -> CGSize? {
  proposal.replacingUnspecifiedDimensions()
}

Returning the proposed size makes the UIView fill whatever space the parent proposes, regardless of where it's nested. This is one of those fixes where the code change is four lines and the debug session was four hours.

Bug 2 — the outer DragGesture stole the first touch

The second symptom was more interesting and shipped further before being caught. On iOS 26, the renderers that had been working suddenly started behaving like single-touch was sticky: the first finger landed correctly, but subsequent finger-downs didn't trigger touchesBegan on the UIView. The user saw one drag working, the rest of the gesture going nowhere.

The cause was upstream of the UIView. Remember the chrome wrapper's DragGesture(minimumDistance: 0)? In iOS 26, SwiftUI's DragGesture is backed by a UIPanGestureRecognizer that aggressively claims ownership of the touch stream the moment a finger lands. It would fire gestureRecognizerShouldBegin before the inner MultiTouchTrackingUIView got touchesBegan — and once the outer pan claimed the touch, the UIView never saw the rest of it.

The fix is on the UIView side. UIKit lets a view veto pan recognizers it's nested inside:

override func gestureRecognizerShouldBegin(
  _ gestureRecognizer: UIGestureRecognizer
) -> Bool {
  guard gestureRecognizer is UIPanGestureRecognizer else {
    return super.gestureRecognizerShouldBegin(gestureRecognizer)
  }
  // UIScrollView pan (feed navigation): allowed when no touches are
  // active so the user can still swipe between experiences;
  // vetoed once a touch is in progress so a mid-draw flick can't
  // kick them to the next item.
  if gestureRecognizer.view is UIScrollView {
    return active.isEmpty
  }
  // All other pan recognizers (chrome DragGesture, etc.): always vetoed.
  return false
}

Two-tier veto. Scroll-view pans are allowed unless the user is mid-draw — so feed swiping still works when the canvas is idle. Every other pan (including the chrome's DragGesture(minimumDistance: 0)) is vetoed unconditionally, because if the UIView is in the hierarchy at all, it's claiming the touch surface.

After this fix, Drones, Predator/Prey, Bees, and the rest of the multi-finger swarm pack worked. The chrome tap-to-reveal still worked, because tap is handled by a different recognizer than the backing pan. Feed swiping still worked. Multi-touch was real.

The bug I shipped anyway, and the rollback

Two experiences still didn't behave: Curtain and Ragdoll. Both put MultiTouchView in a losing position by layout — Curtain's Canvas was hit-test-disabled and Ragdoll wrapped its UIView in an .overlay() — and the gesture arbitration still managed to drop the first touch in those two cases, even after the veto. Other MultiTouchView users got lucky with their layouts (plain ZStack, UIView above Canvas, no overlay).

The root cause was specific to those two layouts and would need a custom UIGestureRecognizer subclass with its own delegate to handle properly. With v1 launch a couple of weeks out, the trade-off was:

• Single-touch Curtain and Ragdoll, lose multi-touch grip points — Curtain's core gesture is "grab the cloth and drag it." Ragdoll's is "yank a limb." Both are completely satisfying with one finger.
• Hold the line for multi-touch, risk shipping a broken first-touch on two experiences — first-touch breakage is the worst kind of bug. The user taps, nothing happens, they conclude the app is broken and uninstall.

I picked the single-touch rollback for both. The renderers reverted to the POC's pattern: a SwiftUI DragGesture(minimumDistance: 0) directly on the Canvas, with .contentShape(Rectangle()) so empty pixels still register as hits.

Canvas { ctx, _ in
  // … verlet step, draw cloth …
}
.contentShape(Rectangle())
.gesture(
  DragGesture(minimumDistance: 0)
    .onChanged { v in
      if dragged == nil {
        dragged = closest(to: v.startLocation)
      }
      if let d = dragged, pts.indices.contains(d), !pts[d].pinned {
        pts[d].x = v.location.x; pts[d].y = v.location.y
        pts[d].prevX = v.location.x; pts[d].prevY = v.location.y
      }
    }
    .onEnded { _ in dragged = nil }
)

Single-touch wins arbitration cleanly because it's the renderer's own gesture (deepest in the hierarchy beats outer). One drag, one cloth point, no UIKit. The diff also caught a separate bug where Curtain's sliders (cols, rows, wind, gravity) were declared in the schema but hardcoded as let constants in the renderer — the controls bar showed them but they did nothing. Wiring them through ControlsState rebuilt the mesh on cols/rows change via a topology-key check.

What I'd do differently next time

Two things, in order:

1. Treat single-touch and multi-touch as separate renderer base types from day one. Trying to make MultiTouchView the default for everything was the architectural mistake. Most of the catalog is single-touch and benefits from DragGesture's clean SwiftUI integration. Only the swarm pack and a couple of cloth experiences genuinely need N fingers. Picking the right primitive per renderer is cheaper than building a single primitive that handles both.
2. Test gesture arbitration in the real player wrapper, not the harness. The bugs only show up when the outer chrome DragGesture(minimumDistance: 0) is in the hierarchy. The test harness without chrome was a poor proxy.

The Compose / Android port had none of this drama, because Compose's pointerInput { awaitPointerEventScope { … } } already delivers multi-touch as a list of PointerInputChange per frame. The mistake to avoid on that side is calling ev.changes.first() — which silently drops you back to single-touch — instead of iterating ev.changes.filter { it.pressed }.map { it.position }. Different platform, different trap door.

If you've shipped something similar — a SwiftUI gesture story or a Compose pointer-input one — I'd love to read it. Mail me at support@motecraftstudio.com.

The full catalog of 261 experiences is live on iOS and Android. The multi-touch swarms are in the Swarm category. Curtain and Ragdoll are single-touch, on purpose, for now.