Gradient Inversion of Multimodal Models

Omri Ben Hemo*1, Alon Zolfi*1, Oryan Yehezkel1, Omer Hofman2, Roman Vainshtein2, Hisashi Kojima3, Yuval Elovici1, Asaf Shabtai1
1Ben-Gurion University of the Negev, Israel | 2Fujitsu Research of Europe | 3Fujitsu Limited, Japan

Abstract

Federated Learning enables training without exposing raw data but remains vulnerable to gradient inversion attacks. We introduce GI-DQA, a pioneering approach that reconstructs sensitive Document QA inputs, demonstrating critical vulnerabilities in multimodal models.

Key Contributions

Novel Attack

First multimodal gradient inversion attack targeting Document QA models.

Hybrid Methodology

Combines visual pixel optimization and analytical QA reconstruction.

Thorough Evaluation

Validated against Donut and LayoutLMv3 using the privacy-aware PFL-DocVQA dataset.

Methodology

GI-DQA leverages leaked gradients and public templates, analytically reconstructing QA tokens, then optimizing visual content to align with client gradients and visual priors.

Methodology Overview

Benign Client

  • Private document xD + question xQ → DQA model
  • Model returns answer y, sends gradients ∇θclient

Attacker Setup

  • Starts from public template ẑD
  • Analytically reconstructs discrete question xQ
  • Computes gradients ∇θadv

Optimization Loop

  • Gradient-matching loss Lgrad(∇θadv, ∇θclient)
  • Adds priors for smoothness, layout preservation, sharp text edges
  • Iteratively updates template pixels until gradients align

Outcome

  • Template morphs into high-fidelity replica, exposing private content

Results

Significantly outperforms existing inversion methods, recovering detailed personal data effectively.

Results Comparison

Resources

GitHub Repository Read Paper Watch Video View Poster