Inpainting forgery detection and localization
Can a small pretrained forgery detector find AI-inpainted regions it was never trained on?
Summary
Out of the box, PSCC-Net cannot tell edited images from real ones, but it partly finds where the edit is. It works much better on Photoshop edits than on Stable Diffusion edits.
Predicted mask compared with the true edit
- Found: edited area the model marked correctly
- False alarm: marked as edited, but untouched
- Missed: edited area the model didn't mark
Key results
- 0.50Detection ROC-AUC: chance level
- 0.335Localization IoU (threshold 0.2)
- 0.45 vs 0.22IoU on Photoshop vs Stable Diffusion edits
The problem
Inpainting tools such as Stable Diffusion or Photoshop can replace part of a photo so that the edit is hard to see. In a course project on data readiness and AI evaluation, our team of five benchmarked small models for spotting these edits, all under the same protocol. Each model gives two answers: is the image edited (detection), and where (a pixel mask, localization).
I was responsible for PSCC-Net, a CNN with 3.67M parameters built for image manipulation detection and localization. My question was simple: how well does it work out of the box, on a kind of edit it was never trained on?
What I did
- Found and prepared the dataset. I located and downloaded TGIF, which pairs MS-COCO photos with inpainted versions and ground-truth masks, and checked it for data leakage.
- Adapted the official PSCC-Net inference code to work with TGIF and ran it on my laptop CPU: 5,145 images per setting, about 5 seconds per image, with a script that can resume after interruptions.
- Measured detection (accuracy, precision, recall, F1, ROC-AUC) and compared it with a baseline that always answers “edited”.
- Experimented with localization settings: the official mask threshold (0.5) against a lower one (0.2), and an open gate (always produce a mask) against a closed gate (only produce a mask when the detector says “edited”).
- Checked how stable the numbers are with three resamples of the test set.
Results
Detection is at chance level. ROC-AUC is 0.498. Accuracy 0.657 and F1 0.786 look reasonable on their own, but a baseline that always says “edited” reaches F1 0.960, because most test images are edited.
Localization is partly useful. With a mask threshold of 0.2, macro IoU (overlap between the predicted and the real edited area, from 0 to 1) is 0.335, against 0.210 with the official 0.5. PSCC-Net does much better on Photoshop edits (IoU 0.451) than on Stable Diffusion edits (0.218). By object category, IoU ranges from 0.12 (skis) to 0.56 (zebra). Gating the mask on the detector lowers IoU to 0.242.
What I learned
- Accuracy alone is misleading on an imbalanced set. 92% of the test images are edited, so F1 0.786 only made sense next to a baseline that always says “edited” (F1 0.960).
- Detection and localization are different problems. The model can be almost certain an image is edited and still point at the wrong area.
- The type of edit matters most. Generator type (Photoshop vs Stable Diffusion), edit size and object category changed the results more than any setting I tried.