Can AI Protect Privacy and Still Explain Its Decisions?

When we train models on sensitive data, we often ask how much accuracy we lose by protecting privacy. But what happens to the explanations?

This is the question behind our new paper, Pushing the (Decision) Boundaries, with Michael Khavkin and Eran Toch, and with Kichang Lee, Jaeho Jin, and JeongGil Ko from Yonsei University.

We study federated learning, where organizations train a shared model while keeping their data locally. Keeping data local is not enough: model updates can still expose information. Differential privacy helps by adding noise during training. That noise can also affect the explanations we use to inspect a model’s predictions.

Our approach, XCal-FL, uses explanation-related signals to adjust the noise during training. We mask regions highlighted by the model’s heatmap and check how its prediction changes. We also measure how concentrated the heatmap is. Together, these signals guide how much noise to add at each step, subject to a privacy budget. The signals themselves receive privacy protection too.

We evaluated the method on blood cell classification, pneumonia detection, and melanoma classification. In one blood cell experiment, F1 improved from 0.654 to 0.716, while explanation fidelity improved roughly fivefold over fixed-noise training. The gains varied across tasks, and our method did not outperform the alternatives everywhere. We found it interesting that accuracy and explanation quality behaved differently during training. Continuing to train could improve predictions without improving explanations. Accuracy alone would miss that.

The paper suggests that explanation quality needs its own place in evaluating private AI. It also offers a way to use explanations during training, rather than only inspecting them afterward.