New AI Dupes Humans into Believing Synthesized Sound Effects Are Real

Slashdot reader shirappu writes: A study published on June 25th in IEEE Transactions on Multimedia looked at creating an automated program to analyze the movements in video frames and create sound effects to match the scene. In movies and television, this work is called Foley and is considered an important part of crafting the experience, but is also time consuming and sometimes costly. The AI Foley system created in the study works by extracting image features from video frames to determine appropriate SFX. It then analyzes the action taking place in the video, and attempts to synthesize SFX to match what is happening in the video. In a survey where students were shown automated sound effects, 73% though the automated effects were more genuine than the original sound effects. It’s worth noting that the automated system performed best when timing was less important (e.g. general weather) versus when timing was key (e.g. typing, thunderstorms).