In the United States, the neural network was trained to pick up sounds for silent video

American researchers have developed an algorithm that independently selects sound for silent video

What's happening A group of researchers from Carnegie Mellon University (Pennsylvania, USA) and Runway have created an algorithm for voicing videos: depending on the picture in the frame, the neural network independently selects the necessary sounds. The development was called Soundify. Its work is divided into three stages: first, the algorithm detects the sources of sounds and classifies them — these can be specific objects or places with a characteristic background sound (road, cafe, and so on). The algorithm then uses the Epidemic Sound database, which contains about 90,000 sounds, to search for the desired sound. For each scene, Soundify picks up the five most likely sound effects: one of them is set by default, but the user can turn on additional ones. At the second stage, the algorithm sets the sound time intervals of each effect depending on how long the object is in the frame. In the last stage, the neural network breaks down each scene...