Use AI to sync post production dialogue?

[RKadin]

For numerous reasons (not the least being budgetary), Welles resorted to re-recording dialogue post-filming. Sometimes the results could be distractingly “art housey,” with an actor’s lip movement and the subsequent audio recording going noticeably out of sync, thus intruding on the viewer’s engagement in the scene.

Of course, this issue didn’t arise in his Hollywood studio enabled efforts (Kane, Ambersons, Lady from Shanghai, The Stranger, Touch of Evil [well, much of it, at least]) making viewing them a bit more seamless to broader audiences.

While I appreciate that Welles aficionados can find the post-sync soundtracks part of his independent works’ special charm, part of me wonders what an AI-assisted improved marriage of the sounds and the visuals might accomplish in terms of enhancing their accessibility.

Just curious. Any thoughts?

[Le Chiffre]

I'm impressed by AI, and it's really just getting started. I think AI has some interesting potential in terms of a possible Ambersons restoration, although I'm still very much looking forward to Brian Rose's completion of his own TMA project. AI may be even useful for some of the unfinished Welles works, like THE DEEP.

[RKadin]

Thanks, Le Chiffre.

Ah, yes, the long-awaited TMA rescue project can’t surface soon enough for me, either.

I managed to get Roger Ryan’s ambitious reassembly version into Roger Ebert’s hands many years ago. Among its real strengths was the restoration of Welles’ continuity, including its unsentimental ending.

I hope it led to something for the former, but never heard one way or the other.

[jbrooks]

While I appreciate that Welles aficionados can find the post-sync soundtracks part of his independent works’ special charm, part of me wonders what an AI-assisted improved marriage of the sounds and the visuals might accomplish in terms of enhancing their accessibility.

As you may know, the 1992 "restoration" of "Othello" attempted to re-synch the dialogue (and otherwise update the soundtrack). It was a controversial choice in the world of Welles fandom. Jonathan Rosenbaum has written about it in detail. It was said that the "restoration" actually lengthened some shots to try to make the words match better. I shudder to think what AI might do with the same task.

Back in 1992, I had never seen (or rather heard) the original soundtrack version, so it was impossible for me to judge whether it was an improvement. When I finally saw the original version (on Criterion laserdisc), I was pleasantly surprised to see (hear) that it sounded just about as good as -- as sometimes better than -- the 1992 "restoration." Today, I don't think one can even find the 1992 version.

[Roger Ryan]

... Back in 1992, I had never seen (or rather heard) the original soundtrack version, so it was impossible for me to judge whether it was an improvement. When I finally saw the original version (on Criterion laserdisc), I was pleasantly surprised to see (hear) that it sounded just about as good as -- as sometimes better than -- the 1992 "restoration."...

Part of the problem with the '92 restoration was the decision to re-record the score in stereo which meant that virtually the entire soundtrack needed to be remixed to accommodate the new score. That meant that all the dialogue needed to exist as separate stems... and some of the '55 edit's dialogue did not exist as separate stems. I believe this is the reason for why some lines are missing altogether from the '92 version (one particularly egregious example happens on a medium shot of Iago as Othello arrives at Cypress where we see his mouth speaking a complete sentence that is absent from the soundtrack).