February update
Added 2024-03-01 16:38:37 +0000 UTCHey everyone,
I'm always disappointed if the end of a month rolls around and I don't have a video for you. I've been chugging along at the animations for the transformers project I mentioned in the last post. It's a big one, and in my current plans, I have it split up into three separate videos, which will be added as chapters to the neural network series I posted several years ago. The aim is to have at least two of those out to you in March, with the third not far behind, so if all goes well the 2024 videos-per-month rate should pop up to be much closer to 1.
---
Much of the reason I'm a bit behind there is that I got further sucked into the translation project I mentioned in the last post. We now have a preliminary interface to enable people to submit corrections/reviews of translations (more details here).
Ever since YouTube disabled community contributions to captions, I've had no good solution for how to let people submit subtitles, so if nothing else it feels nice to have a clear place to point people for that.
The real motivation of the project, though, is to make dubbings easier, and in particular to experiment with AI dubbings. For a sample, take a look at last year's video on Moser's circle problem, click the gear icon, and try out some of the alternate audio tracks.
My feelings on AI dubs have evolved a bit in the last month, though. It reminds me a bit of my feelings towards AI art in 2022. The first impression is a huge wow factor. Look! It's me, but speaking German! Think of what this unlocks! And yes, it is impressive, probably pretty useful, and almost certainly better than having nothing. But once the wow factor wears off, the deviations from what feels human become more noticeable, and also a bit more obnoxious. The intonation is just a bit strange just a little bit too often, and the fact that your brain knows it's not a human limits how friendly it can feel.
By contrast, one of the translation efforts I feel most pleased with is the Japanese channel, something done with care by a community member, and where channel revenue is shared with them. The problem is that doing dubs manually often takes way more time than people expect it will, and too often they don't garner the reach that justifies the time spent.
What about you? What are your feelings about AI-created dubs?
One thought I'm currently toying with is having a system in place where we can create an AI dub as a first pass, but then reuse a lot of the tooling involved in that which syncs the dubbing to the original video to streamline the process of human-narrated dubs. If the dubbing process was smooth enough, taking say an hour for one video rather than a day, I'd be much more comfortable opening applications for community members who might be interested in that and paying them for the efforts.
All of this, from the coordination to the software development involved, would be something I fully intend to hire out for while my own time mainly goes to new videos. To a good extent, that's what I've been doing so far (many thanks to Vince Rubinetti, Jordan Balke, and Shihab Sarar Ahmed). But at least during early experiments, it's a rabbit hole I haven't been able to fully escape.
Comments
That is so interesting! I've been studying Japanese for about 18 months now, and I'm about at that reckoning stage where I've come to understand what a long journey I've started. I watched part of the ?probability 0 impossible? video and man, I get about 20% of that and the speech is fast. I use GPT-4 as kind of a tutor for Japanese, and I know I have to double check the answers sometimes, so...I would say that Japanese is going to be the longest stretch for using AI...first, probably a Japanese native speaker who understands English math terms might beat machine translation. Maybe AI dubbing might work; it'd be freaky to hear Japanese in your voice. Pitch accent is both held up as a necessity....and then someone says don't worry, native speakers get pitch accent wrong all the time. Anyway, that's my non-expert opinion.
Kevin Mitchell
2024-04-02 02:48:28 +0000 UTCHey Grant! Thank you for the update. I hope I'm not repeating a conversation that's already in the comments (did not read all of them), but I just wanted to point out the experience for European viewers that are very used to subtitles (thinking of Scandinavian countries, NL, BE,..) . We are very used to subtitles while the main audio is in English, and I would never watch a dubbed version, regardless of how it was made. For me it all comes down to good human-made subtitles. If those are available, I'll always have those on. If only English is available I'll have that on, as it just helps understanding to both read&listen simultaneously. I would generally skip Auto-Translate to my own language, but it's definitely an option as long as the original English was high quality (not Auto-Translated!). So what I want to stress is that in the end proper human-made English subtitles are still the backbone of all language support. Thank you!
flekkie
2024-03-03 23:22:57 +0000 UTCA follow-up to my previous message: I’ve just submitted a PR for the "better bayes" video french captions. I reviewed the whole (auto-)translated script and tried to make something consistent, so as to maintain a real "talk" feel throughout the video. This was fun, as I literally talked over the original video to try and get a feel of what it would sound like with the prompts I'd written, but it took me around 6 hours in total I think (for a 21 min video). As a whole, the automatic translation really is OK and understandable, but there still are lots of small grammatical flaws — especially when context from previous captions is missing I guess — "compressed" vocabulary (odds and probability both translated to "probabilité" or "chances", which is a bit of an issue !), and it is too formal overall compared to the original. I ended up correcting / reformulating whole sentences more often than not. Cheers !
Raphaël
2024-03-03 14:10:36 +0000 UTCI sent some feedback just yesterday regarding Moser's circle problem test video and I second your feeling about intonation. But I feel that a first AI pass + community contributions through the translating interface connected to github PRs may work well. I'm about to try the new interface right now !
Raphaël
2024-03-01 20:25:58 +0000 UTCIntonation quibbles aside, I don't care what you say: Hearing you speak Hindi is mind-blowing
Don Sanderson
2024-03-01 17:06:08 +0000 UTCIf it is larger language (Spanish, Hindi, Mandarin), thus a large potential audience, then it would be better to hire someone to do the dubbing, using the translated script. However, for smaller languages it could be beneficial to just use an AI-generated dubbing to keep things simpler and probably cheaper. Or just start out with the generated ones and redo them as needed. They are already very useful for people who aren't fluent enough in English to follow abstract mathematical concepts.
Imre Polik
2024-03-01 17:03:03 +0000 UTCA huge boon for the AI dubs is to have your voice narrate. Maybe some kind of manual tuning is possible for the intonation errors you mention? Personally, I would not think it takes away from the "human-ness", after all, it is you behind it in the end.
Martin Manscher
2024-03-01 17:02:28 +0000 UTCYou're amazing, Grant! My offhand notion, as an active Wikimedia volunteer, is that Wikipedia and related projects are deeply involved in both reaching people no matter what language they speak, and also supporting more multi-media content as a way to match people's learning modalities. So I wonder if there might be a good collaboration opportunity between what you're working on and their efforts.
Neal McBurnett
2024-03-01 17:00:04 +0000 UTC
