this post was submitted on 09 Jul 2025
164 points (100.0% liked)

Selfhosted

49336 readers
538 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

I think a lot of people have heard of OpenAI’s local-friendly Whisper model, but I don’t see enough self-hosters talking about WhisperX, so I’ll hop on the soapbox:

Whisper is extremely good when you have lots of audio with one person talking, but fails hard in a conversational setting with people talking over each other. It’s also hard to sync up transcripts with the original audio.

Enter WhisperX: WhisperX is an improved whisper implementation that automatically tags who is talking, and tags each line of speech with a timestamp.

I’ve found it great for DMing TTRPGs — simply record your session with a conference mic, run a transcript with WhisperX, and pass the output to a long-context LLM for easy session summaries. It’s a great way to avoid slowing down the game by taking notes on minor events and NPCs.

I’ve also used it in a hacky script pipeline to bulk download podcast episodes with yt-dlp, create searchable transcripts, and scrub ads by having an LLM sniff out timestamps to cut with ffmpeg.

Privacy-friendly, modest hardware requirements, and good at what it does. WhisperX, apply directly to the forehead.

top 29 comments
sorted by: hot top controversial new old
[–] Squizzy@lemmy.world 2 points 14 hours ago

Just finished a thesis, I used OtterAI which was user friendly and expensive. It got the job done but required some revisions and corrections.

[–] Bluefruit@lemmy.world 4 points 19 hours ago

Man where was this post when in was DMing? lol.

This is super cool though. Rn I'm doing some film editing work for my friend, and this could probably be useful for subtitles too. Thanks for sharing.

[–] homesweethomeMrL@lemmy.world 24 points 1 day ago (1 children)

I’ve also used it in a hacky script pipeline to bulk download podcast episodes with yt-dlp, create searchable transcripts, and scrub ads by having an LLM sniff out timestamps to cut with ffmpeg.

This is genius. Could you appify this and I'll pay you in real or pretend currency as you prefer

I’ve found it great for DMing TTRPGs — simply record your session with a conference mic, run a transcript with WhisperX, and pass the output to a long-context LLM for easy session summaries. It’s a great way to avoid slowing down the game by taking notes on minor events and NPCs.

Okay that's just crazy. ;)

[–] jlh@lemmy.jlh.name 6 points 1 day ago* (last edited 1 day ago)

Probably not that hard to build a simple flask frontend around it.

Automatically processing files in an S3/WebDAV directory would also be useful.

[–] wheezy@lemmy.ml 6 points 1 day ago (1 children)

Nice. I learned about different applications of whisper because I'm a degenerate.

https://github.com/meizhong986/WhisperJAV

[–] RaivoKulli@sopuli.xyz 1 points 16 hours ago (1 children)

Can't say I've ever wanted to turn on the subtitles for porn lol

[–] wheezy@lemmy.ml 1 points 15 hours ago

Sometimes in JAV you really just get curious what the fuck is happening.

[–] irmadlad@lemmy.world 8 points 1 day ago (4 children)

What would be some use cases for WhisperX? I'm struggling to envision how I would use that in a selfhosting/homelabbing environment.

[–] fatalicus@lemmy.world 3 points 1 day ago

I'm personally looking at setting up whisper or whisperx with bazarr, to get subtitles for movies and series that I can't find any to download.

[–] TheFogan@programming.dev 13 points 1 day ago* (last edited 1 day ago) (2 children)

half sarcastic but the overall premise of rigging something in to a local voice assistant, when an arguement starts "Ok nabu record this conversation". then 2 weeks later on another arguement... "OK nabu search our last arguement for the cabinet". Would be like having a court transcriber on call.

[–] irmadlad@lemmy.world 3 points 1 day ago (1 children)

I have a lady friend that does quite a good enough job of that. LOL

'You remember back in 1979......it was a Friday at 2:11 PM, and you said.......' 'Babe, I don't remember what I had for breakfast yesterday.'

[–] onslaught545@lemmy.zip 3 points 1 day ago (1 children)

Does she do it for her fuckups, though?

[–] irmadlad@lemmy.world 3 points 1 day ago

What kind of stupid-ass question is that? LOL All kidding aside, she's a good soul. We're not married, we've just know each other for 45+ years. It just kind of clicked. She lives in her house, and I in mine, and we get together as often as possible.

[–] hendrik@palaver.p3x.de 3 points 1 day ago (2 children)

Hmm... Would be interesting to find out what kind of effect that has on the average marriage or relationship 😅

[–] e0qdk@reddthat.com 4 points 1 day ago (1 children)

"You love the robot more than me!" 💔️

[–] RaivoKulli@sopuli.xyz 3 points 16 hours ago

"WELL AT LEAST THE ROBOT LISTENS TO ME"

[–] TheFogan@programming.dev 2 points 1 day ago (1 children)

I mean, I'd imagine probably not a good one :) Somehow I imagine asking the AI to record a conversation, is an instant arguement escalator... as is asking to read the facts back, and usually the topic would be switched rather than one side admitting their fault in the conversation.

Actually I think there's a black mirror episode on roughly that (not a device for recording audio when asked, but everyone having a chip in their head that automatically records their memories, and a huge fight when a husband discovers his wife deleted a few hours of recordings.

[–] faberfedor@lemmy.world 2 points 15 hours ago

That was a great episode!

[–] hendrik@palaver.p3x.de 12 points 1 day ago (1 children)

Likely everyday stuff... Meeting minutes, phone or video conferences and such...

[–] irmadlad@lemmy.world 4 points 1 day ago (1 children)

I guess that's why I am having difficulty coming up with a use case. I mean, I walk around the lab talking to myself all day long, but I think it'd be a bad idea to have a record of all those conversations. lol

[–] onslaught545@lemmy.zip 3 points 1 day ago

If you don't have to sit through a bunch of 'meetings that could have been emails' on a daily basis, you likely won't have a use case for it.

But in my last job I was a systems engineer for a web development company. I had to be included on all of the dev calls in case an infrastructure question came up that I needed to answer, and so I was vaguely aware of what the devs were doing.

This software would have been a lifesaver, because my ADHD doesn't let me listen to stuff like that for a straight hour or two.

[–] hoshikarakitaridia@lemmy.world 3 points 1 day ago (1 children)

Long videos or voice notes where you're usually just looking for a small snippet.

[–] irmadlad@lemmy.world 2 points 21 hours ago (2 children)

Now that's an interesting angle. I am a mediocre musician on my best day, but sometimes I incorporate phrases and lyric snippits in a piece. I wonder if I could use WhisperX to find those words or phrases from a stack of songs. For instance, I did a piece that used a line from Jimi Hendrix's 'If 6 were 9' where he says 'I'm the one who's gotta die when it's time for me to die. So let me live my life the way I want to.' I wonder if WhisperX could pick that out of a stack of Jimi Hendrix songs.

[–] dgdft@lemmy.world 2 points 18 hours ago* (last edited 18 hours ago) (1 children)

You should be able to get decent results if you pipe your tracks through demucs first to isolate the vocals.

https://github.com/adefossez/demucs

Vanilla whisper will probably be better than whisperX for that use case though.

Depending on how esoteric your music library is, you can also build a lyrics DB with beets: https://beets.readthedocs.io/en/stable/plugins/lyrics.html

[–] irmadlad@lemmy.world 2 points 17 hours ago

I use UVR for vocal isolation. It just works, but that shouldn't be a problem. I'll check it out. At the worst, I'll learn something.

[–] hoshikarakitaridia@lemmy.world 2 points 20 hours ago (1 children)

It might take a while, but when your PC is working on it you are not and searching for words might be easier ^^

I'm excited to hear how well it works ^^

[–] irmadlad@lemmy.world 2 points 15 hours ago

I'm always excited to try new stuff. You never know. A use case might develop that you didn't think of.

[–] wise_pancake@lemmy.ca 5 points 1 day ago (1 children)

That is cool! I've been wanting I've wanted to use a model like this but haven't really looked.

Are you self hosting the long context llm, of do what are you using?

Context lengths are what kill a lot of my local llm experiments.

[–] dgdft@lemmy.world 1 points 1 day ago* (last edited 1 day ago)

Are you self hosting the long context llm, of do what are you using?

I did a lot of my exploration back when GPT4 128K over API was the only long-context game in town.

I imagine the options are much better these days between Llama 3/4, Deepseek, and Qwen — but haven’t tried them locally myself.