starred/audiobookshelf
1
0
Fork 0
mirror of https://github.com/advplyr/audiobookshelf.git synced 2026-05-30 23:40:40 +02:00
Code Issues 1.0k Packages Projects Releases 10 Wiki Activity

[Enhancement]: Add Whisper support #1106

New Issue
Open
opened 2026-04-24 23:32:41 +02:00 by adam · 14 comments
adam commented 2026-04-24 23:32:41 +02:00
Owner
Copy Link
Copy Source

Originally created by @apiweb on GitHub (Apr 25, 2023).

Describe the feature/enhancement

Hi there!

I've been using AudioBookShelf for a while now, and I love the platform. I was thinking about how it could be improved, and I had an idea that I wanted to share with you all.

I think it would be great if AudioBookShelf could integrate with Whisper speech-to-text model to automatically generate subtitles for audiobooks. This could be an external tool like Tone and ffmpeg that the user could enable or disable as needed.

With Whisper, it would be possible to transcribe speech across dozens of languages and even handle poor audio quality or excessive background noise. It would make it easier for people who are hard of hearing or have difficulty understanding accents to enjoy audiobooks.

Here are some tips on how to integrate this feature into the AudioBookShelf flow:

  • Use the metadata language tag to automatically set the Whisper language.
  • Automatically save the srt using the Title Folder Naming structure.

I hope you will consider this suggestion for future updates to AudioBookShelf. Let me know if you have any questions or concerns.

Thank you for all your hard work making AudioBookShelf a great platform!

Originally created by @apiweb on GitHub (Apr 25, 2023). ### Describe the feature/enhancement Hi there! I've been using AudioBookShelf for a while now, and I love the platform. I was thinking about how it could be improved, and I had an idea that I wanted to share with you all. I think it would be great if AudioBookShelf could integrate with [Whisper](https://github.com/openai/whisper) speech-to-text model to automatically generate subtitles for audiobooks. This could be an external tool like Tone and ffmpeg that the user could enable or disable as needed. With Whisper, it would be possible to transcribe speech across dozens of languages and even handle poor audio quality or excessive background noise. It would make it easier for people who are hard of hearing or have difficulty understanding accents to enjoy audiobooks. Here are some tips on how to integrate this feature into the AudioBookShelf flow: - Use the metadata language tag to automatically set the Whisper language. - Automatically save the srt using the [Title Folder Naming](https://www.audiobookshelf.org/docs#book-title-folder-naming) structure. I hope you will consider this suggestion for future updates to AudioBookShelf. Let me know if you have any questions or concerns. Thank you for all your hard work making AudioBookShelf a great platform!
adam added the enhancement label 2026-04-24 23:32:41 +02:00
adam commented 2026-04-24 23:32:42 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Apr 27, 2023):

Maybe we first add support for an srt subtitle file.

The opposite of this was also requested for ebooks #601

@advplyr commented on GitHub (Apr 27, 2023): Maybe we first add support for an `srt` subtitle file. The opposite of this was also requested for ebooks #601
adam commented 2026-04-24 23:32:42 +02:00
Author
Owner
Copy Link
Copy Source

@tehguitarist commented on GitHub (May 4, 2023):

I guess a natural extension (and this WOULD make audiobookshelf an Audible killer) would be a whispersync like syncing with ebooks. I guess that would just be shifting the bookmark position of the ebook/audiobook whenever one or the other is progressed.

Not a lot of work specifically, but some from a quick google: https://github.com/readbeyond/aeneas/ and https://github.com/r4victor/syncabook.

Though any library that can match up audio based on a text file, obviously there's a bunch of work to find the start of the chapter, and the start of the audiobook and match that up but that's more of a pipe dream. Aens probably shows the most reasonable promise, but obvious difference between the formats (any preamble by narrators, or table of contents with ebooks) would all be factors. Without digging into the libraries, as long as there was enough error handling to wait til both files had matches (and skip over extras in one or the other) that may go quite smoothly.

@tehguitarist commented on GitHub (May 4, 2023): I guess a natural extension (and this WOULD make audiobookshelf an Audible killer) would be a whispersync like syncing with ebooks. I guess that would just be shifting the bookmark position of the ebook/audiobook whenever one or the other is progressed. Not a lot of work specifically, but some from a quick google: https://github.com/readbeyond/aeneas/ and https://github.com/r4victor/syncabook. Though any library that can match up audio based on a text file, obviously there's a bunch of work to find the start of the chapter, and the start of the audiobook and match that up but that's more of a pipe dream. Aens probably shows the most reasonable promise, but obvious difference between the formats (any preamble by narrators, or table of contents with ebooks) would all be factors. Without digging into the libraries, as long as there was enough error handling to wait til both files had matches (and skip over extras in one or the other) that may go quite smoothly.
adam commented 2026-04-24 23:32:43 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (May 4, 2023):

There is also this issue https://github.com/advplyr/audiobookshelf/issues/189

@advplyr commented on GitHub (May 4, 2023): There is also this issue https://github.com/advplyr/audiobookshelf/issues/189
adam commented 2026-04-24 23:32:45 +02:00
Author
Owner
Copy Link
Copy Source

@damajor commented on GitHub (May 5, 2023):

I tested Whisper on my setup and results are kind of good but far from perfect.

Using base model it took me around 4 minutes to transcribe 1 hour audiobook.
Using large model it took me a bit more than 1 hour to transcribe 1 hour audiobook.

Those tests were done on 5950X with 12 parallel threads (no GPU involved).

@damajor commented on GitHub (May 5, 2023): I tested Whisper on my setup and results are kind of good but far from perfect. Using base model it took me around 4 minutes to transcribe 1 hour audiobook. Using large model it took me a bit more than 1 hour to transcribe 1 hour audiobook. Those tests were done on 5950X with 12 parallel threads (no GPU involved).
adam commented 2026-04-24 23:32:45 +02:00
Author
Owner
Copy Link
Copy Source

@turnercore commented on GitHub (Sep 13, 2023):

I think Whisper (or some kind of speach-to-text) integration could be really nice to be able to transcribe audiobooks if people wanted subtitles. It would be a nice accessibility feature for someone listening in a second language, for example.

@turnercore commented on GitHub (Sep 13, 2023): I think Whisper (or some kind of speach-to-text) integration could be really nice to be able to transcribe audiobooks if people wanted subtitles. It would be a nice accessibility feature for someone listening in a second language, for example.
adam commented 2026-04-24 23:32:47 +02:00
Author
Owner
Copy Link
Copy Source

@Ed1ks commented on GitHub (Oct 31, 2023):

👍thumbs up. Would be very nice for self recorded audiofiles.

@Ed1ks commented on GitHub (Oct 31, 2023): 👍thumbs up. Would be very nice for self recorded audiofiles.
adam commented 2026-04-24 23:32:48 +02:00
Author
Owner
Copy Link
Copy Source

@rounakdatta commented on GitHub (Nov 2, 2023):

Should we consider working on this? I can volunteer to start contributing.
Currently Snipd is the only app which gets AI-generated subtitles UX perfectly right, and this could be our chance to make Audiobookshelf the FOSS alternative to it.

@rounakdatta commented on GitHub (Nov 2, 2023): Should we consider working on this? I can volunteer to start contributing. Currently [Snipd](https://www.snipd.com/) is the only app which gets AI-generated subtitles UX perfectly right, and this could be our chance to make Audiobookshelf the FOSS alternative to it.
adam commented 2026-04-24 23:32:50 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Nov 2, 2023):

Yeah I'm interested in this but I can't put much attention towards it now. We were talking about it in Discord the other day. If anyone wants to start putting something together or setup a proof of concept that would be great. We can chat about it in Discord

@advplyr commented on GitHub (Nov 2, 2023): Yeah I'm interested in this but I can't put much attention towards it now. We were talking about it in Discord the other day. If anyone wants to start putting something together or setup a proof of concept that would be great. We can chat about it in Discord
adam commented 2026-04-24 23:32:53 +02:00
Author
Owner
Copy Link
Copy Source

@yuchen-lea commented on GitHub (Nov 21, 2023):

Maybe we first add support for an srt subtitle file.

The opposite of this was also requested for ebooks #601

@advplyr I agree with what you said about adding support for SRT subtitle files first. I have now used Whisper to generate corresponding subtitles for my local podcasts. On my computer, I can search and view them. Displaying subtitles while playing on the ABS mobile app is the final piece of the jigsaw.

I think these two things can share the same UI: LRC files #817 (external LRC files with the same name as the audio file or ID3 information embedded in the audio file) and SRT files #2257 (external SRT files with the same name as the audio file).

@yuchen-lea commented on GitHub (Nov 21, 2023): > Maybe we first add support for an `srt` subtitle file. > > The opposite of this was also requested for ebooks #601 @advplyr I agree with what you said about adding support for SRT subtitle files first. I have now used Whisper to generate corresponding subtitles for my local podcasts. On my computer, I can search and view them. Displaying subtitles while playing on the ABS mobile app is the final piece of the jigsaw. I think these two things can share the same UI: LRC files #817 (external LRC files with the same name as the audio file or ID3 information embedded in the audio file) and SRT files #2257 (external SRT files with the same name as the audio file).
adam commented 2026-04-24 23:32:54 +02:00
Author
Owner
Copy Link
Copy Source

@iamhenry commented on GitHub (Apr 19, 2024):

Should we consider working on this? I can volunteer to start contributing. Currently Snipd is the only app which gets AI-generated subtitles UX perfectly right, and this could be our chance to make Audiobookshelf the FOSS alternative to it.

this is exactly what would be great with ABS. I have the paid feature for Snipd and now it's hard to take notes without it using audiobooks.

@iamhenry commented on GitHub (Apr 19, 2024): > Should we consider working on this? I can volunteer to start contributing. Currently [Snipd](https://www.snipd.com/) is the only app which gets AI-generated subtitles UX perfectly right, and this could be our chance to make Audiobookshelf the FOSS alternative to it. this is exactly what would be great with ABS. I have the paid feature for Snipd and now it's hard to take notes without it using audiobooks.
adam commented 2026-04-24 23:32:55 +02:00
Author
Owner
Copy Link
Copy Source

@turnercore commented on GitHub (Apr 20, 2024):

I agree, and it isn't hard to generate the .srt files from audio now. Maybe there should be a branch to work on this, I'd propose doing it in this order:

  1. Getting ABS to recognize .srt files next to audio files and displaying that in the UI in some way, like Snipd (I agree they do a great job with the UI).
  2. Adding .srt upload option for the files in the UI
  3. Creating a function that can use a whisper url to transcribe the files automatically if set up
  4. Adding settings for whisper api url and options to auto-transcribe new files & transcribe button

Honestly as a further extension I would LOVE if you could do audio-clips like Snipd that could export to Obsidian or something, but I think having the ability and UI set up for transcriptions would be the first hurdle for that. We could add audio clips on back button like Snipd does after that.

@turnercore commented on GitHub (Apr 20, 2024): I agree, and it isn't hard to generate the .srt files from audio now. Maybe there should be a branch to work on this, I'd propose doing it in this order: 1. Getting ABS to recognize .srt files next to audio files and displaying that in the UI in some way, like Snipd (I agree they do a great job with the UI). 2. Adding .srt upload option for the files in the UI 3. Creating a function that can use a whisper url to transcribe the files automatically if set up 4. Adding settings for whisper api url and options to auto-transcribe new files & transcribe button Honestly as a further extension I would LOVE if you could do audio-clips like Snipd that could export to Obsidian or something, but I think having the ability and UI set up for transcriptions would be the first hurdle for that. We could add audio clips on back button like Snipd does after that.
adam commented 2026-04-24 23:32:58 +02:00
Author
Owner
Copy Link
Copy Source

@zkvsky commented on GitHub (Apr 27, 2024):

just FYI there are several implementations of whisper specifically tailored to subtitle generation. This one for example https://github.com/jianfch/stable-ts can not only generate srt, but also ssa/ass karaoke style subs [meaning that the current spoken word is highlighted] bringing us even closer to snipd. From my experience base and small models are enough with it.

@zkvsky commented on GitHub (Apr 27, 2024): just FYI there are several implementations of whisper specifically tailored to subtitle generation. This one for example https://github.com/jianfch/stable-ts can not only generate srt, but also ssa/ass karaoke style subs [meaning that the current spoken word is highlighted] bringing us even closer to snipd. From my experience base and small models are enough with it.
adam commented 2026-04-24 23:32:58 +02:00
Author
Owner
Copy Link
Copy Source

@turnercore commented on GitHub (May 1, 2024):

just FYI there are several implementations of whisper specifically tailored to subtitle generation. This one for example https://github.com/jianfch/stable-ts can not only generate srt, but also ssa/ass karaoke style subs [meaning that the current spoken word is highlighted] bringing us even closer to snipd. From my experience base and small models are enough with it.

Thanks for the heads up, hopefully I can get some time to work on a PR for this type of thing. I haven't contributed yet though so I imagine it will take me a bit to get familiar with the code base and what needs to be updated for this kind of feature.

@turnercore commented on GitHub (May 1, 2024): > just FYI there are several implementations of whisper specifically tailored to subtitle generation. This one for example https://github.com/jianfch/stable-ts can not only generate srt, but also ssa/ass karaoke style subs [meaning that the current spoken word is highlighted] bringing us even closer to snipd. From my experience base and small models are enough with it. Thanks for the heads up, hopefully I can get some time to work on a PR for this type of thing. I haven't contributed yet though so I imagine it will take me a bit to get familiar with the code base and what needs to be updated for this kind of feature.
adam commented 2026-04-24 23:32:59 +02:00
Author
Owner
Copy Link
Copy Source

@iamhenry commented on GitHub (May 2, 2024):

@turnercore that would be awesome!

@iamhenry commented on GitHub (May 2, 2024): @turnercore that would be awesome!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
authentication
backlog
bug
chapter editor
config-issue
ebooks
encoding/embedding
enhancement
help wanted
listening sessions & progress
planned
possible plugin
progress sync
pull-request
Mirrored from GitHub Pull Request
 sorting/filtering/searching
unable to reproduce
upload
users & permissions
waiting
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam (Adam Melkus)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/audiobookshelf#1106
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?