mirror of
https://github.com/advplyr/audiobookshelf.git
synced 2026-05-30 23:40:40 +02:00
[Bug]: Changing library path wipes metadata #2426
Assignees
adam (Adam Melkus)
Clear assignees
adam (Adam Melkus)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: starred/audiobookshelf#2426
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @teotikalki on GitHub (Dec 18, 2024).
What happened?
I was trying to organize my library, so I added a /fiction and /nonfiction section. I then moved the former contents of /audiobooks (library and submissions) to /fiction/library and /fiction/submissions.
My library became empty, and then every single book had to be scanned from scratch. This is severely inconvenient but recoverable.
All books which I have listened to lost their listen state. All books which I had (very carefully) manually set proper chapters on, adding 2-3 missing chapters per 'chapter' the m4b came with, have lost their chapter data. WTF!?
I thought that when I hit 'save' every time I made a change it actually saved it. I ASSUMED it was saving to the m4b itself and that my file was now 'fixed'.
PLEASE tell me that is saved it SOMEWHERE and that I can recover it! Several hundred hours of listening, which I manually paused every time a chapter was announced so I could mark them, would require many hours of bullshit skipping to refind the chapter markers and it would be all agonizing pain, unlike the initial listening.
Audiobookshelf is shutdown while I try to figure this out. I currently feel like I've wasted the initial time I invested in curating my library and my joy in listening is wiped.
Please also help myself and others in the future to avoid this pain. Please add tooltips that clearly state whether your metadata edits are ephemeral or persistent and how to make them the desired sate.
Please allow users to change the path to their data without needing to invalidate and replace the entire library. I had previously moved files around (from submissions to library) and 'scan' from within the UI on the relevant title fixed it's path while keeping the library record. That was what I expected this time (I CERTAINLY did NOT expect to have every action I have taken since I started curating audiobooks to be wiped).
What did you expect to happen?
I expected to change the paths to some files, hit 'rescan' in the UI, update the location of the files, and continue listening to the book I was halfway through from where I left off.
Steps to reproduce the issue
Audiobookshelf version
2.17.1
How are you running audiobookshelf?
Docker
What OS is your Audiobookshelf server hosted from?
Linux
If the issue is being seen in the UI, what browsers are you seeing the problem on?
None
Logs
Additional Notes
I didn't actually really enjoy audiobooks until I found Audiobookshelf. Properly chapterizing my library was a labor of love. I ABSOLUTELY did NOT want that efffort removed, and if any action I took had come with a warning over the consequences I WOULD NOT have taken it. I firmly believe that no action I took SHOULD have had these consequences and the fact that something did is a critical flaw in the usability of this product.
@nichwall commented on GitHub (Dec 18, 2024):
Your media files are not edited unless you specifically use the "Embed metadata" tool. Everything else is purely within ABS. Note that not all of the ABS metadata is embedded back in to the file due to different container formats and how metadata tags work (not supporting multiple values consistently, naming, etc).
All of the ABS metadata is stored in the SQLite database and in the
metadata.jsonfiles in the/metadata/itemsdirectory or with the media item if you have the server settings "Store metadata with item". All of the metadata changes you have made will be present in those JSON files. User specific information (listening sessions and progress) is only within the database.If you delete a library from ABS (which it looks like you did based on the logs to create new libraries with the new paths?) everything for that library in the database is removed (books, podcasts, listen progress, etc). You can restore from a backup to get the previous library data.
@teotikalki commented on GitHub (Dec 18, 2024):
@nichwall I pretty much just discovered that. Digging into my installation shows that items get hashed to /metatadata/items/hash and looking at the metadata.json for an item I had just finished listening to yesterday (with all of my chapter edits) showed a file edited a few minutes ago at the same time as the rest of the library with the bad chapters I had replaced.
I did NOT delete a library. ABSOLUTELY NOT.
I had a library named 'audiobooks'. I renamed it 'audiobooks-fiction' and then removed the old PATHS after I had added the new PATHS.
I then created a new library named 'audiobooks-light novels' and another named 'audiobooks-nonfiction'. Both of those got new paths.
I unfortunately don't have a backup of my library metadata, since I didn't find that option in a menu until just now when (after) this happened. It was off by default (which I will call out as another 'request to fix' error) and there was never a prompt to suggest that it should be looked at that I'm aware of (I turn all backup options on, always, ESPECIALLY when it involves personal manual effort).
I will be looking into the 'Embed metadata' tool. Please make the situation with this more clear within the UI: I completely thought that I was 'embedding metadata' every time I hit 'Edit Chapters->Save'.
@nichwall commented on GitHub (Dec 18, 2024):
Backups are enabled by default and stored in
/metadata/backups@teotikalki commented on GitHub (Dec 18, 2024):
/metadata/backups is empty and the item in Settings->Backups->Enable automatic backups was disabled when I found it.
If it is supposed to be enabled by default, could a permissions issue that has since been fixed have caused it to become disabled after it was tried and failed? The manual backup I just made worked, so I'm assuming that automatic backups will from now on.
I'll note that ABS is the first real usage I've made of Docker. I previously had a Proxmox server and set up everything I wanted to run in LXC containers there, but right now I'm a few thousand km from where that's in storage and so... Docker. I've only just gotten into audiobooks and it didn't take me long to crave a better interface for them than my local music player.
Where is this 'Embed metadata' tool located? I feel like I've perused every menu multiple times now and I can't find it.
Edit: I would have expected Settings->Item Metadata Utils to be the starting point...
@teotikalki commented on GitHub (Dec 18, 2024):
Ah, automatic backups would also be hampered by the fact that ABS isn't always running at a specific time.... although it USUALLY is, so there would be at least SOME automatic backups if it was a time thing.
@nichwall commented on GitHub (Dec 18, 2024):
Backups do not get disabled if backups fail. The server will try and make the backup at the specified schedule, unless the server is not running.
You can embed metadata from the Tools tab on the edit modal of a library item. I personally don't embed metadata into files at all, so I don't know exactly what fields map to what or how long it should take. If you need other fields or have a specific way you want metadata embedded, I would recommend writing a custom script to embed metadata into files from the
metadata.jsonthat ABS creates.@teotikalki commented on GitHub (Dec 18, 2024):
I found it, thank you.
I would suggest that the tool have checkboxes beside the fields it DOES have (like the modal for choosing which fields of metatadata to update). Right now the first item I'm looking at has 'comment' and 'description' and they're identical. I have looked over the UI and seen nowhere to interact with a 'comment' field as of yet, so I'm not sure where that's coming from or why it's identical to 'description'. Being able to deselect something like that to not embed useless metadata seems like it would have universal appeal.
I also note that there's no 'narrator' field and that the narrator is listed as 'composer'. Is this because there is no 'narrator' field in the embedded metadata and using 'composer' is a concession to/custom for this circumstance?
@advplyr commented on GitHub (Dec 19, 2024):
We had a discussion on Discord recently about embedding meta tags. In that discussion me and @nichwall mention some of the main reasons that we haven't leaned in to embedding everything in audio files.
https://discord.com/channels/942908292873723984/981321213882282035/1311749736780529774
The preferred method of keeping your Abs metadata is using the metadata.json sidecar file that you can optionally store alongside your audio files with the server setting.
Here is a recent comment on why Abs relies on file paths: https://discord.com/channels/942908292873723984/954760207131615264/1318622958859714571
@teotikalki commented on GitHub (Dec 19, 2024):
@advplyr I don't use Discord and cannot follow those links.
As an open source project I ask you to kindly consider users who insist on their communication software being open source as well.
@advplyr commented on GitHub (Dec 19, 2024):
No problem. We had a matrix server for a while but only a few people of several thousand actually used it so I shut it down.
@nichwall commented on GitHub (Dec 19, 2024):
Copying and pasting parts of the discussion from Discord: (may have missed some messages due to taking place over time)
Edit to add: the Discord/matrix bridge broke pretty frequently, sometimes for multiple weeks at a time before somebody on either end noticed due to so few matrix users.
Part 1
One of the main reasons I don't try to get everything in ID3 tags is there is no standard that can be applied, especially across audio file formats.
Abs uses ffmpeg for reading/writing meta tags. It looks like ffmpeg doesn't have a defined tag for grouping with mp3 files but it does for mp4 files. https://wiki.multimedia.cx/index.php/FFmpeg_Metadata
For mp3 files ffmpeg will still allow embedding custom tags which is why grouping still shows up in mp3tag. For mp4 files ffmpeg does not allow custom tags so if you are using m4b files the meta tags will be much more limited
That's part of why the sidecar file can be stored with the media item. If you want to embed metadata back in to the file with specific values in specific fields, you can use an external tool/script to handle that for your specific media to make it all in one file. While the fields themselves are "standard", they aren't all labeled the same for different containers and are used inconsistently across software, so the usage still isn't standard. This is also more complicated when taking single/multi file content into account, because those are not applied consistently by different software/publishers either
There is more information about the book in Abs than there are meta tags. There is also no standard for specifying multiples. For example, multiple series we are doing as a semicolon delimited string since commas can exist in the series name.
For podcast episodes we store the description as HTML. We may do this for audiobooks eventually as well. There is no good way to store HTML in a meta tag.
The ID3 defined meta tag names don't fit neatly with audiobooks. We use composer for narrator (also semicolon delimited), artist for author.
Depending on the software you can get different results on what meta tags are embedded.
I spent a lot of time trying to figure out standardized meta tags for audiobooks and as it stands now it's insufficient. Having a sidecar file be able to cleanly store all of the data outweighs the negative of having to keep an extra file.
The first metadata file I tried to make a standardized audiobook metadata file. It was similar to ffmetadata and thought a more human readable sidecar file would be nice.
It turned out to be more trouble than it was worth so we switched to JSON
Part 2
The main issue is that ABS doesn't have a "global source of truth for metadata" due to various languages, book metadata sources, etc. When you add a new book, ABS creates the new library item and gets metadata from your local file system. A lot of people match against online providers to get better/updated information, but this is not consistent across books/editions. For example, if you have an old audiobook of Lord of the Rings, but all of the online metadata providers have been updated so your local metadata is no longer searchable online. So even though you have the "same files", it gets difficult with multiple editions of books, audiobook versus ebook metadata, etc. There isn't a good single source of truth with a public API that works like there is for movies and TV shows.
@teotikalki commented on GitHub (Dec 19, 2024):
Wouldn't you know it but I WISH I did... I had a boxed set of LotR on 13 cassettes when I was a child and I loved it. I want THAT version.
Back on topic, I'd like to clearly state that my goal here is almost entirely to properly set chapters. I don't hate the idea of your sidecar files (although I do hate that the data I generated in them was thoughtlessly removed without my knowledge and to my dismay). What I am compulsive about is that if a file HAS chapters they should be CORRECT. If they aren't then I feel like my file is WRONG and that makes me deeply unhappy. My best comparison is that I've used Kodi for years and I'm used too it generating nfo files to store metadata and that makes me happy. It doesn't mean for a second that I won't feel compelled to edit the original files if there are errors in subtitles or the like. What IS in the file should be RIGHT.
If I want to do complex metadata editing I'll use a different app that specializes in doing just that. I only know if a chapter marker is correct or not when actually listening, however, so setting those as I go and saving the updated chapter list to the m4b is the only step I feel a strong desire for ABS to perform.
Also, thanks for... having at least given Matrix a chance and responding both kindly and helpfully.