mirror of
https://github.com/advplyr/audiobookshelf.git
synced 2026-05-30 23:40:40 +02:00
[Bug]: Including same folder in two different libraries sometimes produces duplicate books in older library #581
Assignees
adam (Adam Melkus)
Clear assignees
adam (Adam Melkus)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: starred/audiobookshelf#581
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @wtanksleyjr on GitHub (Aug 12, 2022).
Describe the issue
While trying to figure out how to use libraries I created a new library that shared a disc folder with my older, original library - specifically, both included my Audible books, while my main included books I'd bought from audiobooks.com and so on.
When I started the scan on the new library, I began seeing duplicates showing up in my original library. Not every book was duplicated, but the key seems to be the presence of BOTH an odm file, AND an abs file with different contents (for example because I ran a match).
It's possible that this kind of folder sharing cannot be done using metadata-in-folders, and that's a fine workaround. But probably the system should catch people doing that and at least warn.
Steps to reproduce the issue
Audiobookshelf version
2.1,2
How are you running audiobookshelf?
Docker
@advplyr commented on GitHub (Aug 13, 2022):
I'm glad the backups are working!
Were the 2 folder paths the exact same in both libraries? Or was the folder path a subfolder of a folder in the other library?
When you say "odm file" do you mean OPF file?
@wtanksleyjr commented on GitHub (Aug 13, 2022):
I know, I LOVE that backups passed this no-stress test! I'm so darn
happy with this app, thank you!
Yes, they were the exact same Linux pathname, no links or anything. Not
even a subfolder -- EXACTLY the same path.
Doh! Yes, sorry. OPF. I don't think it's NECESSARY, but I did in fact
always use them. I do think abs-in-folder is necessary, and I strongly
suspect they're what messed me up (because OF COURSE I shouldn't expect
metadata stored in the folders to be stable).
This won't happen again to me now that I know, but gut feeling is that we
can probably at least warn, and maybe possibly prevent it completely.
-Wm
On Fri, Aug 12, 2022 at 3:08 PM advplyr @.***> wrote:
@advplyr commented on GitHub (Aug 13, 2022):
I tested this and wasn't able to reproduce it. I don't think we should warn the user but rather fix the bug if there is one because this should work fine.
Are you able to reproduce this creating new libraries and new folders with just a few books?
@wtanksleyjr commented on GitHub (Aug 13, 2022):
I'll set this up later today.
On Fri, Aug 12, 2022 at 3:59 PM advplyr @.***> wrote:
@wtanksleyjr commented on GitHub (Aug 14, 2022):
I agree I didn't get the same results (I got worse ones). No duplicates, but ...
Following my instructions: I copied a couple of books into a new folder. I edited the OPFs so they'd stand out, made a library "test1" that used it, and scanned the library. No problem. I made the new library "test2" pointing to the same folder. Here I expected to see dupes in "test1", but there weren't any. I did edit some titles and author names, but I didn't get any dupes.
So I deleted both libraries (from the settings screen), and was about to post how I'd tried but failed.
But for some reason I went to my main library and found an "Issue" -- one of the two books thinks it's missing every 4th file. It's not -- nothing changed, all of the files are there and all are right there in the list of files. Nothing at all changed, nothing is wrong EXCEPT that ABS somehow is convinced something used to be there (but it never was).
This is very strange.
@advplyr commented on GitHub (Aug 14, 2022):
I'm not sure those things are related. The goal needs to be to reproduce the bug consistently with as few steps as possible.
I may have asked you this already or a few times, are you using a network file system? What OS are you running docker on and what OS are your files stored? Does the watcher work for you?
@wtanksleyjr commented on GitHub (Aug 14, 2022):
Oh, I should have put those into the initial report!
partition on SSD (I forget how big, but we shouldn't be touching it).
Side comment: is there any way to export my book info (especially after a
match) aside from keeping ABS files in the book folders? I'm 99% sure this
is causing this problem (I kind of think it's a race condition).
It's probably notable that the shared folder I originally exported had some
5000 books (I have an addiction to Audible), so the fact that only SOME of
my books were duplicated might have only proved that it's going to be hard
to recreate this.
-Wm
On Sun, Aug 14, 2022 at 7:27 AM advplyr @.***> wrote:
@wtanksleyjr commented on GitHub (Aug 18, 2022):
I just got a tiny bit of evidence that the watcher is not consistently working for me -- specifically I copied in a new Audible book, and it didn't get automatically added until I manually scanned the next day.
Then, when I manually scanned, the new book got added, an old book got a new duplicate made with half of the metadata of the new book and half of its own metadata and all the files of the old book, and the old book's old entry got marked as missing even though all of its files are present.
@advplyr commented on GitHub (Sep 1, 2022):
If you don't keep the abs files inside your library item folders then they are stored in
/metadata/itemsOn the original issue:
One possible reason this could happen is if the watcher triggered a scan on a new library item then while it is being scanned in the watcher is triggered again before the first finishes. That could cause a duplicate but would be very uncommon.
The watcher waits 4 seconds after detecting a change before it triggers a scan, so the first scan would need to be taking an unusually long time for an overlap to occur.
That being said it IS possible and may be what you encountered so I updated the scanner to queue up file updates if a watcher scan is currently running.
@wtanksleyjr commented on GitHub (Sep 2, 2022):
That makes a lot of sense ... a few books out of about 5000 seems "rare" :)
. I'll leave my settings the same for now, and watch for when this code
change gets pushed so I can hammer things around...
I have seen a RARE few books changed since I posted my last update, so
although we'll never be certain (sorry, just can't make this 100%) I think
I can make it sure enough.
-Wm
On Wed, Aug 31, 2022 at 3:44 PM advplyr @.***> wrote:
@advplyr commented on GitHub (Sep 5, 2022):
Updated in v2.1.5
Adding lock to prevent 2 watcher scans running at the same time.
@wtanksleyjr commented on GitHub (Sep 6, 2022):
OK, I backed up and recreated the situation with 2.1.4 (pre-fix) and got
some of this error; I reverted, upgraded, backed up, and confirmed we DO
NOT see the error.
I'm optimistic about this fix.
On Mon, Sep 5, 2022 at 2:29 PM advplyr @.***> wrote: