mirror of
https://github.com/advplyr/audiobookshelf.git
synced 2026-05-30 23:40:40 +02:00
[Bug]: New library scan crashes server #1814
Assignees
adam (Adam Melkus)
Clear assignees
adam (Adam Melkus)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: starred/audiobookshelf#1814
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ghost on GitHub (Mar 15, 2024).
Describe the issue
This is on a fresh install of audiobookshelf.
I added a new library for ebooks. I pointed the folder to my calibre data location and started the first scan. This maxes out the ram and eventually the server becomes unresponsive and I have to reboot the VM.
At first I tried with 4GB of RAM, and it crashed. I gave it 8GB and it crashed again. This is with around ~150 ebooks, so nothing big (5.4GB in total).
I have seen similar issues where the culprit is ffprobe spawning without limit, but it seems it is not the case here. All the ram is taken up by the node process, only 1/2 instances of ffprobe are running.
Steps to reproduce the issue
Audiobookshelf version
v2.8.0
How are you running audiobookshelf?
Docker
@advplyr commented on GitHub (Mar 15, 2024):
What are the file formats of the ebooks you are scanning in? If you check the scan log is there a place where it stops that we can see?
@ghost commented on GitHub (Mar 15, 2024):
These are the logs from the container:
That last part keeps repeating until I kill the process
@ghost commented on GitHub (Mar 15, 2024):
also, regarding the file formats:
The
.dbare calibre databases. The json is also for calibre. So it's mostly epub, mobi, pdf and azw3. The jpgs are covers, opf is calibre book metadata. Some comics too (cbr, cbz).Is any of these formats known to cause issues?
@advplyr commented on GitHub (Mar 16, 2024):
I think the
cbrcomics are having issues right now I'm still investigating. Can you take those out for a test scan?@ghost commented on GitHub (Mar 16, 2024):
yup that fixes it. both cbr and cbz seem to be causing issues.
one thing to note though, they do seem to be getting partially processed. If i start a scan with a cbr/cbz file present, it goes into the wasm error loop. if i then move the files out of the directory, this shows up in the logs:
So they do seem to be somewhat processed. I do not know much about this project so no clue why this is 😅
@Maran123 commented on GitHub (Mar 20, 2024):
I think i'm also encountering this issue with cbz files. It works if you have enough RAM and/or swap-space for the entire library of cbz files. Otherwise the memory fills up and the scan is slow/halted.
@ghost commented on GitHub (Mar 20, 2024):
this seems unfeasible for moderately big libraries.
are all book files buffered in memory at once?? 😬
also, does this happen with audiobooks too? or just cbz?
@nichwall commented on GitHub (Mar 20, 2024):
It's only with some cbz/cbr and ebook files.
@Tsamsiyu1 commented on GitHub (Mar 20, 2024):
i have the same problem. i just created a new Library and added about 40 cbz files. the first 20 where scanned but it took ages. after that it just crashes the server.
@maxcovergold commented on GitHub (Apr 11, 2024):
Tested adding comics for the first time and ran into this issue. System froze then crashed. Tried creating a small <1GB library and that completed, but noticed memory wasn't released on completion of successful scan. ABS required restart to resolve.
I also see the below error, both when memory completely fills and crashes and when scaning a small library so no crash, but memory is not released:
'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
EDIT - I subsequently also tried adding a couple of issues at a time after successfully scanning in and restarting the ABS container to release memory, but after a couple of times, hit the ceiling where it would crash again. So files already scanned into library also contribute to the memory usage issue on new scan. Which means cannot add to library incrementally either.
@maxcovergold commented on GitHub (Apr 23, 2024):
Unfortunately fixing of error #2856 doesn't resolve this issue. It possibly is stopping the application from crashing, but just makes my entire Synology grind to a halt indefinitely
@Codex966 commented on GitHub (Jun 11, 2024):
Still running into this issue as well.
I added a Comics library last week, primarily with pdf, cbr, and cbz files in it and once I got to about 20~ items any scan crashes the app, requiring a restart (running Docker on UnRaid).
I've tried removing new entries to back-track if there might be a corrupt cbz file causing it, but haven't pinpointed a specific file for it yet.
My ebook/audiobook library is about 2 years old and is up to about 600 items and has no problems. Still think it might be something with how ABS is trying to scan these cbz/cbr.
(Not sure about the RAM troubleshooting though, I don't see an abnormal uptick in usage when scanning (aside from a normal amount) and I've got 128Gb of RAM on this server, my Comics library only being 13Gb so far.)
@pekempy commented on GitHub (Aug 5, 2024):
I also have this issue with .cbr files.
I've noticed that 64/75 of
The Sandmanhave processed properly, but it's having issues with the same sets of files after I delete and re-create the library (issue 1, issues 10-19). I've confirmed those files are not corrupted and have no differences to the other issues.Docker logs:
@pierrejo commented on GitHub (Aug 6, 2024):
Hi, i've the same issue. I'm on a raspberry pi with 4Gb of RAM. And i can't add my CBZ files.
I'm on the
V2.12.1of AudiobookshelfError :
I've also try to limit my container with :
Still get the error .... 😒
@danielb2 commented on GitHub (Aug 7, 2024):
I'm also having the same issues with cbr files:
Init v2.12.1
Node.js Version: v20.16.0
Platform: linux
Arch: x64
@Atfynt commented on GitHub (Aug 8, 2024):
It looks like the issue is in libarchive.js preventing it from automatically closing the archive once it has been opened adding archive.close() wherever we have Archive.open() just before returning would prevent having everything in memory all at the same time. at the very least parseComicMetadata.js parse function and extractCoverImage. it probably also needs to be added wherever the comic reader is extracting files but I have not really looked into that.
@nichwall commented on GitHub (Aug 20, 2024):
Thanks for helping track that down. I attempted a fix here https://github.com/advplyr/audiobookshelf/pull/3312 for the server crashing during a scan if you want to help test it (I don't actually have any comic files and am testing with homemade CBR/CBZ).
@advplyr commented on GitHub (Nov 23, 2024):
Is this still an issue?
@danielb2 commented on GitHub (Nov 23, 2024):
works for me 👍
another issue now that I can read it is that i can't zoom out to see all of it, but I have to scroll a lot
@danielb2 commented on GitHub (Nov 23, 2024):
also seeing this for some
@maxcovergold commented on GitHub (Jan 2, 2025):
Just to confirm that the change has resolved the problem for me. Thank you!