mirror of
https://github.com/advplyr/audiobookshelf.git
synced 2026-05-30 23:40:40 +02:00
[Bug]: ABS crashes with giant ebook libraries #2166
Assignees
adam (Adam Melkus)
Clear assignees
adam (Adam Melkus)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: starred/audiobookshelf#2166
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @hardwareadictos on GitHub (Aug 9, 2024).
What happened?
Server collapses temproary with big ebook libraries
What did you expect to happen?
ABS wotking as smooth as before adding high amounts of books.
Steps to reproduce the issue
Audiobookshelf version
v2.12.2
How are you running audiobookshelf?
Docker
What OS is your Audiobookshelf server hosted from?
Linux
If the issue is being seen in the UI, what browsers are you seeing the problem on?
None
Logs
@hardwareadictos commented on GitHub (Aug 10, 2024):
Update: Still happening on 2.12.3
@nichwall commented on GitHub (Aug 10, 2024):
Related to https://github.com/advplyr/audiobookshelf/issues/2750, https://github.com/advplyr/audiobookshelf/issues/2522, and https://github.com/advplyr/audiobookshelf/issues/2526.
@advplyr commented on GitHub (Aug 10, 2024):
Is there an actual crash happening or is it getting hung up? Can you check /metadata/logs/crash_logs.txt to see if there are any crashes in there?
I can see that the query for Newest Author is taking 243.39s which is a problem that needs to be fixed but I want to make sure you are not hitting other crashes related to ebooks that we've seen.
@hardwareadictos commented on GitHub (Aug 10, 2024):
That's what i get, some sqlite crashes on constraints:
It's weird because now i forced it 2 or 3 times and it didn't get refected on crash_log, but again the server collapsed when i get into books library and the container restarted by itself.
@hardwareadictos commented on GitHub (Aug 23, 2024):
Is there anything i can do?
That's making ABS tottally unusable, as my Books library is the first one to show upon app starts, Container collapses inmediatelly. Could be nice to add an option on library section to disable a concrete library instead of only deleting it :)
@nichwall commented on GitHub (Aug 23, 2024):
Can you try using the
edgedocker tag? That uses the latest commit from GitHub and I think the giant ebook library issue is the same as what I tried to fix in https://github.com/advplyr/audiobookshelf/pull/3312.To clarify, your server is crashing at boot or when you try to scan the library?
@hardwareadictos commented on GitHub (Aug 23, 2024):
Okey! I will fire an edge ABS container on my docker test machine. Will take at least 2 days to generate all metadata for that amount of books. Will report back when ready!
Container is crashing with any task related to Books library, container boots fine and works until you do something with that library.
@hardwareadictos commented on GitHub (Aug 24, 2024):
Well, books, podcasts and audiobooks metadata synced really fast on my test deployment. Now with 57000 books, books library isn't making the container to crash, so seems that this fix worked really well.
So go ahead, waiting patiently for the next release :D
EDIT: Half an hour later, container just crashed on Books library, as i see now, just when a library scan started.
That's what i saw on the log:
So not sure if the container crashed or simply the socket went down. There's nothing on crash_logs.txt
I'm going to follow if this thing comes up again during the weekend.
EDIT2: I had a crash rear a scan regarding with a Java OOM:
@nichwall commented on GitHub (Aug 24, 2024):
Thanks for helping test! Can you change your log level to Debug on the test server and then try running the scan again? That should help narrow down what is happening.
@hardwareadictos commented on GitHub (Aug 24, 2024):
So the same happened again:
I am on debug mode but that's the only thing appearing related to the crash.
@advplyr commented on GitHub (Aug 24, 2024):
You can try increasing the max memory size. To do this you pass an environment variable in your docker command.
The environment variables is:
NODE_OPTIONS=--max-old-space-size=8192Where 8192 is size in MB so you can lower that if you need to.
@hardwareadictos commented on GitHub (Aug 24, 2024):
Sorry for my clumsiness, now i saw on library settings that i set up the autoscan to work every 15 minutes accidentally this morning when i created the library. Is ABS able to abort scans if theres already one on progress? Maybe those scan overlaps were causing those OOMs.
Anyways i added the env var to my compose file and im testing it, for the moment 0 crashes. Let me keep testing the container.
@hardwareadictos commented on GitHub (Aug 26, 2024):
Well, rear one and a half day of testing (no more OOMs once automatic scan each 15 mins was disabled), i didn't see any crash, i got one "Socket disconection", and the interface was not responding, but it happened only one time, researching i didn't find anything related to it on container logs and on crash_log.txt.
My conclusion is: If the Edge branch didn't fully fix the issue, it probably fixed it by 99%.
Another thing to take into account, general ABS is really slow performant on the UI side since i added those 57000 ebooks, it responds, but once you get into ebooks library, performance is slowish. What can be tuned for this? I understand that those are lot's of files to process inside ffmpeg for retrieving miniatures and for a SQLite database, but what can be done?
Thank you :)
@hardwareadictos commented on GitHub (Aug 27, 2024):
Just an update: I can confirm that without the Java Memory bar, container doesn't crash also. 😃
@advplyr commented on GitHub (Aug 27, 2024):
Thanks for testing with a large library. We still have a ways to go in fully migrating to the new data models on sqlite (we previously used JSON files to store data). So, queries and indexes are far from optimized.
I could look at your specific database to see where the slow points are and try to improve them.
We are working on setting up some scripts to seed large databases that we can use for testing. Right now I only have a large audiobook db and a large podcast db.
@hardwareadictos commented on GitHub (Aug 28, 2024):
Glad to know! I can send you on private my DB if you want. Let's hope that helps other with big libraries also.
What about let users to use 3rd party DBs like MySQL/Postgres?I know is a pain to make those type of integrities, but for big libraries maybe it's the key. Other services like for example Kavita/Jellyfin have also issues when dealing with big DBs and SQLite.
Only an idea :D
@advplyr commented on GitHub (Aug 28, 2024):
Yeah you can send it via Discord or email dev@advplyr.com.
The reason I chose Sequelize ORM is so that users could eventually supply their own db. There is still a long way to go before that would be feasible. In order to make the migration in some reasonable timeline some of the queries were written raw for sqlite.
@advplyr commented on GitHub (Aug 28, 2024):
I also don't think the bottleneck we are hitting has to do with sqlite. Sqlite can handle much more data than that.
@hardwareadictos commented on GitHub (Aug 28, 2024):
Sent you a friend request on DIscord!
@hardwareadictos commented on GitHub (Aug 31, 2024):
UPDATE: Rear a week of testing (excluding the OOMs which were solved), Edge tag with those fixes is way more stable than the last release for me. Right now i cannot switch my v2.12.3 Production image to edge because i don't know which countereffects it would produce, but at least edge solved the crashes rear a week of testing.
@hardwareadictos commented on GitHub (Sep 1, 2024):
UPDATE: Deployed on production v2.13.0 and for the moment problem seems solved, even ABS performance increased a lot! Will report back here if i see that the problem reproduces :)
Thanks a lot for the hard work. Most projects doesn't have all the great support we have on ABS by the devs. That's really appreciated.
@hardwareadictos commented on GitHub (Sep 1, 2024):
Update: Sadly, this afternoon, now with 2.13.0 deployed on "Production" container i got various "crashes" using the aplication and trying to open Books library.
Same sympthoms as always. My monitoring throwing service down errors (Uptime Kuma, receiving alerts because the URL doesnt respond with a 200 code), app unusable throwing Socket disconnection errors / ping errors and nothing on logs.
Container doesn't crash, it simply stops responding and recovers itself on 2/3 minutes.
Maybe Test container doesn't have those issues because there's only one user created and online.
Let me know what can i provide when you can.
@nichwall commented on GitHub (Sep 1, 2024):
Can you update to 2.13.1? There was a problem with 2.13.0 which was fixed a few hours ago.
@hardwareadictos commented on GitHub (Sep 2, 2024):
Tried with 2.13.1 and 2.13.2, problem persists.
@hardwareadictos commented on GitHub (Sep 3, 2024):
Updated to 2.13.3, same thing
@hardwareadictos commented on GitHub (Sep 7, 2024):
hey! Is anything more i can debug for myself?
The problem is (rear deoploying edge branch on test and latest stable on production):
ABS hangs (container doesnt stop or reboot but just freezes) when opening Books library, which contains about 57000 ebooks (before adding this library it worked fine, even i have an audiobook library with 10000 audiobooks), UptimeKuma reports the service down as it doesn't report http 200 code and on about 2/3 minutes it goes live again. Sometimes it doesn't hang, but it does on a 9/10 ratio more or less.
Noting appears on logs, even on container logs or crash_log, i'm on debug mode.
What can i do? I love this app but for now is unusable on my case, had to go back to AntennaPod again.
And again: Thanks a lot for the hard work :)
@nichwall commented on GitHub (Sep 7, 2024):
Do you have periodic library scans or the watcher enabled? I would try disabling those.
@hardwareadictos commented on GitHub (Sep 7, 2024):
That was the first thing I switched off a month ago when it was reported:
@hardwareadictos commented on GitHub (Sep 10, 2024):
Updated to 2.13.4: Issue still present.
@hardwareadictos commented on GitHub (Sep 10, 2024):
To give some more info:
That's the last thing i see on logs rear the hangs and disconnections. Then ABS is tottally inresponsibe for 2 mins and recovers itself:
@hardwareadictos commented on GitHub (Sep 25, 2024):
Hey! Any clue on this? What can i do to debug it even further? Right now it's nearly imposible to open Books library without freezing the app. Remember: 57000 files:
@hardwareadictos commented on GitHub (Sep 28, 2024):
To give some more light on the issue, when the "hang" happens, once ABS tries to process Books library, CPU loads tends to go up, but when ABS gets unresponsive, it "hijacks" the first CPU core, with 100% load for those 2 minutes, ending to the container revovering:
Would be nice to get some feedback. Thanks :)
@nichwall commented on GitHub (Sep 28, 2024):
What is the first log line after the server recovers from hanging? It may just be struggling with the large library.
@hardwareadictos commented on GitHub (Sep 28, 2024):
That's what i suspect, something makes ABS to strugle on reading ultra-large libraries. I started the listing of Books library on 21:11:45, that's the log until it recovers and lists the library:
Seems that it hangs on loading authors (?)
@nichwall commented on GitHub (Sep 29, 2024):
Thanks. Yeah, it looks like you're running into https://github.com/advplyr/audiobookshelf/issues/3237.
Series and authors are filtered to ignore ones which contain no books, such as if you have an author where a user is not able to see if their books because they're explicit or has a tag preventing them from seeing it.
The queries and data model needs to be cleaned up a bit to make this work better. If you want to try manually removing the source code for the authors row on the main page, you could get rid of that.
@hardwareadictos commented on GitHub (Sep 30, 2024):
Well, it worked. Now ABS doesnt hang but at the cost of loosing a feature. Is there anything planed on edge tag for fixing this? I will be very pleased to help testing any workaround :)
@nichwall commented on GitHub (Sep 30, 2024):
It may be a while before it is fully fixed. The two main things that are happening:
Could you try adding an index to the database for author creation time? That will speed up the second step (not sure if the time is mostly being spent in the first or the second step). I think you can do this by just adding a
createdAtindex to the Author model.https://github.com/advplyr/audiobookshelf/blob/562c30cff45f648dc02281468fb736543ddfa042/server/models/Author.js#L115-L159
If you add the index, can you check the logs again to make sure that you're still getting 18935 authors in the log and report what the time for that query is?
@hardwareadictos commented on GitHub (Oct 1, 2024):
Nice, added the index creation to code:
Give me a couple of days to test it on edge and on stable branch.
@hardwareadictos commented on GitHub (Oct 1, 2024):
First "cold test" done on Edge:
So seems that at least at first try it doesn't make any difference, maybe because it's the first try rear creating the index, let's see in the comming days how it goes.
@hardwareadictos commented on GitHub (Oct 1, 2024):
So, in the latest hour i did some tests with the same results, even restarting the container some times, access time to books library is the same as without index implementation:
Maybe i made a mistake adding the code?
@nichwall commented on GitHub (Oct 1, 2024):
Thanks for helping to test the idea. Yeah, the code edit looks correct. You can double check if the index was created by using something like https://sqlitebrowser.org/ to open the database (while ABS is not running) and the looking in the lower right corner for the indexes and see if it was created for the authors table. It should be, but an easy way to sanity check.
It looks like the slow down is in step 1, which will be a lot more work to fix than just asking an index.
@hardwareadictos commented on GitHub (Oct 1, 2024):
Well, bad news then as it seems that the index was created:
@hardwareadictos commented on GitHub (Oct 13, 2024):
https://github.com/advplyr/audiobookshelf/pull/3487
🤔🤔🤔🤔🤔
@nichwall commented on GitHub (Oct 13, 2024):
Did that fix the problem?
@hardwareadictos commented on GitHub (Oct 14, 2024):
Don't know, I'm fighting with this now without success:
https://github.com/advplyr/audiobookshelf/issues/3276
@friral commented on GitHub (Oct 14, 2024):
The Newest Author load time is still (v2.15) very high for my 40K Library. Smaller libraries (1-2K) are fine.
Is there any Quickfix to achieve here to make the library usable again? Maybe make the query disable-able in the settings for now?
`2024-10-14 08:49:38.557
DEBUG
Loaded 8 of 8 items for "Continue Listening/Reading" in 0.72s
2024-10-14 08:49:43.263
DEBUG
Loaded 0 of 0 items for "Continue Series" in 4.71s
2024-10-14 08:49:43.672
DEBUG
Loaded 10 of 41478 items for "Recently Added" in 0.41s
2024-10-14 08:49:43.720
DEBUG
Loaded 5 of 2254 series for "Recent Series" in 0.05s
2024-10-14 08:49:47.160
DEBUG
Loaded 10 of 34816 items for "Discover" in 3.44s
2024-10-14 08:49:47.625
DEBUG
Loaded 4 of 4 items for "Listen/Read Again" in 0.47s
2024-10-14 08:52:07.970
DEBUG
Loaded 10 of 18309 authors for "Newest Authors" in 140.34s
2024-10-14 08:52:07.974
DEBUG
Loaded 6 personalized shelves in 150.13s`
@hardwareadictos commented on GitHub (Oct 16, 2024):
I think that's the correct way for temporary fixing this till the model improves.
@hardwareadictos commented on GitHub (Oct 19, 2024):
Well, updated to v2.15.1 and results are really promising!!!! @advplyr
This: DEBUG: Loaded 20 of 57001 items for libary page in 0.45s
Let me allow users to use the service again for testing it for a week to take a better conclusion. But there's a notable difference!! :D
@advplyr commented on GitHub (Oct 19, 2024):
We haven't added the indexes in yet for the performance improvement so it may be a fluke that query is fast for you. We did fix the issue with db corruption you were having though.
It shouldn't be long before those performance indexes are added
@hardwareadictos commented on GitHub (Oct 26, 2024):
Well, one week later now i didn't see any more crash!!! Myabe that could partially fixed it?? https://github.com/advplyr/audiobookshelf/pull/3487
Anyways! It doesn't crah now, but performance will improve with new performance index. Waiting patiently for the next release :D
@github-actions[bot] commented on GitHub (Oct 27, 2024):
Fixed in v2.16.0.
@hardwareadictos commented on GitHub (Oct 30, 2024):
Well rear some days of testing v2.16.X i can say that performance now is FINE. I have to say also that i notice a bigger impact since this https://github.com/advplyr/audiobookshelf/pull/3487 was implemented than with new indexes, anyways new indexes improves the situation a bit more :)