starred/audiobookshelf
1
0
Fork 0
mirror of https://github.com/advplyr/audiobookshelf.git synced 2026-05-30 23:40:40 +02:00
Code Issues 1.0k Packages Projects Releases 10 Wiki Activity

[Bug]: JavaScript heap out of memory #1143

New Issue
Closed
opened 2026-04-24 23:34:00 +02:00 by adam · 12 comments
adam commented 2026-04-24 23:34:00 +02:00
Owner
Copy Link
Copy Source

Originally created by @czahoi on GitHub (May 18, 2023).

Describe the issue

server down when scanning large folders

root@debian:~#` docker logs --tail 20 audiobookshelf
Config /config /metadata
[2023-05-18 14:39:55] INFO: === Starting Server ===
[2023-05-18 14:39:55] INFO: [Server] Init v2.2.20
[2023-05-18 14:39:56] INFO: [DB] 0 Collections Loaded
[2023-05-18 14:39:56] INFO: [DB] 0 Playlists Loaded
[2023-05-18 14:39:56] INFO: [DB] 1 Users Loaded
[2023-05-18 14:39:56] INFO: [DB] 1 Libraries Loaded
[2023-05-18 14:39:56] INFO: [DB] 1292 Authors Loaded
[2023-05-18 14:39:56] INFO: [DB] 12 Series Loaded

<--- Last few GCs --->

[7:0x7f373a1723f0] 32906 ms: Mark-sweep (reduce) 2046.9 (2091.1) -> 2046.6 (2083.3) MB, 760.0 / 0.0 ms (+ 548.6 ms in 109 steps since start of marking, biggest step 14.6 ms, walltime since start of marking 1335 ms) (average mu = 0.262, current mu = 0.[7:0x7f373a1723f0] 33734 ms: Mark-sweep (reduce) 2047.6 (2083.3) -> 2047.5 (2084.3) MB, 768.6 / 0.0 ms (+ 46.6 ms in 15 steps since start of marking, biggest step 4.0 ms, walltime since start of marking 828 ms) (average mu = 0.170, current mu = 0.016)

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Steps to reproduce the issue

Audiobookshelf version

v2.2.20

How are you running audiobookshelf?

Docker

Originally created by @czahoi on GitHub (May 18, 2023). ### Describe the issue server down when scanning large folders root@debian:~#` docker logs --tail 20 audiobookshelf Config /config /metadata [2023-05-18 14:39:55] INFO: === Starting Server === [2023-05-18 14:39:55] INFO: [Server] Init v2.2.20 [2023-05-18 14:39:56] INFO: [DB] 0 Collections Loaded [2023-05-18 14:39:56] INFO: [DB] 0 Playlists Loaded [2023-05-18 14:39:56] INFO: [DB] 1 Users Loaded [2023-05-18 14:39:56] INFO: [DB] 1 Libraries Loaded [2023-05-18 14:39:56] INFO: [DB] 1292 Authors Loaded [2023-05-18 14:39:56] INFO: [DB] 12 Series Loaded <--- Last few GCs ---> [7:0x7f373a1723f0] 32906 ms: Mark-sweep (reduce) 2046.9 (2091.1) -> 2046.6 (2083.3) MB, 760.0 / 0.0 ms (+ 548.6 ms in 109 steps since start of marking, biggest step 14.6 ms, walltime since start of marking 1335 ms) (average mu = 0.262, current mu = 0.[7:0x7f373a1723f0] 33734 ms: Mark-sweep (reduce) 2047.6 (2083.3) -> 2047.5 (2084.3) MB, 768.6 / 0.0 ms (+ 46.6 ms in 15 steps since start of marking, biggest step 4.0 ms, walltime since start of marking 828 ms) (average mu = 0.170, current mu = 0.016) <--- JS stacktrace ---> FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory ### Steps to reproduce the issue 1. ### Audiobookshelf version v2.2.20 ### How are you running audiobookshelf? Docker
adam added the bug label 2026-04-24 23:34:00 +02:00
adam closed this issue 2026-04-24 23:34:00 +02:00
adam commented 2026-04-24 23:34:01 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (May 18, 2023):

This may be a limitation with your system. How big of a folder are we talking about? I'm sure there are users with larger folders running fine.

@advplyr commented on GitHub (May 18, 2023): This may be a limitation with your system. How big of a folder are we talking about? I'm sure there are users with larger folders running fine.
adam commented 2026-04-24 23:34:02 +02:00
Author
Owner
Copy Link
Copy Source

@czahoi commented on GitHub (May 19, 2023):

2093 folders 1268285 files 12.2TB
a vm running on proxmox :
8 cores i5-12400f \ 10G RAM \ nvme ssd
zfs file system

@czahoi commented on GitHub (May 19, 2023): 2093 folders 1268285 files 12.2TB a vm running on proxmox : 8 cores i5-12400f \ 10G RAM \ nvme ssd zfs file system
adam commented 2026-04-24 23:34:03 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (May 19, 2023):

That actually is the largest folder I've heard being used in Abs. I could see a memory issue happening because right now the data for the library items are stored in memory.

@advplyr commented on GitHub (May 19, 2023): That actually is the largest folder I've heard being used in Abs. I could see a memory issue happening because right now the data for the library items are stored in memory.
adam commented 2026-04-24 23:34:03 +02:00
Author
Owner
Copy Link
Copy Source

@czahoi commented on GitHub (May 20, 2023):

oh，it's only 40 percent of my whole audiobook library，and all audiobooks in this part are optimized for Audiobookshelf server.
Will it be improved in the next release?

@czahoi commented on GitHub (May 20, 2023): oh，it's only 40 percent of my whole audiobook library，and all audiobooks in this part are optimized for Audiobookshelf server. Will it be improved in the next release?
adam commented 2026-04-24 23:34:04 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (May 20, 2023):

This won't be improved anytime soon since it is a full rewrite of the database. I have put some effort into it and do plan on switching to a sqlite database but it is a huge amount of work.
You would need to use Abs with a subset of your books or drastically increase the memory you are allowing Abs to use.

To clarify for anyone that might come across this issue, this issue should not impact you unless either
a) you have a very large amount of books/audio files. (e.g. several Terabytes)
b) the server only has access to a small amount of memory

In general, Abs doesn't use much memory in comparison to other media servers.

@advplyr commented on GitHub (May 20, 2023): This won't be improved anytime soon since it is a full rewrite of the database. I have put some effort into it and do plan on switching to a sqlite database but it is a huge amount of work. You would need to use Abs with a subset of your books or drastically increase the memory you are allowing Abs to use. To clarify for anyone that might come across this issue, this issue should not impact you unless either a) you have a very large amount of books/audio files. (e.g. several Terabytes) b) the server only has access to a small amount of memory In general, Abs doesn't use much memory in comparison to other media servers.
adam commented 2026-04-24 23:34:04 +02:00
Author
Owner
Copy Link
Copy Source

@mayli commented on GitHub (Jun 4, 2023):

hmm, njodb actually claims

njodb is a persistent, partitioned, concurrency-controlled, Node JSON object database. Data is written to the file system and distributed across multiple files that are protected by read and write locks. By default, all methods are asynchronous and use read/write streams to improve performance and reduce memory requirements (this should be particularly useful for large databases).

@mayli commented on GitHub (Jun 4, 2023): hmm, njodb actually claims > njodb is a persistent, partitioned, concurrency-controlled, Node JSON object database. Data is written to the file system and distributed across multiple files that are protected by read and write locks. By default, all methods are asynchronous and use read/write streams to improve performance and **reduce memory requirement**s (this should be particularly useful for large databases).
adam commented 2026-04-24 23:34:05 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Jun 4, 2023):

Yes but we are loading everything into memory on initialization. I think the goal should be to implement a database where we can query only the data we need. There are a number of issues we have come up against with the current JSON object db, see #1326

There is also an issue that can occur if the server gets terminated while writing resulting in loss of data.

@advplyr commented on GitHub (Jun 4, 2023): Yes but we are loading everything into memory on initialization. I think the goal should be to implement a database where we can query only the data we need. There are a number of issues we have come up against with the current JSON object db, see #1326 There is also an issue that can occur if the server gets terminated while writing resulting in loss of data.
adam commented 2026-04-24 23:34:06 +02:00
Author
Owner
Copy Link
Copy Source

@mayli commented on GitHub (Jun 11, 2023):

Yes but we are loading everything into memory on initialization. I think the goal should be to implement a database where we can query only the data we need. There are a number of issues we have come up against with the current JSON object db, see #1326

There is also an issue that can occur if the server gets terminated while writing resulting in loss of data.

Or use sqlite with indexes same as plex/emby, I think there is a plan already.
I think it might be interesting to try the sqlite-json support since it could be easier than go relational all the way.

https://antonz.org/json-virtual-columns/

@mayli commented on GitHub (Jun 11, 2023): > Yes but we are loading everything into memory on initialization. I think the goal should be to implement a database where we can query only the data we need. There are a number of issues we have come up against with the current JSON object db, see #1326 > > There is also an issue that can occur if the server gets terminated while writing resulting in loss of data. Or use sqlite with indexes same as plex/emby, I think there is a plan already. I think it might be interesting to try the sqlite-json support since it could be easier than go relational all the way. https://antonz.org/json-virtual-columns/
adam commented 2026-04-24 23:34:06 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Jun 11, 2023):

I think it might be interesting to try the sqlite-json support since it could be easier than go relational all the way.

This is what I was thinking also

@advplyr commented on GitHub (Jun 11, 2023): > I think it might be interesting to try the sqlite-json support since it could be easier than go relational all the way. This is what I was thinking also
adam commented 2026-04-24 23:34:07 +02:00
Author
Owner
Copy Link
Copy Source

@Timtam commented on GitHub (Jun 19, 2023):

I'd like to second that. My audio book archive is much smaller (only rougly 3.5 TB in size), but i've got only 1 GB of RAM available on my Synology NAS for now and I already hit the memory restriction. I'd love to be able to use audiobookshelf, as only feeding in a subset of my audio books isn't the goal either, I want ABS because I want to have all my books in there and organize them properly.

@Timtam commented on GitHub (Jun 19, 2023): I'd like to second that. My audio book archive is much smaller (only rougly 3.5 TB in size), but i've got only 1 GB of RAM available on my Synology NAS for now and I already hit the memory restriction. I'd love to be able to use audiobookshelf, as only feeding in a subset of my audio books isn't the goal either, I want ABS because I want to have all my books in there and organize them properly.
adam commented 2026-04-24 23:34:07 +02:00
Author
Owner
Copy Link
Copy Source

@paxchristos commented on GitHub (Jul 16, 2023):

Running into this with a ~30gb ebook folder + ~1.7 of audiobooks on 2.3.0

Did not run into heap limit allocation failed on previous release.

i3 10100
16gb ram
m2 nvme for /config & /metadata

crash happens on both native linux install and docker compose install (using same config and metadata folders for both.)

edit: rolling back to 2.2.23 allows audiobookshelf to work again.

@paxchristos commented on GitHub (Jul 16, 2023): Running into this with a ~30gb ebook folder + ~1.7 of audiobooks on 2.3.0 Did not run into heap limit allocation failed on previous release. i3 10100 16gb ram m2 nvme for /config & /metadata crash happens on both native linux install and docker compose install (using same config and metadata folders for both.) edit: rolling back to 2.2.23 allows audiobookshelf to work again.
adam commented 2026-04-24 23:34:07 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Sep 13, 2023):

Abs now uses sqlite and minimal data is loaded into memory. There are still performance issues to workout with the sql queries and if you have a podcast with thousands of episodes this could potentially cause an OOM crash. However, this issue is resolved as of v2.4.x

@advplyr commented on GitHub (Sep 13, 2023): Abs now uses sqlite and minimal data is loaded into memory. There are still performance issues to workout with the sql queries and if you have a podcast with thousands of episodes this could potentially cause an OOM crash. However, this issue is resolved as of v2.4.x
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
authentication
backlog
bug
chapter editor
config-issue
ebooks
encoding/embedding
enhancement
help wanted
listening sessions & progress
planned
possible plugin
progress sync
pull-request
Mirrored from GitHub Pull Request
 sorting/filtering/searching
unable to reproduce
upload
users & permissions
waiting
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam (Adam Melkus)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/audiobookshelf#1143
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?