starred/audiobookshelf
1
0
Fork 0
mirror of https://github.com/advplyr/audiobookshelf.git synced 2026-05-30 23:40:40 +02:00
Code Issues 1.0k Packages Projects Releases 10 Wiki Activity

[Bug]: "Match All Authors" with a big library will hit rate limits on Audible API and start getting 429s #994

New Issue
Closed
opened 2026-04-24 23:28:35 +02:00 by adam · 12 comments
adam commented 2026-04-24 23:28:35 +02:00
Owner
Copy Link
Copy Source

Originally created by @FreedomBen on GitHub (Mar 1, 2023).

Describe the issue

Expected behaviour
Ideally the requests would be queued server-side to throttle them to avoid exceeding the rate limit threshold

Actual behaviour
All lookups start to fail with 429s from Audible API. If your library is big enough, you may even get an IP ban

Steps to reproduce the issue

  1. Load hundreds of books from different authors into a library
  2. Visit the "authors" page
  3. Click "Match All Authors"
  4. Observe backend 429 errors from Audible API after several dozen lookups

Audiobookshelf version

v2.2.4. I can update and re-test in a little while but for the next week or so I need it to be perfectly stable as I'm traveling

How are you running audiobookshelf?

Docker

Originally created by @FreedomBen on GitHub (Mar 1, 2023). ### Describe the issue **Expected behaviour** Ideally the requests would be queued server-side to throttle them to avoid exceeding the rate limit threshold **Actual behaviour** All lookups start to fail with 429s from Audible API. If your library is big enough, you may even get an IP ban ### Steps to reproduce the issue 1. Load hundreds of books from different authors into a library 2. Visit the "authors" page 3. Click "Match All Authors" 4. Observe backend 429 errors from Audible API after several dozen lookups ### Audiobookshelf version v2.2.4. I can update and re-test in a little while but for the next week or so I need it to be perfectly stable as I'm traveling ### How are you running audiobookshelf? Docker
adam added the bug label 2026-04-24 23:28:35 +02:00
adam closed this issue 2026-04-24 23:28:36 +02:00
adam commented 2026-04-24 23:28:37 +02:00
Author
Owner
Copy Link
Copy Source

@FreedomBen commented on GitHub (Mar 1, 2023):

Would a client-side fix in client/components/app/BookShelfToolbar.vue method matchAllAuthors() to add a small delay between POST requests be a reasonable (and acceptable for a PR) place to implement this?

Given that the endpoint is currently synchronous, it seems like a much bigger (and contract breaking) effort to turn it async and update the client accordingly, but I'm quite unfamiliar with this codebase so maybe there's something with the web socket that could be done without too much effort.

@FreedomBen commented on GitHub (Mar 1, 2023): Would a client-side fix in `client/components/app/BookShelfToolbar.vue` method `matchAllAuthors()` to add a small delay between POST requests be a reasonable (and acceptable for a PR) place to implement this? Given that the endpoint is currently synchronous, it seems like a much bigger (and contract breaking) effort to turn it async and update the client accordingly, but I'm quite unfamiliar with this codebase so maybe there's something with the web socket that could be done without too much effort.
adam commented 2026-04-24 23:28:38 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Mar 1, 2023):

Yeah I think a client-side fix is best here for the reason you pointed out. PR would be great.

We can do it server side with web socket events but probably more effort then it is worth at this stage. We're working on a new model to handle long running tasks so in the future we could make a match all authors route.

@advplyr commented on GitHub (Mar 1, 2023): Yeah I think a client-side fix is best here for the reason you pointed out. PR would be great. We can do it server side with web socket events but probably more effort then it is worth at this stage. We're working on a new model to handle long running tasks so in the future we could make a match all authors route.
adam commented 2026-04-24 23:28:38 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Mar 1, 2023):

One issue that just occurred to me is if the user navigates while the loop is running

@advplyr commented on GitHub (Mar 1, 2023): One issue that just occurred to me is if the user navigates while the loop is running
adam commented 2026-04-24 23:28:38 +02:00
Author
Owner
Copy Link
Copy Source

@sebclark commented on GitHub (Mar 6, 2023):

Perhaps a second button "Search Missing Authors". To lower the search volume.

@sebclark commented on GitHub (Mar 6, 2023): Perhaps a second button "Search Missing Authors". To lower the search volume.
adam commented 2026-04-24 23:28:39 +02:00
Author
Owner
Copy Link
Copy Source

@Dr-Blank commented on GitHub (Mar 19, 2023):

To lower the api calls further, maybe implement a maintenance task to search only the newly added authors, for e.g after a library scan, queue an author's update scan if new authors were added.

It will be high volume for the initial scan, but only searching for authors that have been added recently, probably after the last update, should only be queried, and also only authors not previously match in case they got an update. Similar to scan and force re-scan feature separation in case of library.

And I do not know how the code structure is right now, but shouldn't this make more sense as a server-side feature instead of client side? I mean server is responsible for maintaining the db of books, scanning and updating, why not authors too, just send an update call to server and let it handle. For larger libraries, you have to keep the tab open, also it uses a lot of resources, not to mention all the notifications of foo author found, bar author not found freezing up my system briefly every now and then.

@Dr-Blank commented on GitHub (Mar 19, 2023): To lower the api calls further, maybe implement a maintenance task to search only the newly added authors, for e.g after a library scan, queue an author's update scan if new authors were added. It will be high volume for the initial scan, but only searching for authors that have been added recently, probably after the last update, should only be queried, and also only authors not previously match in case they got an update. Similar to scan and force re-scan feature separation in case of library. And I do not know how the code structure is right now, but shouldn't this make more sense as a server-side feature instead of client side? I mean server is responsible for maintaining the db of books, scanning and updating, why not authors too, just send an update call to server and let it handle. For larger libraries, you have to keep the tab open, also it uses a lot of resources, not to mention all the notifications of foo author found, bar author not found freezing up my system briefly every now and then.
adam commented 2026-04-24 23:28:39 +02:00
Author
Owner
Copy Link
Copy Source

@TheSylus commented on GitHub (May 1, 2023):

Any news on this? My library is pretty large so doing a rescan is already pretty impossible.

@TheSylus commented on GitHub (May 1, 2023): Any news on this? My library is pretty large so doing a rescan is already pretty impossible.
adam commented 2026-04-24 23:28:40 +02:00
Author
Owner
Copy Link
Copy Source

@ZLoth commented on GitHub (May 5, 2024):

Describe the issue

In some of my AudioBookShelf library, I have hundreds of authors. When I attempt to do an author match using Authors → Match All Authors, I get a match on some authors, but errors on others, including well-known authors. A rescan results in some of those missed authors being matched up. When I review the servers logs, I see the following:

2024-05-05 12:44:57.626 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors/B004AIN4W4?region=us"
2024-05-05 12:44:57.705 - ERROR - [Audnexus] Author request failed for B004AIN4W4 AxiosError: Request failed with status code 429
2024-05-05 12:44:57.730 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors?name=David+Javerbaum&region=us"
2024-05-05 12:44:57.821 - ERROR - [Audnexus] Author ASIN request failed for David Javerbaum AxiosError: Request failed with status code 429
2024-05-05 12:44:58.060 - DEBUG - [Audnexus] Looking up author by name Day Keene
2024-05-05 12:44:58.061 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors?name=Day+Keene&region=us"
2024-05-05 12:44:58.133 - ERROR - [Audnexus] Author ASIN request failed for Day Keene AxiosError: Request failed with status code 429

This tells me that AudioBookShelf is sending too many requests to api.audnex.us within a short period of time and needs to slow down and be rate limited to maybe one request every .25 to .5 seconds. Having author's being matched up is not a time-critical function, and waiting several minutes for the matching to complete should not be an unexpected condition. It may warrant a warning message when the administrator clicks on the Match All Authors that it may take some time to complete.

Screenshot from logs screen:
image

Steps to reproduce the issue

  1. Go to a library with large number of authors
  2. Go to Authors
  3. Click on March All Authors
  4. Wait until red error messages appear.
  5. Go to Settings → Logs
  6. Filter for Warn-level messages

Audiobookshelf version

2.9.0

How are you running audiobookshelf?

Docker

@ZLoth commented on GitHub (May 5, 2024): ### Describe the issue In some of my AudioBookShelf library, I have hundreds of authors. When I attempt to do an author match using Authors → Match All Authors, I get a match on some authors, but errors on others, including well-known authors. A rescan results in some of those missed authors being matched up. When I review the servers logs, I see the following: ``` 2024-05-05 12:44:57.626 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors/B004AIN4W4?region=us" 2024-05-05 12:44:57.705 - ERROR - [Audnexus] Author request failed for B004AIN4W4 AxiosError: Request failed with status code 429 2024-05-05 12:44:57.730 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors?name=David+Javerbaum&region=us" 2024-05-05 12:44:57.821 - ERROR - [Audnexus] Author ASIN request failed for David Javerbaum AxiosError: Request failed with status code 429 2024-05-05 12:44:58.060 - DEBUG - [Audnexus] Looking up author by name Day Keene 2024-05-05 12:44:58.061 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors?name=Day+Keene&region=us" 2024-05-05 12:44:58.133 - ERROR - [Audnexus] Author ASIN request failed for Day Keene AxiosError: Request failed with status code 429 ``` This tells me that AudioBookShelf is sending too many requests to api.audnex.us within a short period of time and needs to slow down and be rate limited to maybe one request every .25 to .5 seconds. Having author's being matched up is not a time-critical function, and waiting several minutes for the matching to complete should not be an unexpected condition. It may warrant a warning message when the administrator clicks on the Match All Authors that it may take some time to complete. Screenshot from logs screen: ![image](https://github.com/advplyr/audiobookshelf/assets/6700159/e622c051-2d2e-449c-ace3-d405d7a8bad8) ### Steps to reproduce the issue 1. Go to a library with large number of authors 2. Go to Authors 3. Click on March All Authors 4. Wait until red error messages appear. 5. Go to Settings → Logs 6. Filter for Warn-level messages ### Audiobookshelf version 2.9.0 ### How are you running audiobookshelf? Docker
adam commented 2026-04-24 23:28:41 +02:00
Author
Owner
Copy Link
Copy Source

@ZLoth commented on GitHub (Jun 9, 2024):

Thank you. Looking forward to the release! @advplyr

@ZLoth commented on GitHub (Jun 9, 2024): Thank you. Looking forward to the release! @advplyr
adam commented 2026-04-24 23:28:41 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Jul 9, 2024):

Fixed in v2.11.0

@advplyr commented on GitHub (Jul 9, 2024): Fixed in [v2.11.0](https://github.com/advplyr/audiobookshelf/releases/tag/v2.11.0)
adam commented 2026-04-24 23:28:42 +02:00
Author
Owner
Copy Link
Copy Source

@ZLoth commented on GitHub (Jul 22, 2024):

Thank you!

@ZLoth commented on GitHub (Jul 22, 2024): Thank you!
adam commented 2026-04-24 23:28:42 +02:00
Author
Owner
Copy Link
Copy Source

@Death916 commented on GitHub (Nov 17, 2024):

Not sure if this is fully fixed. Ran a match today after updating to 2.16 and got rate limited. Getting too many requewsts in the docker logs
` url: 'https://openlibrary.org//works/OL19340214W.json',

  data: undefined

},

request: <ref *1> ClientRequest {

  _events: [Object: null prototype],

  _eventsCount: 7,

  _maxListeners: undefined,

  outputData: [],

  outputSize: 0,

  writable: true,

  destroyed: true,

  _last: true,

  chunkedEncoding: false,

  shouldKeepAlive: true,

  maxRequestsOnConnectionReached: false,

  _defaultKeepAlive: true,

  useChunkedEncodingByDefault: false,

  sendDate: false,

  _removedConnection: false,

  _removedContLen: false,

  _removedTE: false,

  strictContentLength: false,

  _contentLength: 0,

  _hasBody: true,

  _trailer: '',

  finished: true,

  _headerSent: true,

  _closed: true,

  socket: [TLSSocket],

  _header: 'GET /works/OL19340214W.json HTTP/1.1\r\n' +

    'Accept: application/json, text/plain, */*\r\n' +

    'User-Agent: axios/0.27.2\r\n' +

    'Host: openlibrary.org\r\n' +

    'Connection: keep-alive\r\n' +

    '\r\n',

  _keepAliveTimeout: 0,

  _onPendingData: [Function: nop],

  agent: [Agent],

  socketPath: undefined,

  method: 'GET',

  maxHeaderSize: undefined,

  insecureHTTPParser: undefined,

  joinDuplicateHeaders: undefined,

  path: '/works/OL19340214W.json',

  _ended: true,

  res: [IncomingMessage],

  aborted: false,

  timeoutCb: null,

  upgradeOrConnect: false,

  parser: null,

  maxHeadersCount: null,

  reusedSocket: true,

  host: 'openlibrary.org',

  protocol: 'https:',

  _redirectable: [Writable],

  [Symbol(shapeMode)]: false,

  [Symbol(kCapture)]: false,

  [Symbol(kBytesWritten)]: 0,

  [Symbol(kNeedDrain)]: false,

  [Symbol(corked)]: 0,

  [Symbol(kOutHeaders)]: [Object: null prototype],

  [Symbol(errored)]: null,

  [Symbol(kHighWaterMark)]: 16384,

  [Symbol(kRejectNonStandardBodyWrites)]: false,

  [Symbol(kUniqueHeaders)]: null

},

data: {

  status: 429,

  message: 'Too Many Requests. Consider using https://openlibrary.org/developers/dumps.'

}

}

}`

@Death916 commented on GitHub (Nov 17, 2024): Not sure if this is fully fixed. Ran a match today after updating to 2.16 and got rate limited. Getting too many requewsts in the docker logs ` url: 'https://openlibrary.org//works/OL19340214W.json', data: undefined }, request: <ref *1> ClientRequest { _events: [Object: null prototype], _eventsCount: 7, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: true, chunkedEncoding: false, shouldKeepAlive: true, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: false, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 0, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: true, socket: [TLSSocket], _header: 'GET /works/OL19340214W.json HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'User-Agent: axios/0.27.2\r\n' + 'Host: openlibrary.org\r\n' + 'Connection: keep-alive\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'GET', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/works/OL19340214W.json', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: true, host: 'openlibrary.org', protocol: 'https:', _redirectable: [Writable], [Symbol(shapeMode)]: false, [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(errored)]: null, [Symbol(kHighWaterMark)]: 16384, [Symbol(kRejectNonStandardBodyWrites)]: false, [Symbol(kUniqueHeaders)]: null }, data: { status: 429, message: 'Too Many Requests. Consider using https://openlibrary.org/developers/dumps.' } } }`
adam commented 2026-04-24 23:28:43 +02:00
Author
Owner
Copy Link
Copy Source

@advplyr commented on GitHub (Nov 17, 2024):

Open library isn't used for matching authors so that is unrelated to this

@advplyr commented on GitHub (Nov 17, 2024): Open library isn't used for matching authors so that is unrelated to this
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
authentication
backlog
bug
chapter editor
config-issue
ebooks
encoding/embedding
enhancement
help wanted
listening sessions & progress
planned
possible plugin
progress sync
pull-request
Mirrored from GitHub Pull Request
 sorting/filtering/searching
unable to reproduce
upload
users & permissions
waiting
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam (Adam Melkus)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/audiobookshelf#994
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?