[Bug]: Restoring from large AudioBookShelf backup results in broken Author images and "Internal Server Error" #1931

New Issue

@Mega-Cookie commented on GitHub (Jun 12, 2024):

Had a similar Issue where the metadata/authors/ directory was not created at all after restoring from backup.

I'm also running a Docker container

MY WORKARROUND:

mkdir authors/ in metadata/
extract metadata-authors/ from the backup file (with WinRAR in my case)
copy contents of metadata-authors/ over to metadata/authors/

@Mega-Cookie commented on GitHub (Jun 12, 2024): Had a similar Issue where the metadata/authors/ directory was not created at all after restoring from backup. I'm also running a Docker container MY WORKARROUND: 1. mkdir authors/ in metadata/ 2. extract metadata-authors/ from the backup file (with WinRAR in my case) 3. copy contents of metadata-authors/ over to metadata/authors/

adam commented

@ZLoth commented on GitHub (Jul 12, 2024):

Had a similar Issue where the metadata/authors/ directory was not created at all after restoring from backup.

Issue is still occurring in 2.11 . But, I was able to restore the authors per your workaround.

@ZLoth commented on GitHub (Jul 12, 2024): > Had a similar Issue where the metadata/authors/ directory was not created at all after restoring from backup. Issue is still occurring in 2.11 . But, I was able to restore the authors per your workaround.

adam commented

@nichwall commented on GitHub (Jul 14, 2024):

Are you able to get any logs of the backup restore? I'm not able to reproduce the issue of author images not being restored using Docker version 2.11.0

My debug steps:

Made backup
Delete all images in /metadata/authors
Ensured author images were missing after purging cache in main server settings (checked /metadata/cache to make sure images were missing and that they did not show up in the UI)
Used the "Restore" button next to a backup
Verified no errors in logs, author images show up in /metadata/authors, and author images appear in web interface

I also tried:

Made backup
Changed an author image
Used the "Restore" button next to a backup
Verified no errors in logs, author images show up in /metadata/authors, and author images appear in web interface. The author image I changed after the backup was made did correctly restore to the original version

@nichwall commented on GitHub (Jul 14, 2024): Are you able to get any logs of the backup restore? I'm not able to reproduce the issue of author images not being restored using Docker version `2.11.0` My debug steps: - Made backup - Delete all images in `/metadata/authors` - Ensured author images were missing after purging cache in main server settings (checked `/metadata/cache` to make sure images were missing and that they did not show up in the UI) - Used the "Restore" button next to a backup - Verified no errors in logs, author images show up in `/metadata/authors`, and author images appear in web interface I also tried: - Made backup - Changed an author image - Used the "Restore" button next to a backup - Verified no errors in logs, author images show up in `/metadata/authors`, and author images appear in web interface. The author image I changed after the backup was made did correctly restore to the original version

adam commented

@ZLoth commented on GitHub (Jul 20, 2024):

@nichwall : My apologies for the delay as I was not able to attempt a reproduction of the issue until this weekend. My configuration is that I'm running TrueNAS Scale running Dragonfish-24.04.2, and I'm using a Docker container. My testing is that I set up a second test instance of ABS, with the only difference is that my production Config Storage and Metadata Storage, configured to use Host Path, pointed to /mnt/pool/app-config/abs-test instead of my production directory of /mnt/pool/app-config/audiobookshelf , and the port used was different from production.

STEPS TO REPLICATE:

Created a second ABS instance with the name of audiobookshelf-test.
Created a login account.
Went into Settings → Backup.
Restored from my backup. During the import, the socket got disconnected and reconnected.
Refreshed my screen and re-logged in.
Went into Authors.

WHAT SHOULD HAPPEN: Pictures should have been restored.
WHAT REALLY HAPPENS: Multiple errors thrown in the logs, and no pictures in the authors directory.

Here is a recording of the issue. My apologies for the big yellow box, as I don't want my users to be displayed.
https://github.com/user-attachments/assets/4a7dc434-2f7a-4a5c-a457-bf7b9c7dcc4e

Here are the log files: Logs.zip

@ZLoth commented on GitHub (Jul 20, 2024): @nichwall : My apologies for the delay as I was not able to attempt a reproduction of the issue until this weekend. My configuration is that I'm running TrueNAS Scale running Dragonfish-24.04.2, and I'm using a Docker container. My testing is that I set up a second test instance of ABS, with the only difference is that my production Config Storage and Metadata Storage, configured to use Host Path, pointed to /mnt/pool/app-config/abs-test instead of my production directory of /mnt/pool/app-config/audiobookshelf , and the port used was different from production. STEPS TO REPLICATE: 1. Created a second ABS instance with the name of audiobookshelf-test. 2. Created a login account. 3. Went into Settings → Backup. 4. Restored from my backup. During the import, the socket got disconnected and reconnected. 5. Refreshed my screen and re-logged in. 6. Went into Authors. WHAT SHOULD HAPPEN: Pictures should have been restored. WHAT REALLY HAPPENS: Multiple errors thrown in the logs, and no pictures in the authors directory. Here is a recording of the issue. My apologies for the big yellow box, as I don't want my users to be displayed. https://github.com/user-attachments/assets/4a7dc434-2f7a-4a5c-a457-bf7b9c7dcc4e Here are the log files: [Logs.zip](https://github.com/user-attachments/files/16321874/Logs.zip)

adam commented

@nichwall commented on GitHub (Jul 20, 2024):

Thanks for providing more information. I just tried following your steps (using the same hardware, but an entirely new directory path/container). I am not seeing the socket disconnect and reconnect or any errors in my logs. Granted, the largest backup file I have on hand for this is only about 200 MB.

I noticed in your video that your backup is 1.4 GB, do you have any smaller backups you could try?
Based on the log file showing that /metadata/authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg cannot be found while restoring the backup, I'm wondering if it could be:

permissions (does not make much sense since other things are restored)
file is missing (does not make sense if you can extract the backup and restore images manually)
backup file is too large or something is not copied correctly (also does not make sense because StreamZip is used)

Can you answer/do the following:

How many author images are in the backup?
How big/what percentage of the backup size is author images?
How big/what percentage of the backup size is from metadata-items?
Verify that metadata-authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg exists in the backup.
Look at /metadata/authors after the extraction and see if any files exist in the file system?

@nichwall commented on GitHub (Jul 20, 2024): Thanks for providing more information. I just tried following your steps (using the same hardware, but an entirely new directory path/container). I am not seeing the socket disconnect and reconnect or any errors in my logs. Granted, the largest backup file I have on hand for this is only about 200 MB. I noticed in your video that your backup is 1.4 GB, do you have any smaller backups you could try? Based on the log file showing that `/metadata/authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg` cannot be found while restoring the backup, I'm wondering if it could be: - permissions (does not make much sense since other things are restored) - file is missing (does not make sense if you can extract the backup and restore images manually) - backup file is too large or something is not copied correctly (also does not make sense because StreamZip is used) Can you answer/do the following: 1. How many author images are in the backup? 2. How big/what percentage of the backup size is author images? 3. How big/what percentage of the backup size is from `metadata-items`? 4. Verify that `metadata-authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg` exists in the backup. 5. Look at `/metadata/authors` after the extraction and see if any files exist in the file system?

adam commented

https://github.com/user-attachments/assets/410d5c42-9287-41af-baa1-09b13c03b5d5

@ZLoth commented on GitHub (Jul 21, 2024):

I noticed in your video that your backup is 1.4 GB, do you have any smaller backups you could try?

Negative. This is my daily backup of my production instance, and has over 4,400 books (thanks Plus catalog)

How many author images are in the backup?

How big/what percentage of the backup size is author images?

How big/what percentage of the backup size is from metadata-items?

Verify that metadata-authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg exists in the backup.

Look at /metadata/authors after the extraction and see if any files exist in the file system?

As you can see from the following five-minute recording where I went from adding the app in TrueNAS scale as a second instance, setting up the paths, restoring the backup, etc , both /metadata/items and /metadata/authors do not exist until I attempt restore from backup. After restoration from backup, items exist, but authors does not exist. It's only when I update the photo does the authors directory exist.

Log files: Logs.zip

@ZLoth commented on GitHub (Jul 21, 2024): > I noticed in your video that your backup is 1.4 GB, do you have any smaller backups you could try? Negative. This is my daily backup of my production instance, and has over 4,400 books (thanks Plus catalog) > 1. How many author images are in the backup? ![image](https://github.com/user-attachments/assets/3afec287-c918-4e5b-95f5-54640bcc4c32) > 2. How big/what percentage of the backup size is author images? > 3. How big/what percentage of the backup size is from `metadata-items`? ![image](https://github.com/user-attachments/assets/c99b826d-5809-4f18-a531-14a6668693e6) > 4. Verify that `metadata-authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg` exists in the backup. ![image](https://github.com/user-attachments/assets/9b1c9ef1-d7d8-425d-8dfd-382c56fb392b) > 5. Look at `/metadata/authors` after the extraction and see if any files exist in the file system? As you can see from the following five-minute recording where I went from adding the app in TrueNAS scale as a second instance, setting up the paths, restoring the backup, etc , both /metadata/items and /metadata/authors do not exist until I attempt restore from backup. After restoration from backup, items exist, but authors does not exist. It's only when I update the photo does the authors directory exist. https://github.com/user-attachments/assets/410d5c42-9287-41af-baa1-09b13c03b5d5 Log files: [Logs.zip](https://github.com/user-attachments/files/16322744/Logs.zip)

adam commented

@nichwall commented on GitHub (Jul 21, 2024):

Why is absdatabase.sqlite in your metadata folder (at 2:27)? Having config and metadata at the same mount point may be causing some of the problems.

@nichwall commented on GitHub (Jul 21, 2024): Why is `absdatabase.sqlite` in your metadata folder (at 2:27)? Having config and metadata at the same mount point may be causing some of the problems.

adam commented

https://github.com/user-attachments/assets/3728bc04-e08a-471d-9da8-1fc1d699f962

@ZLoth commented on GitHub (Jul 21, 2024):

Here is a recording where I created the subdirectories config and metadata as subdirectories under /mnt/pool/app-config/abs-test:

Here is a recording where I created directories /mnt/pool/app-config/abs-config and /mnt/pool/app-config/abs-metadata:

https://github.com/user-attachments/assets/de0d6df3-ca9f-4cb7-9ac2-2febd59342e3

Still same issue.

@ZLoth commented on GitHub (Jul 21, 2024): Here is a recording where I created the subdirectories config and metadata as subdirectories under /mnt/pool/app-config/abs-test: https://github.com/user-attachments/assets/3728bc04-e08a-471d-9da8-1fc1d699f962 Here is a recording where I created directories /mnt/pool/app-config/abs-config and /mnt/pool/app-config/abs-metadata: https://github.com/user-attachments/assets/de0d6df3-ca9f-4cb7-9ac2-2febd59342e3 Still same issue.

adam commented

https://github.com/user-attachments/assets/7fc44168-7ace-4862-823e-8dc4b00a82a5

@ZLoth commented on GitHub (Jul 21, 2024):

And, just for fun, I'm using the IX defaults for the metadata and config:

@ZLoth commented on GitHub (Jul 21, 2024): And, just for fun, I'm using the IX defaults for the metadata and config: https://github.com/user-attachments/assets/7fc44168-7ace-4862-823e-8dc4b00a82a5

adam commented

@ZLoth commented on GitHub (Jul 21, 2024):

What is the order of the restoration of the backup? Is it items, then the sqlite DB, then authors?

@ZLoth commented on GitHub (Jul 21, 2024): What is the order of the restoration of the backup? Is it items, then the sqlite DB, then authors?

adam commented

@nichwall commented on GitHub (Jul 21, 2024):

SQLite, items, then authors
https://github.com/advplyr/audiobookshelf/blob/master/server%2Fmanagers%2FBackupManager.js#L171-L233

@nichwall commented on GitHub (Jul 21, 2024): SQLite, items, then authors https://github.com/advplyr/audiobookshelf/blob/master/server%2Fmanagers%2FBackupManager.js#L171-L233

adam commented

@nichwall commented on GitHub (Jul 21, 2024):

If you're willing to DM a download link for the backup file, I can try and recreate the issue specifically from your backup. (Though, maybe change the admin user password to "password" or something so it's easier for me to actually log in and verify things)

@nichwall commented on GitHub (Jul 21, 2024): If you're willing to DM a download link for the backup file, I can try and recreate the issue specifically from your backup. (Though, maybe change the admin user password to "password" or something so it's easier for me to actually log in and verify things)

adam commented

@ZLoth commented on GitHub (Jul 21, 2024):

@nichwall : As the file is 1.5 GB in size, a WeTransfer link has been sent to you directly. The link will expire on July 24th.

@ZLoth commented on GitHub (Jul 21, 2024): @nichwall : As the file is 1.5 GB in size, a WeTransfer link has been sent to you directly. The link will expire on July 24th.

adam commented

@nichwall commented on GitHub (Jul 21, 2024):

Thanks, I was able to reproduce the server crash during restoring the backup with your backup file and was able to log in using the new password you created.
I'll see if I can find more information.

@nichwall commented on GitHub (Jul 21, 2024): Thanks, I was able to reproduce the server crash during restoring the backup with your backup file and was able to log in using the new password you created. I'll see if I can find more information.

adam commented

@nichwall commented on GitHub (Jul 21, 2024):

Okay, pretty sure I found the problem.

In your /metadata/authors folder, there are only files (no directories). Backups from my main server (running on Synology) have a hidden folder @eaDir in /metadata/authors. It looks like folders are only created during extraction if there is a folder in the path, so when metadata-authors/ is extracted from the backup, you don't have a folder under metadata-authors/, so the authors directory is not created in the file system. This does not matter for the database because it is extracting a single file to an existing folder, and the metadata-items/ part of the backup is a bunch of folders so /metadata/items gets created.

I added a dummy folder (tried different names to make sure alphabetical sorting didn't matter) to your /metadata/authors folder, made a backup, and then was able to successfully restore the backup on a new server. I also verified that deleting the hidden folder from my backup caused restoring from backup to fail.

@nichwall commented on GitHub (Jul 21, 2024): Okay, pretty sure I found the problem. In your `/metadata/authors` folder, there are only files (no directories). Backups from my main server (running on Synology) have a hidden folder `@eaDir` in `/metadata/authors`. It looks like folders are only created during extraction if there is a folder in the path, so when `metadata-authors/` is extracted from the backup, you don't have a folder under `metadata-authors/`, so the `authors` directory is not created in the file system. This does not matter for the database because it is extracting a single file to an existing folder, and the `metadata-items/` part of the backup is a bunch of folders so `/metadata/items` gets created. I added a dummy folder (tried different names to make sure alphabetical sorting didn't matter) to your `/metadata/authors` folder, made a backup, and then was able to successfully restore the backup on a new server. I also verified that deleting the hidden folder from my backup caused restoring from backup to fail.

adam commented

https://github.com/user-attachments/assets/54cb0b9b-980f-4dc8-9703-6d5311e5a990

@ZLoth commented on GitHub (Jul 21, 2024):

As a test, I manually created the authors and items folders, chown-ed them to be the apps owner, then performed the restore. This time, no crash, and the authors were restored. So, per your pull request, ensuring that authors and Items folders are created prior to restore is a good fix.

And all of the authors were restored.

https://github.com/user-attachments/assets/0b2b7995-103e-4f19-b1c9-a8738fff7d3f

Thank you for tracking this issue down. I look forward to the fix, although hopefully, I won't have to do a restore for quite a while.

@ZLoth commented on GitHub (Jul 21, 2024): As a test, I manually created the authors and items folders, chown-ed them to be the apps owner, then performed the restore. This time, no crash, and the authors were restored. So, per your pull request, ensuring that authors and Items folders are created prior to restore is a good fix. https://github.com/user-attachments/assets/54cb0b9b-980f-4dc8-9703-6d5311e5a990 And all of the authors were restored. https://github.com/user-attachments/assets/0b2b7995-103e-4f19-b1c9-a8738fff7d3f Thank you for tracking this issue down. I look forward to the fix, although hopefully, I won't have to do a restore for quite a while.

adam commented

@ZLoth commented on GitHub (Jul 23, 2024):

Thank you!

@ZLoth commented on GitHub (Jul 23, 2024): Thank you!

adam commented