starred/CoreStore-JohnEstropia
1
0
Fork 0
mirror of https://github.com/JohnEstropia/CoreStore.git synced 2026-01-14 07:33:28 +01:00
Code Issues 95 Packages Projects Releases 10 Wiki Activity

Support for Synchronous Lightweight Migration (adding a new entity) #424

New Issue
Closed
opened 2025-12-29 18:26:49 +01:00 by adam · 5 comments
adam commented 2025-12-29 18:26:49 +01:00
Owner
Copy Link

Originally created by @ptrkstr on GitHub (Jan 3, 2024).

Hey @JohnEstropia, thank you for CoreStore ❤️

I am following the Synchronous Lightweight Migration guide from the readme.
My new model has a new entity which is supported by lightweight migration:

You can add, remove, and rename entities in the hierarchy.

This is code that is live (CoreStore 9.2.0):

class Dog: CoreStoreObject, Identifiable {

    @Field.Stored("identifier", dynamicInitialValue: { UUID() })
    var id: UUID

    @Field.Stored("name")
    var name: String = ""
}

CoreStoreDefaults.dataStack = DataStack(
    CoreStoreSchema(
        modelVersion: "V1",
        entities: [
            Entity<Dog>("Dog")
        ],
        versionLock: [
            "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3]
        ]
    )
)

try! CoreStoreDefaults.dataStack.addStorageAndWait()

This is a change in the code to introduce the entity Cat (no change to Dog):

// struct Dog...

class Cat: CoreStoreObject, Identifiable {

    @Field.Stored("identifier", dynamicInitialValue: { UUID() })
    var id: UUID

    @Field.Stored("name")
    var name: String = ""
}

CoreStoreDefaults.dataStack = DataStack(
    CoreStoreSchema(
        modelVersion: "V1",
        entities: [
            Entity<Dog>("Dog")
        ],
        versionLock: [
            "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3] // Same version lock
        ]
    ),
    CoreStoreSchema(
        modelVersion: "V2",
        entities: [
            Entity<Dog>("Dog"),
            Entity<Cat>("Cat")
        ],
        versionLock: [
            "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3], // Same version lock
            "Cat": [0x303f93c3bcb0c108, 0x3ca1f5d4fdc3e42b, 0xbcb7e18668333b8e, 0x4c26e8e6a7327eb3],
        ]
    ),
    migrationChain: ["V1", "V2"]
)

try! CoreStoreDefaults.dataStack.addStorageAndWait(
    SQLiteStore(
        fileURL: SQLiteStore().fileURL,
        localStorageOptions: .allowSynchronousLightweightMigration
    )
)

Errors when running

CoreData

DataStack.swift:544 calls coordinator.addPersistentStore which triggers the error to console:

CoreData: error: addPersistentStoreWithType:configuration:URL:options:error: returned error NSCocoaErrorDomain (134130)
CoreData: error: userInfo:
CoreData: error: 	URL : file:///...
CoreData: error: 	metadata : {
    NSPersistenceFrameworkVersion = 1338;
    NSStoreModelVersionChecksumKey = "SeJK6kdLCNX3AK7GOZrsIs6fsaQIUdP+LE1FNjyBDEU=";
    NSStoreModelVersionHashes =     {
        Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 };
    };
    NSStoreModelVersionHashesDigest = "2KMBV9An/E0/QE6gU2uS7lysEJgk05M/NCNlG2OCR0bOF31qLgZXvvcOBWtOGtSTs+0sXr0xHQPCO97ibCouyw==";
    NSStoreModelVersionHashesVersion = 3;
    NSStoreModelVersionIdentifiers =     (
    );
    NSStoreType = SQLite;
    NSStoreUUID = "D5463488-853C-45E3-A1F4-076FD8EEE28D";
    "_NSAutoVacuumLevel" = 2;
}
CoreData: error: 	reason : Can't find model for source store
CoreData: error: storeType: SQLite
CoreData: error: configuration: (null)
CoreData: error: URL: file:///...
CoreData: annotation: options:
CoreData: annotation: 	NSInferMappingModelAutomaticallyOption : 1
CoreData: annotation: 	NSMigratePersistentStoresAutomaticallyOption : 1
CoreData: annotation: 	_NSBinaryStoreInsecureDecodingCompatibilityOption : 1
CoreData: annotation: 	NSSQLitePragmasOption : {
    "journal_mode" = WAL;
}
CoreData: error: NSPersistentStoreCoordinator's current model hashes are {
    Cat = {length = 32, bytes = 0x08c1b0bc c3933f30 2be4c3fd d4f5a13c ... b37e32a7 e6e8264c };
    Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 };
}
CoreStore error
⚠️ [CoreStore: Error] DataStack.swift:378 addStorageAndWait(_:)
  ↪︎ Failed to add 'CoreStore.SQLiteStore' to the stack.
    (CoreStore.CoreStoreError) .asynchronousMigrationRequired (
    .errorDomain = "com.corestore.error";
    .errorCode = 4;
    .localStoreURL = "file:///...";
    .NSError = (
        .domain = "NSCocoaErrorDomain";
        .code = 134130;
        .userInfo = 3 key-value(s) [
            "reason" = "Can't find model for source store";
            "URL" = "file:///...";
            "metadata" = {
                NSPersistenceFrameworkVersion = 1338;
                NSStoreModelVersionChecksumKey = "SeJK6kdLCNX3AK7GOZrsIs6fsaQIUdP+LE1FNjyBDEU=";
                NSStoreModelVersionHashes =     {
                    Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 };
                };
                NSStoreModelVersionHashesDigest = "2KMBV9An/E0/QE6gU2uS7lysEJgk05M/NCNlG2OCR0bOF31qLgZXvvcOBWtOGtSTs+0sXr0xHQPCO97ibCouyw==";
                NSStoreModelVersionHashesVersion = 3;
                NSStoreModelVersionIdentifiers =     (
                );
                NSStoreType = SQLite;
                NSStoreUUID = "D5463488-853C-45E3-A1F4-076FD8EEE28D";
                "_NSAutoVacuumLevel" = 2;
            };
        ];
    );
)

Questions

Is this a design decision by CoreStore to not allow synchronous lightweight migration? (my understanding is this is achievable in CoreData directly)

Sample Project

💡CoreStore.zip

Originally created by @ptrkstr on GitHub (Jan 3, 2024). Hey @JohnEstropia, thank you for CoreStore ❤️ I am following the Synchronous Lightweight Migration guide from the [readme](https://github.com/JohnEstropia/CoreStore#starting-migrations). My new model has a new entity which is [supported by lightweight migration](https://developer.apple.com/documentation/coredata/migrating_your_data_model_automatically#2903990): > You can add, remove, and rename entities in the hierarchy. #### This is code that is live (CoreStore 9.2.0): ```swift class Dog: CoreStoreObject, Identifiable { @Field.Stored("identifier", dynamicInitialValue: { UUID() }) var id: UUID @Field.Stored("name") var name: String = "" } CoreStoreDefaults.dataStack = DataStack( CoreStoreSchema( modelVersion: "V1", entities: [ Entity<Dog>("Dog") ], versionLock: [ "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3] ] ) ) try! CoreStoreDefaults.dataStack.addStorageAndWait() ``` #### This is a change in the code to introduce the entity `Cat` (no change to `Dog`): ```swift // struct Dog... class Cat: CoreStoreObject, Identifiable { @Field.Stored("identifier", dynamicInitialValue: { UUID() }) var id: UUID @Field.Stored("name") var name: String = "" } CoreStoreDefaults.dataStack = DataStack( CoreStoreSchema( modelVersion: "V1", entities: [ Entity<Dog>("Dog") ], versionLock: [ "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3] // Same version lock ] ), CoreStoreSchema( modelVersion: "V2", entities: [ Entity<Dog>("Dog"), Entity<Cat>("Cat") ], versionLock: [ "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3], // Same version lock "Cat": [0x303f93c3bcb0c108, 0x3ca1f5d4fdc3e42b, 0xbcb7e18668333b8e, 0x4c26e8e6a7327eb3], ] ), migrationChain: ["V1", "V2"] ) try! CoreStoreDefaults.dataStack.addStorageAndWait( SQLiteStore( fileURL: SQLiteStore().fileURL, localStorageOptions: .allowSynchronousLightweightMigration ) ) ``` #### Errors when running ##### CoreData DataStack.swift:544 calls `coordinator.addPersistentStore` which triggers the error to console: ``` CoreData: error: addPersistentStoreWithType:configuration:URL:options:error: returned error NSCocoaErrorDomain (134130) CoreData: error: userInfo: CoreData: error: URL : file:///... CoreData: error: metadata : { NSPersistenceFrameworkVersion = 1338; NSStoreModelVersionChecksumKey = "SeJK6kdLCNX3AK7GOZrsIs6fsaQIUdP+LE1FNjyBDEU="; NSStoreModelVersionHashes = { Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 }; }; NSStoreModelVersionHashesDigest = "2KMBV9An/E0/QE6gU2uS7lysEJgk05M/NCNlG2OCR0bOF31qLgZXvvcOBWtOGtSTs+0sXr0xHQPCO97ibCouyw=="; NSStoreModelVersionHashesVersion = 3; NSStoreModelVersionIdentifiers = ( ); NSStoreType = SQLite; NSStoreUUID = "D5463488-853C-45E3-A1F4-076FD8EEE28D"; "_NSAutoVacuumLevel" = 2; } CoreData: error: reason : Can't find model for source store CoreData: error: storeType: SQLite CoreData: error: configuration: (null) CoreData: error: URL: file:///... CoreData: annotation: options: CoreData: annotation: NSInferMappingModelAutomaticallyOption : 1 CoreData: annotation: NSMigratePersistentStoresAutomaticallyOption : 1 CoreData: annotation: _NSBinaryStoreInsecureDecodingCompatibilityOption : 1 CoreData: annotation: NSSQLitePragmasOption : { "journal_mode" = WAL; } CoreData: error: NSPersistentStoreCoordinator's current model hashes are { Cat = {length = 32, bytes = 0x08c1b0bc c3933f30 2be4c3fd d4f5a13c ... b37e32a7 e6e8264c }; Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 }; } ``` ##### CoreStore error ``` ⚠️ [CoreStore: Error] DataStack.swift:378 addStorageAndWait(_:) ↪︎ Failed to add 'CoreStore.SQLiteStore' to the stack. (CoreStore.CoreStoreError) .asynchronousMigrationRequired ( .errorDomain = "com.corestore.error"; .errorCode = 4; .localStoreURL = "file:///..."; .NSError = ( .domain = "NSCocoaErrorDomain"; .code = 134130; .userInfo = 3 key-value(s) [ "reason" = "Can't find model for source store"; "URL" = "file:///..."; "metadata" = { NSPersistenceFrameworkVersion = 1338; NSStoreModelVersionChecksumKey = "SeJK6kdLCNX3AK7GOZrsIs6fsaQIUdP+LE1FNjyBDEU="; NSStoreModelVersionHashes = { Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 }; }; NSStoreModelVersionHashesDigest = "2KMBV9An/E0/QE6gU2uS7lysEJgk05M/NCNlG2OCR0bOF31qLgZXvvcOBWtOGtSTs+0sXr0xHQPCO97ibCouyw=="; NSStoreModelVersionHashesVersion = 3; NSStoreModelVersionIdentifiers = ( ); NSStoreType = SQLite; NSStoreUUID = "D5463488-853C-45E3-A1F4-076FD8EEE28D"; "_NSAutoVacuumLevel" = 2; }; ]; ); ) ``` #### Questions Is this a design decision by CoreStore to not allow synchronous lightweight migration? (my understanding is this is achievable in CoreData directly) #### Sample Project [💡CoreStore.zip](https://github.com/JohnEstropia/CoreStore/files/13815679/CoreStore.zip)
adam closed this issue 2025-12-29 18:26:49 +01:00
adam commented 2025-12-29 18:26:49 +01:00
Author
Owner
Copy Link

@ptrkstr commented on GitHub (Jan 5, 2024):

I was able to create a synchronous version of:

func addStorage<T: LocalStorage>(_ storage: T, completion: @escaping (SetupResult<T>) -> Void) -> Progress?

This didn't work with the non-progress function

func addStorage<T>(_ storage: T, completion: @escaping (SetupResult<T>) -> Void)

which caused the following CoreData error:

Exception was caught during NSPersistentStoreCoordinator -performBlock: Cannot create an SQL store with a nil URL.

You can see the code changes for this here:

Which contains:

  • Removing dispatches to the main thread of completion handlers, in storage setup code
  • Calling addStorage on a background thread
  • Blocking the main thread with a semaphore until addStorage completes
@ptrkstr commented on GitHub (Jan 5, 2024): I was able to create a synchronous version of: ```swift func addStorage<T: LocalStorage>(_ storage: T, completion: @escaping (SetupResult<T>) -> Void) -> Progress? ``` This didn't work with the non-progress function ```swift func addStorage<T>(_ storage: T, completion: @escaping (SetupResult<T>) -> Void) ``` which caused the following CoreData error: ``` Exception was caught during NSPersistentStoreCoordinator -performBlock: Cannot create an SQL store with a nil URL. ``` You can see the code changes for this here: - https://github.com/JohnEstropia/CoreStore/compare/develop...ptrkstr:CoreStore:develop?expand=1 Which contains: - Removing dispatches to the main thread of completion handlers, in storage setup code - Calling `addStorage` on a background thread - Blocking the main thread with a semaphore until `addStorage` completes
adam commented 2025-12-29 18:26:50 +01:00
Author
Owner
Copy Link

@JohnEstropia commented on GitHub (Jan 9, 2024):

@ptrkstr Does your update above actually resolve the lightweight migration issue?

@JohnEstropia commented on GitHub (Jan 9, 2024): @ptrkstr Does your update above actually resolve the lightweight migration issue?
adam commented 2025-12-29 18:26:50 +01:00
Author
Owner
Copy Link

@ptrkstr commented on GitHub (Jan 9, 2024):

If by lightweight migration issue you're referring to being able to perform a lightweight migration synchronously, then yes, attached is a video and project (which points to the above branch) demonstrating this.
Given lightweight migration performed synchronously is possible with vanilla CoreData, could it be supported by CoreStore?

💡CoreStore 2.zip

https://github.com/JohnEstropia/CoreStore/assets/11362913/c60508be-13a3-41cf-a5a1-3db8c74657d3

@ptrkstr commented on GitHub (Jan 9, 2024): If by lightweight migration issue you're referring to being able to perform a lightweight migration synchronously, then yes, attached is a video and project (which points to the above branch) demonstrating this. Given lightweight migration performed synchronously is possible with vanilla CoreData, could it be supported by CoreStore? [💡CoreStore 2.zip](https://github.com/JohnEstropia/CoreStore/files/13877310/CoreStore.2.zip) https://github.com/JohnEstropia/CoreStore/assets/11362913/c60508be-13a3-41cf-a5a1-3db8c74657d3
adam commented 2025-12-29 18:26:50 +01:00
Author
Owner
Copy Link

@JohnEstropia commented on GitHub (Jan 11, 2024):

Thanks for creating a demo app! It helped clarify what's going on.

It's been raised before and I just completely forgot about it, so I'm so sorry about the delays.
Core Data actually cannot execute a "lightweight migration" on this model, that is, addPersistentStore() fails on both the async and sync versions of CoreStore's addStorage* methods.

What does succeed is the proceeding "Inferred Migration", where CoreStore's async method initializes an inferred mapping model and calls NSMigrationManager.migrateStore() for you. This is not Apple's advertized "lightweight migration" that requires only the addPersistentStore() call.

Admittedly, I never researched in detail what schema changes this "lightweight migration" does succeed at. I guess at this point I can consider lifting this discrepancy for CoreStore's API consumers and just treat .allowSynchronousLightweightMigration as something like .allowSynchronousInferredMigration instead, but I never supported this from the start because of one reason:

  • The use of Mapping Models (even inferred) means that there is a possibility that all records will be loaded into memory at some point. If so then this would be significantly slower and memory intensive for a synchronous blocking method. Note that a lot of developers call addStorageAndWait() from the AppDelegate's didLaunchWithOptions or something like your Demo's App.init that blocks the app's launch process and will increase likelihood to get terminated by the OS's watchdog timer
@JohnEstropia commented on GitHub (Jan 11, 2024): Thanks for creating a demo app! It helped clarify what's going on. [It's been raised before](https://github.com/JohnEstropia/CoreStore/issues/277#issuecomment-430069780) and I just completely forgot about it, so I'm so sorry about the delays. Core Data actually cannot execute a **"lightweight migration"** on this model, that is, `addPersistentStore()` fails on both the async and sync versions of CoreStore's `addStorage*` methods. What does succeed is the proceeding **"Inferred Migration"**, where CoreStore's async method initializes an inferred mapping model and calls `NSMigrationManager.migrateStore()` for you. This is not Apple's advertized **"lightweight migration"** that requires only the `addPersistentStore()` call. Admittedly, I never researched in detail what schema changes this "lightweight migration" does succeed at. I guess at this point I can consider lifting this discrepancy for CoreStore's API consumers and just treat `.allowSynchronousLightweightMigration` as something like `.allowSynchronousInferredMigration` instead, but I never supported this from the start because of one reason: - The use of Mapping Models (even inferred) means that there is a possibility that all records will be loaded into memory at some point. If so then this would be significantly slower and memory intensive for a synchronous blocking method. Note that a lot of developers call `addStorageAndWait()` from the AppDelegate's `didLaunchWithOptions` or something like your Demo's `App.init` that blocks the app's launch process and will increase likelihood to get **terminated by the OS's [watchdog timer](https://developer.apple.com/documentation/xcode/addressing-watchdog-terminations)**
adam commented 2025-12-29 18:26:50 +01:00
Author
Owner
Copy Link

@ptrkstr commented on GitHub (Jan 13, 2024):

Hey @JohnEstropia thanks so much for spending more time explaining this 🙏
I did do a search for this situation and did come across that post but I must have missed that comment.
Your point about memory use in inferred migrations is a good reason to avoid it, I will look into restructuring my application to support async migrations.

what schema changes this "lightweight migration" does succeed at

Perhaps this is the takeaway from this whole issue 😅

Regardless I will close the issue as CoreStore's API does enforce best practises and it may lead to issues if they are modified to support synchronous inferred migration in a simpler way.

Thank you for being an active maintainer @JohnEstropia, I have sent a one-time sponsorship in exchange for the time you have spent responding to me.

@ptrkstr commented on GitHub (Jan 13, 2024): Hey @JohnEstropia thanks so much for spending more time explaining this 🙏 I did do a search for this situation and did come across that post but I must have missed that comment. Your point about memory use in inferred migrations is a good reason to avoid it, I will look into restructuring my application to support async migrations. > what schema changes this "lightweight migration" does succeed at Perhaps this is the takeaway from this whole issue 😅 Regardless I will close the issue as CoreStore's API does enforce best practises and it may lead to issues if they are modified to support synchronous inferred migration in a simpler way. Thank you for being an active maintainer @JohnEstropia, I have sent a one-time sponsorship in exchange for the time you have spent responding to me.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
abandoned
corestore bug
discussion
duplicate
enhancement
fixed
fixed
help wanted
in progress
investigating
ios/compiler bug
irreproducible
looking for workarounds
not a bug
pending
pending docs update
pull-request
Mirrored from GitHub Pull Request
 question
regression
user bug
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/CoreStore-JohnEstropia#424
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?