Support for Synchronous Lightweight Migration (adding a new entity) #425

New Issue

Closed

opened 2025-12-29 15:31:32 +01:00 by adam · 5 comments

adam commented 2025-12-29 15:31:32 +01:00

Owner

Copy Link

Originally created by @ptrkstr on GitHub (Jan 3, 2024).

Hey @JohnEstropia, thank you for CoreStore ❤️

I am following the Synchronous Lightweight Migration guide from the readme.
My new model has a new entity which is supported by lightweight migration:

You can add, remove, and rename entities in the hierarchy.

This is code that is live (CoreStore 9.2.0):

class Dog: CoreStoreObject, Identifiable {

    @Field.Stored("identifier", dynamicInitialValue: { UUID() })
    var id: UUID

    @Field.Stored("name")
    var name: String = ""
}

CoreStoreDefaults.dataStack = DataStack(
    CoreStoreSchema(
        modelVersion: "V1",
        entities: [
            Entity<Dog>("Dog")
        ],
        versionLock: [
            "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3]
        ]
    )
)

try! CoreStoreDefaults.dataStack.addStorageAndWait()

This is a change in the code to introduce the entity Cat (no change to Dog):

// struct Dog...

class Cat: CoreStoreObject, Identifiable {

    @Field.Stored("identifier", dynamicInitialValue: { UUID() })
    var id: UUID

    @Field.Stored("name")
    var name: String = ""
}

CoreStoreDefaults.dataStack = DataStack(
    CoreStoreSchema(
        modelVersion: "V1",
        entities: [
            Entity<Dog>("Dog")
        ],
        versionLock: [
            "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3] // Same version lock
        ]
    ),
    CoreStoreSchema(
        modelVersion: "V2",
        entities: [
            Entity<Dog>("Dog"),
            Entity<Cat>("Cat")
        ],
        versionLock: [
            "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3], // Same version lock
            "Cat": [0x303f93c3bcb0c108, 0x3ca1f5d4fdc3e42b, 0xbcb7e18668333b8e, 0x4c26e8e6a7327eb3],
        ]
    ),
    migrationChain: ["V1", "V2"]
)

try! CoreStoreDefaults.dataStack.addStorageAndWait(
    SQLiteStore(
        fileURL: SQLiteStore().fileURL,
        localStorageOptions: .allowSynchronousLightweightMigration
    )
)

Errors when running

CoreData

DataStack.swift:544 calls coordinator.addPersistentStore which triggers the error to console:

CoreData: error: addPersistentStoreWithType:configuration:URL:options:error: returned error NSCocoaErrorDomain (134130)
CoreData: error: userInfo:
CoreData: error: 	URL : file:///...
CoreData: error: 	metadata : {
    NSPersistenceFrameworkVersion = 1338;
    NSStoreModelVersionChecksumKey = "SeJK6kdLCNX3AK7GOZrsIs6fsaQIUdP+LE1FNjyBDEU=";
    NSStoreModelVersionHashes =     {
        Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 };
    };
    NSStoreModelVersionHashesDigest = "2KMBV9An/E0/QE6gU2uS7lysEJgk05M/NCNlG2OCR0bOF31qLgZXvvcOBWtOGtSTs+0sXr0xHQPCO97ibCouyw==";
    NSStoreModelVersionHashesVersion = 3;
    NSStoreModelVersionIdentifiers =     (
    );
    NSStoreType = SQLite;
    NSStoreUUID = "D5463488-853C-45E3-A1F4-076FD8EEE28D";
    "_NSAutoVacuumLevel" = 2;
}
CoreData: error: 	reason : Can't find model for source store
CoreData: error: storeType: SQLite
CoreData: error: configuration: (null)
CoreData: error: URL: file:///...
CoreData: annotation: options:
CoreData: annotation: 	NSInferMappingModelAutomaticallyOption : 1
CoreData: annotation: 	NSMigratePersistentStoresAutomaticallyOption : 1
CoreData: annotation: 	_NSBinaryStoreInsecureDecodingCompatibilityOption : 1
CoreData: annotation: 	NSSQLitePragmasOption : {
    "journal_mode" = WAL;
}
CoreData: error: NSPersistentStoreCoordinator's current model hashes are {
    Cat = {length = 32, bytes = 0x08c1b0bc c3933f30 2be4c3fd d4f5a13c ... b37e32a7 e6e8264c };
    Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 };
}

CoreStore error

⚠️ [CoreStore: Error] DataStack.swift:378 addStorageAndWait(_:)
  ↪︎ Failed to add 'CoreStore.SQLiteStore' to the stack.
    (CoreStore.CoreStoreError) .asynchronousMigrationRequired (
    .errorDomain = "com.corestore.error";
    .errorCode = 4;
    .localStoreURL = "file:///...";
    .NSError = (
        .domain = "NSCocoaErrorDomain";
        .code = 134130;
        .userInfo = 3 key-value(s) [
            "reason" = "Can't find model for source store";
            "URL" = "file:///...";
            "metadata" = {
                NSPersistenceFrameworkVersion = 1338;
                NSStoreModelVersionChecksumKey = "SeJK6kdLCNX3AK7GOZrsIs6fsaQIUdP+LE1FNjyBDEU=";
                NSStoreModelVersionHashes =     {
                    Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 };
                };
                NSStoreModelVersionHashesDigest = "2KMBV9An/E0/QE6gU2uS7lysEJgk05M/NCNlG2OCR0bOF31qLgZXvvcOBWtOGtSTs+0sXr0xHQPCO97ibCouyw==";
                NSStoreModelVersionHashesVersion = 3;
                NSStoreModelVersionIdentifiers =     (
                );
                NSStoreType = SQLite;
                NSStoreUUID = "D5463488-853C-45E3-A1F4-076FD8EEE28D";
                "_NSAutoVacuumLevel" = 2;
            };
        ];
    );
)

Questions

Is this a design decision by CoreStore to not allow synchronous lightweight migration? (my understanding is this is achievable in CoreData directly)

Sample Project

💡CoreStore.zip

Originally created by @ptrkstr on GitHub (Jan 3, 2024). Hey @JohnEstropia, thank you for CoreStore ❤️ I am following the Synchronous Lightweight Migration guide from the [readme](https://github.com/JohnEstropia/CoreStore#starting-migrations). My new model has a new entity which is [supported by lightweight migration](https://developer.apple.com/documentation/coredata/migrating_your_data_model_automatically#2903990): > You can add, remove, and rename entities in the hierarchy. #### This is code that is live (CoreStore 9.2.0): ```swift class Dog: CoreStoreObject, Identifiable { @Field.Stored("identifier", dynamicInitialValue: { UUID() }) var id: UUID @Field.Stored("name") var name: String = "" } CoreStoreDefaults.dataStack = DataStack( CoreStoreSchema( modelVersion: "V1", entities: [ Entity<Dog>("Dog") ], versionLock: [ "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3] ] ) ) try! CoreStoreDefaults.dataStack.addStorageAndWait() ``` #### This is a change in the code to introduce the entity `Cat` (no change to `Dog`): ```swift // struct Dog... class Cat: CoreStoreObject, Identifiable { @Field.Stored("identifier", dynamicInitialValue: { UUID() }) var id: UUID @Field.Stored("name") var name: String = "" } CoreStoreDefaults.dataStack = DataStack( CoreStoreSchema( modelVersion: "V1", entities: [ Entity<Dog>("Dog") ], versionLock: [ "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3] // Same version lock ] ), CoreStoreSchema( modelVersion: "V2", entities: [ Entity<Dog>("Dog"), Entity<Cat>("Cat") ], versionLock: [ "Dog": [0x318f24569ce1979d, 0xc617bd516e8b8676, 0x5d49bce952f15e16, 0xe8f2f5c52286f1d3], // Same version lock "Cat": [0x303f93c3bcb0c108, 0x3ca1f5d4fdc3e42b, 0xbcb7e18668333b8e, 0x4c26e8e6a7327eb3], ] ), migrationChain: ["V1", "V2"] ) try! CoreStoreDefaults.dataStack.addStorageAndWait( SQLiteStore( fileURL: SQLiteStore().fileURL, localStorageOptions: .allowSynchronousLightweightMigration ) ) ``` #### Errors when running ##### CoreData DataStack.swift:544 calls `coordinator.addPersistentStore` which triggers the error to console: ``` CoreData: error: addPersistentStoreWithType:configuration:URL:options:error: returned error NSCocoaErrorDomain (134130) CoreData: error: userInfo: CoreData: error: URL : file:///... CoreData: error: metadata : { NSPersistenceFrameworkVersion = 1338; NSStoreModelVersionChecksumKey = "SeJK6kdLCNX3AK7GOZrsIs6fsaQIUdP+LE1FNjyBDEU="; NSStoreModelVersionHashes = { Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 }; }; NSStoreModelVersionHashesDigest = "2KMBV9An/E0/QE6gU2uS7lysEJgk05M/NCNlG2OCR0bOF31qLgZXvvcOBWtOGtSTs+0sXr0xHQPCO97ibCouyw=="; NSStoreModelVersionHashesVersion = 3; NSStoreModelVersionIdentifiers = ( ); NSStoreType = SQLite; NSStoreUUID = "D5463488-853C-45E3-A1F4-076FD8EEE28D"; "_NSAutoVacuumLevel" = 2; } CoreData: error: reason : Can't find model for source store CoreData: error: storeType: SQLite CoreData: error: configuration: (null) CoreData: error: URL: file:///... CoreData: annotation: options: CoreData: annotation: NSInferMappingModelAutomaticallyOption : 1 CoreData: annotation: NSMigratePersistentStoresAutomaticallyOption : 1 CoreData: annotation: _NSBinaryStoreInsecureDecodingCompatibilityOption : 1 CoreData: annotation: NSSQLitePragmasOption : { "journal_mode" = WAL; } CoreData: error: NSPersistentStoreCoordinator's current model hashes are { Cat = {length = 32, bytes = 0x08c1b0bc c3933f30 2be4c3fd d4f5a13c ... b37e32a7 e6e8264c }; Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 }; } ``` ##### CoreStore error ``` ⚠️ [CoreStore: Error] DataStack.swift:378 addStorageAndWait(_:) ↪︎ Failed to add 'CoreStore.SQLiteStore' to the stack. (CoreStore.CoreStoreError) .asynchronousMigrationRequired ( .errorDomain = "com.corestore.error"; .errorCode = 4; .localStoreURL = "file:///..."; .NSError = ( .domain = "NSCocoaErrorDomain"; .code = 134130; .userInfo = 3 key-value(s) [ "reason" = "Can't find model for source store"; "URL" = "file:///..."; "metadata" = { NSPersistenceFrameworkVersion = 1338; NSStoreModelVersionChecksumKey = "SeJK6kdLCNX3AK7GOZrsIs6fsaQIUdP+LE1FNjyBDEU="; NSStoreModelVersionHashes = { Dog = {length = 32, bytes = 0x9d97e19c 56248f31 76868b6e 51bd17c6 ... d3f18622 c5f5f2e8 }; }; NSStoreModelVersionHashesDigest = "2KMBV9An/E0/QE6gU2uS7lysEJgk05M/NCNlG2OCR0bOF31qLgZXvvcOBWtOGtSTs+0sXr0xHQPCO97ibCouyw=="; NSStoreModelVersionHashesVersion = 3; NSStoreModelVersionIdentifiers = ( ); NSStoreType = SQLite; NSStoreUUID = "D5463488-853C-45E3-A1F4-076FD8EEE28D"; "_NSAutoVacuumLevel" = 2; }; ]; ); ) ``` #### Questions Is this a design decision by CoreStore to not allow synchronous lightweight migration? (my understanding is this is achievable in CoreData directly) #### Sample Project [💡CoreStore.zip](https://github.com/JohnEstropia/CoreStore/files/13815679/CoreStore.zip)

adam closed this issue 2025-12-29 15:31:33 +01:00

adam commented 2025-12-29 15:31:33 +01:00

Author

Owner

Copy Link

@ptrkstr commented on GitHub (Jan 5, 2024):

I was able to create a synchronous version of:

func addStorage<T: LocalStorage>(_ storage: T, completion: @escaping (SetupResult<T>) -> Void) -> Progress?

This didn't work with the non-progress function

func addStorage<T>(_ storage: T, completion: @escaping (SetupResult<T>) -> Void)

which caused the following CoreData error:

Exception was caught during NSPersistentStoreCoordinator -performBlock: Cannot create an SQL store with a nil URL.

You can see the code changes for this here:

• https://github.com/JohnEstropia/CoreStore/compare/develop...ptrkstr:CoreStore:develop?expand=1

Which contains:

• Removing dispatches to the main thread of completion handlers, in storage setup code
• Calling addStorage on a background thread
• Blocking the main thread with a semaphore until addStorage completes

@ptrkstr commented on GitHub (Jan 5, 2024): I was able to create a synchronous version of: ```swift func addStorage<T: LocalStorage>(_ storage: T, completion: @escaping (SetupResult<T>) -> Void) -> Progress? ``` This didn't work with the non-progress function ```swift func addStorage<T>(_ storage: T, completion: @escaping (SetupResult<T>) -> Void) ``` which caused the following CoreData error: ``` Exception was caught during NSPersistentStoreCoordinator -performBlock: Cannot create an SQL store with a nil URL. ``` You can see the code changes for this here: - https://github.com/JohnEstropia/CoreStore/compare/develop...ptrkstr:CoreStore:develop?expand=1 Which contains: - Removing dispatches to the main thread of completion handlers, in storage setup code - Calling `addStorage` on a background thread - Blocking the main thread with a semaphore until `addStorage` completes

adam commented 2025-12-29 15:31:34 +01:00

Author

Owner

Copy Link

@JohnEstropia commented on GitHub (Jan 9, 2024):

@ptrkstr Does your update above actually resolve the lightweight migration issue?

@JohnEstropia commented on GitHub (Jan 9, 2024): @ptrkstr Does your update above actually resolve the lightweight migration issue?

adam commented 2025-12-29 15:31:34 +01:00

Author

Owner

Copy Link

@ptrkstr commented on GitHub (Jan 9, 2024):

If by lightweight migration issue you're referring to being able to perform a lightweight migration synchronously, then yes, attached is a video and project (which points to the above branch) demonstrating this.
Given lightweight migration performed synchronously is possible with vanilla CoreData, could it be supported by CoreStore?

💡CoreStore 2.zip

https://github.com/JohnEstropia/CoreStore/assets/11362913/c60508be-13a3-41cf-a5a1-3db8c74657d3

@ptrkstr commented on GitHub (Jan 9, 2024): If by lightweight migration issue you're referring to being able to perform a lightweight migration synchronously, then yes, attached is a video and project (which points to the above branch) demonstrating this. Given lightweight migration performed synchronously is possible with vanilla CoreData, could it be supported by CoreStore? [💡CoreStore 2.zip](https://github.com/JohnEstropia/CoreStore/files/13877310/CoreStore.2.zip) https://github.com/JohnEstropia/CoreStore/assets/11362913/c60508be-13a3-41cf-a5a1-3db8c74657d3

adam commented 2025-12-29 15:31:39 +01:00

Author

Owner

Copy Link

@JohnEstropia commented on GitHub (Jan 11, 2024):

Thanks for creating a demo app! It helped clarify what's going on.

It's been raised before and I just completely forgot about it, so I'm so sorry about the delays.
Core Data actually cannot execute a "lightweight migration" on this model, that is, addPersistentStore() fails on both the async and sync versions of CoreStore's addStorage* methods.

What does succeed is the proceeding "Inferred Migration", where CoreStore's async method initializes an inferred mapping model and calls NSMigrationManager.migrateStore() for you. This is not Apple's advertized "lightweight migration" that requires only the addPersistentStore() call.

Admittedly, I never researched in detail what schema changes this "lightweight migration" does succeed at. I guess at this point I can consider lifting this discrepancy for CoreStore's API consumers and just treat .allowSynchronousLightweightMigration as something like .allowSynchronousInferredMigration instead, but I never supported this from the start because of one reason:

• The use of Mapping Models (even inferred) means that there is a possibility that all records will be loaded into memory at some point. If so then this would be significantly slower and memory intensive for a synchronous blocking method. Note that a lot of developers call addStorageAndWait() from the AppDelegate's didLaunchWithOptions or something like your Demo's App.init that blocks the app's launch process and will increase likelihood to get terminated by the OS's watchdog timer

@JohnEstropia commented on GitHub (Jan 11, 2024): Thanks for creating a demo app! It helped clarify what's going on. [It's been raised before](https://github.com/JohnEstropia/CoreStore/issues/277#issuecomment-430069780) and I just completely forgot about it, so I'm so sorry about the delays. Core Data actually cannot execute a **"lightweight migration"** on this model, that is, `addPersistentStore()` fails on both the async and sync versions of CoreStore's `addStorage*` methods. What does succeed is the proceeding **"Inferred Migration"**, where CoreStore's async method initializes an inferred mapping model and calls `NSMigrationManager.migrateStore()` for you. This is not Apple's advertized **"lightweight migration"** that requires only the `addPersistentStore()` call. Admittedly, I never researched in detail what schema changes this "lightweight migration" does succeed at. I guess at this point I can consider lifting this discrepancy for CoreStore's API consumers and just treat `.allowSynchronousLightweightMigration` as something like `.allowSynchronousInferredMigration` instead, but I never supported this from the start because of one reason: - The use of Mapping Models (even inferred) means that there is a possibility that all records will be loaded into memory at some point. If so then this would be significantly slower and memory intensive for a synchronous blocking method. Note that a lot of developers call `addStorageAndWait()` from the AppDelegate's `didLaunchWithOptions` or something like your Demo's `App.init` that blocks the app's launch process and will increase likelihood to get **terminated by the OS's [watchdog timer](https://developer.apple.com/documentation/xcode/addressing-watchdog-terminations)**

adam commented 2025-12-29 15:31:39 +01:00

Author

Owner

Copy Link

@ptrkstr commented on GitHub (Jan 13, 2024):

Hey @JohnEstropia thanks so much for spending more time explaining this 🙏
I did do a search for this situation and did come across that post but I must have missed that comment.
Your point about memory use in inferred migrations is a good reason to avoid it, I will look into restructuring my application to support async migrations.

what schema changes this "lightweight migration" does succeed at

Perhaps this is the takeaway from this whole issue 😅

Regardless I will close the issue as CoreStore's API does enforce best practises and it may lead to issues if they are modified to support synchronous inferred migration in a simpler way.

Thank you for being an active maintainer @JohnEstropia, I have sent a one-time sponsorship in exchange for the time you have spent responding to me.

@ptrkstr commented on GitHub (Jan 13, 2024): Hey @JohnEstropia thanks so much for spending more time explaining this 🙏 I did do a search for this situation and did come across that post but I must have missed that comment. Your point about memory use in inferred migrations is a good reason to avoid it, I will look into restructuring my application to support async migrations. > what schema changes this "lightweight migration" does succeed at Perhaps this is the takeaway from this whole issue 😅 Regardless I will close the issue as CoreStore's API does enforce best practises and it may lead to issues if they are modified to support synchronous inferred migration in a simpler way. Thank you for being an active maintainer @JohnEstropia, I have sent a one-time sponsorship in exchange for the time you have spent responding to me.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

develop

master

prototype/concurrency

minIOS13

prototype/containers

transactionSourceTracking

prototype/swiftui

minIOS11

swift5_3

prototype/propertyWrapperFields

prototype/propertyWrappers

prototype/propertyWrapper

Swift5

throwables

prototype/Swift_4_1

prototype/Swift_4_0

prototype/Swift_3_2

prototype/queryBuilders

temp/develop

corestore4_develop

prototype/mainContextToPSC

master_ios_7_to_9

swift_2_3

gh-pages

_corespotlight-test

9.3.0

9.2.0

9.1.0

9.0.0

8.1.0

8.0.1

8.0.0

7.3.1

7.3.0

7.2.0

7.1.0

7.0.4

7.0.3

7.0.2

7.0.1

7.0.0

6.3.2

6.3.1

6.3.0

6.2.1

6.2.0

6.1.0

6.0.0

5.3.1

5.3.0

5.2.0

5.1.1

5.1.0

5.0.0

4.2.4

4.2.3

4.2.2

4.2.1

4.2.0

4.1.4

4.1.3

4.1.1

4.1.0

4.0.5

4.0.4

4.0.3

4.0.2

4.0.1

4.0.0

4.0.0-beta3

4.0.0-beta2

4.0.0-beta

3.1.1

3.1.0

3.0.3

3.0.2

3.0.1

3.0.0

2.1.3

2.1.2

2.1.1

2.1.0

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.6.10

1.6.9

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.1

1.2.0

1.0.2

1.0.1

1.0.0

0.2.1

0.2.0

0.1.2

0.1.1

0.1.0

Labels

Clear labels

abandoned

corestore bug

discussion

duplicate

enhancement

fixed

fixed

help wanted

in progress

investigating

ios/compiler bug

irreproducible

looking for workarounds

not a bug

pending

pending docs update

pull-request

Mirrored from GitHub Pull Request

question

regression

user bug

wontfix

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

adam

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/CoreStore#425