CustomSchemaMappingProvider hangs and errors #365

New Issue

Closed

opened 2025-12-29 18:26:07 +01:00 by adam · 6 comments

adam commented 2025-12-29 18:26:07 +01:00

Owner

Copy Link

Copy Source

Originally created by @jeanetienne on GitHub (Jun 25, 2021).

Hi John & Contributors, thanks for providing such a valuable tool, making CoreData much more palatable, high level and swifty!

I am using CoreStore 8.0.1 and I can't find a way to make a custom migration work.
It seems to hang for a veeery long time (7+ minutes in iOS Simulator), and I get an error:

[BackgroundTask] Background Task 4 ("CoreData: Adding persistent store"), was created over 30 seconds ago. In applications running in the background, this creates a risk of termination. Remember to call UIApplication.endBackgroundTask(_:) for your task in a timely manner to avoid this.

When I only try to do a V1 to V2 migration and I let CoreStore infer all the migration (deleting a table and deleting a column), it works fine. As soon as I add a V2 to V3 migration with the CustomSchemaMappingProvider, it exhibits this behaviour (hanging for 7 min and erroring).

Let me know if you need more details. I would appreciate any help understanding this problem! Thank you!

(I renamed my entities and table names, but everything else is the same)

Setup Details

Project

• iOS 12.1+
• CoreStore 8.0.1
• Xcode 12.5

Schema

enum DatabaseSchema {

    enum V3 {

        static let name: ModelVersion = "V3"

        static let schema = CoreStoreSchema(
            modelVersion: Self.name,
            entities: [
                Entity<SomeNewEntity>("SomeNewTable"),

                Entity<MyEntity_01>("MyTable_01"),
                Entity<MyEntity_02>("MyTable_02"),
                Entity<MyEntity_03>("MyTable_03"),
                Entity<MyEntity_03>("MyTable_04"),
            ],
            versionLock: [
              // ...
            ]
        )

    }

    enum V2 {

        static let name: ModelVersion = "V2"

        static let schema = CoreStoreSchema(
            modelVersion: Self.name,
            entities: [
              Entity<MyEntity_01>("MyTable_01"),
              Entity<MyEntity_02>("MyTable_02"),
              Entity<MyEntity_03>("MyTable_03"),
              Entity<MyEntity_04>("MyTable_04"), // Removed a column since DatabaseSchema.V1.MyLegacyEntity_04
              // Removing MyLegacyEntity_05 and its associated table "MyTable_05"
            ],
            versionLock: [
              //...
            ]
        )

    }

    enum V1 {

        static let name: ModelVersion = "V1"

        static let schema = CoreStoreSchema(
            modelVersion: Self.name,
            entities: [
                Entity<MyEntity_01>("MyTable_01"),
                Entity<MyEntity_02>("MyTable_02"),
                Entity<MyEntity_03>("MyTable_03"),

                Entity<DatabaseSchema.V1.MyLegacyEntity_04>("MyTable_04"),
                Entity<DatabaseSchema.V1.MyLegacyEntity_05>("MyTable_05"),
            ],
            versionLock: [
              //...
            ]
        )

    }

}

extension DatabaseSchema.V3 {

    class SomeNewEntity: CoreStoreObject {
        @Field.Stored("first_property") var firstProperty: String = ""
        @Field.Stored("second_property") var secondProperty: Date?
        @Field.Stored("third_property") var thirdProperty: Bool = false
    }

}

Database setup

func setup(_ completion: @escaping (Result<Void, Error>) -> Void) {
    Log.info("Setting up the local database")
    CoreStoreDefaults.dataStack = DataStack(
        DatabaseSchema.V1.schema, DatabaseSchema.V2.schema, DatabaseSchema.V3.schema,
        migrationChain: [DatabaseSchema.V1.name, DatabaseSchema.V2.name, DatabaseSchema.V3.name]
    )

    let v2_to_v3_mapping = CustomSchemaMappingProvider(
        from: "V2",
        to: "V3",
        entityMappings: [
            .insertEntity(destinationEntity: "SomeNewTable"),
            .copyEntity(sourceEntity: "MyTable_01", destinationEntity: "MyTable_01"),
            .copyEntity(sourceEntity: "MyTable_02", destinationEntity: "MyTable_02"),
            .copyEntity(sourceEntity: "MyTable_04", destinationEntity: "MyTable_04"),

            .transformEntity(
                sourceEntity: "MyTable_03",
                destinationEntity: "MyTable_03",
                transformer: { (sourceObject: CustomSchemaMappingProvider.UnsafeSourceObject, createDestinationObject: () -> CustomSchemaMappingProvider.UnsafeDestinationObject) in
                    let destinationObject = createDestinationObject()
                    destinationObject.enumerateAttributes { (attribute, sourceAttribute) in
                        if let sourceAttribute = sourceAttribute {
                            destinationObject[attribute] = sourceObject[sourceAttribute]
                        }
                    }
                    // Mark all items as enabled
                    destinationObject["enabled"] = true
                    
                    // Uppercase all items' titles
                    destinationObject["title"] = (destinationObject["title"] as? String)?.uppercased()
            })
        ]
    )

    let storage = SQLiteStore(fileName: Constants.databaseFileName, migrationMappingProviders: [v2_to_v3_mapping])

    // Discarding the progress indicator for now. Will use later.
    _ = CoreStoreDefaults.dataStack.addStorage(storage) { result in
        completion(result.map { _ in }.mapError { $0 as Error })
    }
}

Simulator output

2021-06-25 11:35:42.972661+1000 MyApp[4861:318117] info: Setting up the local database
2021-06-25 11:36:18.790029+1000 MyApp[4861:318117] [BackgroundTask] Background Task 4 ("CoreData: Adding persistent store"), was created over 30 seconds ago. In applications running in the background, this creates a risk of termination. Remember to call UIApplication.endBackgroundTask(_:) for your task in a timely manner to avoid this.
2021-06-25 11:43:43.988169+1000 MyApp[4861:319180] [error] error: addPersistentStoreWithType:configuration:URL:options:error: returned error NSCocoaErrorDomain (134090)
CoreData: error: addPersistentStoreWithType:configuration:URL:options:error: returned error NSCocoaErrorDomain (134090)
CoreData: annotation: userInfo:
CoreData: annotation: 	NSSQLiteErrorDomain : 5
CoreData: annotation: storeType: SQLite
CoreData: annotation: configuration: (null)
CoreData: annotation: URL: file:///Users/jeanetienne/Library/Developer/CoreSimulator/Devices/D7BC2069-1D37-466E-BE36-04A8530AB99E/data/Containers/Data/Application/37A4C44D-BCD1-442D-924E-37694F0367BD/Library/Application%20Support/my_app_bundle_id/my_database.sqlite
CoreData: annotation: options:
CoreData: annotation: 	_NSBinaryStoreInsecureDecodingCompatibilityOption : 1
CoreData: annotation: 	NSSQLitePragmasOption : {
    "journal_mode" = DELETE;
}
⚠️ [CoreStore: Error] DataStack+Migration.swift:449 upgradeStorageIfNeeded(_:metadata:completion:)
  ↪︎ Failed to migrate version model "V1" to version "V2".
    (CoreStore.CoreStoreError) .internalError (
    .errorDomain = "com.corestore.error";
    .errorCode = 5;
    .NSError = (
        .domain = "NSCocoaErrorDomain";
        .code = 134090;
        .userInfo = 1 key-value(s) [
            "NSSQLiteErrorDomain" = 5;
        ];
    );
)

Surprisingly this error output make it look like the migration from V1 to V2 has a problem, when in fact this migration works fine when I comment out anything related to V3 🤔 ...

Thanks!

Originally created by @jeanetienne on GitHub (Jun 25, 2021). Hi John & Contributors, thanks for providing such a valuable tool, making CoreData much more palatable, high level and swifty! I am using CoreStore 8.0.1 and I can't find a way to make a custom migration work. It seems to hang for a veeery long time (7+ minutes in iOS Simulator), and I get an error: ``` [BackgroundTask] Background Task 4 ("CoreData: Adding persistent store"), was created over 30 seconds ago. In applications running in the background, this creates a risk of termination. Remember to call UIApplication.endBackgroundTask(_:) for your task in a timely manner to avoid this. ``` When I only try to do a `V1` to `V2` migration and I let CoreStore infer all the migration (deleting a table and deleting a column), it works fine. As soon as I add a `V2` to `V3` migration with the `CustomSchemaMappingProvider`, it exhibits this behaviour (hanging for 7 min and erroring). Let me know if you need more details. I would appreciate any help understanding this problem! Thank you! _(I renamed my entities and table names, but everything else is the same)_ # Setup Details ## Project - iOS 12.1+ - CoreStore 8.0.1 - Xcode 12.5 ## Schema ```swift enum DatabaseSchema { enum V3 { static let name: ModelVersion = "V3" static let schema = CoreStoreSchema( modelVersion: Self.name, entities: [ Entity<SomeNewEntity>("SomeNewTable"), Entity<MyEntity_01>("MyTable_01"), Entity<MyEntity_02>("MyTable_02"), Entity<MyEntity_03>("MyTable_03"), Entity<MyEntity_03>("MyTable_04"), ], versionLock: [ // ... ] ) } enum V2 { static let name: ModelVersion = "V2" static let schema = CoreStoreSchema( modelVersion: Self.name, entities: [ Entity<MyEntity_01>("MyTable_01"), Entity<MyEntity_02>("MyTable_02"), Entity<MyEntity_03>("MyTable_03"), Entity<MyEntity_04>("MyTable_04"), // Removed a column since DatabaseSchema.V1.MyLegacyEntity_04 // Removing MyLegacyEntity_05 and its associated table "MyTable_05" ], versionLock: [ //... ] ) } enum V1 { static let name: ModelVersion = "V1" static let schema = CoreStoreSchema( modelVersion: Self.name, entities: [ Entity<MyEntity_01>("MyTable_01"), Entity<MyEntity_02>("MyTable_02"), Entity<MyEntity_03>("MyTable_03"), Entity<DatabaseSchema.V1.MyLegacyEntity_04>("MyTable_04"), Entity<DatabaseSchema.V1.MyLegacyEntity_05>("MyTable_05"), ], versionLock: [ //... ] ) } } extension DatabaseSchema.V3 { class SomeNewEntity: CoreStoreObject { @Field.Stored("first_property") var firstProperty: String = "" @Field.Stored("second_property") var secondProperty: Date? @Field.Stored("third_property") var thirdProperty: Bool = false } } ``` ## Database setup ```swift func setup(_ completion: @escaping (Result<Void, Error>) -> Void) { Log.info("Setting up the local database") CoreStoreDefaults.dataStack = DataStack( DatabaseSchema.V1.schema, DatabaseSchema.V2.schema, DatabaseSchema.V3.schema, migrationChain: [DatabaseSchema.V1.name, DatabaseSchema.V2.name, DatabaseSchema.V3.name] ) let v2_to_v3_mapping = CustomSchemaMappingProvider( from: "V2", to: "V3", entityMappings: [ .insertEntity(destinationEntity: "SomeNewTable"), .copyEntity(sourceEntity: "MyTable_01", destinationEntity: "MyTable_01"), .copyEntity(sourceEntity: "MyTable_02", destinationEntity: "MyTable_02"), .copyEntity(sourceEntity: "MyTable_04", destinationEntity: "MyTable_04"), .transformEntity( sourceEntity: "MyTable_03", destinationEntity: "MyTable_03", transformer: { (sourceObject: CustomSchemaMappingProvider.UnsafeSourceObject, createDestinationObject: () -> CustomSchemaMappingProvider.UnsafeDestinationObject) in let destinationObject = createDestinationObject() destinationObject.enumerateAttributes { (attribute, sourceAttribute) in if let sourceAttribute = sourceAttribute { destinationObject[attribute] = sourceObject[sourceAttribute] } } // Mark all items as enabled destinationObject["enabled"] = true // Uppercase all items' titles destinationObject["title"] = (destinationObject["title"] as? String)?.uppercased() }) ] ) let storage = SQLiteStore(fileName: Constants.databaseFileName, migrationMappingProviders: [v2_to_v3_mapping]) // Discarding the progress indicator for now. Will use later. _ = CoreStoreDefaults.dataStack.addStorage(storage) { result in completion(result.map { _ in }.mapError { $0 as Error }) } } ``` # Simulator output ```bash 2021-06-25 11:35:42.972661+1000 MyApp[4861:318117] info: Setting up the local database 2021-06-25 11:36:18.790029+1000 MyApp[4861:318117] [BackgroundTask] Background Task 4 ("CoreData: Adding persistent store"), was created over 30 seconds ago. In applications running in the background, this creates a risk of termination. Remember to call UIApplication.endBackgroundTask(_:) for your task in a timely manner to avoid this. 2021-06-25 11:43:43.988169+1000 MyApp[4861:319180] [error] error: addPersistentStoreWithType:configuration:URL:options:error: returned error NSCocoaErrorDomain (134090) CoreData: error: addPersistentStoreWithType:configuration:URL:options:error: returned error NSCocoaErrorDomain (134090) CoreData: annotation: userInfo: CoreData: annotation: NSSQLiteErrorDomain : 5 CoreData: annotation: storeType: SQLite CoreData: annotation: configuration: (null) CoreData: annotation: URL: file:///Users/jeanetienne/Library/Developer/CoreSimulator/Devices/D7BC2069-1D37-466E-BE36-04A8530AB99E/data/Containers/Data/Application/37A4C44D-BCD1-442D-924E-37694F0367BD/Library/Application%20Support/my_app_bundle_id/my_database.sqlite CoreData: annotation: options: CoreData: annotation: _NSBinaryStoreInsecureDecodingCompatibilityOption : 1 CoreData: annotation: NSSQLitePragmasOption : { "journal_mode" = DELETE; } ⚠️ [CoreStore: Error] DataStack+Migration.swift:449 upgradeStorageIfNeeded(_:metadata:completion:) ↪︎ Failed to migrate version model "V1" to version "V2". (CoreStore.CoreStoreError) .internalError ( .errorDomain = "com.corestore.error"; .errorCode = 5; .NSError = ( .domain = "NSCocoaErrorDomain"; .code = 134090; .userInfo = 1 key-value(s) [ "NSSQLiteErrorDomain" = 5; ]; ); ) ``` Surprisingly this error output make it look like the migration from `V1` to `V2` has a problem, when in fact this migration works fine when I comment out anything related to `V3` 🤔 ... - Thanks!

adam closed this issue 2025-12-29 18:26:07 +01:00

adam commented 2025-12-29 18:26:08 +01:00

Author

Owner

Copy Link

Copy Source

@JohnEstropia commented on GitHub (Jun 25, 2021):

@jeanetienne I see. I think your data is simply too large and is hitting memory performance limits. Note that NSCocoaErrorDomain (134090) is NSPersistentStoreTimeoutError.

Unfortunately, Core Data's migration methods don't do well with large migrations like this since Core Data runs migrations in one transaction pass, which means the migrating NSManagedObjectContext will eat up large amounts of memory.

I had a project where I had to handle something like this. What I did was create a new empty DataStack for the new model (your V3), and manually move over chunks of data from the original DataStack (your V2). I don't think there's much we can do on CoreStore's side, since migration logic depends on how models need to be transformed to the new model.

@JohnEstropia commented on GitHub (Jun 25, 2021): @jeanetienne I see. I think your data is simply too large and is hitting ~memory~ performance limits. Note that `NSCocoaErrorDomain (134090)` is `NSPersistentStoreTimeoutError`. Unfortunately, Core Data's migration methods don't do well with large migrations like this since Core Data runs migrations in one transaction pass, which means the migrating `NSManagedObjectContext` will eat up large amounts of memory. I had a project where I had to handle something like this. What I did was create a new empty `DataStack` for the new model (your `V3`), and manually move over chunks of data from the original `DataStack` (your `V2`). I don't think there's much we can do on CoreStore's side, since migration logic depends on how models need to be transformed to the new model.

adam commented 2025-12-29 18:26:08 +01:00

Author

Owner

Copy Link

Copy Source

@jeanetienne commented on GitHub (Jun 25, 2021):

Thanks @JohnEstropia for the reply.

That's surprising, I only have 11 tables in total, and when running this above example, no table had more than 40 records... The whole sqlite file is 78KB... Could this be caused by something else?

So there's nothing wrong from the setup or the way the schema/migrations are implemented?

@jeanetienne commented on GitHub (Jun 25, 2021): Thanks @JohnEstropia for the reply. That's surprising, I only have 11 tables in total, and when running this above example, no table had more than 40 records... The whole sqlite file is 78KB... Could this be caused by something else? So there's nothing wrong from the setup or the way the schema/migrations are implemented?

adam commented 2025-12-29 18:26:08 +01:00

Author

Owner

Copy Link

Copy Source

@JohnEstropia commented on GitHub (Jun 25, 2021):

I see, then it isn't a memory error.

I took a closer look at your models and it looks like you are reusing entities in multiple versions:

              Entity<MyEntity_01>("MyTable_01"),
              Entity<MyEntity_02>("MyTable_02"),
              Entity<MyEntity_03>("MyTable_03"),
              Entity<MyEntity_04>("MyTable_04"),

The strings ("MyTable_01" etc) should stay the same, but I have never reused the same entity in multiple versions. I don't think that is supported.
In your case, even if MyEntity_01 didn't change across models try to create a new type for it for every new version.
Let me know how that works for you.

@JohnEstropia commented on GitHub (Jun 25, 2021): I see, then it isn't a memory error. I took a closer look at your models and it looks like you are reusing entities in multiple versions: ```swift Entity<MyEntity_01>("MyTable_01"), Entity<MyEntity_02>("MyTable_02"), Entity<MyEntity_03>("MyTable_03"), Entity<MyEntity_04>("MyTable_04"), ``` The strings (`"MyTable_01"` etc) should stay the same, but I have never reused the same entity in multiple versions. I don't think that is supported. In your case, even if `MyEntity_01` didn't change across models try to create a new type for it for every new version. Let me know how that works for you.

adam commented 2025-12-29 18:26:08 +01:00

Author

Owner

Copy Link

Copy Source

@jeanetienne commented on GitHub (Jun 25, 2021):

Oh, interesting! I'll give that a try now. Thanks for the suggestion!

@jeanetienne commented on GitHub (Jun 25, 2021): Oh, interesting! I'll give that a try now. Thanks for the suggestion!

adam commented 2025-12-29 18:26:09 +01:00

Author

Owner

Copy Link

Copy Source

@jeanetienne commented on GitHub (Jun 25, 2021):

@JohnEstropia you were right on target 🎯

That was it. The migration took ~2.4s in the simulator once I reshuffled my schema by separating each entity versions so I don't reuse them between version.

Thanks for the help!
🙏

@jeanetienne commented on GitHub (Jun 25, 2021): @JohnEstropia you were right on target 🎯 That was it. The migration took ~2.4s in the simulator once I reshuffled my schema by separating each entity versions so I don't reuse them between version. Thanks for the help! 🙏

adam commented 2025-12-29 18:26:09 +01:00

Author

Owner

Copy Link

Copy Source

@JohnEstropia commented on GitHub (Jun 25, 2021):

That's great to hear!

@JohnEstropia commented on GitHub (Jun 25, 2021): That's great to hear!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

develop

prototype/swift6

master

prototype/concurrency

minIOS13

prototype/containers

transactionSourceTracking

prototype/swiftui

minIOS11

swift5_3

prototype/propertyWrapperFields

prototype/propertyWrappers

prototype/propertyWrapper

Swift5

throwables

prototype/Swift_4_1

prototype/Swift_4_0

prototype/Swift_3_2

prototype/queryBuilders

temp/develop

corestore4_develop

prototype/mainContextToPSC

master_ios_7_to_9

swift_2_3

gh-pages

_corespotlight-test

9.3.0

9.2.0

9.1.0

9.0.0

8.1.0

8.0.1

8.0.0

7.3.1

7.3.0

7.2.0

7.1.0

7.0.4

7.0.3

7.0.2

7.0.1

7.0.0

6.3.2

6.3.1

6.3.0

6.2.1

6.2.0

6.1.0

6.0.0

5.3.1

5.3.0

5.2.0

5.1.1

5.1.0

5.0.0

4.2.4

4.2.3

4.2.2

4.2.1

4.2.0

4.1.4

4.1.3

4.1.1

4.1.0

4.0.5

4.0.4

4.0.3

4.0.2

4.0.1

4.0.0

4.0.0-beta3

4.0.0-beta2

4.0.0-beta

3.1.1

3.1.0

3.0.3

3.0.2

3.0.1

3.0.0

2.1.3

2.1.2

2.1.1

2.1.0

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.6.10

1.6.9

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.1

1.2.0

1.0.2

1.0.1

1.0.0

0.2.1

0.2.0

0.1.2

0.1.1

0.1.0

Labels

Clear labels

abandoned

corestore bug

discussion

duplicate

enhancement

fixed

fixed

help wanted

in progress

investigating

ios/compiler bug

irreproducible

looking for workarounds

not a bug

pending

pending docs update

pull-request

Mirrored from GitHub Pull Request

question

regression

user bug

wontfix

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

adam (Adam Melkus)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/CoreStore-JohnEstropia#365