How to have performant fetches from a data stack without locking the main thread, but still allow reads concurrently with writes #405

New Issue

2025-12-29T15:31:02+01:00

adam commented

2025-12-29 15:31:02 +01:00

Originally created by @jimbengtsson92 on GitHub (Mar 7, 2023).

In our app all of our writes basically look like this:

dataStack.perform(asynchronous: { (transaction) -> Void in
    // Create, update or delete on transaction
    try transaction.create(Into<MyEntity>()) / deleteAll(From<MyEntity>(.where(...)))
}, completion: { result in
    completion()
})

and our reads like this:

DispatchQueue.main.async { [self] in
    do {
        // Fetch one or many entities as snapshots from dataStack
        // We use the snapshots in order to work with objects that might be deleted while we are reading them
        let snapshots = dataStack.fetchAll(From<MyEntity>(.where(...))).compactMap { $0.asSnapshot(in: dataStack) }
        completion(snapshots.transformToMyModels())
    } catch {
        // Handle error
    }
}

With this setup we can achieve concurrent read and writes, but when performing a read this is locking the main thread. If I do not perform the fetch on the main thread I get a fatalError saying "Attempted to fetch from a 'CoreStore.DataStack' outside the main thread." error from CoreStore.

One way I have found we can do it to perform reads without locking the main thread is to use a transaction for these as well, like this:

dataStack.perform(asynchronous: { (transaction) -> [MyModel] in
    // Fetch one or many entities as snapshots from transaction
    let snapshots = transaction.fetchAll(From<MyEntity>(.where(...))).compactMap { $0.asSnapshot(in: transaction) }
    return snapshots.transformToMyModels()
}, completion: { result in
    completion(result)
})

With this approach I cannot however perform reads concurrently with writes, since they are both run on the same background serial queue that is created from the transaction. This is making the performance way worse in the cases when we are trying to do writes at around the same time as we do reads.

Do you, or anyone else, have any advice on how to do create fetches that do not lock the main thread, but can still run concurrently with writes?

Originally created by @jimbengtsson92 on GitHub (Mar 7, 2023). In our app all of our writes basically look like this: ``` dataStack.perform(asynchronous: { (transaction) -> Void in // Create, update or delete on transaction try transaction.create(Into<MyEntity>()) / deleteAll(From<MyEntity>(.where(...))) }, completion: { result in completion() }) ``` and our reads like this: ``` DispatchQueue.main.async { [self] in do { // Fetch one or many entities as snapshots from dataStack // We use the snapshots in order to work with objects that might be deleted while we are reading them let snapshots = dataStack.fetchAll(From<MyEntity>(.where(...))).compactMap { $0.asSnapshot(in: dataStack) } completion(snapshots.transformToMyModels()) } catch { // Handle error } } ``` With this setup we can achieve concurrent read and writes, but when performing a read this is locking the main thread. If I do not perform the fetch on the main thread I get a fatalError saying "Attempted to fetch from a 'CoreStore.DataStack' outside the main thread." error from CoreStore. One way I have found we can do it to perform reads without locking the main thread is to use a transaction for these as well, like this: ``` dataStack.perform(asynchronous: { (transaction) -> [MyModel] in // Fetch one or many entities as snapshots from transaction let snapshots = transaction.fetchAll(From<MyEntity>(.where(...))).compactMap { $0.asSnapshot(in: transaction) } return snapshots.transformToMyModels() }, completion: { result in completion(result) }) ``` With this approach I cannot however perform reads concurrently with writes, since they are both run on the same [background serial queue](https://github.com/JohnEstropia/CoreStore#asynchronous-transactions) that is created from the transaction. This is making the performance way worse in the cases when we are trying to do writes at around the same time as we do reads. Do you, or anyone else, have any advice on how to do create fetches that do not lock the main thread, but can still run concurrently with writes?

adam commented

2025-12-29 15:31:03 +01:00

@JohnEstropia commented on GitHub (Mar 7, 2023):

The fetches are fine, but the error message indicates you are accessing the fetched objects' properties outside the main thread

@JohnEstropia commented on GitHub (Mar 7, 2023): The fetches are fine, but the error message indicates you are accessing the fetched objects' properties outside the main thread

adam commented

2025-12-29 15:31:04 +01:00

@jimbengtsson92 commented on GitHub (Mar 8, 2023):

@JohnEstropia

The fetches are fine, but the error message indicates you are accessing the fetched objects' properties outside the main thread

The error comes from this method in DataStack+Querying.swift:

public func fetchAll<B: FetchChainableBuilderType>(_ clauseChain: B) throws -> [B.ObjectType] {

        Internals.assert(
            Thread.isMainThread,
            "Attempted to fetch from a \(Internals.typeName(self)) outside the main thread."
        )
        return try self.mainContext.fetchAll(clauseChain)
    }

So it looks like it is the actual fetch that is the problem.

An example of a complete fetch where this error happens is:

let scheduledEventSnapshots = try self.dataStack.fetchAll(
                        From<ScheduledEventEntity>()
                            .where(combineByOr: (\.scheduledUntil >= from && \.scheduledFrom <= to), (\.scheduledFrom < from && \.scheduledUntil > to))
                    ).compactMap { $0.asSnapshot(in: self.dataStack) }

@jimbengtsson92 commented on GitHub (Mar 8, 2023): @JohnEstropia > The fetches are fine, but the error message indicates you are accessing the fetched objects' properties outside the main thread The error comes from this method in `DataStack+Querying.swift`: ``` public func fetchAll<B: FetchChainableBuilderType>(_ clauseChain: B) throws -> [B.ObjectType] { Internals.assert( Thread.isMainThread, "Attempted to fetch from a \(Internals.typeName(self)) outside the main thread." ) return try self.mainContext.fetchAll(clauseChain) } ``` So it looks like it is the actual fetch that is the problem. An example of a complete fetch where this error happens is: ``` let scheduledEventSnapshots = try self.dataStack.fetchAll( From<ScheduledEventEntity>() .where(combineByOr: (\.scheduledUntil >= from && \.scheduledFrom <= to), (\.scheduledFrom < from && \.scheduledUntil > to)) ).compactMap { $0.asSnapshot(in: self.dataStack) } ```

adam commented

2025-12-29 15:31:04 +01:00

@JohnEstropia commented on GitHub (Mar 8, 2023):

@jimbengtsson92 I see, then it's exactly as the method says: fetches from DataStack instances should come from the main thread. If you want to fetch things asynchronously, you can write something like:

DispatchQueue.global(qos: .utility).async {
    withExtendedLifetime(dataStack.beginUnsafe()) { transaction in
        let snapshots = try? transaction.fetchAll(...).compactMap({ $0.asSnapshot()  })
        completion(snapshots)
    }
}
// code written by hand, so there might be compiler errors

Just be careful not to access the relationship objects directly from the snapshots, as those need to be fetched again from either a DataStack or a transaction.

Then when you need to edit the objects for these snapshots inside transactions, convert them back into the transaction instance by doing snapshot.asEditable(in: transaction)

@JohnEstropia commented on GitHub (Mar 8, 2023): @jimbengtsson92 I see, then it's exactly as the method says: fetches from `DataStack` instances should come from the main thread. If you want to fetch things asynchronously, you can write something like: ```swift DispatchQueue.global(qos: .utility).async { withExtendedLifetime(dataStack.beginUnsafe()) { transaction in let snapshots = try? transaction.fetchAll(...).compactMap({ $0.asSnapshot() }) completion(snapshots) } } // code written by hand, so there might be compiler errors ``` Just be careful not to access the relationship objects directly from the snapshots, as those need to be fetched again from either a `DataStack` or a transaction. Then when you need to edit the objects for these snapshots inside transactions, convert them back into the transaction instance by doing `snapshot.asEditable(in: transaction)`

Sign in to join this conversation.

Branches Tags

develop

prototype/swift6

master

prototype/concurrency

minIOS13

prototype/containers

transactionSourceTracking

prototype/swiftui

minIOS11

swift5_3

prototype/propertyWrapperFields

prototype/propertyWrappers

prototype/propertyWrapper

Swift5

throwables

prototype/Swift_4_1

prototype/Swift_4_0

prototype/Swift_3_2

prototype/queryBuilders

temp/develop

corestore4_develop

prototype/mainContextToPSC

master_ios_7_to_9

swift_2_3

gh-pages

_corespotlight-test

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/CoreStore#405