Long make calls lock table metadata #1170
Description
Bug Report
Description
A client locks table metadata for the entire duration of a make function call. When other clients attempt to drop or declare child tables, the call is blocked until the first client finishes make. This approach scales poorly with number of clients and number of child tables.
Reproducibility
Include:
- OS: Any
- Python Version: Any
- MySQL Version: Any
- MySQL Deployment Strategy: Any
- DataJoint Version: 0.14.1
- Minimum steps required to reproduce:
- See the test case presented Table Metadata Lock by long populate calls LorenFrankLab/spyglass#1030
Proposed Solution
As an alternative to writing a Computed.make function, allow user to write three functions:
make_fetchfor reading inputsmake_compute, which is not run in a transaction, and is passed the return value ofmake_fetchmake_insert, which inserts computed values using the same transaction semantics asmake.
In pseudocode, these three functions will be used in the following routine as such:
if hasattr(table, "make"):
return make()
else:
assert hasattr(table, "make_fetch")
assert hasattr(table, "make_compute")
assert hasattr(table, "make_insert")
input = make_fetch()
conn.disconnect() # I assume this disconnect step is to ensure that make_compute cannot insert?
result = make_compute(input)
tx = conn.start_transaction()
input2 = make_fetch()
if hash(serialize(input2)) == hash(serialize(input)):
result = make_insert(result)
tx.commit()
return result
else:
print("ERROR: inputs have changed")
tx.abort()
return NoneAdditional Research and Context
Related Issues
- Table Metadata Lock by long populate calls LorenFrankLab/spyglass#1030
- Allow disable transaction for select populates LorenFrankLab/spyglass#1067
cc: @dimitri-yatsenko @ttngu207 @CBroz1 @samuelbray32 @peabody124
Metadata
Metadata
Assignees
Labels
No labels
Activity
dimitri-yatsenko commentedon Aug 21, 2024
This will be inside populate and will follow all the conventions of
populate.Yes, it looks correct. If we want to be fancy, we can prohibit
insertcalls inmake_fetch,insertandfetchcalls frommake_compute, andfetchoperators frommake_insert.dimitri-yatsenko commentedon Aug 21, 2024
@ethho, our blob serialization serializes most types of data into binary strings. You can use a hash on the serialized data for comparing
inputtoinput2horsto commentedon Aug 30, 2024
I am following this. I see the #1171. Can this issue here be updated regularly when this is implemented / in a testable state? Thanks for taking care of this!
dimitri-yatsenko commentedon Sep 7, 2024
This is a high priority for multiple labs.
horsto commentedon Jan 16, 2025
This has not been merged / solved yet, right?
CBroz1 commentedon Jan 17, 2025
Looks like the last commit was 5m ago
We gave users a 'check threads' tool to check for hold-ups and see whose process might be slowing things down
implement tri-partite make (fix datajoint#1170)
Merge pull request #1204 from dimitri-yatsenko/master