Skip to content

CASSJAVA-130: Fix Result-Metadata Race Causing Corrupt SKIP_METADATA Row Decoding - #2094

Open
yifan-c wants to merge 2 commits into
apache:trunkfrom
yifan-c:CASSJAVA-130/fix-result-metadata-race
Open

yifan-c wants to merge 2 commits into
apache:trunkfrom
yifan-c:CASSJAVA-130/fix-result-metadata-race

Conversation

@yifan-c

@yifan-c yifan-c commented Jul 17, 2026

Copy link
Copy Markdown

Patch by Yifan Cai; Reviewed by TBD for CASSJAVA-130

…Row Decoding

Patch by Yifan Cai; Reviewed by TBD for CASSJAVA-130
@yifan-c
yifan-c requested a review from SiyaoIsHiding July 17, 2026 22:05

@tejalchak tejalchak left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid fix. One follow-up question inline around toMessage worth deciding whether to thread the snapshot there too before/after merge.

Comment on lines 304 to 306
logPrefixJoiner.join(this.sessionName, nodeRequestId, currentExecutionIndex),
resultMetadataSnapshot(statement));
Message message = Conversions.toMessage(statement, executionProfile, context);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was looking at the race condition window here and wanted to get a second pair of eyes ...

While the callback now safely uses the snapshot, toMessage still reads the live metadata state. I'm wondering if a concurrent schema update between capturing the snapshot and the toMessage execution could cause a mismatch (where the request sends the new metadata ID, but the callback decodes using the old snapshot). Should toMessage accept the snapshot to close any chance of discrepancy?

Since ConversionsMetadataRaceTest calls getResultDefinitions directly rather than driving through toMessage, so it might not catch this without some change.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense. We should share the same metadata snapshot in toMessage

Comment on lines +223 to +238
static class ResultMetadata {
private final ByteBuffer resultMetadataId;
private final ColumnDefinitions resultSetDefinitions;

private ResultMetadata(ByteBuffer resultMetadataId, ColumnDefinitions resultSetDefinitions) {
this.resultMetadataId = resultMetadataId;
this.resultSetDefinitions = resultSetDefinitions;
}

ByteBuffer getResultMetadataId() {
return resultMetadataId;
}

ColumnDefinitions getResultSetDefinitions() {
return resultSetDefinitions;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A minor thought: if this branch is strictly Java 17+, we could consider using a record to avoid the boilerplate code
static record ResultMetadata(ByteBuffer resultMetadataId, ColumnDefinitions resultSetDefinitions) {}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the compiler is pinned to java 8. It cannot use the language feature in jdk 17.

@SiyaoIsHiding SiyaoIsHiding left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of having resultMetadata as a field of BoundStatement?
In DefaultBoundStatement constructor, resultMetadata will basically be a snapshot of the preparedStatement.getCurrentResultMetadata() at that moment.
Then at response decode time, as long as rowsMetadata.columnSpecs.isEmpty(), use the one in BountStatement.resultMetadata, otherwise, update the PreparedStatement.resultMetadata.

@yifan-c

yifan-c commented Jul 24, 2026

Copy link
Copy Markdown
Author

What do you think of having resultMetadata as a field of BoundStatement? In DefaultBoundStatement constructor, resultMetadata will basically be a snapshot of the preparedStatement.getCurrentResultMetadata() at that moment. Then at response decode time, as long as rowsMetadata.columnSpecs.isEmpty(), use the one in BountStatement.resultMetadata, otherwise, update the PreparedStatement.resultMetadata.

@SiyaoIsHiding , thanks for the suggestion! Snapshotting on constructor could make the code more clean.

There are three drawbacks:

  1. When the bound statement is reused, after schema change, it is going to send stable metadata ID on every execution.
  2. Snapshotting becomes a hidden contract that every other BoundStatement implementation to follow. Maybe there is no other implementation at all, but BoundStatement is a public interface.
  3. (2) can be addressed with adding the getResultMetadata public API in the BoundStatement. However, public API change requires more discussion, a larger size of decision. For the same reason, the getCurrentResultMetadata is added to DefaultPreparedStatement, not in PreparedStatement.

Therefore, I'd like to stay with the current implementation. Let me know if the above makes sense.

@SiyaoIsHiding

Copy link
Copy Markdown
Contributor

On second thought, what is the race condition that we are trying to fix again?
Is this the current driver's behavior?

  1. SELECT * ... statement is prepared in Cassandra with md5 abc123, and result set (b, c) that hashes to cde456
  2. column a gets added to the table, server updates the result set to (a, b, c) which hashes to fff789
  3. client sends a DefaultBoundStatement for (statementId=abc123, resultId=cde456) and skip_metadata flag
  4. client receives the response from another DefaultBoundStatement, so it updates the PreparedStatement's resultMetadataId to fff789
  5. On the server side, cde456!=fff789, so C* responds with ROWS(..., no_metadata=false, metadata_changed=true, new_metadata_id=fff789,col specs for (a,b,c))
  6. client updates its column specifications again, and decodes the response with (a, b, c).

Under what circumstances does the driver corrupt result metadata?

@yifan-c

yifan-c commented Jul 29, 2026

Copy link
Copy Markdown
Author

On second thought, what is the race condition that we are trying to fix again? Is this the current driver's behavior?

1. SELECT * ... statement is prepared in Cassandra with md5 abc123, and result set (b, c) that hashes to cde456

2. column `a` gets added to the table,  server updates the result set to (a, b, c) which hashes to fff789

3. client sends a DefaultBoundStatement for (statementId=abc123, resultId=cde456) and skip_metadata flag

4. client receives the response from another DefaultBoundStatement, so it updates the PreparedStatement's resultMetadataId to fff789

5. On the server side, cde456!=fff789, so C* responds with ROWS(..., no_metadata=false, metadata_changed=true, new_metadata_id=fff789,col specs for (a,b,c))

6. client updates its column specifications again, and decodes the response with (a, b, c).

Under what circumstances does the driver corrupt result metadata?

Not really, corruption requires two different bound statements of the same prepared statement in flight around a schema change, where the older one gets a SKIP_METADATA reply and its decode is delayed past a newer one's full-metadata reply that updates the shared cache first. The following is the corruption scenario modified from your example.

  1. SELECT * ... prepared as abc123, current result metadata cde456 → (b, c).
  2. Client binds two statements against the same prepared statement — A and B — both while the cache still says cde456/(b,c).
  3. Client sends A's Execute(abc123, resultId=cde456, skip_metadata). Server's current metadata is still cde456 at the moment it processes A (schema hasn't changed yet), so it replies NO_METADATA — just row bytes for (b,c), no column specs, no new id.
  4. Column a is added; server's live metadata for abc123 becomes fff789 → (a,b,c).
  5. Client sends B's Execute(abc123, resultId=cde456, skip_metadata). Now the id mismatches (cde456 != fff789), so server replies with full metadata: column specs (a,b,c), new_metadata_id=fff789.
  6. B's response arrives and is decoded first. Since rowsMetadata.columnSpecs is non-empty, the driver updates the shared PreparedStatement's cached metadata to fff789/(a,b,c).
  7. A's response arrives next. It's the NO_METADATA reply from step 3 — empty columnSpecs, just 2 columns' worth of row bytes laid out as (b,c). The driver decodes it via the 3-column (a,b,c) layout swapped in step 6. The decoder now applies a 3-column definition to 2-columns'-worth of bytes → wrong types/offsets against real data, i.e. corruption (or an exception, depending on type sizes).

@SiyaoIsHiding

Copy link
Copy Markdown
Contributor

Got it. So the protocol is that if the server replies NO_METADATA, the result uses the metadata attached in the request message. The driver can corrupt when it violates this protocol by using the result metadata in the prepared statement instead.
My solution proposed above about using the BoundStatement to hold the result metadata won't work because a BoundStatement doesn't necessarily correspond to one request message.
I think your solution of using the NodeResponseCallback to hold the result metadata and pass from the request to the response decoding will work because each NodeResponseCallback indeed corresponds to one request message and one response message.
I was looking for a cleaner way for passing info from a request message to a response message decoding and didn't find a better way.
I'd give my +1 but I hope there are better ways tho.

@SiyaoIsHiding

Copy link
Copy Markdown
Contributor

Thank you for the fix!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the changes that preserve request-time result metadata.

No open issues 🎉

What changed in this PR

Addresses CASSJAVA-130 in the driver’s CQL execution path by preserving request-time result metadata, preventing concurrent metadata updates from corrupting SKIP_METADATA row decoding.

Changes:

  • Exposes an atomic snapshot of a prepared statement’s result metadata ID and column definitions.
  • Uses the same snapshot for request encoding and response decoding.
  • Adds deterministic regression tests for encode-side and decode-side races and updates a reactive test call.
File Description
core/​src/​test/​java/​com/​datastax/​oss/​driver/​internal/​core/​cql/​ConversionsMetadataRaceTest.java Adds regression tests for concurrent metadata updates.
core/​src/​test/​java/​com/​datastax/​dse/​driver/​internal/​core/​cql/​reactive/​CqlRequestReactiveProcessorTest.java Updates the result-conversion call for the snapshot parameter.
core/​src/​main/​java/​com/​datastax/​oss/​driver/​internal/​core/​cql/​DefaultPreparedStatement.java Exposes the metadata snapshot and makes its fields final.
core/​src/​main/​java/​com/​datastax/​oss/​driver/​internal/​core/​cql/​CqlRequestHandler.java Captures and retains metadata for each request attempt.
core/​src/​main/​java/​com/​datastax/​oss/​driver/​internal/​core/​cql/​Conversions.java Uses captured metadata for encoding and decoding.

Review effort: Balanced

@yifan-c

yifan-c commented Nov 9, 2026

Copy link
Copy Markdown
Author

It feels a bit odd to see AI bot giving recommendation on PR approval, but thanks for the reminder.

@SiyaoIsHiding, anything else pending before merging the fix?

@SiyaoIsHiding

Copy link
Copy Markdown
Contributor

Hi @yifan-c ! Sorry for the confusion. I'm still not 100% convinced this is the best solution we got, so I still want to spend more time on this PR.
Bret didn't have much capacity recently, so I tried several AI code review tools on this PR, that's why you saw the copilot's review. Claude pointed out some problems that I want to take a deeper look at. I might also invite other java driver side developers for another opinion, too.
I will get back to you!

@SiyaoIsHiding

Copy link
Copy Markdown
Contributor

@yifan-c
What do you think of a new class

public class BoundExecute extends Execute {

  private final ColumnDefinitions resultSetDefinitions;

So each message actually holds its own result metadata? A draft: https://github.com/apache/cassandra-java-driver/compare/trunk...SiyaoIsHiding:java-driver:CASSJAVA-130?expand=1

@SiyaoIsHiding

SiyaoIsHiding commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Bret mentioned (tho he didn't look into this PR deeply) that we can just add a field to Execute in the native protocol lib.

public class Execute extends Message {

  public final byte[] queryId;
  public final byte[] resultMetadataId;
  @Nullable public final Iterable<?> resultMetadata; // a pointer to an immutable Iterable, not a copy. `ColumnDefinitions` implements `Iterable`

So it will be even cleaner and other implementations of PreparedStatement and BoundStatement can make use of it, too. And, we plan to cut a release for the native protocol library anyway.
I like this solution the most.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants