Repository navigation
bigquery: ARROW query_and_wait returns empty result on cached large-result queries #18935
Copy link
Copy link
Open
Labels
priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Description
I've been seeing falures on pyarrow tests, that seem related to the new feature (#18650)
________ test_query_and_wait_arrow_cached_results[too_large_for_inline] ________
[gw1] linux -- Python 3.12.12 /tmpfs/src/github/google-cloud-python/packages/google-cloud-bigquery/.nox/system-3-12/bin/python
bigquery_client =
priority = None, page_size = None, num_rows = 20000, payload_size = 1024
@pytest.mark.parametrize(
("priority", "page_size", "num_rows", "payload_size"),
[
(None, 100, 1000, 0),
# The results are too large to return inline, so the cached jobs.query
# response has no rows and totalRows is 0.
(None, None, 20000, 1024),
(enums.QueryPriority.INTERACTIVE, 100, 1000, 0),
],
ids=["some_rows_inline", "too_large_for_inline", "no_rows_inline"],
)
def test_query_and_wait_arrow_cached_results(
bigquery_client, priority, page_size, num_rows, payload_size
):
# No other query uses this literal, so only the second run can hit the
# cache.
query = (
f"SELECT x, '{uuid.uuid4().hex}' AS key, REPEAT('x', {payload_size}) AS p "
f"FROM UNNEST(GENERATE_ARRAY(1, {num_rows})) AS x ORDER BY x"
)
job_config = bigquery.QueryJobConfig(use_query_cache=True)
if priority is not None:
job_config.priority = priority
def read():
rows = bigquery_client.query_and_wait(
query,
job_config=job_config,
page_size=page_size,
query_results_format=enums.QueryResultsFormat.ARROW,
)
table = pyarrow.Table.from_batches(list(rows.to_arrow_iterable()))
return bigquery_client.get_job(rows.job_id, location=rows.location), table
first_job, first = read()
cached_job, cached = read()
assert not first_job.cache_hit
assert cached_job.cache_hit
> assert cached.num_rows == num_rows
E assert 0 == 20000
E + where 0 = pyarrow.Table\n\n----.num_rows
Metadata
Metadata
Assignees
Labels
priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Activity
[-]Pyarrow flakes[/-][+]bigquery: flaky tests related to arrow implementation[/+]ohmayr commentedon Oct 31, 2026
Good catch. This isn't just a flaky test. I believe when we hit a cache, the backend previously sent
totalRowsas 0 (buggy) and now it's not sending it. Probably a recent change. See b/573764383We technically use
getQueryResultsonly to wait (not for results) which kinda depends ontotalRowsand returns an empty table straight up (which we don't want). We can update that logic to usePageTokenfor the arrow path.[-]bigquery: flaky tests related to arrow implementation[/-][+]bigquery: ARROW query_and_wait returns empty result on cached large-result queries[/+]daniel-sanche commentedon Oct 31, 2026
Ok, thanks for the quick response. I'll bump this to P1 for now