Skip to content

bigquery: ARROW query_and_wait returns empty result on cached large-result queries #18935

Open

Description

@daniel-sanche

I've been seeing falures on pyarrow tests, that seem related to the new feature (#18650)

________ test_query_and_wait_arrow_cached_results[too_large_for_inline] ________
[gw1] linux -- Python 3.12.12 /tmpfs/src/github/google-cloud-python/packages/google-cloud-bigquery/.nox/system-3-12/bin/python

bigquery_client = 
priority = None, page_size = None, num_rows = 20000, payload_size = 1024

    @pytest.mark.parametrize(
        ("priority", "page_size", "num_rows", "payload_size"),
        [
            (None, 100, 1000, 0),
            # The results are too large to return inline, so the cached jobs.query
            # response has no rows and totalRows is 0.
            (None, None, 20000, 1024),
            (enums.QueryPriority.INTERACTIVE, 100, 1000, 0),
        ],
        ids=["some_rows_inline", "too_large_for_inline", "no_rows_inline"],
    )
    def test_query_and_wait_arrow_cached_results(
        bigquery_client, priority, page_size, num_rows, payload_size
    ):
        # No other query uses this literal, so only the second run can hit the
        # cache.
        query = (
            f"SELECT x, '{uuid.uuid4().hex}' AS key, REPEAT('x', {payload_size}) AS p "
            f"FROM UNNEST(GENERATE_ARRAY(1, {num_rows})) AS x ORDER BY x"
        )
        job_config = bigquery.QueryJobConfig(use_query_cache=True)
        if priority is not None:
            job_config.priority = priority
    
        def read():
            rows = bigquery_client.query_and_wait(
                query,
                job_config=job_config,
                page_size=page_size,
                query_results_format=enums.QueryResultsFormat.ARROW,
            )
            table = pyarrow.Table.from_batches(list(rows.to_arrow_iterable()))
            return bigquery_client.get_job(rows.job_id, location=rows.location), table
    
        first_job, first = read()
        cached_job, cached = read()
    
        assert not first_job.cache_hit
        assert cached_job.cache_hit
>       assert cached.num_rows == num_rows
E       assert 0 == 20000
E        +  where 0 = pyarrow.Table\n\n----.num_rows

Activity

  1. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    triage meI really want to be triaged.
    on Oct 31, 2026
  2. changed the title [-]Pyarrow flakes[/-] [+]bigquery: flaky tests related to arrow implementation[/+] on Oct 31, 2026
  3. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    and removed
    triage meI really want to be triaged.
    on Oct 31, 2026
  4. ohmayr commented on Oct 31, 2026

    @ohmayr
    Contributor

    Good catch. This isn't just a flaky test. I believe when we hit a cache, the backend previously sent totalRows as 0 (buggy) and now it's not sending it. Probably a recent change. See b/573764383

    We technically use getQueryResults only to wait (not for results) which kinda depends on totalRows and returns an empty table straight up (which we don't want). We can update that logic to use PageToken for the arrow path.

  5. changed the title [-]bigquery: flaky tests related to arrow implementation[/-] [+]bigquery: ARROW query_and_wait returns empty result on cached large-result queries[/+] on Oct 31, 2026
  6. added
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    and removed
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Oct 31, 2026
  7. daniel-sanche commented on Oct 31, 2026

    @daniel-sanche
    ContributorAuthor

    Ok, thanks for the quick response. I'll bump this to P1 for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions