Skip to content

Fix HTTP/2 race condition waiting on stream events.#439

Merged
lovelydinosaur merged 1 commit into
masterfrom
fix-http2-race-condition-waiting-stream-events
Nov 18, 2021
Merged

Fix HTTP/2 race condition waiting on stream events.#439
lovelydinosaur merged 1 commit into
masterfrom
fix-http2-race-condition-waiting-stream-events

Conversation

@lovelydinosaur

@lovelydinosaur lovelydinosaur commented Nov 18, 2021

Copy link
Copy Markdown
Contributor

While looking at https://github.com/encode/httpx/issues/1414 I've come across a race condition in the HTTP/2 handling. It's not something I'm able to figure out any sensible test case for, but can describe how to reproduce locally...

First we'll setup Hypercorn as a locally HTTP/2-capable server...

app.py

async def app(scope, receive, send):
    if scope["type"] != "http":
        raise Exception("Only the HTTP protocol is supported")

    await send({
        'type': 'http.response.start',
        'status': 200,
        'headers': [
            (b'content-type', b'text/plain'),
            (b'content-length', b'5'),
        ],
    })
    await send({
        'type': 'http.response.body',
        'body': b'hello',
    })

Let's run that lil kiddo...

$ hypercorn app:app

Okey dokes, now let's bump into it with a coupla concurrent HTTP/2 requests. We've not setup any certificate on the hypercorn server, so we'll just enforce HTTP/2 support, so that we'll still get HTTP/2 without any https connection.

import asyncio
import httpcore
import traceback


async def download(http, idx):
    try:
        response = await http.request("GET", "http://127.0.0.1:8000")
    except Exception as exc:
        traceback.print_exc()
        raise exc
    else:
        print(idx, response)


async def main():
    async with httpcore.AsyncConnectionPool(http1=False, http2=True) as http:
        tasks = [download(http, idx) for idx in range(2)]
        await asyncio.gather(*tasks)


asyncio.run(main())

If we run that without this proposed change we get the following...

$ venv/bin/python ./example.py 
0 <Response [200]>
... [hangs]

Here's why:

  • Flows A and B both start.
  • Flow A hits while not self._events.get(stream_id) - there's no events yet so we move on to _receive_events().
  • Flow B hits while not self._events.get(stream_id) - there's no events yet so we move on to _receive_events().
  • Flow A reads the response events for both streams, and returns ok.
  • Flow B hangs waiting for network data, despite the fact that there is now actually an event for it to return.

The key to resolving this is to move the read lock, so that we check "have we got an event that we can return" inside the lock.

I'm not wild about this PR, because it's just difficult to be convinced that it's resilient and correct enough, and it's also not obvious how to test case it. but. Working code's gotta be better than code that I can demo as hanging. So.

@lovelydinosaur lovelydinosaur merged commit bd72c1b into master Nov 18, 2021
@lovelydinosaur lovelydinosaur deleted the fix-http2-race-condition-waiting-stream-events branch November 18, 2021 12:03
@lovelydinosaur lovelydinosaur mentioned this pull request Jan 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant