Repository navigation
Pubsub 0.37.2: Observed non-recoverable stream error 400 #5913
Description
Hello! Firstly, thank you for this awesome library!
I am running around 40 pubsub client workers in an EKS cluster. After more than a day of running uninterrupted, one of my pubsub workers observed a non-recoverable error. (The rest are still running fine)
OS - ubuntu:16.04
Python version - Python 3.5.2
Pubsub version - google-cloud-pubsub==0.37.2
Stack Trace:
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded
INFO:google.cloud.pubsub_v1.subscriber._protocol.bidi:Re-established stream
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed non-recoverable stream error 400 Invalid arguments provided: the number of ack ids to modify must be equal to the number of ack deadlines.
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:RPC termination has signaled streaming pull manager shutdown.
INFO:google.cloud.pubsub_v1.subscriber._protocol.bidi:Thread-ConsumeBidirectionalStream exiting
INFO:google.cloud.pubsub_v1.subscriber._protocol.leaser:Thread-LeaseMaintainer exiting.
INFO:google.cloud.pubsub_v1.subscriber._protocol.heartbeater:Thread-Heartbeater exiting.
Code snippet
flow_control = pubsub_v1.types.FlowControl(max_messages=500)
subscription = sub_client.subscribe("pubsub_subscription", process_message, flow_control=flow_control)Where process_message batches messages for a few minutes and acks them all when their payloads have been inserted into a database.
Any help would be appreciated here!
Activity