[REQUIRED] Step 2: Describe your environment
- Operating System version: GCP Cloud Run (Java 8 runtime). US-Central region.
- Firebase SDK version: 6.8.1, 6.8.0, 6.7.0
- Library version: 6.8.1, 6.8.0, 6.7.0
- Firebase Product: database
[REQUIRED] Step 3: Describe the problem
I experience intermittent connection problems with Firebase Realtime Database from a Spring Boot App running on Google Cloud Run (Managed) service. It works without any problem locally (tried multiple networks and computers as well), but once deployed as a container on Cloud Run - queries response time to Firebase become unpredictable or sometimes no response at all.
I can constantly reproduce this problem during last 48+ hours. Not sure if it was the case before. I am under impression that it worked well in the past.
In order to isolate the problem I came up with a simple controller that queries ONE object from a database and waits for the result. Here is the controller code:
@GetMapping("/me")
public String me() throws Exception{
DatabaseReference ref = FirebaseDatabase.getInstance().getReference().child("PromoCodes");
logger.info("DatabaseReference: " + ref.getKey());
ref.addListenerForSingleValueEvent(new ValueEventListener() {
@Override
public void onDataChange(DataSnapshot snapshot) {
logger.info("-- onDataChange: " + snapshot.getChildrenCount());
}
@Override
public void onCancelled(DatabaseError error) {
logger.info("! onCancelled: " + error);
}
});
return "Firebase path: " + ref.getPath();
}
Here is one sample response from log files:
2019-07-01 20:22:20.689 PDTGET200 247 B 68 ms Safari 12 /me
2019-07-01 20:27:15.886 PDT2019-07-02 03:27:15.885 WARN 1 --- [database-worker] io.netty.util.internal.MacAddressUtil : Failed to find a usable hardware address from the network interfaces; using random bytes: 8c:e7:67:82:bd:a7:0c:1b
2019-07-01 20:28:28.887 PDT2019-07-02 03:28:28.886 ERROR 1 --- [ebsocket-worker] c.g.f.d.connection.WebsocketConnection : [ws_0] WebSocket error
2019-07-01 20:28:28.887 PDT
2019-07-01 20:28:28.887 PDTjava.lang.IllegalStateException: executor not accepting a task at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:60) ~[netty-resolver-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:200) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:49) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:188) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:174) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:502) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:476) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:415) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:540) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:529) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:101) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:1009) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:517) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:487) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_202] at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_202] at java.lang.Thread.run(Unknown Source) [na:1.8.0_202]
2019-07-01 20:28:28.887 PDT
2019-07-01 20:29:59.885 PDT2019-07-02 03:29:59.685 INFO 1 --- [se-event-target] ApiApplication : -- onDataChange: 1
In this case I got the response 7+ minutes after the request, including [database-worker] and [ebsocket-worker] exceptions prior to that. This is pretty much a pattern.
Sometimes I get response in a matter of milliseconds like expected, other times it takes 5-15 minutes to get it eventually with a bunch of similar connectivity exceptions in the logs.
Steps to reproduce:
Deploy simple Spring Boot app with a controller that queries small object from Firebase to Cloud Run.
Let me know if I can provide any other details to help resolve it.
[REQUIRED] Step 2: Describe your environment
[REQUIRED] Step 3: Describe the problem
I experience intermittent connection problems with Firebase Realtime Database from a Spring Boot App running on Google Cloud Run (Managed) service. It works without any problem locally (tried multiple networks and computers as well), but once deployed as a container on Cloud Run - queries response time to Firebase become unpredictable or sometimes no response at all.
I can constantly reproduce this problem during last 48+ hours. Not sure if it was the case before. I am under impression that it worked well in the past.
In order to isolate the problem I came up with a simple controller that queries ONE object from a database and waits for the result. Here is the controller code:
@GetMapping("/me")
public String me() throws Exception{
Here is one sample response from log files:
2019-07-01 20:22:20.689 PDTGET200 247 B 68 ms Safari 12 /me
2019-07-01 20:27:15.886 PDT2019-07-02 03:27:15.885 WARN 1 --- [database-worker] io.netty.util.internal.MacAddressUtil : Failed to find a usable hardware address from the network interfaces; using random bytes: 8c:e7:67:82:bd:a7:0c:1b
2019-07-01 20:28:28.887 PDT2019-07-02 03:28:28.886 ERROR 1 --- [ebsocket-worker] c.g.f.d.connection.WebsocketConnection : [ws_0] WebSocket error
2019-07-01 20:28:28.887 PDT
2019-07-01 20:28:28.887 PDTjava.lang.IllegalStateException: executor not accepting a task at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:60) ~[netty-resolver-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:200) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:49) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:188) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:174) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:502) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:476) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:415) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:540) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:529) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:101) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:1009) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:517) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:487) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) [netty-transport-4.1.34.Final.jar!/:4.1.34.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) [netty-common-4.1.34.Final.jar!/:4.1.34.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_202] at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_202] at java.lang.Thread.run(Unknown Source) [na:1.8.0_202]
2019-07-01 20:28:28.887 PDT
2019-07-01 20:29:59.885 PDT2019-07-02 03:29:59.685 INFO 1 --- [se-event-target] ApiApplication : -- onDataChange: 1
In this case I got the response 7+ minutes after the request, including [database-worker] and [ebsocket-worker] exceptions prior to that. This is pretty much a pattern.
Sometimes I get response in a matter of milliseconds like expected, other times it takes 5-15 minutes to get it eventually with a bunch of similar connectivity exceptions in the logs.
Steps to reproduce:
Deploy simple Spring Boot app with a controller that queries small object from Firebase to Cloud Run.
Let me know if I can provide any other details to help resolve it.