http2 GOAWAY stopping client from reconnecting

[joshforbes]

We're getting this error:

Which is stopping the client and leaving our agent in an offline state. The token endpoint is nothing special running behind an nginx server. Not sure if there is something we should change or if something is going on with the client. Any tips appreciated.

Thanks!

┆Issue is synchronized with this Jira Task by Unito

[sacOO7]

Hi @joshforbes, thanks for raising the issue!
I have some questions before this issue is resolved

1. ably-go version
2. go version
3. Which type of auth mechanism are you using?
4. Is this only happening with the latest version of ably-go or do previous versions also have the same issue?
5. If you have a small working snippet that would be able to reproduce this issue, it would be super useful for us to resolve it asap!

[joshforbes]

1. ably-go 1.2.12
2. Go 1.20
3. Token
4. We upgraded from ably-go 1.1.3 🫣. Things were a lot different and we had years of workarounds to problems.
5. Sorry, I don't. This one is a bit more common than the other issue I opened. After running for two days on 83 devices we have 3 of them stuck in this state.

We're considering replacing the HTTP client in the Ably client to disable HTTP/2.

[joshforbes]

I appreciate that the Ably client allowed us to so easily replace the HTTP client. We decided to move forward with swapping to an HTTP/1.1 client. I think we'll probably stick with that unless you have some ideas about how to fix the HTTP/2 errors.

[sacOO7]

Sure, if HTTP/2 is causing errors, you can keep HTTP/1.1 as a default client. We will need to see HTTP2 support for ably-go

[sacOO7]

Related issue -> olivere/elastic#1443

[sync-by-unito]

➤ Automation for Jira commented:

The link to the corresponding Jira issue is https://ably.atlassian.net/browse/SDK-4120

[sacOO7]

We need to add retry mechanism to fallback endpoint when this error occurs. More details for the issue given here
connectrpc/connect-go#672
Need to modify canFallback method to check same error in the response, if it's GOAWAY then return true

ably-go/ably/rest_client.go

Lines 801 to 806 in dd8236f

	func canFallBack(err error, res *http.Response) bool {
	return isStatusCodeBetween500_504(res) || // RSC15l3
	isCloudFrontError(res) || //RSC15l4
	isTimeoutOrDnsErr(err) //RSC15l1, RSC15l2
	}

[sacOO7]

@joshforbes since error is returned from your own server agents.rmm.dev/issue-token-request, you might like to implement authCallback option with custom retry mechanism inside it. This will make sure, even if the tokenRequest fails, it will keep retrying until tokenRequest fetch is successful.
As per RSA4d, connection will transition to failed state when authUrl or authCallback fails.