As per celery#5605, the Consul backend does not cleanly associate responses

from Consul with the outbound Celery request that caused it. This leaves
it prone to mistaking the (final) response from an operation N as the
response to an (early) part of operation N + 1.

This changes fix that by using a separate connection for each request.
That of course has the downside of (a) being relatively expensive and (b)
increasing the rate of connection requests into Consul:

- The former is annoying, but at least the backend works reliably.

- The latter can cause Consul to reject excessive connection attempt, but
  if it does, at least it returns a clear indication of this (IIRC, it
  responds with an HTTP 429"too many connections" indication).

  Additionally, this issue can be ameliorated by enabling retries in the
  python-consul2 (which I believe should be turned on regards less to handle
  transient network issues). This is fixed by the PR in
  https:/github.com/poppyred/python-consul2/pull/31.

Note that we have never seen (b) outside a test specifically trying to hammer
the system, but we see (a) all the time in our normal system tests.

Author: ShaheedHaque

celery/backends/consul.py

@@ -47,8 +47,12 @@ def _init_from_params(self, hostname, port, virtual_host, **params):
         logger.debug('Setting on Consul client to connect to %s:%d',
                      hostname, port)
         self.path = virtual_host
-        self.client = consul.Consul(host=hostname, port=port,
-                                    consistency=self.consistency)
+        self.hostname = hostname
+        self.port = port
+
+    @property
+    def client(self):
+        return consul.Consul(host=self.hostname, port=self.port, consistency=self.consistency)
 
     def _key_to_consul_key(self, key):
         key = bytes_to_str(key)