Getting 3fs.cpp:list_bucket(2632): ListBucketRequest returns with error

[niteenkole]

Using ibmcloud object storage for more then year now and everything was good,recently we see errors like

journalctl -u kubelet -f

[/var/data/kubelet/pods/4a5da436-796c-xxxxxxxx/volumes/ibm~ibmc-s3fs/pvc-23d3c887-0xxxxxb-4812-9711-8exxxxxx5]s3fs.cpp:list_bucket(2632): ListBucketRequest returns with error.

we don't see any issue with PVC or pv and even no errors in describe of pod

kubectl get storageclass |grep s3

ibmc-s3fs-cold-cross-region ibm.io/ibmc-s3fs 118d
ibmc-s3fs-cold-regional ibm.io/ibmc-s3fs 118d
ibmc-s3fs-flex-cross-region ibm.io/ibmc-s3fs 118d
ibmc-s3fs-flex-perf-cross-region ibm.io/ibmc-s3fs 118d
ibmc-s3fs-flex-perf-regional ibm.io/ibmc-s3fs 118d
ibmc-s3fs-flex-regional ibm.io/ibmc-s3fs 118d
ibmc-s3fs-standard-cross-region ibm.io/ibmc-s3fs 118d
ibmc-s3fs-standard-perf-cross-region ibm.io/ibmc-s3fs 118d
ibmc-s3fs-standard-perf-regional ibm.io/ibmc-s3fs 118d
ibmc-s3fs-standard-regional ibm.io/ibmc-s3fs 118d
ibmc-s3fs-vault-cross-region ibm.io/ibmc-s3fs 118d
ibmc-s3fs-vault-regional ibm.io/ibmc-s3fs 118d

kubectl get pods -n kube-system | grep object-storage
ibmcloud-object-storage-driver-64qs4 1/1 Running 3 104d
ibmcloud-object-storage-driver-7s9hj 1/1 Running 2 102d
ibmcloud-object-storage-driver-7zc7m 1/1 Running 3 46d
ibmcloud-object-storage-driver-f7q8q 1/1 Running 2 59d
ibmcloud-object-storage-driver-fw7q5 1/1 Running 4 104d
ibmcloud-object-storage-driver-x7zq7 1/1 Running 3 102d
ibmcloud-object-storage-plugin-6b6cf9c6cb-n57wd 1/1 Running 2 102d