[_499] authenticate native and pam_password schemes using iRODS 4.3+ auth flow

Author: d-w-moore

README.md

@@ -132,48 +132,87 @@ required when they are placed in the environment file.
 Creating PAM or Native Credentials File (.irodsA)
 -------------------------------------------------
 
-Two free functions exist for creating encoded authentication files:
+Two free functions exist which allow the user to create encoded authentication files
+for use in the client's iRODS login environment:
 ```
-irods.client_init.write_native_credentials_to_secrets_file
-irods.client_init.write_pam_credentials_to_secrets_file
+irods.client_init.write_native_irodsA_file
+irods.client_init.write_pam_irodsA_file
 ```
 
+These functions can roughly be described as duplicating the function of iinit (from iCommands),
+once a valid irods_environment.json has already been created.
+
 Each takes a cleartext password and writes an appropriately processed version of it
-into an .irodsA (secrets) file in the login environment.
+into an .irodsA "password" or "secrets" in the appropriate location.
 
-Examples:
-For the `native` authentication scheme, we can use the currently set iRODS password to create the .irodsA file directly:
+That location is  ~/.irods/.irodsA) unless IRODS_AUTHENTICATION_FILE has
+been set with an alternate file path in the OS environment.
 
-```python
-import irods.client_init as iinit
-iinit.write_native_credentials_to_secrets_file(irods_password)
-```
+As an example, for the `native` authentication scheme, it is simple to create the
+.irodsA file directly:
 
-Note, in the `pam_password` case, this involves sending the cleartext password
-to the server (SSL must be enabled!) and then writing the scrambled token that
-is returned from the transaction.
+```bash
+$ echo '{ "irods_user_name":"rods", ...  }'> ~/.irods/irods_environment.json
+$ python -c "import irods.client_init, getpass
+irods.client_init.write_native_irodsA_file(getpass.getpass('Enter iRODS password -> '))"
+```
 
-If an .irodsA file exists already, it will be overwritten by default; however, if these functions'
+If an .irodsA file already exists, it will be overwritten by default; however, if these functions'
 overwrite parameter is set to `False`, an exception of type `irods.client_init.irodsA_already_exists`
-will be raised to indicate the older .irodsA file is present.
+will be raised to warn of an older .irodsA file that might otherwise be overwritten.
 
-For the `pam_password` authentication scheme, we must first ensure an `irods_environment.json` file exists in the 
-client environment (necessary for establishing SSL/TLS connection parameters as well as obtaining a PAM token from the server after connecting)
-and then make the call to write .irodsA using the Bash commands:
+Equivalently, we can issue the following command.
 
 ```bash
-$ cat > ~/.irods/irods_environment.json << EOF
-{
-  "irods_user_name":"rods",
-  "irods_host":"server-hostname",
-  ...  [all other connection settings, including SSL parameters, needed for communication with iRODS] ...
-}
-EOF
-$ python -c "import irods.client_init as iinit; iinit.write_pam_credentials_to_secrets_file(pam_cleartext_password)"
+$ prc_write_irodsA.py native <<<"${MY_CURRENT_IRODS_PASSWORD}"
 ```
 
-PAM logins
-----------
+The redirect may be left off, in which case the user is prompted for the iRODS password
+and echo of the keyboard input will be suppressed.  (Regardless which technique is used,
+no password will be visible on the terminal during or after input.)
+
+For the `pam_password` scheme, typically SSL/TLS must first be enabled to avoid sending data related
+to the password - or even sending the raw password itself - over a network connection in the clear.
+
+Thus, for `pam_password` authentication to work well, we should first ensure when setting up the
+client environment that the `irods_environment.json` file includes the appropriate
+SSL/TLS connection parameters.  If present, `iinit` can be used to verify this condition is fulfilled,
+as of course its invocation would create a valid .irodsA from merely prompting the user for their
+PAM password
+
+But if we wish to use the Python client for this purpose instead, we can run:
+
+```python
+irods.client_init.write_pam_irodsA_file(getpass.getpass('Enter current PAM password -> '))
+```
+
+Or from the Bash command shell, we simply run:
+
+```bash
+$ prc_write_irodsA.py pam_password <<<"${MY_CURRENT_PAM_PASSWORD}"
+```
+
+again leaving out the redirection if password prompting is preferable.
+
+As a final note, in the "pam_password" scheme the default SSL requirement can be disabled (for purposes
+of testing only):
+
+```python
+session = irods.session.iRODSSession(host = "localhost", port = 1247, 
+                                     user = "alice", password = "test123", zone="tempZone",
+                                     authentication_scheme = "pam_password")
+
+session.set_auth_option_for_scheme('pam_password', irods.auth.pam_password.ENSURE_SSL_IS_ACTIVE, False)
+
+# Do something with the session:
+home = session.collections.get('/tempZone/home/alice')
+```
+
+Note however that in future releases of iRODS it is possible that extra SSL checking could be
+implemented server-side, at which point, the above code could not be guaranteed to work.
+
+Legacy (iRODS 4.2-compatible) PAM authentication
+------------------------------------------------
 
 Since v2.0.0, the Python iRODS Client is able to authenticate via PAM using the same file-based client environment as the
 iCommands.

irods/account.py

@@ -28,6 +28,11 @@ def __init__(
                 irods_host = v
 
         self.env_file = env_file
+
+        # The '_auth_file' attribute will be written in the call to iRODSSession.configure,
+        # if an .irodsA file from the client environment is used to load password information.
+        self._auth_file = ""
+
         tuplify = lambda _: _ if isinstance(_, (list, tuple)) else (_,)
         schemes = [_.lower() for _ in tuplify(irods_authentication_scheme)]
 

irods/api_number.py

@@ -179,4 +179,5 @@
     "REPLICA_CLOSE_APN": 20004,
     "TOUCH_APN": 20007,
     "AUTH_PLUG_REQ_AN": 1201,
+    "AUTHENTICATION_APN": 110000,
 }

irods/auth/__init__.py

@@ -1,8 +1,116 @@
+import importlib
+import logging
+import weakref
+from irods.api_number import api_number
+from irods.message import iRODSMessage, JSON_Message
+import irods.password_obfuscation as obf
+import irods.session
+
+
 __all__ = ["pam_password", "native"]
 
+
 AUTH_PLUGIN_PACKAGE = "irods.auth"
 
-import importlib
+
+# Python3 does not have types.NoneType
+_NoneType = type(None)
+
+
+class AuthStorage:
+    """A class that facilitates flexible means password storage.
+
+    Using an instance of this class, passwords may either be
+
+        - directly placed in a member attribute (pw), or
+
+        - they may be written to / read from a specified file path in encoded
+          form, usually in an .irodsA file intended for iRODS client authentication.
+
+    Most typical of this class's utility is the transfer of password information from
+    the pam_password to the native authentication flow.  In this usage, whether the
+    password is stored in RAM or in the filesystem depends on whether it was read
+    originally as a function parameter or from an authentication file, respectively,
+    when the session was created.
+
+    """
+
+    @staticmethod
+    def get_env_password(filename=None):
+        options = dict(irods_authentication_file=filename) if filename else {}
+        return irods.session.iRODSSession.get_irods_password(**options)
+
+    @staticmethod
+    def get_env_password_file():
+        return irods.session.iRODSSession.get_irods_password_file()
+
+    @staticmethod
+    def set_env_password(unencoded_pw, filename=None):
+        if filename is None:
+            filename = AuthStorage.get_env_password_file()
+        from ..client_init import _open_file_for_protected_contents
+
+        with _open_file_for_protected_contents(filename, "w") as irodsA:
+            irodsA.write(obf.encode(unencoded_pw))
+        return filename
+
+    @staticmethod
+    def get_temp_pw_storage(conn):
+        """Fetch the AuthStorage instance associated with this connection object."""
+        return getattr(conn, "auth_storage", lambda: None)()
+
+    @staticmethod
+    def create_temp_pw_storage(conn):
+        """Creates an AuthStorage instance to be cached and associated with this connection object.
+
+        Called multiple times for the same connection, it will return the cached instance.
+
+        The value returned by this call should be stored by the caller into an appropriately scoped
+        variable to ensure the AuthStorage instance endures for the desired lifetime -- that is,
+        for however long we wish to keep the password information around.  This is because the
+        connection object only maintains a weak reference to said instance.
+        """
+
+        # resolve the weakly referenced AuthStorage obj for the connection if there is one.
+        weakref_to_store = getattr(conn, "auth_storage", None)
+        store = weakref_to_store and weakref_to_store()
+
+        # In absence of a persistent AuthStorage object, create one.
+        if store is None:
+            store = AuthStorage(conn)
+            # So that the connection object doesn't hold on to password data too long:
+            conn.auth_storage = weakref.ref(store)
+        return store
+
+    def __init__(self, conn):
+        self.conn = conn
+        self.pw = ""
+        self._auth_file = ""
+
+    @property
+    def auth_file(self):
+        if self._auth_file is None:
+            return ""
+        return self._auth_file or self.conn.account.derived_auth_file
+
+    def use_client_auth_file(self, auth_file):
+        """Set to None to completely suppress use of an .irodsA auth file."""
+        if isinstance(auth_file, (str, _NoneType)):
+            self._auth_file = auth_file
+        else:
+            msg = f"Invalid object in {self.__class__}._auth_file"
+            raise RuntimeError(msg)
+
+    def store_pw(self, pw):
+        if self.auth_file:
+            self.set_env_password(pw, filename=self.auth_file)
+        else:
+            self.pw = pw
+
+    def retrieve_pw(self):
+        if self.auth_file:
+            return self.get_env_password(filename=self.auth_file)
+        return self.pw
 
 
 def load_plugins(subset=set(), _reload=False):
@@ -18,9 +126,74 @@ def load_plugins(subset=set(), _reload=False):
     return dir_
 
 
-# TODO(#499): X models a class which we could define here as a base for various server or client state machines
-#             as appropriate for the various authentication types.
+class REQUEST_IS_MISSING_KEY(Exception):
+    pass
 
 
-class X:
-    pass
+def throw_if_request_message_is_missing_key(request, required_keys):
+    for key in required_keys:
+        if not key in request:
+            raise REQUEST_IS_MISSING_KEY(f"key = {key}")
+
+
+def _auth_api_request(conn, data):
+    message_body = JSON_Message(data, conn.server_version)
+    message = iRODSMessage(
+        "RODS_API_REQ", msg=message_body, int_info=api_number["AUTHENTICATION_APN"]
+    )
+    conn.send(message)
+    response = conn.recv()
+    return response.get_json_encoded_struct()
+
+
+__FLOW_COMPLETE__ = "authentication_flow_complete"
+__NEXT_OPERATION__ = "next_operation"
+
+
+CLIENT_GET_REQUEST_RESULT = "client_get_request_result"
+FORCE_PASSWORD_PROMPT = "force_password_prompt"
+STORE_PASSWORD_IN_MEMORY = "store_password_in_memory"
+
+
+class authentication_base:
+
+    def __init__(self, connection, scheme):
+        self.conn = connection
+        self.loggedIn = 0
+        self.scheme = scheme
+
+    def call(self, next_operation, request):
+        logging.info("next operation = %r", next_operation)
+        old_func = func = next_operation
+        # One level of indirection should be sufficient to get a callable method.
+        if not callable(func):
+            old_func, func = (func, getattr(self, func, None))
+        func = func or old_func
+        if not callable(func):
+            raise RuntimeError("client request contains no callable 'next_operation'")
+        resp = func(request)
+        logging.info("resp = %r", resp)
+        return resp
+
+    def authenticate_client(
+        self, next_operation="auth_client_start", initial_request={}
+    ):
+        to_send = initial_request.copy()
+        to_send["scheme"] = self.scheme
+
+        while True:
+            resp = self.call(next_operation, to_send)
+            if self.loggedIn:
+                break
+            next_operation = resp.get(__NEXT_OPERATION__)
+            if next_operation is None:
+                raise ClientAuthError(
+                    "next_operation key missing; cannot determine next operation"
+                )
+            if next_operation in (__FLOW_COMPLETE__, ""):
+                raise ClientAuthError(
+                    f"authentication flow stopped without success: scheme = {self.scheme}"
+                )
+            to_send = resp
+
+        logging.info("fully authenticated")