Ephemeral mappers

[augustuswm]

Adds the concept of ephemeral mappers. Ephemeral mappers are mappers that exist only within the runtime memory of the server. Their purpose is to solve a pain point around defining mappers by configuration as opposed to database state.

Main differences:

1. Ephemeral mappers are defined at VContext create time.
2. Ephemeral mappers are not persisted to backend storage (i.e. database).
3. Ephemeral mappers do not support activation limits or counts.
4. Ephemeral mappers can not be updated or deleted.

Generally an application will require at least one mapper rule to bootstrap itself. This is something like an admin group with a user that should be the initial admin. From here, that admin can create groups, and additional dynamic mappers that are stored alongside the data of the application. This bootstrapping mapper is a good use case for an ephemeral mapper. The user defines a configuration for an ephemeral mapper that is passed to the v-api context builder at runtime, and that mapper is installed in memory. An admin can then log in and be granted permissions. On restart, the user can decide if they still want to pass this configuration to the v-api context or not. If they don't then the mapper rule acts as if it never existed.

Because these mappers are ephemeral, we also need to include the ability to record when they are processed. This change adds a log table that records the execution results of mappers so that independent of a mapper being ephemeral or dynamic, we still know when it was used to grant permissions

[notpeter]

At first glance this looks fine.

1. As I mentioned in the other review I don't love the Ephemeral term because they can't go away. They are perhaps static, stateless, immutable but ephemeral things are short lived and these are largely immortal. Open to other terms.

2. I think this could use a db transaction. Currently record_mapper_event is triggered before register_api_user (which could fail). Ideally these would occur in the same transaction so if the user create failed we don't have a mapper_event that didn't actually happen. Not sure how difficult that is to thread, but wanted to mention it.

3. No referential integrity for mapper_event.user_id -- often times audit tables intentionally don't have foreign keys (for good reasons) but I think they might be reasonable here.

[notpeter]

1. Should we validate each rule? (e.g. mapping_ctx.validate(&config.rule))
2. Can we catch duplicate names? Since the UUID is deterministic based on name a copy pasta error in config could be wonky.

[augustuswm]

1. Yes, absolutely this should be validating. I'll fix that.
2. Yeah, easy enough to verify if we have duplicate names are fail out early here.

[notpeter]

Year is wrong on these migrations.

[notpeter]

nit: Should MapperEvent.ephemeral and NewMapperEvent.ephemeral actually be MapperSource?

[augustuswm]

Yeah it should. That way we can expand the variants if needed.

[notpeter]

Is this comment outdated? I would expect the upsert in consume_mapping_activation wouldn't result in conflicts.

[augustuswm]

1. So they are immortal in the sense that they remain until they are changed in configuration, but they are never installed into the db, I'll work some more on what they are called, I think the tension I am trying to balance is a term that is clear to the operator of the server (doing configuration and running) as well as an admin type user that might create / read mappers. Specifically avoiding the concept that they are static to the server itself. It might be just something like ServerConfigured vs ClientConfigured.
2. Ideally yes, but I'm not sure it is feasible atm. I would really actually prefer a saga (but that is a whole other issue) that I could run forward and then rollback, as we don't want to try and hold a transaction over the scope of the number of things we need to do. Also to prevent races we need to consume the mapper at the point in time that we evaluate the groups to assign, but this has to happen before we determine if the caller has the necessary permissions to grant the groups as we need the group list to determine that. I think what this looks like in the nearer terms is two different events, one that tracks the consumption of a mapper activation and another indicating the assignment of the properties to the user record. I think it is ultimately find to consume an activation and then error as long as we can then infer from the system why we may have a mismatch.
3. I think this is indicative of an issue with our user registration flow. There are two different branches for existing vs new users. Mapper evaluations occur before that branch, and so we can't have a foreign id as the user id may not yet be persisted. I think this is an indicator that we should be persisting the empty user record first, then preforming mapper evaluations, and assigning permissions. This would then let us have a foreign key.