feat(AIP-4210): add code gen diagram + tweaks (#1148)

Author: noahdietz

aip/client-libraries/4210.md

@@ -2,6 +2,10 @@
 id: 4210
 state: approved
 created: 2018-06-22
+js_scripts:
+  - /assets/js/graphviz/viz.js
+  - /assets/js/graphviz/lite.render.js
+  - /assets/js/aip/aip-graphviz.js
 ---
 
 # Client library generators
@@ -31,25 +35,87 @@ expectation.
 
 ## Guidance
 
+The general flow of code generation for client libraries and supporting code in
+most supported languages is outlined below.
+
+**Note:** Exceptions to this pattern are typically due to use of a unique stack
+e.g. Node.js use of `protobuf.js` and `grpc-node` which don't have code
+generation, or Python using a protobuf wrapper in `proto-plus-python`, but the
+general GAPIC flow remains the same.
+
+```graphviz
+digraph {
+  node [ style="filled,solid" shape=box fontname="Roboto" ];
+  splines=ortho;
+  nodesep=0.3;
+  center=true;
+
+  proto [ label="API Protobuf\nDescriptors" shape=rectancle fillcolor=aliceblue ];
+
+  subgraph cluster_code_generators {
+    rank = same;
+    style = filled;
+    fillcolor = lightgrey;
+    node [ shape=oval ];
+
+    protobuf [ label="protobuf\ngenerator" fillcolor=deepskyblue3 ];
+    grpc [ label="gRPC\ngenerator" fillcolor=gold3 ];
+    gapic [ label="GAPIC\ngenerator" fillcolor=darkseagreen ];
+  }
+
+  proto -> protobuf;
+  proto -> grpc;
+  proto -> gapic;
+
+  subgraph cluster_generated_code {
+    rank = same;
+    style = filled;
+    fillcolor = lightgrey;
+    node [ shape=rectangle ];
+
+    protobuf_output [ label="Message & Enum\nCode" fillcolor=deepskyblue3 ];
+    grpc_output [ label="Server & Client\nStubs" fillcolor=gold3 ];
+    gapic_output [ label="Google API\nClient" fillcolor=darkseagreen ];
+  }
+  
+  protobuf -> protobuf_output;
+  grpc -> grpc_output;
+  gapic -> gapic_output;
+
+  assembly [ label="Package\nassembly" shape=oval fillcolor=aliceblue ];
+  
+  protobuf_output -> assembly
+  grpc_output -> assembly
+  gapic_output -> assembly
+
+  assembled_package [ label="Package of\ngenerated code" fillcolor=aliceblue ];
+
+  assembly -> assembled_package
+}
+```
+
+The following sections focus on the "GAPIC generator" in the above diagram.
+
 ### Protobuf plugins
 
-Code generators **must** be implemented as plugins to `protoc`, the protocol
-buffer compiler. The [plugin system][0] allows plugins to be written in any
-language, and plugins **should** ordinarily be written in the language being
-targeted, in order to take advantage of in-language tooling, and to ensure that
-experts in the target environment are able to meaningfully contribute.
+The protobuf compiler, `protoc`, supports a [plugin system][0] for code
+generation. The plugin system allows plugins to be written _in_ and _for_ any
+language.
+
+Code generators **must** be implemented as `protoc` plugins. The following
+rules apply to the implementation of a client library generator as a `protoc`
+plugin:
 
+- The plugin  **should** be written in the language being targeted for
+  generation.
 - `protoc` expects plugins to be an executable in `$PATH`, and named
   `protoc-gen-{plugin_name}`, corresponding to the `--{plugin_name}_out` option
-  sent to the `protoc` executable.
-  - For a plugin creating client libraries for a specific language, the option
-    name **should** follow the convention `--{lang}_gapic_out` (meaning the
-    corresponding plugin executable is named `protoc-gen-{lang}_gapic`).
-- Plugins **must** accept a serialized `CodeGeneratorRequest` object (defined
-  in [`plugin.proto`][1]) on `stdin`; the bulk of this is a series of
-  `FileDescriptorProto` messages (defined in [`descriptor.proto`][2]).
-- Plugins **must** emit a serialized `CodeGeneratorResponse` object (defined in
-  [`plugin.proto`][1]) on `stdout`.
+  sent to the `protoc` executable. As such:
+  - the plugin executable **should** be named `protoc-gen-{lang}_gapic`
+  - the plugin option  **should** follow the convention `--{lang}_gapic_out`
+- The plugin **must not** leverage `protoc` "insertion points". Despite the
+  `protoc` plugin documentation indicating the existence of insertion points,
+  their use is unsupported and discouraged by the Protobuf team.
 
 ### CLI options
 
@@ -59,6 +125,10 @@ options are required, `protoc` allows them to be passed as
 `--{plugin_name}_opt`, and the string provided here becomes set as the
 `parameter` string on the `CodeGeneratorRequest`.
 
+**Important:** The `CodeGeneratorRequest.parameter` value is a comma-delimited
+string of _all_ associated plugin option values that appear at execution time.
+This means that commas cannot be used to delimit list-like plugin option values.
+
 Code generators **must not** rely on environment variables for configuration.
 
 ## Expected behavior
@@ -68,6 +138,12 @@ client library generator (in other words: the libraries that the generators
 write). Client libraries **must** implement these concepts in order to be
 considered complete.
 
+### Messages and Enums
+
+Client library generators **should not** generate code for `message` or `enum`
+descriptors which are already generated by the Protobuf-provided code
+generators.
+
 ### Services and methods
 
 Each of the `service` and `rpc` directives in the requested protos **must** be
@@ -96,10 +172,12 @@ methods for each RPC.
 - Finally, service classes **must** also accept credentials, which are used
   appropriately when requests are made. (Accepting a custom gRPC channel
   satisfies this requirement.)
+- Code generators **must not** generate client _stub_ classes, that would
+  normally be generated by gRPC, in addition to client library classes.
 
 ### Long-running operations
 
-<!-- TODO(b/126177694): Reference the LRO AIP once we have it. -->
+<!-- TODO(1145): Move to its own client library AIP. -->
 
 An RPC is considered to be a "long-running" RPC if (and only if) the RPC's
 return type is [`google.longrunning.Operation`][3]. Any API which has one or
@@ -135,11 +213,16 @@ in the [`MethodDescriptorProto`][6] message using the `client_streaming` and
 `server_streaming` keys.
 
 <!-- prettier-ignore-start -->
-[0]: https://developers.google.com/protocol-buffers/docs/reference/other
+[0]: https://protobuf.dev/reference/other
 [1]: https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/plugin.proto
 [2]: https://github.com/google/protobuf/blob/master/src/google/protobuf/descriptor.proto
 [3]: https://github.com/googleapis/googleapis/blob/master/google/longrunning/operations.proto#L122
 [4]: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/any.proto
 [5]: https://github.com/googleapis/googleapis/blob/master/google/longrunning/operations.proto#L222
 [6]: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L269
 <!-- prettier-ignore-end -->
+
+## Changelog
+
+- **2023-06-22**: Added code gen diagram, message/enum guidance, and cleaned up
+  plugin & option guidance.