diff --git a/contrib/temporal-aws-lambda/README.md b/contrib/temporal-aws-lambda/README.md new file mode 100644 index 0000000000..0619796fdf --- /dev/null +++ b/contrib/temporal-aws-lambda/README.md @@ -0,0 +1,80 @@ +# Temporal AWS Lambda worker module + +This module provides a direct AWS Lambda Java handler for running a Temporal worker for one Lambda invocation. + +## Usage + +Add `temporal-aws-lambda` next to your Temporal SDK dependency, then expose the returned handler from your Lambda class: + +```java +import com.amazonaws.services.lambda.runtime.Context; +import com.amazonaws.services.lambda.runtime.RequestHandler; +import io.temporal.aws.lambda.LambdaWorker; +import io.temporal.common.WorkerDeploymentVersion; + +public final class Handler implements RequestHandler { + private static final RequestHandler WORKER = + LambdaWorker.define( + new WorkerDeploymentVersion("orders-worker", "2026-06-02"), + builder -> + builder + .setTaskQueue("orders") + .registerWorkflowImplementationTypes(OrderWorkflowImpl.class) + .registerActivitiesImplementations(new OrderActivitiesImpl())); + + @Override + public Void handleRequest(Object input, Context context) { + return WORKER.handleRequest(input, context); + } +} +``` + +`TEMPORAL_TASK_QUEUE` can provide the task queue. If it is not set, call `setTaskQueue`. + +Connection options are loaded with `temporal-envconfig` when the handler is constructed during Lambda cold start. The Lambda worker checks `TEMPORAL_CONFIG_FILE` first, then `$LAMBDA_TASK_ROOT/temporal.toml`, then `./temporal.toml`, then falls back to the envconfig defaults and Temporal environment variables. The `configure` callback, passed as the second parameter to `LambdaWorker.define`, also runs during handler construction, so non-invocation configuration is prepared once and reused. + +Use the per-invocation configuration overload when final options depend on the Lambda `Context` or on resources opened for one invocation. The cold-start `configure` callback passed as the second parameter to `LambdaWorker.define` still runs once; the invocation callback runs before Temporal service stubs, client, and worker are created: + +```java +private static final RequestHandler WORKER = + LambdaWorker.define( + new WorkerDeploymentVersion("orders-worker", "2026-06-02"), + builder -> builder.registerWorkflowImplementationTypes(OrderWorkflowImpl.class), + (builder, context) -> { + builder.setTaskQueue(taskQueueFor(context)); + builder.addShutdownHook(() -> cleanupInvocationResources(context)); + }); +``` + +Each invocation receives a fresh copy of the base options, so mutations and shutdown hooks added by the invocation callback do not leak across warm invocations. If the invocation callback throws after adding shutdown hooks, those hooks still run for cleanup. Reusable resources such as AWS SDK clients should usually be created during cold start and reused across warm invocations. + +If you need to assemble options outside the `define` callback, call `LambdaWorkerOptions.newBuilderFromEnvironment()`, configure the returned builder, call `build()`, and pass the options to `LambdaWorker.newHandler(...)`. + +Dynamic workflow and activity implementations can be registered with `registerDynamicWorkflowImplementationType(...)` and `registerDynamicActivityImplementation(...)`. Java SDK worker rules still apply: only one dynamic workflow implementation type and one dynamic activity implementation can be registered per worker. + +The handler creates one worker per invocation, starts the worker, shuts it down before the Lambda deadline, runs shutdown hooks in order, and closes service stubs. Worker deployment versioning is always enabled for the supplied `WorkerDeploymentVersion`. If neither client nor worker identity is set by the user, each invocation uses `@` as the Temporal identity. + +`shutdownDeadlineBuffer` is the full shutdown window reserved at the end of the Lambda invocation. The default is 7 seconds: 5 seconds for `gracefulShutdownTimeout` and 2 seconds for hooks and service stubs. The worker runs until `remainingTime - shutdownDeadlineBuffer`, then stops and awaits termination for `gracefulShutdownTimeout`. If you change `gracefulShutdownTimeout` without explicitly setting `shutdownDeadlineBuffer`, the buffer is recomputed as `gracefulShutdownTimeout + 2s`. +If you explicitly set `shutdownDeadlineBuffer`, it must be greater than or equal to `gracefulShutdownTimeout`. + +## OpenTelemetry + +`OtelLambdaWorkerConfigurationHelper.configure(builder)` is a Lambda-specific facade over `temporal-opentelemetry`. It creates an `OpenTelemetryPlugin`, configures it with AWS X-Ray-compatible trace ID generation, installs it on service stubs options so it propagates to the workflow client and worker factory, and registers the plugin's timed flush hook for each Lambda invocation. The hook reports buffered Tally values before the OpenTelemetry provider hook force-flushes exporters. To enable it, call the helper from the handler initializer: + +```java +private static final RequestHandler WORKER = + LambdaWorker.define( + new WorkerDeploymentVersion("orders-worker", "2026-06-02"), + builder -> { + OtelLambdaWorkerConfigurationHelper.configure(builder); + builder + .setTaskQueue("orders") + .registerWorkflowImplementationTypes(OrderWorkflowImpl.class); + }); +``` + +The helper defaults the OTLP endpoint from `OTEL_EXPORTER_OTLP_ENDPOINT`, then `http://localhost:4317`. It defaults the service name from `OTEL_SERVICE_NAME`, then `AWS_LAMBDA_FUNCTION_NAME`, then `temporal-lambda-worker`, and sets it on the OpenTelemetry resource. To use an application-owned provider, call `builder.setOpenTelemetry(...)`; in that path, no exporters are created and the helper only installs the plugin and per-invocation flush hook. Providers and scopes are not closed after each invocation. + +Use `OtelLambdaWorkerConfigurationHelper.configureMetrics(...)`, `OtelLambdaWorkerConfigurationHelper.configureTracing(...)`, and `OtelLambdaWorkerConfigurationHelper.configureFlushHook(...)` when you want to compose metrics, tracing, or provider flushing separately around an application-owned OpenTelemetry instance. Use `temporal-opentelemetry` directly for non-Lambda serverless adapters or long-running workers. + +For Java logging, this module depends on `slf4j-api` only. It does not bundle a runtime logging binding, so Lambda log formatting remains owned by the application. diff --git a/contrib/temporal-aws-lambda/build.gradle b/contrib/temporal-aws-lambda/build.gradle new file mode 100644 index 0000000000..de4203109b --- /dev/null +++ b/contrib/temporal-aws-lambda/build.gradle @@ -0,0 +1,29 @@ +description = '''Temporal Java SDK AWS Lambda Worker Support Module''' + +ext { + awsLambdaJavaCoreVersion = '1.4.0' + otelVersion = '1.25.0' +} + +dependencies { + api platform("io.opentelemetry:opentelemetry-bom:$otelVersion") + + // This module shouldn't carry temporal-sdk with it, especially for situations when users may + // be using a shaded artifact. + compileOnly project(':temporal-sdk') + compileOnly "javax.annotation:javax.annotation-api:$annotationApiVersion" + + api "com.amazonaws:aws-lambda-java-core:$awsLambdaJavaCoreVersion" + api project(':temporal-opentelemetry') + + implementation project(':temporal-envconfig') + api "io.opentelemetry:opentelemetry-api" + implementation "io.opentelemetry.contrib:opentelemetry-aws-xray:$otelVersion" + implementation "org.slf4j:slf4j-api:$slf4jVersion" + + testImplementation project(':temporal-sdk') + testImplementation "io.opentelemetry:opentelemetry-sdk" + testImplementation "junit:junit:${junitVersion}" + + testRuntimeOnly group: 'ch.qos.logback', name: 'logback-classic', version: "${logbackVersion}" +} diff --git a/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/DefaultLambdaWorkerRuntime.java b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/DefaultLambdaWorkerRuntime.java new file mode 100644 index 0000000000..56efc4a7fd --- /dev/null +++ b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/DefaultLambdaWorkerRuntime.java @@ -0,0 +1,149 @@ +package io.temporal.aws.lambda; + +import io.temporal.client.WorkflowClient; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.common.converter.EncodedValues; +import io.temporal.serviceclient.WorkflowServiceStubs; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.Worker; +import io.temporal.worker.WorkerFactory; +import io.temporal.worker.WorkerFactoryOptions; +import io.temporal.worker.WorkerOptions; +import io.temporal.worker.WorkflowImplementationOptions; +import io.temporal.workflow.Functions; +import java.time.Duration; +import java.util.concurrent.TimeUnit; + +final class DefaultLambdaWorkerRuntime implements LambdaWorkerRuntime { + @Override + public Invocation create( + WorkflowServiceStubsOptions serviceStubsOptions, + WorkflowClientOptions clientOptions, + WorkerFactoryOptions workerFactoryOptions, + String taskQueue, + WorkerOptions workerOptions) { + WorkflowServiceStubs stubs = WorkflowServiceStubs.newServiceStubs(serviceStubsOptions); + WorkerFactory factory = null; + try { + WorkflowClient client = WorkflowClient.newInstance(stubs, clientOptions); + factory = WorkerFactory.newInstance(client, workerFactoryOptions); + Worker worker = factory.newWorker(taskQueue, workerOptions); + return new DefaultInvocation(stubs, factory, worker); + } catch (RuntimeException e) { + if (factory != null) { + try { + factory.shutdownNow(); + } catch (RuntimeException shutdownException) { + e.addSuppressed(shutdownException); + } + } + try { + stubs.shutdownNow(); + } catch (RuntimeException shutdownException) { + e.addSuppressed(shutdownException); + } + throw e; + } + } + + private static final class DefaultInvocation implements Invocation { + private final WorkflowServiceStubs stubs; + private final WorkerFactory factory; + private final WorkerRegistrar registrar; + + private DefaultInvocation(WorkflowServiceStubs stubs, WorkerFactory factory, Worker worker) { + this.stubs = stubs; + this.factory = factory; + this.registrar = new DefaultWorkerRegistrar(worker); + } + + @Override + public WorkerRegistrar getWorkerRegistrar() { + return registrar; + } + + @Override + public void start() { + factory.start(); + } + + @Override + public void shutdown() { + factory.shutdown(); + } + + @Override + public void shutdownNow() { + factory.shutdownNow(); + } + + @Override + public void awaitTermination(Duration timeout) { + factory.awaitTermination(timeout.toMillis(), TimeUnit.MILLISECONDS); + } + + @Override + public boolean isTerminated() { + return factory.isTerminated(); + } + + @Override + public void closeStubs(Duration timeout) { + stubs.shutdown(); + if (!stubs.awaitTermination(timeout.toMillis(), TimeUnit.MILLISECONDS)) { + stubs.shutdownNow(); + } + } + } + + private static final class DefaultWorkerRegistrar implements WorkerRegistrar { + private final Worker worker; + + private DefaultWorkerRegistrar(Worker worker) { + this.worker = worker; + } + + @Override + public void registerWorkflowImplementationTypes(Class... workflowImplementationClasses) { + worker.registerWorkflowImplementationTypes(workflowImplementationClasses); + } + + @Override + public void registerWorkflowImplementationTypes( + WorkflowImplementationOptions options, Class... workflowImplementationClasses) { + worker.registerWorkflowImplementationTypes(options, workflowImplementationClasses); + } + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, Functions.Func factory) { + worker.registerWorkflowImplementationFactory(workflowInterface, factory); + } + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, + Functions.Func1 factory, + WorkflowImplementationOptions options) { + worker.registerWorkflowImplementationFactory(workflowInterface, factory, options); + } + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, + Functions.Func factory, + WorkflowImplementationOptions options) { + worker.registerWorkflowImplementationFactory(workflowInterface, factory, options); + } + + @Override + public void registerActivitiesImplementations(Object... activityImplementations) { + worker.registerActivitiesImplementations(activityImplementations); + } + + @Override + public void registerNexusServiceImplementation(Object... nexusServiceImplementations) { + worker.registerNexusServiceImplementation(nexusServiceImplementations); + } + } +} diff --git a/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorker.java b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorker.java new file mode 100644 index 0000000000..7badec476a --- /dev/null +++ b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorker.java @@ -0,0 +1,531 @@ +package io.temporal.aws.lambda; + +import com.amazonaws.services.lambda.runtime.Context; +import com.amazonaws.services.lambda.runtime.RequestHandler; +import io.temporal.common.WorkerDeploymentVersion; +import io.temporal.opentelemetry.TimedShutdownHook; +import java.io.IOException; +import java.time.Duration; +import java.util.List; +import java.util.Objects; +import java.util.function.Consumer; +import javax.annotation.Nonnull; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** Creates AWS Lambda handlers that run one Temporal worker per Lambda invocation. */ +public final class LambdaWorker { + private static final Logger log = LoggerFactory.getLogger(LambdaWorker.class); + + private static final Duration MINIMUM_AVAILABLE_RUNTIME = Duration.ofSeconds(1); + private static final Duration LOW_AVAILABLE_RUNTIME_WARNING = Duration.ofSeconds(5); + + private LambdaWorker() {} + + /** + * Configures options for one Lambda invocation before Temporal service stubs, client, and worker + * are created. + * + *

The supplied builder is a fresh copy of the base options for the current invocation. + * Shutdown hooks added by this callback run for only that invocation, including when this + * callback throws. + */ + @FunctionalInterface + public interface InvocationConfigurator { + void configure(@Nonnull LambdaWorkerOptions.Builder builder, @Nonnull Context context); + } + + /** + * Returns an AWS Lambda Java handler that creates, starts, and shuts down one Temporal worker per + * invocation. + * + * @param version worker deployment version to advertise for this worker. + * @param configure callback invoked once while the Lambda handler is constructed. + */ + public static RequestHandler define( + @Nonnull WorkerDeploymentVersion version, + @Nonnull Consumer configure) { + LambdaWorkerOptions.validateVersion(version); + Objects.requireNonNull(configure, "configure"); + try { + LambdaWorkerOptions.Builder builder = + LambdaWorkerOptions.newBuilderFromEnvironment(System.getenv()); + configure.accept(builder); + return newHandler(version, builder.build()); + } catch (IOException e) { + throw new RuntimeException("Unable to load Temporal client configuration", e); + } + } + + /** + * Returns an AWS Lambda Java handler with both cold-start and per-invocation configuration. + * + * @param version worker deployment version to advertise for this worker. + * @param configure callback invoked once while the Lambda handler is constructed. + * @param invocationConfigure callback invoked for each Lambda invocation before Temporal service + * stubs, client, and worker are created. Required fields may be supplied by this callback. + */ + public static RequestHandler define( + @Nonnull WorkerDeploymentVersion version, + @Nonnull Consumer configure, + @Nonnull InvocationConfigurator invocationConfigure) { + LambdaWorkerOptions.validateVersion(version); + Objects.requireNonNull(configure, "configure"); + Objects.requireNonNull(invocationConfigure, "invocationConfigure"); + try { + LambdaWorkerOptions.Builder builder = + LambdaWorkerOptions.newBuilderFromEnvironment(System.getenv()); + configure.accept(builder); + return newHandler(version, builder.build(), invocationConfigure); + } catch (IOException e) { + throw new RuntimeException("Unable to load Temporal client configuration", e); + } + } + + /** Returns an AWS Lambda Java handler using already-configured Lambda worker options. */ + public static RequestHandler newHandler( + @Nonnull WorkerDeploymentVersion version, @Nonnull LambdaWorkerOptions options) { + return newHandler( + version, options, new DefaultLambdaWorkerRuntime(), sleep(), systemMonotonicClock()); + } + + /** + * Returns an AWS Lambda Java handler using base options and per-invocation configuration. + * + *

The supplied options are copied for each invocation before {@code invocationConfigure} runs. + * Required fields may be supplied by {@code invocationConfigure}. + */ + public static RequestHandler newHandler( + @Nonnull WorkerDeploymentVersion version, + @Nonnull LambdaWorkerOptions options, + @Nonnull InvocationConfigurator invocationConfigure) { + return newHandler( + version, + options, + invocationConfigure, + new DefaultLambdaWorkerRuntime(), + sleep(), + systemMonotonicClock()); + } + + static RequestHandler newHandler( + WorkerDeploymentVersion version, + LambdaWorkerOptions options, + LambdaWorkerRuntime runtime, + Sleeper sleeper) { + // This overload exists to let tests inject a fake runtime and sleeper. + return newHandler(version, options, runtime, sleeper, systemMonotonicClock()); + } + + static RequestHandler newHandler( + WorkerDeploymentVersion version, + LambdaWorkerOptions options, + InvocationConfigurator invocationConfigure, + LambdaWorkerRuntime runtime, + Sleeper sleeper) { + // This overload exists to let tests inject a fake runtime and sleeper. + return newHandler( + version, options, invocationConfigure, runtime, sleeper, systemMonotonicClock()); + } + + static RequestHandler newHandler( + WorkerDeploymentVersion version, + LambdaWorkerOptions options, + LambdaWorkerRuntime runtime, + Sleeper sleeper, + MonotonicClock clock) { + // This overload exists to let tests inject a deterministic clock. + return new Handler( + Objects.requireNonNull(options, "options").prepare(version), + Objects.requireNonNull(runtime, "runtime"), + Objects.requireNonNull(sleeper, "sleeper"), + Objects.requireNonNull(clock, "clock"), + false); + } + + static RequestHandler newHandler( + WorkerDeploymentVersion version, + LambdaWorkerOptions options, + InvocationConfigurator invocationConfigure, + LambdaWorkerRuntime runtime, + Sleeper sleeper, + MonotonicClock clock) { + LambdaWorkerOptions.validateVersion(version); + Objects.requireNonNull(options, "options"); + Objects.requireNonNull(invocationConfigure, "invocationConfigure"); + return new Handler( + context -> { + LambdaWorkerOptions.Builder builder = options.toBuilder(); + try { + invocationConfigure.configure(builder, context); + return builder.build().prepare(version).materialize(identityFor(context)); + } catch (RuntimeException e) { + throw new InvocationConfigurationException(e, builder); + } + }, + Objects.requireNonNull(runtime, "runtime"), + Objects.requireNonNull(sleeper, "sleeper"), + Objects.requireNonNull(clock, "clock"), + true); + } + + private static Sleeper sleep() { + return duration -> Thread.sleep(duration.toMillis()); + } + + interface Sleeper { + void sleep(Duration duration) throws InterruptedException; + } + + interface MonotonicClock { + long nanoTime(); + } + + private static MonotonicClock systemMonotonicClock() { + return System::nanoTime; + } + + private static String identityFor(Context context) { + return emptyToUnknown(context.getAwsRequestId()) + + "@" + + emptyToUnknown(context.getInvokedFunctionArn()); + } + + private static String emptyToUnknown(String value) { + return value == null || value.isEmpty() ? "unknown" : value; + } + + private interface OptionsMaterializer { + LambdaWorkerOptions.Materialized materialize(Context context); + } + + private static final class Handler implements RequestHandler { + private final OptionsMaterializer optionsMaterializer; + private final LambdaWorkerRuntime runtime; + private final Sleeper sleeper; + private final MonotonicClock clock; + private final boolean runShutdownHooksBeforeRuntimeCreation; + + private Handler( + LambdaWorkerOptions.Materialized preparedOptions, + LambdaWorkerRuntime runtime, + Sleeper sleeper, + MonotonicClock clock, + boolean runShutdownHooksBeforeRuntimeCreation) { + this( + context -> preparedOptions.materialize(identityFor(context)), + runtime, + sleeper, + clock, + runShutdownHooksBeforeRuntimeCreation); + Objects.requireNonNull(preparedOptions, "preparedOptions"); + } + + private Handler( + OptionsMaterializer optionsMaterializer, + LambdaWorkerRuntime runtime, + Sleeper sleeper, + MonotonicClock clock) { + this(optionsMaterializer, runtime, sleeper, clock, false); + } + + private Handler( + OptionsMaterializer optionsMaterializer, + LambdaWorkerRuntime runtime, + Sleeper sleeper, + MonotonicClock clock, + boolean runShutdownHooksBeforeRuntimeCreation) { + this.optionsMaterializer = Objects.requireNonNull(optionsMaterializer, "optionsMaterializer"); + this.runtime = runtime; + this.sleeper = sleeper; + this.clock = clock; + this.runShutdownHooksBeforeRuntimeCreation = runShutdownHooksBeforeRuntimeCreation; + } + + @Override + public Void handleRequest(Object input, Context context) { + Objects.requireNonNull(context, "context"); + + LambdaWorkerOptions.Materialized options = null; + LambdaWorkerRuntime.Invocation invocation = null; + try { + options = optionsMaterializer.materialize(context); + validateRemainingTime(context, options.shutdownDeadlineBuffer); + + invocation = + runtime.create( + options.serviceStubsOptions, + options.clientOptions, + options.workerFactoryOptions, + options.taskQueue, + options.workerOptions); + + for (LambdaWorkerOptions.Registration registration : options.registrations) { + registration.apply(invocation.getWorkerRegistrar()); + } + + invocation.start(); + log.info( + "Temporal Lambda worker started awsRequestId={} invokedFunctionArn={} taskQueue={} identity={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue, + options.workerOptions.getIdentity()); + + sleepUntilShutdownWindow(context, options); + return null; + } catch (InvocationConfigurationException e) { + runShutdownHooks( + context, + e.taskQueue, + e.shutdownHooks, + cleanupDeadlineNanos(context, e.gracefulShutdownTimeout, e.shutdownDeadlineBuffer)); + throw e.failure; + } catch (RuntimeException e) { + if (invocation == null && options != null && runShutdownHooksBeforeRuntimeCreation) { + runShutdownHooks( + context, + options.taskQueue, + options.shutdownHooks, + cleanupDeadlineNanos(context, options)); + } + throw e; + } finally { + if (invocation != null) { + shutdownInvocation(context, invocation, options); + } + } + } + + private void sleepUntilShutdownWindow( + Context context, LambdaWorkerOptions.Materialized options) { + Duration runDuration = durationUntilShutdownWindow(context, options); + if (runDuration.isZero() || runDuration.isNegative()) { + return; + } + + try { + sleeper.sleep(runDuration); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted while running Temporal Lambda worker", e); + } + } + + private void shutdownInvocation( + Context context, + LambdaWorkerRuntime.Invocation invocation, + LambdaWorkerOptions.Materialized options) { + Long cleanupDeadlineNanos = null; + if (invocation != null) { + try { + invocation.shutdown(); + invocation.awaitTermination(options.gracefulShutdownTimeout); + } catch (RuntimeException e) { + log.error( + "Temporal Lambda worker shutdown failed awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue, + e); + } + + cleanupDeadlineNanos = cleanupDeadlineNanos(context, options); + boolean terminated = isTerminated(context, invocation, options); + if (!terminated) { + terminated = forceShutdownInvocation(context, invocation, options, cleanupDeadlineNanos); + } + if (terminated) { + log.info( + "Temporal Lambda worker stopped awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue); + } + } + + if (cleanupDeadlineNanos == null) { + cleanupDeadlineNanos = cleanupDeadlineNanos(context, options); + } + runShutdownHooks(context, options, cleanupDeadlineNanos.longValue()); + + try { + invocation.closeStubs(remainingCleanupTime(cleanupDeadlineNanos.longValue())); + } catch (RuntimeException e) { + log.error( + "Temporal Lambda worker service stubs close failed awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue, + e); + } + } + + private boolean forceShutdownInvocation( + Context context, + LambdaWorkerRuntime.Invocation invocation, + LambdaWorkerOptions.Materialized options, + long cleanupDeadlineNanos) { + log.warn( + "Temporal Lambda worker did not stop before graceful shutdown timeout; forcing stop awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue); + try { + invocation.shutdownNow(); + } catch (RuntimeException e) { + log.error( + "Temporal Lambda worker forced shutdown failed awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue, + e); + } + + try { + invocation.awaitTermination(remainingCleanupTime(cleanupDeadlineNanos)); + } catch (RuntimeException e) { + log.error( + "Temporal Lambda worker forced shutdown wait failed awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue, + e); + } + + boolean terminated = isTerminated(context, invocation, options); + if (!terminated) { + log.warn( + "Temporal Lambda worker did not terminate after forced shutdown awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue); + } + return terminated; + } + + private boolean isTerminated( + Context context, + LambdaWorkerRuntime.Invocation invocation, + LambdaWorkerOptions.Materialized options) { + try { + return invocation.isTerminated(); + } catch (RuntimeException e) { + log.error( + "Temporal Lambda worker termination check failed awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + options.taskQueue, + e); + return false; + } + } + + private void runShutdownHooks( + Context context, LambdaWorkerOptions.Materialized options, long cleanupDeadlineNanos) { + runShutdownHooks(context, options.taskQueue, options.shutdownHooks, cleanupDeadlineNanos); + } + + private void runShutdownHooks( + Context context, + String taskQueue, + List shutdownHooks, + long cleanupDeadlineNanos) { + for (Runnable hook : shutdownHooks) { + try { + if (hook instanceof TimedShutdownHook) { + ((TimedShutdownHook) hook).run(remainingCleanupTime(cleanupDeadlineNanos)); + } else { + hook.run(); + } + } catch (RuntimeException e) { + log.error( + "Temporal Lambda worker shutdown hook failed awsRequestId={} invokedFunctionArn={} taskQueue={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + taskQueue, + e); + } + } + } + + private Duration durationUntilShutdownWindow( + Context context, LambdaWorkerOptions.Materialized options) { + return Duration.ofMillis(context.getRemainingTimeInMillis()) + .minus(options.shutdownDeadlineBuffer); + } + + private void validateRemainingTime(Context context, Duration shutdownDeadlineBuffer) { + Duration available = + Duration.ofMillis(context.getRemainingTimeInMillis()).minus(shutdownDeadlineBuffer); + if (available.compareTo(MINIMUM_AVAILABLE_RUNTIME) <= 0) { + throw new IllegalStateException( + "Insufficient Lambda invocation time remaining after shutdown buffer: " + + available.toMillis() + + "ms"); + } + if (available.compareTo(LOW_AVAILABLE_RUNTIME_WARNING) < 0) { + log.warn( + "Temporal Lambda worker has low remaining time awsRequestId={} invokedFunctionArn={} availableRuntimeMs={} shutdownDeadlineBufferMs={}", + context.getAwsRequestId(), + context.getInvokedFunctionArn(), + available.toMillis(), + shutdownDeadlineBuffer.toMillis()); + } + } + + private long cleanupDeadlineNanos(Context context, LambdaWorkerOptions.Materialized options) { + return cleanupDeadlineNanos( + context, options.gracefulShutdownTimeout, options.shutdownDeadlineBuffer); + } + + private long cleanupDeadlineNanos( + Context context, Duration gracefulShutdownTimeout, Duration shutdownDeadlineBuffer) { + return clock.nanoTime() + + cleanupWindow(context, gracefulShutdownTimeout, shutdownDeadlineBuffer).toNanos(); + } + + private Duration cleanupWindow(Context context, LambdaWorkerOptions.Materialized options) { + return cleanupWindow( + context, options.gracefulShutdownTimeout, options.shutdownDeadlineBuffer); + } + + private Duration cleanupWindow( + Context context, Duration gracefulShutdownTimeout, Duration shutdownDeadlineBuffer) { + Duration configuredWindow = + nonNegative(shutdownDeadlineBuffer.minus(gracefulShutdownTimeout)); + Duration remaining = remainingInvocationTime(context); + return configuredWindow.compareTo(remaining) <= 0 ? configuredWindow : remaining; + } + + private Duration remainingCleanupTime(long cleanupDeadlineNanos) { + return nonNegative(Duration.ofNanos(cleanupDeadlineNanos - clock.nanoTime())); + } + + private Duration remainingInvocationTime(Context context) { + return nonNegative(Duration.ofMillis(context.getRemainingTimeInMillis())); + } + + private static Duration nonNegative(Duration duration) { + return duration.isNegative() ? Duration.ZERO : duration; + } + } + + private static final class InvocationConfigurationException extends RuntimeException { + private final RuntimeException failure; + private final String taskQueue; + private final Duration gracefulShutdownTimeout; + private final Duration shutdownDeadlineBuffer; + private final List shutdownHooks; + + private InvocationConfigurationException( + RuntimeException failure, LambdaWorkerOptions.Builder builder) { + super(failure); + this.failure = failure; + this.taskQueue = builder.getTaskQueue(); + this.gracefulShutdownTimeout = builder.getGracefulShutdownTimeout(); + this.shutdownDeadlineBuffer = builder.getShutdownDeadlineBuffer(); + this.shutdownHooks = builder.getShutdownHooks(); + } + } +} diff --git a/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorkerOptions.java b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorkerOptions.java new file mode 100644 index 0000000000..7a74309368 --- /dev/null +++ b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorkerOptions.java @@ -0,0 +1,646 @@ +package io.temporal.aws.lambda; + +import io.temporal.activity.DynamicActivity; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.common.VersioningBehavior; +import io.temporal.common.WorkerDeploymentVersion; +import io.temporal.common.converter.EncodedValues; +import io.temporal.envconfig.ClientConfigProfile; +import io.temporal.envconfig.LoadClientConfigProfileOptions; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.WorkerDeploymentOptions; +import io.temporal.worker.WorkerFactoryOptions; +import io.temporal.worker.WorkerOptions; +import io.temporal.worker.WorkflowImplementationOptions; +import io.temporal.workflow.DynamicWorkflow; +import io.temporal.workflow.Functions; +import java.io.File; +import java.io.IOException; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import javax.annotation.Nonnull; + +/** + * Configuration for Temporal workers running inside AWS Lambda invocations. + * + *

Instances are immutable snapshots. Lambda handlers copy snapshots for each invocation so + * invocation identity and optional per-invocation configuration can be applied without mutating the + * base options. + */ +public final class LambdaWorkerOptions { + public static final String TEMPORAL_TASK_QUEUE = "TEMPORAL_TASK_QUEUE"; + public static final String TEMPORAL_CONFIG_FILE = "TEMPORAL_CONFIG_FILE"; + public static final String LAMBDA_TASK_ROOT = "LAMBDA_TASK_ROOT"; + + static final Duration DEFAULT_GRACEFUL_SHUTDOWN_TIMEOUT = Duration.ofSeconds(5); + static final Duration DEFAULT_SHUTDOWN_HOOKS_AND_STUBS_TIMEOUT = Duration.ofSeconds(2); + static final Duration DEFAULT_SHUTDOWN_DEADLINE_BUFFER = + DEFAULT_GRACEFUL_SHUTDOWN_TIMEOUT.plus(DEFAULT_SHUTDOWN_HOOKS_AND_STUBS_TIMEOUT); + + private static final int DEFAULT_MAX_CONCURRENT_ACTIVITY_EXECUTION_SIZE = 2; + private static final int DEFAULT_MAX_CONCURRENT_WORKFLOW_TASK_EXECUTION_SIZE = 10; + private static final int DEFAULT_MAX_CONCURRENT_LOCAL_ACTIVITY_EXECUTION_SIZE = 2; + private static final int DEFAULT_MAX_CONCURRENT_NEXUS_EXECUTION_SIZE = 5; + private static final int DEFAULT_MAX_CONCURRENT_WORKFLOW_TASK_POLLERS = 2; + private static final int DEFAULT_MAX_CONCURRENT_ACTIVITY_TASK_POLLERS = 1; + private static final int DEFAULT_MAX_CONCURRENT_NEXUS_TASK_POLLERS = 1; + private static final int DEFAULT_WORKFLOW_CACHE_SIZE = 30; + private static final int DEFAULT_MAX_WORKFLOW_THREAD_COUNT = 30; + + private final WorkflowServiceStubsOptions workflowServiceStubsOptions; + private final WorkflowClientOptions workflowClientOptions; + private final WorkerFactoryOptions workerFactoryOptions; + private final WorkerOptions workerOptions; + private final List registrations; + private final List shutdownHooks; + private final String taskQueue; + private final Duration gracefulShutdownTimeout; + private final Duration shutdownDeadlineBuffer; + private final boolean shutdownDeadlineBufferExplicit; + + private LambdaWorkerOptions( + WorkflowServiceStubsOptions workflowServiceStubsOptions, + WorkflowClientOptions workflowClientOptions, + WorkerFactoryOptions workerFactoryOptions, + WorkerOptions workerOptions, + List registrations, + List shutdownHooks, + String taskQueue, + Duration gracefulShutdownTimeout, + Duration shutdownDeadlineBuffer, + boolean shutdownDeadlineBufferExplicit) { + this.workflowServiceStubsOptions = workflowServiceStubsOptions; + this.workflowClientOptions = workflowClientOptions; + this.workerFactoryOptions = workerFactoryOptions; + this.workerOptions = workerOptions; + this.registrations = Collections.unmodifiableList(new ArrayList<>(registrations)); + this.shutdownHooks = Collections.unmodifiableList(new ArrayList<>(shutdownHooks)); + this.taskQueue = taskQueue; + this.gracefulShutdownTimeout = gracefulShutdownTimeout; + this.shutdownDeadlineBuffer = shutdownDeadlineBuffer; + this.shutdownDeadlineBufferExplicit = shutdownDeadlineBufferExplicit; + } + + public static Builder newBuilder() { + return new Builder(); + } + + public static Builder newBuilder(@Nonnull LambdaWorkerOptions options) { + return new Builder(options); + } + + /** Loads Temporal client configuration from the process environment into a new builder. */ + public static Builder newBuilderFromEnvironment() throws IOException { + return newBuilderFromEnvironment(System.getenv()); + } + + /** + * Loads Temporal client configuration from the provided environment values into a new builder. + */ + public static Builder newBuilderFromEnvironment(Map env) throws IOException { + return newBuilderFromEnvironment(env, new File(".")); + } + + static Builder newBuilderFromEnvironment(Map env, File cwd) throws IOException { + ClientConfigProfile profile = + ClientConfigProfile.load( + LoadClientConfigProfileOptions.newBuilder() + .setConfigFilePath(resolveConfigFilePath(env, cwd)) + .setEnvOverrides(env) + .build()); + return new Builder(profile, env); + } + + static String resolveConfigFilePath(Map env) { + return resolveConfigFilePath(env, new File(".")); + } + + static String resolveConfigFilePath(Map env, File cwd) { + String configured = nonEmptyEnv(env, TEMPORAL_CONFIG_FILE); + if (configured != null) { + return configured; + } + + String taskRoot = nonEmptyEnv(env, LAMBDA_TASK_ROOT); + if (taskRoot != null) { + File lambdaConfig = new File(taskRoot, "temporal.toml"); + if (isReadableFile(lambdaConfig)) { + return lambdaConfig.getAbsolutePath(); + } + } + + File cwdConfig = new File(Objects.requireNonNull(cwd, "cwd"), "temporal.toml"); + return isReadableFile(cwdConfig) ? cwdConfig.getAbsolutePath() : null; + } + + public Builder toBuilder() { + return new Builder(this); + } + + public WorkflowServiceStubsOptions getWorkflowServiceStubsOptions() { + return workflowServiceStubsOptions; + } + + public WorkflowClientOptions getWorkflowClientOptions() { + return workflowClientOptions; + } + + public WorkerFactoryOptions getWorkerFactoryOptions() { + return workerFactoryOptions; + } + + public WorkerOptions getWorkerOptions() { + return workerOptions; + } + + public String getTaskQueue() { + return taskQueue; + } + + public Duration getGracefulShutdownTimeout() { + return gracefulShutdownTimeout; + } + + public Duration getShutdownDeadlineBuffer() { + return shutdownDeadlineBuffer; + } + + Materialized materialize( + @Nonnull WorkerDeploymentVersion version, @Nonnull String invocationIdentity) { + return prepare(version).materialize(invocationIdentity); + } + + Materialized prepare(@Nonnull WorkerDeploymentVersion version) { + validateVersion(version); + if (isNullOrEmpty(taskQueue)) { + throw new IllegalStateException( + "Task queue must be set with LambdaWorkerOptions.Builder#setTaskQueue or TEMPORAL_TASK_QUEUE"); + } + validateShutdownConfiguration(); + + WorkflowClientOptions rawClientOptions = workflowClientOptions; + WorkerOptions rawWorkerOptions = workerOptions; + WorkerFactoryOptions rawFactoryOptions = workerFactoryOptions; + + WorkflowClientOptions.Builder clientOptionsBuilder = + WorkflowClientOptions.newBuilder(rawClientOptions); + WorkerOptions.Builder workerOptionsBuilder = WorkerOptions.newBuilder(rawWorkerOptions); + WorkerFactoryOptions.Builder factoryOptionsBuilder = + WorkerFactoryOptions.newBuilder(rawFactoryOptions); + + if (rawClientOptions.getIdentity() == null && rawWorkerOptions.getIdentity() != null) { + clientOptionsBuilder.setIdentity(rawWorkerOptions.getIdentity()); + } else if (rawWorkerOptions.getIdentity() == null && rawClientOptions.getIdentity() != null) { + workerOptionsBuilder.setIdentity(rawClientOptions.getIdentity()); + } + + applyLambdaWorkerDefaults(rawWorkerOptions, workerOptionsBuilder, version); + applyLambdaFactoryDefaults(rawFactoryOptions, factoryOptionsBuilder); + + return new Materialized( + WorkflowServiceStubsOptions.newBuilder(workflowServiceStubsOptions) + .validateAndBuildWithDefaults(), + clientOptionsBuilder.build(), + factoryOptionsBuilder.validateAndBuildWithDefaults(), + taskQueue, + workerOptionsBuilder.validateAndBuildWithDefaults(), + gracefulShutdownTimeout, + shutdownDeadlineBuffer, + new ArrayList<>(registrations), + new ArrayList<>(shutdownHooks)); + } + + private void validateShutdownConfiguration() { + if (shutdownDeadlineBuffer.compareTo(gracefulShutdownTimeout) < 0) { + throw new IllegalStateException( + "shutdownDeadlineBuffer must be greater than or equal to gracefulShutdownTimeout"); + } + } + + private static void applyLambdaWorkerDefaults( + WorkerOptions rawOptions, WorkerOptions.Builder builder, WorkerDeploymentVersion version) { + if (rawOptions.getWorkerTuner() == null) { + if (rawOptions.getMaxConcurrentActivityExecutionSize() == 0) { + builder.setMaxConcurrentActivityExecutionSize( + DEFAULT_MAX_CONCURRENT_ACTIVITY_EXECUTION_SIZE); + } + if (rawOptions.getMaxConcurrentWorkflowTaskExecutionSize() == 0) { + builder.setMaxConcurrentWorkflowTaskExecutionSize( + DEFAULT_MAX_CONCURRENT_WORKFLOW_TASK_EXECUTION_SIZE); + } + if (rawOptions.getMaxConcurrentLocalActivityExecutionSize() == 0) { + builder.setMaxConcurrentLocalActivityExecutionSize( + DEFAULT_MAX_CONCURRENT_LOCAL_ACTIVITY_EXECUTION_SIZE); + } + if (rawOptions.getMaxConcurrentNexusExecutionSize() == 0) { + builder.setMaxConcurrentNexusExecutionSize(DEFAULT_MAX_CONCURRENT_NEXUS_EXECUTION_SIZE); + } + } + + if (rawOptions.getWorkflowTaskPollersBehavior() == null + && rawOptions.getMaxConcurrentWorkflowTaskPollers() == 0) { + builder.setMaxConcurrentWorkflowTaskPollers(DEFAULT_MAX_CONCURRENT_WORKFLOW_TASK_POLLERS); + } + if (rawOptions.getActivityTaskPollersBehavior() == null + && rawOptions.getMaxConcurrentActivityTaskPollers() == 0) { + builder.setMaxConcurrentActivityTaskPollers(DEFAULT_MAX_CONCURRENT_ACTIVITY_TASK_POLLERS); + } + if (rawOptions.getNexusTaskPollersBehavior() == null + && rawOptions.getMaxConcurrentNexusTaskPollers() == 0) { + builder.setMaxConcurrentNexusTaskPollers(DEFAULT_MAX_CONCURRENT_NEXUS_TASK_POLLERS); + } + + builder.setDisableEagerExecution(true); + builder.setDeploymentOptions( + forcedDeploymentOptions(rawOptions.getDeploymentOptions(), version)); + } + + private static void applyLambdaFactoryDefaults( + WorkerFactoryOptions rawOptions, WorkerFactoryOptions.Builder builder) { + if (rawOptions.getWorkflowCacheSize() == 0) { + builder.setWorkflowCacheSize(DEFAULT_WORKFLOW_CACHE_SIZE); + } + if (rawOptions.getMaxWorkflowThreadCount() == 0) { + builder.setMaxWorkflowThreadCount(DEFAULT_MAX_WORKFLOW_THREAD_COUNT); + } + } + + private static WorkerDeploymentOptions forcedDeploymentOptions( + WorkerDeploymentOptions existing, WorkerDeploymentVersion version) { + VersioningBehavior behavior = VersioningBehavior.PINNED; + if (existing != null + && existing.getDefaultVersioningBehavior() != VersioningBehavior.UNSPECIFIED) { + behavior = existing.getDefaultVersioningBehavior(); + } + + return WorkerDeploymentOptions.newBuilder() + .setUseVersioning(true) + .setVersion(version) + .setDefaultVersioningBehavior(behavior) + .build(); + } + + static void validateVersion(@Nonnull WorkerDeploymentVersion version) { + Objects.requireNonNull(version, "version"); + if (isNullOrEmpty(version.getDeploymentName())) { + throw new IllegalArgumentException("Worker deployment name must be non-empty"); + } + if (isNullOrEmpty(version.getBuildId())) { + throw new IllegalArgumentException("Worker deployment build ID must be non-empty"); + } + } + + private static Duration requireNonNegative(Duration value, String name) { + Objects.requireNonNull(value, name); + if (value.isNegative()) { + throw new IllegalArgumentException(name + " must not be negative"); + } + return value; + } + + private static boolean isReadableFile(File file) { + return file.isFile() && file.canRead(); + } + + private static String nonEmptyEnv(Map env, String name) { + if (env == null) { + return null; + } + String value = env.get(name); + return isNullOrEmpty(value) ? null : value; + } + + private static boolean isNullOrEmpty(String value) { + return value == null || value.trim().isEmpty(); + } + + private static Class[] copyClasses(Class... classes) { + Objects.requireNonNull(classes, "classes"); + for (Class workflowImplementationClass : classes) { + Objects.requireNonNull(workflowImplementationClass, "workflowImplementationClass"); + } + return Arrays.copyOf(classes, classes.length); + } + + private static Object[] copyObjects(Object[] objects, String name) { + Objects.requireNonNull(objects, name); + return Arrays.copyOf(objects, objects.length); + } + + public static final class Builder { + private final WorkflowServiceStubsOptions.Builder workflowServiceStubsOptionsBuilder; + private final WorkflowClientOptions.Builder workflowClientOptionsBuilder; + private final WorkerFactoryOptions.Builder workerFactoryOptionsBuilder; + private final WorkerOptions.Builder workerOptionsBuilder; + private final List registrations = new ArrayList<>(); + private final List shutdownHooks = new ArrayList<>(); + + private String taskQueue; + private Duration gracefulShutdownTimeout = DEFAULT_GRACEFUL_SHUTDOWN_TIMEOUT; + private Duration shutdownDeadlineBuffer = DEFAULT_SHUTDOWN_DEADLINE_BUFFER; + private boolean shutdownDeadlineBufferExplicit; + + private Builder() { + this.workflowServiceStubsOptionsBuilder = WorkflowServiceStubsOptions.newBuilder(); + this.workflowClientOptionsBuilder = WorkflowClientOptions.newBuilder(); + this.workerFactoryOptionsBuilder = WorkerFactoryOptions.newBuilder(); + this.workerOptionsBuilder = WorkerOptions.newBuilder(); + } + + private Builder(ClientConfigProfile profile, Map env) { + this.workflowServiceStubsOptionsBuilder = + WorkflowServiceStubsOptions.newBuilder(profile.toWorkflowServiceStubsOptions()); + this.workflowClientOptionsBuilder = + WorkflowClientOptions.newBuilder(profile.toWorkflowClientOptions()); + this.workerFactoryOptionsBuilder = WorkerFactoryOptions.newBuilder(); + this.workerOptionsBuilder = WorkerOptions.newBuilder(); + this.taskQueue = nonEmptyEnv(env, TEMPORAL_TASK_QUEUE); + } + + private Builder(LambdaWorkerOptions options) { + Objects.requireNonNull(options, "options"); + this.workflowServiceStubsOptionsBuilder = + WorkflowServiceStubsOptions.newBuilder(options.workflowServiceStubsOptions); + this.workflowClientOptionsBuilder = + WorkflowClientOptions.newBuilder(options.workflowClientOptions); + this.workerFactoryOptionsBuilder = + WorkerFactoryOptions.newBuilder(options.workerFactoryOptions); + this.workerOptionsBuilder = WorkerOptions.newBuilder(options.workerOptions); + this.registrations.addAll(options.registrations); + this.shutdownHooks.addAll(options.shutdownHooks); + this.taskQueue = options.taskQueue; + this.gracefulShutdownTimeout = options.gracefulShutdownTimeout; + this.shutdownDeadlineBuffer = options.shutdownDeadlineBuffer; + this.shutdownDeadlineBufferExplicit = options.shutdownDeadlineBufferExplicit; + } + + /** Returns the builder used to prepare {@link WorkflowServiceStubsOptions}. */ + public WorkflowServiceStubsOptions.Builder getWorkflowServiceStubsOptionsBuilder() { + return workflowServiceStubsOptionsBuilder; + } + + /** Returns the builder used to prepare {@link WorkflowClientOptions}. */ + public WorkflowClientOptions.Builder getWorkflowClientOptionsBuilder() { + return workflowClientOptionsBuilder; + } + + /** Returns the builder used to prepare {@link WorkerFactoryOptions}. */ + public WorkerFactoryOptions.Builder getWorkerFactoryOptionsBuilder() { + return workerFactoryOptionsBuilder; + } + + /** Returns the builder used to prepare {@link WorkerOptions}. */ + public WorkerOptions.Builder getWorkerOptionsBuilder() { + return workerOptionsBuilder; + } + + public String getTaskQueue() { + return taskQueue; + } + + /** Sets the Temporal task queue polled by the per-invocation worker. */ + public Builder setTaskQueue(String taskQueue) { + this.taskQueue = taskQueue; + return this; + } + + public Duration getGracefulShutdownTimeout() { + return gracefulShutdownTimeout; + } + + /** + * Sets how long worker shutdown waits for pollers and executions to stop. + * + *

If {@link #setShutdownDeadlineBuffer(Duration)} has not been called, the shutdown deadline + * buffer is recomputed as this timeout plus a 2 second hook and service stubs margin. + */ + public Builder setGracefulShutdownTimeout(@Nonnull Duration gracefulShutdownTimeout) { + this.gracefulShutdownTimeout = + requireNonNegative(gracefulShutdownTimeout, "gracefulShutdownTimeout"); + if (!shutdownDeadlineBufferExplicit) { + shutdownDeadlineBuffer = + this.gracefulShutdownTimeout.plus(DEFAULT_SHUTDOWN_HOOKS_AND_STUBS_TIMEOUT); + } + return this; + } + + public Duration getShutdownDeadlineBuffer() { + return shutdownDeadlineBuffer; + } + + List getShutdownHooks() { + return new ArrayList<>(shutdownHooks); + } + + /** + * Sets the full shutdown window reserved at the end of the Lambda invocation. + * + *

The worker stops when remaining invocation time reaches this buffer. The default is 7 + * seconds, made up of the 5 second graceful shutdown timeout and a 2 second hook and service + * stubs margin. This buffer must be greater than or equal to {@link + * #getGracefulShutdownTimeout()}. + */ + public Builder setShutdownDeadlineBuffer(@Nonnull Duration shutdownDeadlineBuffer) { + this.shutdownDeadlineBuffer = + requireNonNegative(shutdownDeadlineBuffer, "shutdownDeadlineBuffer"); + shutdownDeadlineBufferExplicit = true; + return this; + } + + public Builder registerWorkflowImplementationTypes( + @Nonnull Class... workflowImplementationClasses) { + final Class[] classes = copyClasses(workflowImplementationClasses); + registrations.add(registrar -> registrar.registerWorkflowImplementationTypes(classes)); + return this; + } + + public Builder registerWorkflowImplementationTypes( + @Nonnull WorkflowImplementationOptions options, + @Nonnull Class... workflowImplementationClasses) { + Objects.requireNonNull(options, "options"); + final Class[] classes = copyClasses(workflowImplementationClasses); + registrations.add( + registrar -> registrar.registerWorkflowImplementationTypes(options, classes)); + return this; + } + + /** + * Registers a dynamic workflow implementation type. + * + *

Only one dynamic workflow implementation type can be registered per worker. + */ + public Builder registerDynamicWorkflowImplementationType( + @Nonnull Class workflowImplementationClass) { + final Class implementationClass = + Objects.requireNonNull(workflowImplementationClass, "workflowImplementationClass"); + registrations.add( + registrar -> registrar.registerWorkflowImplementationTypes(implementationClass)); + return this; + } + + /** + * Registers a dynamic workflow implementation type with custom workflow implementation options. + * + *

Only one dynamic workflow implementation type can be registered per worker. + */ + public Builder registerDynamicWorkflowImplementationType( + @Nonnull WorkflowImplementationOptions options, + @Nonnull Class workflowImplementationClass) { + Objects.requireNonNull(options, "options"); + final Class implementationClass = + Objects.requireNonNull(workflowImplementationClass, "workflowImplementationClass"); + registrations.add( + registrar -> registrar.registerWorkflowImplementationTypes(options, implementationClass)); + return this; + } + + public Builder registerWorkflowImplementationFactory( + @Nonnull Class workflowInterface, @Nonnull Functions.Func factory) { + Objects.requireNonNull(workflowInterface, "workflowInterface"); + Objects.requireNonNull(factory, "factory"); + registrations.add( + registrar -> registrar.registerWorkflowImplementationFactory(workflowInterface, factory)); + return this; + } + + public Builder registerWorkflowImplementationFactory( + @Nonnull Class workflowInterface, + @Nonnull Functions.Func factory, + @Nonnull WorkflowImplementationOptions options) { + Objects.requireNonNull(workflowInterface, "workflowInterface"); + Objects.requireNonNull(factory, "factory"); + Objects.requireNonNull(options, "options"); + registrations.add( + registrar -> + registrar.registerWorkflowImplementationFactory(workflowInterface, factory, options)); + return this; + } + + public Builder registerWorkflowImplementationFactory( + @Nonnull Class workflowInterface, + @Nonnull Functions.Func1 factory, + @Nonnull WorkflowImplementationOptions options) { + Objects.requireNonNull(workflowInterface, "workflowInterface"); + Objects.requireNonNull(factory, "factory"); + Objects.requireNonNull(options, "options"); + registrations.add( + registrar -> + registrar.registerWorkflowImplementationFactory(workflowInterface, factory, options)); + return this; + } + + public Builder registerActivitiesImplementations(@Nonnull Object... activityImplementations) { + final Object[] implementations = + copyObjects(activityImplementations, "activityImplementations"); + registrations.add(registrar -> registrar.registerActivitiesImplementations(implementations)); + return this; + } + + /** + * Registers a dynamic activity implementation. + * + *

Only one dynamic activity implementation can be registered per worker. + */ + public Builder registerDynamicActivityImplementation( + @Nonnull DynamicActivity activityImplementation) { + final DynamicActivity implementation = + Objects.requireNonNull(activityImplementation, "activityImplementation"); + registrations.add(registrar -> registrar.registerActivitiesImplementations(implementation)); + return this; + } + + public Builder registerNexusServiceImplementation( + @Nonnull Object... nexusServiceImplementations) { + final Object[] implementations = + copyObjects(nexusServiceImplementations, "nexusServiceImplementations"); + registrations.add(registrar -> registrar.registerNexusServiceImplementation(implementations)); + return this; + } + + /** + * Adds a shutdown hook that runs after the worker has stopped and before service stubs close. + */ + public Builder addShutdownHook(@Nonnull Runnable hook) { + shutdownHooks.add(Objects.requireNonNull(hook, "hook")); + return this; + } + + public LambdaWorkerOptions build() { + return new LambdaWorkerOptions( + workflowServiceStubsOptionsBuilder.build(), + workflowClientOptionsBuilder.build(), + workerFactoryOptionsBuilder.build(), + workerOptionsBuilder.build(), + registrations, + shutdownHooks, + taskQueue, + gracefulShutdownTimeout, + shutdownDeadlineBuffer, + shutdownDeadlineBufferExplicit); + } + } + + interface Registration { + void apply(WorkerRegistrar registrar); + } + + static final class Materialized { + final WorkflowServiceStubsOptions serviceStubsOptions; + final WorkflowClientOptions clientOptions; + final WorkerFactoryOptions workerFactoryOptions; + final String taskQueue; + final WorkerOptions workerOptions; + final Duration gracefulShutdownTimeout; + final Duration shutdownDeadlineBuffer; + final List registrations; + final List shutdownHooks; + + private Materialized( + WorkflowServiceStubsOptions serviceStubsOptions, + WorkflowClientOptions clientOptions, + WorkerFactoryOptions workerFactoryOptions, + String taskQueue, + WorkerOptions workerOptions, + Duration gracefulShutdownTimeout, + Duration shutdownDeadlineBuffer, + List registrations, + List shutdownHooks) { + this.serviceStubsOptions = serviceStubsOptions; + this.clientOptions = clientOptions; + this.workerFactoryOptions = workerFactoryOptions; + this.taskQueue = taskQueue; + this.workerOptions = workerOptions; + this.gracefulShutdownTimeout = gracefulShutdownTimeout; + this.shutdownDeadlineBuffer = shutdownDeadlineBuffer; + this.registrations = Collections.unmodifiableList(registrations); + this.shutdownHooks = Collections.unmodifiableList(shutdownHooks); + } + + Materialized materialize(@Nonnull String invocationIdentity) { + WorkflowClientOptions.Builder clientOptionsBuilder = + WorkflowClientOptions.newBuilder(clientOptions); + WorkerOptions.Builder workerOptionsBuilder = WorkerOptions.newBuilder(workerOptions); + + if (clientOptions.getIdentity() == null && workerOptions.getIdentity() == null) { + clientOptionsBuilder.setIdentity(invocationIdentity); + workerOptionsBuilder.setIdentity(invocationIdentity); + } + + return new Materialized( + serviceStubsOptions, + clientOptionsBuilder.validateAndBuildWithDefaults(), + workerFactoryOptions, + taskQueue, + workerOptionsBuilder.validateAndBuildWithDefaults(), + gracefulShutdownTimeout, + shutdownDeadlineBuffer, + new ArrayList<>(registrations), + new ArrayList<>(shutdownHooks)); + } + } +} diff --git a/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorkerRuntime.java b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorkerRuntime.java new file mode 100644 index 0000000000..e296d4bc00 --- /dev/null +++ b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/LambdaWorkerRuntime.java @@ -0,0 +1,32 @@ +package io.temporal.aws.lambda; + +import io.temporal.client.WorkflowClientOptions; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.WorkerFactoryOptions; +import io.temporal.worker.WorkerOptions; +import java.time.Duration; + +interface LambdaWorkerRuntime { + Invocation create( + WorkflowServiceStubsOptions serviceStubsOptions, + WorkflowClientOptions clientOptions, + WorkerFactoryOptions workerFactoryOptions, + String taskQueue, + WorkerOptions workerOptions); + + interface Invocation { + WorkerRegistrar getWorkerRegistrar(); + + void start(); + + void shutdown(); + + void shutdownNow(); + + void awaitTermination(Duration timeout); + + boolean isTerminated(); + + void closeStubs(Duration timeout); + } +} diff --git a/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/OtelLambdaWorkerConfigurationHelper.java b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/OtelLambdaWorkerConfigurationHelper.java new file mode 100644 index 0000000000..235352b7ac --- /dev/null +++ b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/OtelLambdaWorkerConfigurationHelper.java @@ -0,0 +1,280 @@ +package io.temporal.aws.lambda; + +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator; +import io.opentelemetry.sdk.trace.IdGenerator; +import io.temporal.opentelemetry.OpenTelemetryPlugin; +import io.temporal.opentelemetry.OpenTelemetryWorker; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.serviceclient.WorkflowServiceStubsPlugin; +import java.time.Duration; +import java.util.Arrays; +import java.util.Map; +import java.util.Objects; +import java.util.function.Consumer; +import javax.annotation.Nonnull; + +/** OpenTelemetry configuration helper for {@link LambdaWorker}. */ +public final class OtelLambdaWorkerConfigurationHelper { + public static final String OTEL_EXPORTER_OTLP_ENDPOINT = + OpenTelemetryWorker.OTEL_EXPORTER_OTLP_ENDPOINT; + public static final String OTEL_SERVICE_NAME = OpenTelemetryWorker.OTEL_SERVICE_NAME; + public static final String AWS_LAMBDA_FUNCTION_NAME = "AWS_LAMBDA_FUNCTION_NAME"; + public static final String DEFAULT_OTLP_ENDPOINT = OpenTelemetryWorker.DEFAULT_OTLP_ENDPOINT; + public static final String DEFAULT_SERVICE_NAME = "temporal-lambda-worker"; + + private static final Duration DEFAULT_METRICS_REPORT_INTERVAL = Duration.ofSeconds(1); + + private OtelLambdaWorkerConfigurationHelper() {} + + public static Builder newBuilder() { + return new Builder(System.getenv()); + } + + static Builder newBuilder(Map env) { + return new Builder(env); + } + + public static String getDefaultEndpoint() { + return OpenTelemetryWorker.getDefaultEndpoint(); + } + + public static String getDefaultServiceName() { + return resolveServiceName(System.getenv()); + } + + public static void configure(@Nonnull LambdaWorkerOptions.Builder options) { + configure(options, builder -> {}); + } + + /** + * Configures metrics, tracing interceptors, and per-invocation flushing for a Lambda worker. + * + *

By default this method creates an OpenTelemetry SDK with OTLP trace and metric exporters and + * AWS X-Ray-compatible trace ID generation. If {@link Builder#setOpenTelemetry(OpenTelemetry)} is + * used, the provided instance is used instead and exporters are not created. + */ + public static void configure( + @Nonnull LambdaWorkerOptions.Builder options, @Nonnull Consumer configure) { + Objects.requireNonNull(options, "options"); + Builder builder = newBuilder(); + Objects.requireNonNull(configure, "configure").accept(builder); + builder.apply(options); + } + + /** + * Configures Temporal metrics with the default service name and reporting interval. + * + *

This helper installs the metrics scope and registers a hook that reports buffered Tally + * metrics before provider flushing. It does not configure tracing interceptors or an + * OpenTelemetry provider flush hook. + */ + public static void configureMetrics( + @Nonnull LambdaWorkerOptions.Builder options, @Nonnull OpenTelemetry openTelemetry) { + configureMetrics( + options, openTelemetry, getDefaultServiceName(), DEFAULT_METRICS_REPORT_INTERVAL); + } + + /** + * Configures Temporal metrics with an application-owned OpenTelemetry provider. + * + *

This helper installs the metrics scope and registers a hook that reports buffered Tally + * metrics before provider flushing. It does not configure tracing interceptors or an + * OpenTelemetry provider flush hook. + */ + public static void configureMetrics( + @Nonnull LambdaWorkerOptions.Builder options, + @Nonnull OpenTelemetry openTelemetry, + @Nonnull String serviceName, + @Nonnull Duration reportInterval) { + Objects.requireNonNull(options, "options"); + OpenTelemetryWorker.configureMetrics( + options.getWorkflowServiceStubsOptionsBuilder(), + options::addShutdownHook, + openTelemetry, + serviceName, + reportInterval); + } + + /** + * Configures Temporal tracing interceptors with an application-owned OpenTelemetry provider. + * + *

This helper only installs tracing interceptors. It does not configure metrics or register a + * flush hook. + */ + public static void configureTracing( + @Nonnull LambdaWorkerOptions.Builder options, @Nonnull OpenTelemetry openTelemetry) { + Objects.requireNonNull(options, "options"); + OpenTelemetryWorker.configureTracing( + options.getWorkflowClientOptionsBuilder(), + options.getWorkerFactoryOptionsBuilder(), + openTelemetry); + } + + /** + * Registers a per-invocation OpenTelemetry force-flush hook. + * + *

This helper only registers the flush hook. It does not configure metrics or tracing. + */ + public static void configureFlushHook( + @Nonnull LambdaWorkerOptions.Builder options, + @Nonnull OpenTelemetry openTelemetry, + @Nonnull Duration flushTimeout) { + Objects.requireNonNull(options, "options"); + OpenTelemetryWorker.configureFlushHook(options::addShutdownHook, openTelemetry, flushTimeout); + } + + static String resolveServiceName(Map env) { + String serviceName = nonEmptyEnv(env, OTEL_SERVICE_NAME); + if (serviceName != null) { + return serviceName; + } + serviceName = nonEmptyEnv(env, AWS_LAMBDA_FUNCTION_NAME); + return serviceName == null ? DEFAULT_SERVICE_NAME : serviceName; + } + + public static final class Builder { + private final Map env; + private final OpenTelemetryPlugin.Builder delegate; + private OpenTelemetry openTelemetry; + private String serviceName; + private Duration metricsReportInterval = DEFAULT_METRICS_REPORT_INTERVAL; + private Duration flushTimeout = Duration.ofSeconds(10); + private TelemetryFactory telemetryFactory; + + private Builder(Map env) { + this.env = Objects.requireNonNull(env, "env"); + this.delegate = + OpenTelemetryPlugin.newBuilder(env) + .setIdGenerator(AwsXrayIdGenerator.getInstance()) + .setFlushOnWorkerFactoryShutdown(false); + } + + /** + * Uses an application-owned OpenTelemetry instance instead of creating an SDK and exporters. + */ + public Builder setOpenTelemetry(@Nonnull OpenTelemetry openTelemetry) { + this.openTelemetry = Objects.requireNonNull(openTelemetry, "openTelemetry"); + delegate.setOpenTelemetry(openTelemetry); + return this; + } + + /** Sets the OTLP metric and trace exporter endpoint used by the default SDK setup. */ + public Builder setEndpoint(@Nonnull String endpoint) { + delegate.setEndpoint(endpoint); + return this; + } + + /** Sets the service name used by the default SDK resource and Temporal metrics reporter. */ + public Builder setServiceName(@Nonnull String serviceName) { + this.serviceName = Objects.requireNonNull(serviceName, "serviceName"); + delegate.setServiceName(serviceName); + return this; + } + + /** Sets the interval used by the Tally metrics scope and periodic metric reader. */ + public Builder setMetricsReportInterval(@Nonnull Duration metricsReportInterval) { + delegate.setMetricsReportInterval(metricsReportInterval); + this.metricsReportInterval = metricsReportInterval; + return this; + } + + /** Sets how long the per-invocation OpenTelemetry flush hook waits for provider flushing. */ + public Builder setFlushTimeout(@Nonnull Duration flushTimeout) { + delegate.setFlushTimeout(flushTimeout); + this.flushTimeout = flushTimeout; + return this; + } + + /** Overrides the per-invocation flush hook. */ + public Builder setFlushHook(@Nonnull Runnable flushHook) { + delegate.setFlushHook(flushHook); + return this; + } + + public String getEndpoint() { + return delegate.getEndpoint(); + } + + public String getServiceName() { + return serviceName == null ? resolveServiceName(env) : serviceName; + } + + Builder setTelemetryFactory(TelemetryFactory telemetryFactory) { + this.telemetryFactory = Objects.requireNonNull(telemetryFactory, "telemetryFactory"); + return this; + } + + OpenTelemetry createOpenTelemetry() { + applyLambdaServiceNameDefault(); + if (telemetryFactory != null) { + return openTelemetry == null ? getOrCreateTestOpenTelemetry() : openTelemetry; + } + return delegate.createOpenTelemetry(); + } + + void apply(LambdaWorkerOptions.Builder options) { + Objects.requireNonNull(options, "options"); + applyLambdaDefaults(); + OpenTelemetryPlugin plugin = delegate.build(); + appendServiceStubsPlugin(options.getWorkflowServiceStubsOptionsBuilder(), plugin); + options.addShutdownHook(plugin.newFlushHook()); + } + + private void applyLambdaDefaults() { + applyLambdaServiceNameDefault(); + if (telemetryFactory != null && openTelemetry == null) { + delegate.setOpenTelemetry(getOrCreateTestOpenTelemetry()); + } + } + + private void applyLambdaServiceNameDefault() { + delegate.setServiceName(getServiceName()); + } + + private OpenTelemetry getOrCreateTestOpenTelemetry() { + if (openTelemetry == null) { + openTelemetry = createTestOpenTelemetry(); + } + return openTelemetry; + } + + private OpenTelemetry createTestOpenTelemetry() { + return telemetryFactory.create( + getEndpoint(), + getServiceName(), + metricsReportInterval, + flushTimeout, + AwsXrayIdGenerator.getInstance()); + } + } + + interface TelemetryFactory { + OpenTelemetry create( + String endpoint, + String serviceName, + Duration metricsReportInterval, + Duration flushTimeout, + IdGenerator idGenerator); + } + + private static String nonEmptyEnv(Map env, String name) { + if (env == null) { + return null; + } + String value = env.get(name); + return value == null || value.trim().isEmpty() ? null : value; + } + + private static void appendServiceStubsPlugin( + WorkflowServiceStubsOptions.Builder options, WorkflowServiceStubsPlugin plugin) { + WorkflowServiceStubsPlugin[] existing = options.build().getPlugins(); + int existingLength = existing == null ? 0 : existing.length; + WorkflowServiceStubsPlugin[] plugins = + existingLength == 0 + ? new WorkflowServiceStubsPlugin[1] + : Arrays.copyOf(existing, existingLength + 1); + plugins[existingLength] = Objects.requireNonNull(plugin, "plugin"); + options.setPlugins(plugins); + } +} diff --git a/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/WorkerRegistrar.java b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/WorkerRegistrar.java new file mode 100644 index 0000000000..53a47ce6a4 --- /dev/null +++ b/contrib/temporal-aws-lambda/src/main/java/io/temporal/aws/lambda/WorkerRegistrar.java @@ -0,0 +1,27 @@ +package io.temporal.aws.lambda; + +import io.temporal.common.converter.EncodedValues; +import io.temporal.worker.WorkflowImplementationOptions; +import io.temporal.workflow.Functions; + +interface WorkerRegistrar { + void registerWorkflowImplementationTypes(Class... workflowImplementationClasses); + + void registerWorkflowImplementationTypes( + WorkflowImplementationOptions options, Class... workflowImplementationClasses); + + void registerWorkflowImplementationFactory( + Class workflowInterface, Functions.Func factory); + + void registerWorkflowImplementationFactory( + Class workflowInterface, + Functions.Func1 factory, + WorkflowImplementationOptions options); + + void registerWorkflowImplementationFactory( + Class workflowInterface, Functions.Func factory, WorkflowImplementationOptions options); + + void registerActivitiesImplementations(Object... activityImplementations); + + void registerNexusServiceImplementation(Object... nexusServiceImplementations); +} diff --git a/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/LambdaWorkerLifecycleTest.java b/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/LambdaWorkerLifecycleTest.java new file mode 100644 index 0000000000..b4c7640884 --- /dev/null +++ b/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/LambdaWorkerLifecycleTest.java @@ -0,0 +1,848 @@ +package io.temporal.aws.lambda; + +import static org.junit.Assert.*; + +import com.amazonaws.services.lambda.runtime.Context; +import com.amazonaws.services.lambda.runtime.RequestHandler; +import io.temporal.activity.DynamicActivity; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.common.SimplePlugin; +import io.temporal.common.WorkerDeploymentVersion; +import io.temporal.common.converter.EncodedValues; +import io.temporal.opentelemetry.TimedShutdownHook; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.WorkerFactory; +import io.temporal.worker.WorkerFactoryOptions; +import io.temporal.worker.WorkerOptions; +import io.temporal.worker.WorkflowImplementationOptions; +import io.temporal.workflow.DynamicWorkflow; +import io.temporal.workflow.Functions; +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Consumer; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +public class LambdaWorkerLifecycleTest { + private static final WorkerDeploymentVersion VERSION = + new WorkerDeploymentVersion("deployment", "build"); + + @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder(); + + @Test + public void invocationLifecycleRunsInOrder() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> + options + .setTaskQueue("task-queue") + .registerWorkflowImplementationTypes(TestWorkflowImpl.class) + .registerActivitiesImplementations(new Object()) + .registerNexusServiceImplementation(new Object()) + .addShutdownHook(() -> runtime.events.add("hook-1")) + .addShutdownHook(() -> runtime.events.add("hook-2")), + runtime, + duration -> runtime.events.add("sleep:" + duration.toMillis())); + + handler.handleRequest(null, context(20_000)); + + assertEquals( + events( + "create", + "registerWorkflowTypes:1", + "registerActivities:1", + "registerNexus:1", + "start", + "sleep:13000", + "shutdown", + "await:5000", + "hook-1", + "hook-2", + "close:2000"), + runtime.events); + } + + @Test + public void dynamicRegistrationsReplayBeforeWorkerStart() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> + options + .setTaskQueue("task-queue") + .registerDynamicWorkflowImplementationType(TestDynamicWorkflow.class) + .registerDynamicWorkflowImplementationType( + WorkflowImplementationOptions.getDefaultInstance(), + TestDynamicWorkflowWithOptions.class) + .registerDynamicActivityImplementation(new TestDynamicActivity()), + runtime); + + handler.handleRequest(null, context(20_000)); + + assertEquals( + events( + "create", + "registerWorkflowTypes:1", + "registerWorkflowTypesWithOptions:1", + "registerActivities:1", + "start", + "shutdown", + "await:5000", + "close:2000"), + runtime.events); + } + + @Test + public void configureRunsOnceAndRuntimeIsCreatedOncePerInvocation() { + AtomicInteger configureCount = new AtomicInteger(); + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> { + configureCount.incrementAndGet(); + options.setTaskQueue("task-queue"); + }, + runtime); + + assertEquals(1, configureCount.get()); + + handler.handleRequest(null, context(20_000, "request-1", "function-arn-1")); + assertEquals("request-1@function-arn-1", runtime.clientOptions.getIdentity()); + assertEquals("request-1@function-arn-1", runtime.workerOptions.getIdentity()); + + handler.handleRequest(null, context(20_000, "request-2", "function-arn-2")); + assertEquals("request-2@function-arn-2", runtime.clientOptions.getIdentity()); + assertEquals("request-2@function-arn-2", runtime.workerOptions.getIdentity()); + + assertEquals(1, configureCount.get()); + assertEquals(2, runtime.createCount); + } + + @Test + public void perInvocationConfigureRunsOncePerInvocation() { + AtomicInteger coldStartConfigureCount = new AtomicInteger(); + AtomicInteger invocationConfigureCount = new AtomicInteger(); + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> { + coldStartConfigureCount.incrementAndGet(); + options.setTaskQueue("task-queue"); + }, + (options, context) -> { + invocationConfigureCount.incrementAndGet(); + options.getWorkflowClientOptionsBuilder().setNamespace(context.getAwsRequestId()); + }, + runtime); + + assertEquals(1, coldStartConfigureCount.get()); + + handler.handleRequest(null, context(20_000, "request-1", "function-arn-1")); + assertEquals("request-1", runtime.clientOptions.getNamespace()); + + handler.handleRequest(null, context(20_000, "request-2", "function-arn-2")); + assertEquals("request-2", runtime.clientOptions.getNamespace()); + + assertEquals(1, coldStartConfigureCount.get()); + assertEquals(2, invocationConfigureCount.get()); + assertEquals(2, runtime.createCount); + } + + @Test + public void perInvocationConfigureCanSupplyTaskQueue() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> {}, + (options, context) -> options.setTaskQueue(context.getAwsRequestId()), + runtime); + + handler.handleRequest(null, context(20_000, "request-task-queue", "function-arn")); + + assertEquals("request-task-queue", runtime.taskQueue); + assertEquals(1, runtime.createCount); + } + + @Test + public void missingTaskQueueWithPerInvocationConfigureFailsDuringInvocation() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler(options -> {}, (options, context) -> {}, runtime); + + assertThrows(IllegalStateException.class, () -> handler.handleRequest(null, context(20_000))); + + assertEquals(0, runtime.createCount); + } + + @Test + public void perInvocationShutdownHooksDoNotAccumulateAcrossInvocations() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> options.setTaskQueue("task-queue"), + (options, context) -> + options.addShutdownHook( + () -> runtime.events.add("hook:" + context.getAwsRequestId())), + runtime); + + handler.handleRequest(null, context(20_000, "request-1", "function-arn")); + handler.handleRequest(null, context(20_000, "request-2", "function-arn")); + + assertEquals(1, Collections.frequency(runtime.events, "hook:request-1")); + assertEquals(1, Collections.frequency(runtime.events, "hook:request-2")); + } + + @Test + public void perInvocationShutdownHooksRunWhenConfigureThrows() { + FakeRuntime runtime = new FakeRuntime(); + FakeMonotonicClock clock = new FakeMonotonicClock(); + AtomicReference hookTimeout = new AtomicReference<>(); + RequestHandler handler = + handler( + options -> {}, + (options, context) -> { + options.setTaskQueue("task-queue"); + options.addShutdownHook( + new TimedShutdownHook() { + @Override + public void run() { + run(Duration.ZERO); + } + + @Override + public void run(Duration timeout) { + hookTimeout.set(timeout); + runtime.events.add("cleanup"); + } + }); + throw new RuntimeException("configure failed"); + }, + runtime, + duration -> {}, + clock); + + RuntimeException e = + assertThrows(RuntimeException.class, () -> handler.handleRequest(null, context(20_000))); + + assertEquals("configure failed", e.getMessage()); + assertEquals(0, runtime.createCount); + assertEquals(Duration.ofSeconds(2), hookTimeout.get()); + assertEquals(events("cleanup"), runtime.events); + } + + @Test + public void perInvocationShutdownHooksRunWhenFinalValidationFails() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> {}, + (options, context) -> options.addShutdownHook(() -> runtime.events.add("cleanup")), + runtime); + + assertThrows(IllegalStateException.class, () -> handler.handleRequest(null, context(20_000))); + + assertEquals(0, runtime.createCount); + assertEquals(events("cleanup"), runtime.events); + } + + @Test + public void newHandlerSupportsPerInvocationConfigure() throws IOException { + FakeRuntime runtime = new FakeRuntime(); + LambdaWorkerOptions options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()).build(); + RequestHandler handler = + LambdaWorker.newHandler( + VERSION, + options, + (builder, context) -> { + builder.setTaskQueue("task-queue"); + builder.getWorkerOptionsBuilder().setIdentity("worker-" + context.getAwsRequestId()); + }, + runtime, + duration -> {}, + new FakeMonotonicClock()); + + handler.handleRequest(null, context(20_000, "request-1", "function-arn")); + + assertEquals("task-queue", runtime.taskQueue); + assertEquals("worker-request-1", runtime.workerOptions.getIdentity()); + assertEquals("worker-request-1", runtime.clientOptions.getIdentity()); + } + + @Test + public void missingTaskQueueFailsDuringHandlerConstruction() { + FakeRuntime runtime = new FakeRuntime(); + + assertThrows(IllegalStateException.class, () -> handler(options -> {}, runtime)); + + assertEquals(0, runtime.createCount); + } + + @Test + public void envconfigValuesAreVisibleToConfigureBeforeInvocation() throws IOException { + File config = temporaryFolder.newFile("temporal.toml"); + Files.write( + config.toPath(), + "[profile.default]\nnamespace = \"configured\"\n".getBytes(StandardCharsets.UTF_8)); + Map env = new HashMap<>(); + env.put(LambdaWorkerOptions.TEMPORAL_CONFIG_FILE, config.getAbsolutePath()); + AtomicReference namespaceInConfigure = new AtomicReference<>(); + FakeRuntime runtime = new FakeRuntime(); + + RequestHandler handler = + handler( + env, + options -> { + namespaceInConfigure.set( + options.getWorkflowClientOptionsBuilder().build().getNamespace()); + options.setTaskQueue("task-queue"); + }, + runtime, + duration -> {}); + + assertEquals("configured", namespaceInConfigure.get()); + + handler.handleRequest(null, context(20_000)); + + assertEquals("configured", runtime.clientOptions.getNamespace()); + } + + @Test + public void identityUsesAwsRequestIdAndFunctionArn() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler(options -> options.setTaskQueue("task-queue"), runtime); + + handler.handleRequest(null, context(20_000)); + + assertEquals("request-id@function-arn", runtime.clientOptions.getIdentity()); + assertEquals("request-id@function-arn", runtime.workerOptions.getIdentity()); + } + + @Test + public void userProvidedClientIdentityIsPreserved() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> { + options.setTaskQueue("task-queue"); + options.getWorkflowClientOptionsBuilder().setIdentity("custom-client"); + }, + runtime); + + handler.handleRequest(null, context(20_000)); + + assertEquals("custom-client", runtime.clientOptions.getIdentity()); + assertEquals("custom-client", runtime.workerOptions.getIdentity()); + } + + @Test + public void userProvidedWorkerIdentityIsPreserved() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> { + options.setTaskQueue("task-queue"); + options.getWorkerOptionsBuilder().setIdentity("custom-worker"); + }, + runtime); + + handler.handleRequest(null, context(20_000)); + + assertEquals("custom-worker", runtime.workerOptions.getIdentity()); + assertEquals("custom-worker", runtime.clientOptions.getIdentity()); + } + + @Test + public void insufficientRemainingTimeThrowsBeforeRuntimeCreation() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler(options -> options.setTaskQueue("task-queue"), runtime); + + assertThrows(IllegalStateException.class, () -> handler.handleRequest(null, context(8_000))); + + assertEquals(0, runtime.createCount); + } + + @Test + public void lowRemainingTimeStillStartsAndSleepsUntilShutdownBuffer() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> options.setTaskQueue("task-queue"), + runtime, + duration -> runtime.events.add("sleep:" + duration.toMillis())); + + handler.handleRequest(null, context(10_500)); + + assertTrue(runtime.events.contains("start")); + assertTrue(runtime.events.contains("sleep:3500")); + } + + @Test + public void shutdownHookErrorsDoNotSkipLaterHooks() { + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> + options + .setTaskQueue("task-queue") + .addShutdownHook( + () -> { + runtime.events.add("hook-1"); + throw new RuntimeException("hook failed"); + }) + .addShutdownHook(() -> runtime.events.add("hook-2")), + runtime); + + handler.handleRequest(null, context(20_000)); + + assertTrue(runtime.events.contains("hook-1")); + assertTrue(runtime.events.contains("hook-2")); + assertEquals("close:2000", runtime.events.get(runtime.events.size() - 1)); + } + + @Test + public void shutdownHooksAndStubsShareCleanupDeadline() { + FakeRuntime runtime = new FakeRuntime(); + FakeMonotonicClock clock = new FakeMonotonicClock(); + AtomicReference hookTimeout = new AtomicReference<>(); + RequestHandler handler = + handler( + options -> + options + .setTaskQueue("task-queue") + .addShutdownHook( + new TimedShutdownHook() { + @Override + public void run() { + run(Duration.ZERO); + } + + @Override + public void run(Duration timeout) { + hookTimeout.set(timeout); + clock.advance(Duration.ofMillis(1500)); + } + }), + runtime, + duration -> {}, + clock); + + handler.handleRequest(null, context(20_000)); + + assertEquals(Duration.ofSeconds(2), hookTimeout.get()); + assertEquals("close:500", runtime.events.get(runtime.events.size() - 1)); + } + + @Test + public void workerShutdownEscalatesAfterGracefulTimeoutAndSharesCleanupDeadline() { + FakeMonotonicClock clock = new FakeMonotonicClock(); + FakeRuntime runtime = new FakeRuntime(false, true, clock, Duration.ofMillis(1500)); + RequestHandler handler = + handler(options -> options.setTaskQueue("task-queue"), runtime, duration -> {}, clock); + + handler.handleRequest(null, context(20_000)); + + assertEquals( + events( + "create", "start", "shutdown", "await:5000", "shutdownNow", "await:2000", "close:500"), + runtime.events); + } + + @Test + public void workerShutdownEscalatesWhenGracefulAwaitThrows() { + FakeMonotonicClock clock = new FakeMonotonicClock(); + FakeRuntime runtime = new FakeRuntime(false, true, clock, Duration.ZERO, true); + RequestHandler handler = + handler(options -> options.setTaskQueue("task-queue"), runtime, duration -> {}, clock); + + handler.handleRequest(null, context(20_000)); + + assertEquals( + events( + "create", "start", "shutdown", "await:5000", "shutdownNow", "await:2000", "close:2000"), + runtime.events); + } + + @Test + public void defaultRuntimeShutsDownFactoryWhenWorkerCreationFails() { + AtomicInteger factoryShutdowns = new AtomicInteger(); + DefaultLambdaWorkerRuntime runtime = new DefaultLambdaWorkerRuntime(); + + WorkerFactoryOptions factoryOptions = + WorkerFactoryOptions.newBuilder() + .setPlugins( + new SimplePlugin("failing-worker-initializer") { + @Override + public void configureWorker(String taskQueue, WorkerOptions.Builder builder) { + throw new RuntimeException("worker configuration failed"); + } + + @Override + public void shutdownWorkerFactory( + WorkerFactory factory, Consumer next) { + factoryShutdowns.incrementAndGet(); + next.accept(factory); + } + }) + .build(); + + RuntimeException e = + assertThrows( + RuntimeException.class, + () -> + runtime.create( + WorkflowServiceStubsOptions.newBuilder() + .setRpcTimeout(Duration.ofMillis(10)) + .setSystemInfoTimeout(Duration.ofMillis(10)) + .build(), + WorkflowClientOptions.newBuilder().build(), + factoryOptions, + "task-queue", + WorkerOptions.newBuilder().build())); + + assertEquals("worker configuration failed", e.getMessage()); + assertEquals(1, factoryShutdowns.get()); + } + + private RequestHandler handler( + java.util.function.Consumer configure, FakeRuntime runtime) { + return handler(configure, runtime, duration -> {}); + } + + private RequestHandler handler( + java.util.function.Consumer configure, + LambdaWorker.InvocationConfigurator invocationConfigure, + FakeRuntime runtime) { + return handler(configure, invocationConfigure, runtime, duration -> {}); + } + + private RequestHandler handler( + java.util.function.Consumer configure, + FakeRuntime runtime, + LambdaWorker.Sleeper sleeper) { + return handler(baseEnv(), configure, runtime, sleeper, new FakeMonotonicClock()); + } + + private RequestHandler handler( + java.util.function.Consumer configure, + LambdaWorker.InvocationConfigurator invocationConfigure, + FakeRuntime runtime, + LambdaWorker.Sleeper sleeper) { + return handler( + baseEnv(), configure, invocationConfigure, runtime, sleeper, new FakeMonotonicClock()); + } + + private RequestHandler handler( + java.util.function.Consumer configure, + FakeRuntime runtime, + LambdaWorker.Sleeper sleeper, + LambdaWorker.MonotonicClock clock) { + return handler(baseEnv(), configure, runtime, sleeper, clock); + } + + private RequestHandler handler( + java.util.function.Consumer configure, + LambdaWorker.InvocationConfigurator invocationConfigure, + FakeRuntime runtime, + LambdaWorker.Sleeper sleeper, + LambdaWorker.MonotonicClock clock) { + return handler(baseEnv(), configure, invocationConfigure, runtime, sleeper, clock); + } + + private RequestHandler handler( + Map env, + java.util.function.Consumer configure, + FakeRuntime runtime, + LambdaWorker.Sleeper sleeper) { + return handler(env, configure, runtime, sleeper, new FakeMonotonicClock()); + } + + private RequestHandler handler( + Map env, + java.util.function.Consumer configure, + FakeRuntime runtime, + LambdaWorker.Sleeper sleeper, + LambdaWorker.MonotonicClock clock) { + try { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(env); + configure.accept(options); + return LambdaWorker.newHandler(VERSION, options.build(), runtime, sleeper, clock); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + private RequestHandler handler( + Map env, + java.util.function.Consumer configure, + LambdaWorker.InvocationConfigurator invocationConfigure, + FakeRuntime runtime, + LambdaWorker.Sleeper sleeper, + LambdaWorker.MonotonicClock clock) { + try { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(env); + configure.accept(options); + return LambdaWorker.newHandler( + VERSION, options.build(), invocationConfigure, runtime, sleeper, clock); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + private Context context(int remainingTimeMillis) { + return new TestLambdaContext(remainingTimeMillis); + } + + private Context context(int remainingTimeMillis, String awsRequestId, String invokedFunctionArn) { + return new TestLambdaContext(remainingTimeMillis, awsRequestId, invokedFunctionArn); + } + + private Map baseEnv() { + Map env = new HashMap<>(); + env.put(LambdaWorkerOptions.TEMPORAL_CONFIG_FILE, "/nonexistent/temporal.toml"); + return env; + } + + private List events(String... events) { + List result = new ArrayList<>(); + for (String event : events) { + result.add(event); + } + return result; + } + + private static final class FakeMonotonicClock implements LambdaWorker.MonotonicClock { + private long nowNanos; + + @Override + public long nanoTime() { + return nowNanos; + } + + private void advance(Duration duration) { + nowNanos += duration.toNanos(); + } + } + + private static final class FakeRuntime implements LambdaWorkerRuntime { + private final List events = new ArrayList<>(); + private final boolean gracefulTerminates; + private final boolean forcedTerminates; + private final FakeMonotonicClock clock; + private final Duration forcedAwaitDuration; + private final boolean gracefulAwaitThrows; + private int createCount; + private WorkflowClientOptions clientOptions; + private WorkerOptions workerOptions; + private String taskQueue; + + private FakeRuntime() { + this(true, true, null, Duration.ZERO, false); + } + + private FakeRuntime( + boolean gracefulTerminates, + boolean forcedTerminates, + FakeMonotonicClock clock, + Duration forcedAwaitDuration) { + this(gracefulTerminates, forcedTerminates, clock, forcedAwaitDuration, false); + } + + private FakeRuntime( + boolean gracefulTerminates, + boolean forcedTerminates, + FakeMonotonicClock clock, + Duration forcedAwaitDuration, + boolean gracefulAwaitThrows) { + this.gracefulTerminates = gracefulTerminates; + this.forcedTerminates = forcedTerminates; + this.clock = clock; + this.forcedAwaitDuration = forcedAwaitDuration; + this.gracefulAwaitThrows = gracefulAwaitThrows; + } + + @Override + public Invocation create( + WorkflowServiceStubsOptions serviceStubsOptions, + WorkflowClientOptions clientOptions, + WorkerFactoryOptions workerFactoryOptions, + String taskQueue, + WorkerOptions workerOptions) { + createCount++; + events.add("create"); + this.clientOptions = clientOptions; + this.workerOptions = workerOptions; + this.taskQueue = taskQueue; + return new FakeInvocation( + events, + gracefulTerminates, + forcedTerminates, + clock, + forcedAwaitDuration, + gracefulAwaitThrows); + } + } + + private static final class FakeInvocation implements LambdaWorkerRuntime.Invocation { + private final List events; + private final WorkerRegistrar registrar; + private final boolean gracefulTerminates; + private final boolean forcedTerminates; + private final FakeMonotonicClock clock; + private final Duration forcedAwaitDuration; + private final boolean gracefulAwaitThrows; + private boolean terminated; + private int awaitCount; + + private FakeInvocation( + List events, + boolean gracefulTerminates, + boolean forcedTerminates, + FakeMonotonicClock clock, + Duration forcedAwaitDuration, + boolean gracefulAwaitThrows) { + this.events = events; + this.registrar = new FakeWorkerRegistrar(events); + this.gracefulTerminates = gracefulTerminates; + this.forcedTerminates = forcedTerminates; + this.clock = clock; + this.forcedAwaitDuration = forcedAwaitDuration; + this.gracefulAwaitThrows = gracefulAwaitThrows; + } + + @Override + public WorkerRegistrar getWorkerRegistrar() { + return registrar; + } + + @Override + public void start() { + events.add("start"); + } + + @Override + public void shutdown() { + events.add("shutdown"); + terminated = false; + } + + @Override + public void shutdownNow() { + events.add("shutdownNow"); + } + + @Override + public void awaitTermination(Duration timeout) { + events.add("await:" + timeout.toMillis()); + awaitCount++; + if (awaitCount == 1 && gracefulAwaitThrows) { + throw new RuntimeException("await failed"); + } + if (awaitCount == 1 && gracefulTerminates) { + terminated = true; + } else if (awaitCount > 1) { + if (clock != null) { + clock.advance(forcedAwaitDuration); + } + terminated = forcedTerminates; + } + } + + @Override + public boolean isTerminated() { + return terminated; + } + + @Override + public void closeStubs(Duration timeout) { + events.add("close:" + timeout.toMillis()); + } + } + + private static final class FakeWorkerRegistrar implements WorkerRegistrar { + private final List events; + + private FakeWorkerRegistrar(List events) { + this.events = events; + } + + @Override + public void registerWorkflowImplementationTypes(Class... workflowImplementationClasses) { + events.add("registerWorkflowTypes:" + workflowImplementationClasses.length); + } + + @Override + public void registerWorkflowImplementationTypes( + WorkflowImplementationOptions options, Class... workflowImplementationClasses) { + events.add("registerWorkflowTypesWithOptions:" + workflowImplementationClasses.length); + } + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, Functions.Func factory) { + events.add("registerWorkflowFactory"); + } + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, + Functions.Func1 factory, + WorkflowImplementationOptions options) { + events.add("registerWorkflowFactoryWithArgs"); + } + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, + Functions.Func factory, + WorkflowImplementationOptions options) { + events.add("registerWorkflowFactoryWithOptions"); + } + + @Override + public void registerActivitiesImplementations(Object... activityImplementations) { + events.add("registerActivities:" + activityImplementations.length); + } + + @Override + public void registerNexusServiceImplementation(Object... nexusServiceImplementations) { + events.add("registerNexus:" + nexusServiceImplementations.length); + } + } + + private static final class TestWorkflowImpl {} + + private static final class TestDynamicWorkflow implements DynamicWorkflow { + @Override + public Object execute(EncodedValues args) { + return null; + } + } + + private static final class TestDynamicWorkflowWithOptions implements DynamicWorkflow { + @Override + public Object execute(EncodedValues args) { + return null; + } + } + + private static final class TestDynamicActivity implements DynamicActivity { + @Override + public Object execute(EncodedValues args) { + return null; + } + } +} diff --git a/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/LambdaWorkerOptionsTest.java b/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/LambdaWorkerOptionsTest.java new file mode 100644 index 0000000000..39c9cd6547 --- /dev/null +++ b/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/LambdaWorkerOptionsTest.java @@ -0,0 +1,339 @@ +package io.temporal.aws.lambda; + +import static org.junit.Assert.*; + +import io.temporal.activity.DynamicActivity; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.common.VersioningBehavior; +import io.temporal.common.WorkerDeploymentVersion; +import io.temporal.common.converter.EncodedValues; +import io.temporal.worker.WorkerDeploymentOptions; +import io.temporal.worker.WorkerFactoryOptions; +import io.temporal.worker.WorkerOptions; +import io.temporal.worker.WorkflowImplementationOptions; +import io.temporal.workflow.DynamicWorkflow; +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.time.Duration; +import java.util.HashMap; +import java.util.Map; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +public class LambdaWorkerOptionsTest { + private static final WorkerDeploymentVersion VERSION = + new WorkerDeploymentVersion("deployment", "build"); + + @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder(); + + @Test + public void lambdaDefaultsAreAppliedToUnsetOptions() throws IOException { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + options.setTaskQueue("task-queue"); + + LambdaWorkerOptions.Materialized materialized = + options.build().materialize(VERSION, "request@arn"); + + WorkerOptions workerOptions = materialized.workerOptions; + assertEquals(2, workerOptions.getMaxConcurrentActivityExecutionSize()); + assertEquals(10, workerOptions.getMaxConcurrentWorkflowTaskExecutionSize()); + assertEquals(2, workerOptions.getMaxConcurrentLocalActivityExecutionSize()); + assertEquals(5, workerOptions.getMaxConcurrentNexusExecutionSize()); + assertEquals(2, workerOptions.getMaxConcurrentWorkflowTaskPollers()); + assertEquals(1, workerOptions.getMaxConcurrentActivityTaskPollers()); + assertEquals(1, workerOptions.getMaxConcurrentNexusTaskPollers()); + assertTrue(workerOptions.isEagerExecutionDisabled()); + + WorkerFactoryOptions factoryOptions = materialized.workerFactoryOptions; + assertEquals(30, factoryOptions.getWorkflowCacheSize()); + assertEquals(30, factoryOptions.getMaxWorkflowThreadCount()); + + WorkerDeploymentOptions deploymentOptions = workerOptions.getDeploymentOptions(); + assertTrue(deploymentOptions.isUsingVersioning()); + assertEquals(VERSION, deploymentOptions.getVersion()); + assertEquals(VersioningBehavior.PINNED, deploymentOptions.getDefaultVersioningBehavior()); + assertEquals(Duration.ofSeconds(5), materialized.gracefulShutdownTimeout); + assertEquals(Duration.ofSeconds(7), materialized.shutdownDeadlineBuffer); + } + + @Test + public void userOverridesWinOverLambdaDefaults() throws IOException { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + options.setTaskQueue("task-queue"); + options.getWorkerOptionsBuilder().setMaxConcurrentActivityExecutionSize(11); + options.getWorkerOptionsBuilder().setMaxConcurrentWorkflowTaskExecutionSize(12); + options.getWorkerOptionsBuilder().setMaxConcurrentLocalActivityExecutionSize(13); + options.getWorkerOptionsBuilder().setMaxConcurrentNexusExecutionSize(14); + options.getWorkerOptionsBuilder().setMaxConcurrentWorkflowTaskPollers(3); + options.getWorkerOptionsBuilder().setMaxConcurrentActivityTaskPollers(4); + options.getWorkerOptionsBuilder().setMaxConcurrentNexusTaskPollers(6); + options + .getWorkerOptionsBuilder() + .setDeploymentOptions( + WorkerDeploymentOptions.newBuilder() + .setUseVersioning(true) + .setVersion(new WorkerDeploymentVersion("ignored", "ignored")) + .setDefaultVersioningBehavior(VersioningBehavior.AUTO_UPGRADE) + .build()); + options.getWorkerFactoryOptionsBuilder().setWorkflowCacheSize(17); + options.getWorkerFactoryOptionsBuilder().setMaxWorkflowThreadCount(18); + + LambdaWorkerOptions.Materialized materialized = + options.build().materialize(VERSION, "request@arn"); + + WorkerOptions workerOptions = materialized.workerOptions; + assertEquals(11, workerOptions.getMaxConcurrentActivityExecutionSize()); + assertEquals(12, workerOptions.getMaxConcurrentWorkflowTaskExecutionSize()); + assertEquals(13, workerOptions.getMaxConcurrentLocalActivityExecutionSize()); + assertEquals(14, workerOptions.getMaxConcurrentNexusExecutionSize()); + assertEquals(3, workerOptions.getMaxConcurrentWorkflowTaskPollers()); + assertEquals(4, workerOptions.getMaxConcurrentActivityTaskPollers()); + assertEquals(6, workerOptions.getMaxConcurrentNexusTaskPollers()); + assertTrue(workerOptions.isEagerExecutionDisabled()); + assertEquals(17, materialized.workerFactoryOptions.getWorkflowCacheSize()); + assertEquals(18, materialized.workerFactoryOptions.getMaxWorkflowThreadCount()); + assertTrue(workerOptions.getDeploymentOptions().isUsingVersioning()); + assertEquals(VERSION, workerOptions.getDeploymentOptions().getVersion()); + assertEquals( + VersioningBehavior.AUTO_UPGRADE, + workerOptions.getDeploymentOptions().getDefaultVersioningBehavior()); + } + + @Test + public void temporalTaskQueueEnvPopulatesTaskQueue() throws IOException { + Map env = baseEnv(); + env.put(LambdaWorkerOptions.TEMPORAL_TASK_QUEUE, "env-task-queue"); + + LambdaWorkerOptions.Materialized materialized = + LambdaWorkerOptions.newBuilderFromEnvironment(env) + .build() + .materialize(VERSION, "request@arn"); + + assertEquals("env-task-queue", materialized.taskQueue); + } + + @Test + public void missingTaskQueueFailsBeforeRuntimeCreation() throws IOException { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + + IllegalStateException e = + assertThrows( + IllegalStateException.class, () -> options.build().materialize(VERSION, "request@arn")); + + assertTrue(e.getMessage().contains("Task queue must be set")); + } + + @Test + public void gracefulShutdownTimeoutRecomputesDefaultShutdownDeadlineBuffer() throws IOException { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + options.setTaskQueue("task-queue"); + options.setGracefulShutdownTimeout(Duration.ofSeconds(3)); + + LambdaWorkerOptions.Materialized materialized = + options.build().materialize(VERSION, "request@arn"); + + assertEquals(Duration.ofSeconds(3), materialized.gracefulShutdownTimeout); + assertEquals(Duration.ofSeconds(5), materialized.shutdownDeadlineBuffer); + } + + @Test + public void explicitShutdownDeadlineBufferIsNotRecomputed() throws IOException { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + options.setTaskQueue("task-queue"); + options.setShutdownDeadlineBuffer(Duration.ofSeconds(9)); + options.setGracefulShutdownTimeout(Duration.ofSeconds(3)); + + LambdaWorkerOptions.Materialized materialized = + options.build().materialize(VERSION, "request@arn"); + + assertEquals(Duration.ofSeconds(3), materialized.gracefulShutdownTimeout); + assertEquals(Duration.ofSeconds(9), materialized.shutdownDeadlineBuffer); + } + + @Test + public void buildCreatesImmutableSnapshot() throws IOException { + LambdaWorkerOptions.Builder builder = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + builder.setTaskQueue("first-task-queue"); + + LambdaWorkerOptions options = builder.build(); + builder.setTaskQueue("second-task-queue"); + + assertEquals("first-task-queue", options.getTaskQueue()); + assertEquals("first-task-queue", options.materialize(VERSION, "request@arn").taskQueue); + } + + @Test + public void toBuilderPreservesExplicitShutdownDeadlineBuffer() throws IOException { + LambdaWorkerOptions options = + LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()) + .setTaskQueue("task-queue") + .setShutdownDeadlineBuffer(Duration.ofSeconds(9)) + .setGracefulShutdownTimeout(Duration.ofSeconds(3)) + .build(); + + LambdaWorkerOptions.Materialized materialized = + options.toBuilder() + .setGracefulShutdownTimeout(Duration.ofSeconds(4)) + .build() + .materialize(VERSION, "request@arn"); + + assertEquals(Duration.ofSeconds(4), materialized.gracefulShutdownTimeout); + assertEquals(Duration.ofSeconds(9), materialized.shutdownDeadlineBuffer); + } + + @Test + public void shutdownDeadlineBufferMustCoverGracefulShutdownTimeout() throws IOException { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + options.setTaskQueue("task-queue"); + options.setShutdownDeadlineBuffer(Duration.ofSeconds(2)); + + IllegalStateException e = + assertThrows( + IllegalStateException.class, () -> options.build().materialize(VERSION, "request@arn")); + + assertTrue(e.getMessage().contains("shutdownDeadlineBuffer")); + } + + @Test + public void temporalConfigFileTakesPrecedenceOverLambdaTaskRoot() throws IOException { + File explicitConfig = temporaryFolder.newFile("explicit.toml"); + Files.write( + explicitConfig.toPath(), + "[profile.default]\nnamespace = \"explicit\"\n".getBytes(StandardCharsets.UTF_8)); + File taskRoot = temporaryFolder.newFolder("task-root"); + Files.write( + new File(taskRoot, "temporal.toml").toPath(), + "[profile.default]\nnamespace = \"lambda-root\"\n".getBytes(StandardCharsets.UTF_8)); + + Map env = new HashMap<>(); + env.put(LambdaWorkerOptions.TEMPORAL_CONFIG_FILE, explicitConfig.getAbsolutePath()); + env.put(LambdaWorkerOptions.LAMBDA_TASK_ROOT, taskRoot.getAbsolutePath()); + + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(env); + options.setTaskQueue("task-queue"); + WorkflowClientOptions clientOptions = + options.build().materialize(VERSION, "request@arn").clientOptions; + + assertEquals("explicit", clientOptions.getNamespace()); + } + + @Test + public void lambdaTaskRootTemporalTomlWinsOverCwdTemporalToml() throws IOException { + File taskRoot = temporaryFolder.newFolder("task-root"); + writeConfig(taskRoot, "lambda-root"); + File cwd = temporaryFolder.newFolder("cwd"); + writeConfig(cwd, "cwd"); + + Map env = new HashMap<>(); + env.put(LambdaWorkerOptions.LAMBDA_TASK_ROOT, taskRoot.getAbsolutePath()); + + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(env, cwd); + options.setTaskQueue("task-queue"); + + assertEquals( + "lambda-root", + options.build().materialize(VERSION, "request@arn").clientOptions.getNamespace()); + } + + @Test + public void cwdTemporalTomlIsUsedWhenLambdaTaskRootIsUnset() throws IOException { + File cwd = temporaryFolder.newFolder("cwd"); + writeConfig(cwd, "cwd"); + + LambdaWorkerOptions.Builder options = + LambdaWorkerOptions.newBuilderFromEnvironment(new HashMap<>(), cwd); + options.setTaskQueue("task-queue"); + + assertEquals( + "cwd", options.build().materialize(VERSION, "request@arn").clientOptions.getNamespace()); + } + + @Test + public void cwdTemporalTomlIsUsedWhenLambdaTaskRootHasNoTemporalToml() throws IOException { + File taskRoot = temporaryFolder.newFolder("task-root"); + File cwd = temporaryFolder.newFolder("cwd"); + writeConfig(cwd, "cwd"); + Map env = new HashMap<>(); + env.put(LambdaWorkerOptions.LAMBDA_TASK_ROOT, taskRoot.getAbsolutePath()); + + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(env, cwd); + options.setTaskQueue("task-queue"); + + assertEquals( + "cwd", options.build().materialize(VERSION, "request@arn").clientOptions.getNamespace()); + } + + @Test + public void envconfigDefaultsAreUsedWhenNoConfigFileCandidateExists() throws IOException { + File cwd = temporaryFolder.newFolder("cwd"); + + LambdaWorkerOptions.Builder options = + LambdaWorkerOptions.newBuilderFromEnvironment(new HashMap<>(), cwd); + options.setTaskQueue("task-queue"); + + assertEquals( + "default", + options.build().materialize(VERSION, "request@arn").clientOptions.getNamespace()); + } + + @Test + public void deploymentVersionMustHaveDeploymentNameAndBuildId() { + assertThrows( + IllegalArgumentException.class, + () -> LambdaWorkerOptions.validateVersion(new WorkerDeploymentVersion("", "build"))); + assertThrows( + IllegalArgumentException.class, + () -> LambdaWorkerOptions.validateVersion(new WorkerDeploymentVersion("deployment", ""))); + } + + @Test + public void dynamicRegistrationMethodsRejectNullInputs() throws IOException { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + + assertThrows( + NullPointerException.class, + () -> + options.registerDynamicWorkflowImplementationType( + (Class) null)); + assertThrows( + NullPointerException.class, + () -> options.registerDynamicWorkflowImplementationType(null, TestDynamicWorkflow.class)); + assertThrows( + NullPointerException.class, + () -> + options.registerDynamicWorkflowImplementationType( + WorkflowImplementationOptions.getDefaultInstance(), null)); + assertThrows( + NullPointerException.class, () -> options.registerDynamicActivityImplementation(null)); + } + + private Map baseEnv() { + Map env = new HashMap<>(); + env.put(LambdaWorkerOptions.TEMPORAL_CONFIG_FILE, "/nonexistent/temporal.toml"); + return env; + } + + private void writeConfig(File directory, String namespace) throws IOException { + Files.write( + new File(directory, "temporal.toml").toPath(), + ("[profile.default]\nnamespace = \"" + namespace + "\"\n") + .getBytes(StandardCharsets.UTF_8)); + } + + private static final class TestDynamicWorkflow implements DynamicWorkflow { + @Override + public Object execute(EncodedValues args) { + return null; + } + } + + private static final class TestDynamicActivity implements DynamicActivity { + @Override + public Object execute(EncodedValues args) { + return null; + } + } +} diff --git a/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/OtelLambdaWorkerConfigurationHelperTest.java b/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/OtelLambdaWorkerConfigurationHelperTest.java new file mode 100644 index 0000000000..3adac8a55a --- /dev/null +++ b/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/OtelLambdaWorkerConfigurationHelperTest.java @@ -0,0 +1,458 @@ +package io.temporal.aws.lambda; + +import static org.junit.Assert.*; + +import com.amazonaws.services.lambda.runtime.Context; +import com.amazonaws.services.lambda.runtime.RequestHandler; +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.api.metrics.MeterBuilder; +import io.opentelemetry.api.metrics.MeterProvider; +import io.opentelemetry.api.trace.Span; +import io.opentelemetry.api.trace.Tracer; +import io.opentelemetry.api.trace.TracerProvider; +import io.opentelemetry.context.propagation.ContextPropagators; +import io.opentelemetry.sdk.OpenTelemetrySdk; +import io.opentelemetry.sdk.common.CompletableResultCode; +import io.opentelemetry.sdk.trace.IdGenerator; +import io.temporal.common.WorkerDeploymentVersion; +import io.temporal.opentelemetry.OpenTelemetryPlugin; +import io.temporal.opentelemetry.TimedShutdownHook; +import io.temporal.serviceclient.WorkflowServiceStubsPlugin; +import java.io.Closeable; +import java.time.Duration; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import org.junit.Test; + +public class OtelLambdaWorkerConfigurationHelperTest { + private static final WorkerDeploymentVersion VERSION = + new WorkerDeploymentVersion("deployment", "build"); + + @Test + public void defaultsResolveEndpointAndServiceName() { + assertEquals( + "http://localhost:4317", + OtelLambdaWorkerConfigurationHelper.newBuilder(new HashMap<>()).getEndpoint()); + assertEquals( + "temporal-lambda-worker", + OtelLambdaWorkerConfigurationHelper.newBuilder(new HashMap<>()).getServiceName()); + + Map env = new HashMap<>(); + env.put( + OtelLambdaWorkerConfigurationHelper.OTEL_EXPORTER_OTLP_ENDPOINT, "http://collector:4317"); + env.put(OtelLambdaWorkerConfigurationHelper.AWS_LAMBDA_FUNCTION_NAME, "function-name"); + assertEquals( + "http://collector:4317", OtelLambdaWorkerConfigurationHelper.newBuilder(env).getEndpoint()); + assertEquals( + "function-name", OtelLambdaWorkerConfigurationHelper.newBuilder(env).getServiceName()); + + env.put(OtelLambdaWorkerConfigurationHelper.OTEL_SERVICE_NAME, "explicit-service"); + assertEquals( + "explicit-service", OtelLambdaWorkerConfigurationHelper.newBuilder(env).getServiceName()); + } + + @Test + public void defaultFactoryCreatesXRayTraceIds() { + OpenTelemetry openTelemetry = + OtelLambdaWorkerConfigurationHelper.newBuilder(new HashMap<>()) + .setFlushTimeout(Duration.ofMillis(10)) + .createOpenTelemetry(); + long beforeSeconds = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis()); + + assertTrue(openTelemetry instanceof OpenTelemetrySdk); + + OpenTelemetrySdk sdk = (OpenTelemetrySdk) openTelemetry; + Span span = sdk.getSdkTracerProvider().get("test").spanBuilder("test").startSpan(); + try { + String traceId = span.getSpanContext().getTraceId(); + long afterSeconds = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis()); + long xrayTimestampSeconds = Long.parseLong(traceId.substring(0, 8), 16); + + assertTrue(xrayTimestampSeconds >= beforeSeconds); + assertTrue(xrayTimestampSeconds <= afterSeconds); + } finally { + sdk.shutdown().join(1, TimeUnit.SECONDS); + span.end(); + } + } + + @Test + public void exporterFactoryReceivesResolvedEndpointServiceNameAndIdGenerator() throws Exception { + Map env = new HashMap<>(); + env.put( + OtelLambdaWorkerConfigurationHelper.OTEL_EXPORTER_OTLP_ENDPOINT, "http://collector:4317"); + env.put(OtelLambdaWorkerConfigurationHelper.AWS_LAMBDA_FUNCTION_NAME, "function-name"); + RecordingTelemetryFactory factory = new RecordingTelemetryFactory(); + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + + OtelLambdaWorkerConfigurationHelper.newBuilder(env).setTelemetryFactory(factory).apply(options); + + assertEquals(1, factory.creates.get()); + assertEquals("http://collector:4317", factory.endpoint); + assertEquals("function-name", factory.serviceName); + assertNotNull(factory.idGenerator); + } + + @Test + public void customOpenTelemetryBypassesExporterCreation() throws Exception { + RecordingTelemetryFactory factory = new RecordingTelemetryFactory(); + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + + OtelLambdaWorkerConfigurationHelper.newBuilder(new HashMap<>()) + .setTelemetryFactory(factory) + .setOpenTelemetry(OpenTelemetry.noop()) + .apply(options); + + assertEquals(0, factory.creates.get()); + assertTrue(installedPlugin(options) instanceof OpenTelemetryPlugin); + } + + @Test + public void pluginInstallsMetricsScopeAndTracingInterceptors() throws Exception { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + + OtelLambdaWorkerConfigurationHelper.configure( + options, builder -> builder.setOpenTelemetry(OpenTelemetry.noop()).setFlushHook(() -> {})); + OpenTelemetryPlugin plugin = installedPlugin(options); + + plugin.configureServiceStubs(options.getWorkflowServiceStubsOptionsBuilder()); + plugin.configureWorkflowClient(options.getWorkflowClientOptionsBuilder()); + plugin.configureWorkerFactory(options.getWorkerFactoryOptionsBuilder()); + + assertNotNull(options.getWorkflowServiceStubsOptionsBuilder().build().getMetricsScope()); + assertEquals(1, options.getWorkflowClientOptionsBuilder().build().getInterceptors().length); + assertEquals( + 1, options.getWorkerFactoryOptionsBuilder().build().getWorkerInterceptors().length); + } + + @Test + public void configureRegistersPluginAndPerInvocationFlushHook() throws Exception { + LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + + OtelLambdaWorkerConfigurationHelper.configure( + options, builder -> builder.setOpenTelemetry(OpenTelemetry.noop())); + options.setTaskQueue("task-queue"); + + List hooks = options.build().prepare(VERSION).materialize("identity").shutdownHooks; + assertTrue(installedPlugin(options) instanceof OpenTelemetryPlugin); + assertEquals(1, hooks.size()); + assertTrue(hooks.get(0) instanceof TimedShutdownHook); + } + + @Test + public void flushHookReceivesLambdaCleanupTimeoutAndDoesNotCloseProviders() { + TimeoutRecordingOpenTelemetry openTelemetry = new TimeoutRecordingOpenTelemetry(); + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> { + options.setTaskQueue("task-queue"); + OtelLambdaWorkerConfigurationHelper.configureFlushHook( + options, openTelemetry, Duration.ofSeconds(10)); + }, + runtime, + duration -> {}); + + handler.handleRequest(null, context()); + + assertTrue(openTelemetry.tracerProvider.joinTimeoutMillis.get() > 0); + assertTrue(openTelemetry.tracerProvider.joinTimeoutMillis.get() < 10_000); + assertTrue(openTelemetry.meterProvider.joinTimeoutMillis.get() > 0); + assertTrue(openTelemetry.meterProvider.joinTimeoutMillis.get() < 10_000); + assertEquals(0, openTelemetry.tracerProvider.closes.get()); + assertEquals(0, openTelemetry.meterProvider.closes.get()); + } + + @Test + public void flushHookRunsOncePerInvocationAndDoesNotCloseProviders() { + CountingOpenTelemetry openTelemetry = new CountingOpenTelemetry(); + FakeRuntime runtime = new FakeRuntime(); + RequestHandler handler = + handler( + options -> { + options.setTaskQueue("task-queue"); + OtelLambdaWorkerConfigurationHelper.configure( + options, builder -> builder.setOpenTelemetry(openTelemetry)); + }, + runtime, + duration -> {}); + + handler.handleRequest(null, context()); + handler.handleRequest(null, context()); + + assertEquals(2, openTelemetry.tracerProvider.flushes.get()); + assertEquals(2, openTelemetry.meterProvider.flushes.get()); + assertEquals(0, openTelemetry.tracerProvider.closes.get()); + assertEquals(0, openTelemetry.meterProvider.closes.get()); + } + + private Context context() { + return new TestLambdaContext(20_000); + } + + private Map baseEnv() { + Map env = new HashMap<>(); + env.put(LambdaWorkerOptions.TEMPORAL_CONFIG_FILE, "/nonexistent/temporal.toml"); + return env; + } + + private RequestHandler handler( + java.util.function.Consumer configure, + FakeRuntime runtime, + LambdaWorker.Sleeper sleeper) { + try { + LambdaWorkerOptions.Builder options = + LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv()); + configure.accept(options); + return LambdaWorker.newHandler(VERSION, options.build(), runtime, sleeper); + } catch (java.io.IOException e) { + throw new RuntimeException(e); + } + } + + private OpenTelemetryPlugin installedPlugin(LambdaWorkerOptions.Builder options) { + WorkflowServiceStubsPlugin[] plugins = + options.getWorkflowServiceStubsOptionsBuilder().build().getPlugins(); + assertNotNull(plugins); + assertEquals(1, plugins.length); + assertTrue(plugins[0] instanceof OpenTelemetryPlugin); + return (OpenTelemetryPlugin) plugins[0]; + } + + private static final class RecordingTelemetryFactory + implements OtelLambdaWorkerConfigurationHelper.TelemetryFactory { + private final AtomicInteger creates = new AtomicInteger(); + private String endpoint; + private String serviceName; + private IdGenerator idGenerator; + + @Override + public OpenTelemetry create( + String endpoint, + String serviceName, + Duration metricsReportInterval, + Duration flushTimeout, + IdGenerator idGenerator) { + creates.incrementAndGet(); + this.endpoint = endpoint; + this.serviceName = serviceName; + this.idGenerator = idGenerator; + return OpenTelemetry.noop(); + } + } + + private static final class CountingOpenTelemetry implements OpenTelemetry { + private final CountingTracerProvider tracerProvider = new CountingTracerProvider(); + private final CountingMeterProvider meterProvider = new CountingMeterProvider(); + + @Override + public TracerProvider getTracerProvider() { + return tracerProvider; + } + + @Override + public MeterProvider getMeterProvider() { + return meterProvider; + } + + @Override + public ContextPropagators getPropagators() { + return ContextPropagators.noop(); + } + } + + private static final class TimeoutRecordingOpenTelemetry implements OpenTelemetry { + private final TimeoutRecordingTracerProvider tracerProvider = + new TimeoutRecordingTracerProvider(); + private final TimeoutRecordingMeterProvider meterProvider = new TimeoutRecordingMeterProvider(); + + @Override + public TracerProvider getTracerProvider() { + return tracerProvider; + } + + @Override + public MeterProvider getMeterProvider() { + return meterProvider; + } + + @Override + public ContextPropagators getPropagators() { + return ContextPropagators.noop(); + } + } + + public static final class TimeoutRecordingTracerProvider implements TracerProvider, Closeable { + private final AtomicLong joinTimeoutMillis = new AtomicLong(-1); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public Tracer get(String instrumentationName) { + return TracerProvider.noop().get(instrumentationName); + } + + @Override + public Tracer get(String instrumentationName, String instrumentationVersion) { + return TracerProvider.noop().get(instrumentationName, instrumentationVersion); + } + + public TimeoutRecordingResult forceFlush() { + return new TimeoutRecordingResult(joinTimeoutMillis); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } + + public static final class TimeoutRecordingMeterProvider implements MeterProvider, Closeable { + private final AtomicLong joinTimeoutMillis = new AtomicLong(-1); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public MeterBuilder meterBuilder(String instrumentationName) { + return MeterProvider.noop().meterBuilder(instrumentationName); + } + + public TimeoutRecordingResult forceFlush() { + return new TimeoutRecordingResult(joinTimeoutMillis); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } + + public static final class TimeoutRecordingResult { + private final AtomicLong joinTimeoutMillis; + + private TimeoutRecordingResult(AtomicLong joinTimeoutMillis) { + this.joinTimeoutMillis = joinTimeoutMillis; + } + + public TimeoutRecordingResult join(long timeout, TimeUnit unit) { + joinTimeoutMillis.set(unit.toMillis(timeout)); + return this; + } + } + + public static final class CountingTracerProvider implements TracerProvider, Closeable { + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public Tracer get(String instrumentationName) { + return TracerProvider.noop().get(instrumentationName); + } + + @Override + public Tracer get(String instrumentationName, String instrumentationVersion) { + return TracerProvider.noop().get(instrumentationName, instrumentationVersion); + } + + public CompletableResultCode forceFlush() { + flushes.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } + + public static final class CountingMeterProvider implements MeterProvider, Closeable { + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public MeterBuilder meterBuilder(String instrumentationName) { + return MeterProvider.noop().meterBuilder(instrumentationName); + } + + public CompletableResultCode forceFlush() { + flushes.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } + + private static final class FakeRuntime implements LambdaWorkerRuntime { + @Override + public Invocation create( + io.temporal.serviceclient.WorkflowServiceStubsOptions serviceStubsOptions, + io.temporal.client.WorkflowClientOptions clientOptions, + io.temporal.worker.WorkerFactoryOptions workerFactoryOptions, + String taskQueue, + io.temporal.worker.WorkerOptions workerOptions) { + return new Invocation() { + @Override + public WorkerRegistrar getWorkerRegistrar() { + return new NoopRegistrar(); + } + + @Override + public void start() {} + + @Override + public void shutdown() {} + + @Override + public void shutdownNow() {} + + @Override + public void awaitTermination(java.time.Duration timeout) {} + + @Override + public boolean isTerminated() { + return true; + } + + @Override + public void closeStubs(java.time.Duration timeout) {} + }; + } + } + + private static final class NoopRegistrar implements WorkerRegistrar { + @Override + public void registerWorkflowImplementationTypes(Class... workflowImplementationClasses) {} + + @Override + public void registerWorkflowImplementationTypes( + io.temporal.worker.WorkflowImplementationOptions options, + Class... workflowImplementationClasses) {} + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, io.temporal.workflow.Functions.Func factory) {} + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, + io.temporal.workflow.Functions.Func1 factory, + io.temporal.worker.WorkflowImplementationOptions options) {} + + @Override + public void registerWorkflowImplementationFactory( + Class workflowInterface, + io.temporal.workflow.Functions.Func factory, + io.temporal.worker.WorkflowImplementationOptions options) {} + + @Override + public void registerActivitiesImplementations(Object... activityImplementations) {} + + @Override + public void registerNexusServiceImplementation(Object... nexusServiceImplementations) {} + } +} diff --git a/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/TestLambdaContext.java b/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/TestLambdaContext.java new file mode 100644 index 0000000000..1d58d6f8b1 --- /dev/null +++ b/contrib/temporal-aws-lambda/src/test/java/io/temporal/aws/lambda/TestLambdaContext.java @@ -0,0 +1,96 @@ +package io.temporal.aws.lambda; + +import com.amazonaws.services.lambda.runtime.ClientContext; +import com.amazonaws.services.lambda.runtime.CognitoIdentity; +import com.amazonaws.services.lambda.runtime.Context; +import com.amazonaws.services.lambda.runtime.LambdaLogger; +import java.util.function.IntSupplier; + +final class TestLambdaContext implements Context { + private static final LambdaLogger NOOP_LOGGER = + new LambdaLogger() { + @Override + public void log(String message) {} + + @Override + public void log(byte[] message) {} + }; + + private final IntSupplier remainingTimeMillis; + private final String awsRequestId; + private final String invokedFunctionArn; + + TestLambdaContext(int remainingTimeMillis) { + this(remainingTimeMillis, "request-id", "function-arn"); + } + + TestLambdaContext(int remainingTimeMillis, String awsRequestId, String invokedFunctionArn) { + this(() -> remainingTimeMillis, awsRequestId, invokedFunctionArn); + } + + TestLambdaContext(IntSupplier remainingTimeMillis) { + this(remainingTimeMillis, "request-id", "function-arn"); + } + + TestLambdaContext( + IntSupplier remainingTimeMillis, String awsRequestId, String invokedFunctionArn) { + this.remainingTimeMillis = remainingTimeMillis; + this.awsRequestId = awsRequestId; + this.invokedFunctionArn = invokedFunctionArn; + } + + @Override + public String getAwsRequestId() { + return awsRequestId; + } + + @Override + public String getLogGroupName() { + return "log-group"; + } + + @Override + public String getLogStreamName() { + return "log-stream"; + } + + @Override + public String getFunctionName() { + return "function"; + } + + @Override + public String getFunctionVersion() { + return "1"; + } + + @Override + public String getInvokedFunctionArn() { + return invokedFunctionArn; + } + + @Override + public CognitoIdentity getIdentity() { + return null; + } + + @Override + public ClientContext getClientContext() { + return null; + } + + @Override + public int getRemainingTimeInMillis() { + return remainingTimeMillis.getAsInt(); + } + + @Override + public int getMemoryLimitInMB() { + return 128; + } + + @Override + public LambdaLogger getLogger() { + return NOOP_LOGGER; + } +} diff --git a/contrib/temporal-opentelemetry/README.md b/contrib/temporal-opentelemetry/README.md new file mode 100644 index 0000000000..ab315279eb --- /dev/null +++ b/contrib/temporal-opentelemetry/README.md @@ -0,0 +1,65 @@ +# Temporal OpenTelemetry module + +This module provides OpenTelemetry integration helpers for Temporal Java SDK workers. + +The Java SDK metrics API is based on Tally `Scope`. `temporal-opentelemetry` bridges that Tally surface into OpenTelemetry metrics, configures tracing through the SDK OpenTracing interceptor path, and provides shutdown hooks that flush Tally and OpenTelemetry providers without closing application-owned providers. + +## Usage + +Add `temporal-opentelemetry` next to your Temporal SDK dependency, then install the plugin on service stubs options before creating clients and workers: + +```java +OpenTelemetryPlugin plugin = OpenTelemetryPlugin.newBuilder().build(); + +WorkflowServiceStubs service = + WorkflowServiceStubs.newServiceStubs( + WorkflowServiceStubsOptions.newBuilder() + .setPlugins(plugin) + .build()); +WorkflowClient client = WorkflowClient.newInstance(service); +WorkerFactory factory = WorkerFactory.newInstance(client); +``` + +Plugins configured on service stubs propagate to workflow clients and worker factories. By default, the plugin creates an OpenTelemetry SDK with OTLP metric and trace exporters, W3C trace context propagation, an OpenTelemetry-backed Tally metrics scope, OpenTracing-shim client and worker interceptors, and a worker-factory shutdown flush. Buffered Tally metrics are reported before OpenTelemetry providers are force-flushed. Providers are not closed. + +The plugin defaults the OTLP endpoint from `OTEL_EXPORTER_OTLP_ENDPOINT`, then `http://localhost:4317`. It defaults the service name from `OTEL_SERVICE_NAME`, then `temporal-worker`. + +To use an application-owned provider, call `builder.setOpenTelemetry(...)`; in that path, no exporters are created and the plugin only installs the metrics scope, interceptors, and flush hook: + +```java +OpenTelemetryPlugin plugin = + OpenTelemetryPlugin.newBuilder() + .setOpenTelemetry(openTelemetry) + .build(); +``` + +Serverless adapters should disable worker-factory shutdown flushing and run `plugin.newFlushHook()` from their per-invocation cleanup path. If a hook implements `TimedShutdownHook`, pass the remaining cleanup timeout to `run(Duration)`. + +For manual composition or existing integrations, `OpenTelemetryWorker.configure(...)` remains available: + +```java +List shutdownHooks = new ArrayList<>(); + +WorkflowServiceStubsOptions.Builder serviceOptions = WorkflowServiceStubsOptions.newBuilder(); +WorkflowClientOptions.Builder clientOptions = WorkflowClientOptions.newBuilder(); +WorkerFactoryOptions.Builder factoryOptions = WorkerFactoryOptions.newBuilder(); + +OpenTelemetryWorker.configure( + serviceOptions, + clientOptions, + factoryOptions, + shutdownHooks::add); +``` + +Use `OpenTelemetryWorker.configureMetrics(...)`, `OpenTelemetryWorker.configureTracing(...)`, and `OpenTelemetryWorker.configureFlushHook(...)` when you want to compose metrics, tracing, or provider flushing separately. + +To use an application-owned provider with the manual helper: + +```java +OpenTelemetryWorker.configure( + serviceOptions, + clientOptions, + factoryOptions, + shutdownHooks::add, + builder -> builder.setOpenTelemetry(openTelemetry)); +``` diff --git a/contrib/temporal-opentelemetry/build.gradle b/contrib/temporal-opentelemetry/build.gradle new file mode 100644 index 0000000000..d49bd04e05 --- /dev/null +++ b/contrib/temporal-opentelemetry/build.gradle @@ -0,0 +1,32 @@ +description = '''Temporal Java SDK OpenTelemetry Support Module''' + +ext { + otelVersion = '1.25.0' + otShimVersion = "${otelVersion}-alpha" +} + +dependencies { + api platform("io.opentelemetry:opentelemetry-bom:$otelVersion") + + // This module shouldn't carry temporal-sdk with it, especially for situations when users may + // be using a shaded artifact. + compileOnly project(':temporal-serviceclient') + compileOnly project(':temporal-sdk') + compileOnly "javax.annotation:javax.annotation-api:$annotationApiVersion" + + api "com.uber.m3:tally-core:$tallyVersion" + api "io.opentelemetry:opentelemetry-api" + api "io.opentelemetry:opentelemetry-sdk-trace" + + implementation project(':temporal-opentracing') + implementation "io.opentelemetry:opentelemetry-exporter-otlp" + implementation "io.opentelemetry:opentelemetry-opentracing-shim:$otShimVersion" + implementation "io.opentelemetry:opentelemetry-sdk" + implementation "org.slf4j:slf4j-api:$slf4jVersion" + + testImplementation project(':temporal-sdk') + testImplementation project(':temporal-serviceclient') + testImplementation "junit:junit:${junitVersion}" + + testRuntimeOnly group: 'ch.qos.logback', name: 'logback-classic', version: "${logbackVersion}" +} diff --git a/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryFlushHook.java b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryFlushHook.java new file mode 100644 index 0000000000..92a100cf63 --- /dev/null +++ b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryFlushHook.java @@ -0,0 +1,120 @@ +package io.temporal.opentelemetry; + +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.sdk.OpenTelemetrySdk; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.time.Duration; +import java.util.Objects; +import java.util.concurrent.TimeUnit; +import javax.annotation.Nonnull; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** Force-flushes OpenTelemetry providers without closing them. */ +public final class OpenTelemetryFlushHook implements TimedShutdownHook { + private static final Logger log = LoggerFactory.getLogger(OpenTelemetryFlushHook.class); + + private final OpenTelemetry openTelemetry; + private final Duration timeout; + private final MonotonicClock clock; + + public OpenTelemetryFlushHook(@Nonnull OpenTelemetry openTelemetry, @Nonnull Duration timeout) { + this(openTelemetry, timeout, System::nanoTime); + } + + OpenTelemetryFlushHook(OpenTelemetry openTelemetry, Duration timeout, MonotonicClock clock) { + this.openTelemetry = Objects.requireNonNull(openTelemetry, "openTelemetry"); + this.timeout = Objects.requireNonNull(timeout, "timeout"); + this.clock = Objects.requireNonNull(clock, "clock"); + } + + @Override + public void run() { + run(timeout); + } + + @Override + public void run(@Nonnull Duration timeout) { + long deadlineNanos = clock.nanoTime() + min(timeout, this.timeout).toNanos(); + forceFlush(tracerProvider(), deadlineNanos); + forceFlush(meterProvider(), deadlineNanos); + } + + interface MonotonicClock { + long nanoTime(); + } + + private Object tracerProvider() { + if (openTelemetry instanceof OpenTelemetrySdk) { + return ((OpenTelemetrySdk) openTelemetry).getSdkTracerProvider(); + } + return openTelemetry.getTracerProvider(); + } + + private Object meterProvider() { + if (openTelemetry instanceof OpenTelemetrySdk) { + return ((OpenTelemetrySdk) openTelemetry).getSdkMeterProvider(); + } + return openTelemetry.getMeterProvider(); + } + + private void forceFlush(Object provider, long deadlineNanos) { + if (provider == null) { + return; + } + + try { + Method forceFlush = provider.getClass().getMethod("forceFlush"); + Object result = forceFlush.invoke(provider); + join(result, remainingFlushTime(deadlineNanos)); + } catch (NoSuchMethodException e) { + // The OpenTelemetry API no-op providers do not expose forceFlush. + } catch (IllegalAccessException | InvocationTargetException | RuntimeException e) { + log.warn("OpenTelemetry forceFlush failed provider={}", provider.getClass().getName(), e); + } + } + + private void join(Object result, Duration timeout) { + if (result == null) { + return; + } + + try { + Method join = result.getClass().getMethod("join", long.class, TimeUnit.class); + join.invoke(result, timeout.toMillis(), TimeUnit.MILLISECONDS); + } catch (NoSuchMethodException e) { + tryJoinMillis(result, timeout); + } catch (IllegalAccessException | InvocationTargetException | RuntimeException e) { + log.warn("OpenTelemetry forceFlush join failed result={}", result.getClass().getName(), e); + } + } + + private void tryJoinMillis(Object result, Duration timeout) { + try { + Method join = result.getClass().getMethod("join", long.class); + join.invoke(result, timeout.toMillis()); + } catch (NoSuchMethodException e) { + // Some forceFlush result implementations do not expose a blocking join method. + } catch (IllegalAccessException | InvocationTargetException | RuntimeException e) { + log.warn("OpenTelemetry forceFlush join failed result={}", result.getClass().getName(), e); + } + } + + private static Duration requireNonNegative(Duration timeout) { + Objects.requireNonNull(timeout, "timeout"); + return timeout.isNegative() ? Duration.ZERO : timeout; + } + + private Duration remainingFlushTime(long deadlineNanos) { + return requireNonNegative(Duration.ofNanos(deadlineNanos - clock.nanoTime())); + } + + private static Duration min(Duration first, Duration second) { + Duration nonNegativeFirst = requireNonNegative(first); + Duration nonNegativeSecond = requireNonNegative(second); + return nonNegativeFirst.compareTo(nonNegativeSecond) <= 0 + ? nonNegativeFirst + : nonNegativeSecond; + } +} diff --git a/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryPlugin.java b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryPlugin.java new file mode 100644 index 0000000000..2604e205f1 --- /dev/null +++ b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryPlugin.java @@ -0,0 +1,252 @@ +package io.temporal.opentelemetry; + +import com.uber.m3.tally.Scope; +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.sdk.trace.IdGenerator; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.common.Experimental; +import io.temporal.common.SimplePlugin; +import io.temporal.opentracing.OpenTracingOptions; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.WorkerFactory; +import io.temporal.worker.WorkerFactoryOptions; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.function.Consumer; +import javax.annotation.Nonnull; + +/** + * OpenTelemetry plugin for Temporal workers. + * + *

The plugin installs an OpenTelemetry-backed Tally metrics scope on service stubs and + * OpenTelemetry-backed tracing interceptors on workflow clients and worker factories. + */ +@Experimental +public final class OpenTelemetryPlugin extends SimplePlugin { + public static final String NAME = "io.temporal.opentelemetry"; + + private final Map env; + private final OpenTelemetryWorker.TelemetryFactory telemetryFactory; + private final String endpoint; + private final String serviceName; + private final Duration metricsReportInterval; + private final Duration flushTimeout; + private final Runnable flushHook; + private final IdGenerator idGenerator; + private final boolean flushOnWorkerFactoryShutdown; + private final List metricsScopes = new ArrayList<>(); + + private OpenTelemetry openTelemetry; + + private OpenTelemetryPlugin(Builder builder) { + super(NAME); + this.env = builder.env; + this.telemetryFactory = builder.telemetryFactory; + this.openTelemetry = builder.openTelemetry; + this.endpoint = builder.endpoint; + this.serviceName = builder.serviceName; + this.metricsReportInterval = builder.metricsReportInterval; + this.flushTimeout = builder.flushTimeout; + this.flushHook = builder.flushHook; + this.idGenerator = builder.idGenerator; + this.flushOnWorkerFactoryShutdown = builder.flushOnWorkerFactoryShutdown; + } + + public static Builder newBuilder() { + return new Builder(System.getenv()); + } + + public static Builder newBuilder(@Nonnull Map env) { + return new Builder(env); + } + + public String getEndpoint() { + return endpoint == null ? OpenTelemetryWorker.resolveEndpoint(env) : endpoint; + } + + public String getServiceName() { + return serviceName == null ? OpenTelemetryWorker.resolveServiceName(env) : serviceName; + } + + public synchronized OpenTelemetry getOpenTelemetry() { + if (openTelemetry == null) { + openTelemetry = + telemetryFactory.create( + getEndpoint(), getServiceName(), metricsReportInterval, flushTimeout, idGenerator); + } + return openTelemetry; + } + + @Override + public void configureServiceStubs(@Nonnull WorkflowServiceStubsOptions.Builder builder) { + Scope scope = + OpenTelemetryWorker.createMetricsScope( + getOpenTelemetry(), getServiceName(), metricsReportInterval); + synchronized (metricsScopes) { + metricsScopes.add(scope); + } + builder.setMetricsScope(scope); + } + + @Override + public void configureWorkflowClient(@Nonnull WorkflowClientOptions.Builder builder) { + OpenTracingOptions tracingOptions = + OpenTelemetryWorker.createOpenTracingOptions(getOpenTelemetry()); + OpenTelemetryWorker.appendClientInterceptor( + builder, new io.temporal.opentracing.OpenTracingClientInterceptor(tracingOptions)); + } + + @Override + public void configureWorkerFactory(@Nonnull WorkerFactoryOptions.Builder builder) { + OpenTracingOptions tracingOptions = + OpenTelemetryWorker.createOpenTracingOptions(getOpenTelemetry()); + OpenTelemetryWorker.appendWorkerInterceptor( + builder, new io.temporal.opentracing.OpenTracingWorkerInterceptor(tracingOptions)); + } + + @Override + public void shutdownWorkerFactory( + @Nonnull WorkerFactory factory, @Nonnull Consumer next) { + next.accept(factory); + if (flushOnWorkerFactoryShutdown) { + newFlushHook().run(); + } + } + + /** + * Creates a flush hook that reports buffered Tally metrics before force-flushing OpenTelemetry + * providers. + */ + public TimedShutdownHook newFlushHook() { + return new TimedShutdownHook() { + @Override + public void run() { + flush(flushTimeout); + } + + @Override + public void run(@Nonnull Duration timeout) { + flush(timeout); + } + }; + } + + private void flush(Duration timeout) { + for (Scope scope : drainMetricsScopes()) { + new TallyScopeFlushHook(scope).run(); + } + if (flushHook == null) { + new OpenTelemetryFlushHook(getOpenTelemetry(), flushTimeout).run(timeout); + } else { + flushHook.run(); + } + } + + private List drainMetricsScopes() { + synchronized (metricsScopes) { + List scopes = new ArrayList<>(metricsScopes); + metricsScopes.clear(); + return scopes; + } + } + + /** Builder for {@link OpenTelemetryPlugin}. */ + public static final class Builder { + private final Map env; + private OpenTelemetry openTelemetry; + private String endpoint; + private String serviceName; + private Duration metricsReportInterval = OpenTelemetryWorker.DEFAULT_METRICS_REPORT_INTERVAL; + private Duration flushTimeout = Duration.ofSeconds(10); + private Runnable flushHook; + private IdGenerator idGenerator; + private OpenTelemetryWorker.TelemetryFactory telemetryFactory = + new OpenTelemetryWorker.DefaultTelemetryFactory(); + private boolean flushOnWorkerFactoryShutdown = true; + + private Builder(Map env) { + this.env = Objects.requireNonNull(env, "env"); + } + + /** + * Uses an application-owned OpenTelemetry instance instead of creating an SDK and exporters. + */ + public Builder setOpenTelemetry(@Nonnull OpenTelemetry openTelemetry) { + this.openTelemetry = Objects.requireNonNull(openTelemetry, "openTelemetry"); + return this; + } + + /** Sets the OTLP metric and trace exporter endpoint used by the default SDK setup. */ + public Builder setEndpoint(@Nonnull String endpoint) { + this.endpoint = Objects.requireNonNull(endpoint, "endpoint"); + return this; + } + + /** Sets the service name used by the default SDK resource and Temporal metrics reporter. */ + public Builder setServiceName(@Nonnull String serviceName) { + this.serviceName = Objects.requireNonNull(serviceName, "serviceName"); + return this; + } + + /** Sets the interval used by the Tally metrics scope and periodic metric reader. */ + public Builder setMetricsReportInterval(@Nonnull Duration metricsReportInterval) { + this.metricsReportInterval = + OpenTelemetryWorker.requirePositive(metricsReportInterval, "metricsReportInterval"); + return this; + } + + /** Sets how long the OpenTelemetry flush hook waits for provider flushing. */ + public Builder setFlushTimeout(@Nonnull Duration flushTimeout) { + this.flushTimeout = OpenTelemetryWorker.requireNonNegative(flushTimeout, "flushTimeout"); + return this; + } + + /** Overrides the OpenTelemetry provider flush hook. */ + public Builder setFlushHook(@Nonnull Runnable flushHook) { + this.flushHook = Objects.requireNonNull(flushHook, "flushHook"); + return this; + } + + /** + * Controls whether the plugin flushes when a {@link WorkerFactory} is shut down. + * + *

Serverless integrations should disable this and use {@link + * OpenTelemetryPlugin#newFlushHook()} from their per-invocation cleanup path. + */ + public Builder setFlushOnWorkerFactoryShutdown(boolean flushOnWorkerFactoryShutdown) { + this.flushOnWorkerFactoryShutdown = flushOnWorkerFactoryShutdown; + return this; + } + + /** Sets the trace ID generator used by the default SDK setup. */ + public Builder setIdGenerator(@Nonnull IdGenerator idGenerator) { + this.idGenerator = Objects.requireNonNull(idGenerator, "idGenerator"); + return this; + } + + public String getEndpoint() { + return endpoint == null ? OpenTelemetryWorker.resolveEndpoint(env) : endpoint; + } + + public String getServiceName() { + return serviceName == null ? OpenTelemetryWorker.resolveServiceName(env) : serviceName; + } + + Builder setTelemetryFactory(OpenTelemetryWorker.TelemetryFactory telemetryFactory) { + this.telemetryFactory = Objects.requireNonNull(telemetryFactory, "telemetryFactory"); + return this; + } + + public OpenTelemetry createOpenTelemetry() { + return telemetryFactory.create( + getEndpoint(), getServiceName(), metricsReportInterval, flushTimeout, idGenerator); + } + + public OpenTelemetryPlugin build() { + return new OpenTelemetryPlugin(this); + } + } +} diff --git a/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryStatsReporter.java b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryStatsReporter.java new file mode 100644 index 0000000000..7b9836eb74 --- /dev/null +++ b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryStatsReporter.java @@ -0,0 +1,160 @@ +package io.temporal.opentelemetry; + +import com.uber.m3.tally.Capabilities; +import com.uber.m3.tally.CapableOf; +import com.uber.m3.tally.StatsReporter; +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.api.common.Attributes; +import io.opentelemetry.api.common.AttributesBuilder; +import io.opentelemetry.api.metrics.DoubleHistogram; +import io.opentelemetry.api.metrics.LongCounter; +import io.opentelemetry.api.metrics.Meter; +import io.opentelemetry.api.metrics.ObservableDoubleGauge; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.atomic.AtomicReference; +import javax.annotation.Nonnull; + +/** Tally reporter that emits Temporal metrics through OpenTelemetry. */ +public final class OpenTelemetryStatsReporter implements StatsReporter { + private final Meter meter; + private final String serviceName; + private final ConcurrentMap counters = new ConcurrentHashMap<>(); + private final ConcurrentMap timers = new ConcurrentHashMap<>(); + private final ConcurrentMap gauges = new ConcurrentHashMap<>(); + + public OpenTelemetryStatsReporter( + @Nonnull OpenTelemetry openTelemetry, @Nonnull String serviceName) { + this.meter = Objects.requireNonNull(openTelemetry, "openTelemetry").getMeter("io.temporal"); + this.serviceName = Objects.requireNonNull(serviceName, "serviceName"); + } + + @Override + public Capabilities capabilities() { + return CapableOf.REPORTING; + } + + @Override + public void flush() { + // OpenTelemetry SDK flushing is handled by OpenTelemetryWorker's shutdown hook. + } + + @Override + public void close() { + flush(); + } + + @Override + public void reportCounter(String name, Map tags, long value) { + LongCounter counter = counters.computeIfAbsent(name, key -> meter.counterBuilder(key).build()); + counter.add(value, attributes(tags)); + } + + @Override + public void reportGauge(String name, Map tags, double value) { + MetricKey key = new MetricKey(name, tags); + GaugeHolder holder = + gauges.computeIfAbsent( + key, + metricKey -> { + AtomicReference current = new AtomicReference<>(0.0); + ObservableDoubleGauge gauge = + meter + .gaugeBuilder(metricKey.name) + .buildWithCallback( + measurement -> + measurement.record(current.get(), attributes(metricKey.tags))); + return new GaugeHolder(current, gauge); + }); + holder.current.set(value); + } + + @Override + public void reportTimer( + String name, Map tags, com.uber.m3.util.Duration interval) { + DoubleHistogram timer = + timers.computeIfAbsent(name, key -> meter.histogramBuilder(key).setUnit("ms").build()); + timer.record(interval.getNanos() / 1_000_000.0, attributes(tags)); + } + + @Override + @SuppressWarnings("deprecation") + public void reportHistogramValueSamples( + String name, + Map tags, + com.uber.m3.tally.Buckets buckets, + double bucketLowerBound, + double bucketUpperBound, + long samples) { + // Tally reports pre-aggregated bucket samples, while the OpenTelemetry API records raw values. + } + + @Override + @SuppressWarnings("deprecation") + public void reportHistogramDurationSamples( + String name, + Map tags, + com.uber.m3.tally.Buckets buckets, + com.uber.m3.util.Duration bucketLowerBound, + com.uber.m3.util.Duration bucketUpperBound, + long samples) { + // Tally reports pre-aggregated bucket samples, while the OpenTelemetry API records raw values. + } + + private Attributes attributes(Map tags) { + AttributesBuilder builder = Attributes.builder(); + builder.put("service.name", serviceName); + if (tags != null) { + for (Map.Entry entry : tags.entrySet()) { + if (entry.getKey() != null && entry.getValue() != null) { + builder.put(entry.getKey(), entry.getValue()); + } + } + } + return builder.build(); + } + + private static final class GaugeHolder { + private final AtomicReference current; + + @SuppressWarnings("unused") + private final ObservableDoubleGauge gauge; + + private GaugeHolder(AtomicReference current, ObservableDoubleGauge gauge) { + this.current = current; + this.gauge = gauge; + } + } + + private static final class MetricKey { + private final String name; + private final Map tags; + + private MetricKey(String name, Map tags) { + this.name = Objects.requireNonNull(name, "name"); + this.tags = + tags == null ? Collections.emptyMap() : Collections.unmodifiableMap(new HashMap<>(tags)); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + MetricKey metricKey = (MetricKey) o; + return name.equals(metricKey.name) && tags.equals(metricKey.tags); + } + + @Override + public int hashCode() { + return Objects.hash(name, tags); + } + } +} diff --git a/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryWorker.java b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryWorker.java new file mode 100644 index 0000000000..90e7697aac --- /dev/null +++ b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/OpenTelemetryWorker.java @@ -0,0 +1,400 @@ +package io.temporal.opentelemetry; + +import com.uber.m3.tally.RootScopeBuilder; +import com.uber.m3.tally.Scope; +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.api.common.AttributeKey; +import io.opentelemetry.api.common.Attributes; +import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator; +import io.opentelemetry.context.propagation.ContextPropagators; +import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter; +import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter; +import io.opentelemetry.opentracingshim.OpenTracingShim; +import io.opentelemetry.sdk.OpenTelemetrySdk; +import io.opentelemetry.sdk.metrics.SdkMeterProvider; +import io.opentelemetry.sdk.metrics.export.MetricExporter; +import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader; +import io.opentelemetry.sdk.resources.Resource; +import io.opentelemetry.sdk.trace.IdGenerator; +import io.opentelemetry.sdk.trace.SdkTracerProvider; +import io.opentelemetry.sdk.trace.SdkTracerProviderBuilder; +import io.opentelemetry.sdk.trace.export.BatchSpanProcessor; +import io.opentelemetry.sdk.trace.export.SpanExporter; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.common.interceptors.WorkerInterceptor; +import io.temporal.common.interceptors.WorkflowClientInterceptor; +import io.temporal.opentracing.OpenTracingClientInterceptor; +import io.temporal.opentracing.OpenTracingOptions; +import io.temporal.opentracing.OpenTracingWorkerInterceptor; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.WorkerFactoryOptions; +import java.time.Duration; +import java.util.Arrays; +import java.util.Map; +import java.util.Objects; +import java.util.function.Consumer; +import javax.annotation.Nonnull; + +/** OpenTelemetry helper for Temporal workers. */ +public final class OpenTelemetryWorker { + public static final String OTEL_EXPORTER_OTLP_ENDPOINT = "OTEL_EXPORTER_OTLP_ENDPOINT"; + public static final String OTEL_SERVICE_NAME = "OTEL_SERVICE_NAME"; + public static final String DEFAULT_OTLP_ENDPOINT = "http://localhost:4317"; + public static final String DEFAULT_SERVICE_NAME = "temporal-worker"; + + static final Duration DEFAULT_METRICS_REPORT_INTERVAL = Duration.ofSeconds(1); + + private static final AttributeKey SERVICE_NAME_ATTRIBUTE = + AttributeKey.stringKey("service.name"); + + private OpenTelemetryWorker() {} + + public static Builder newBuilder() { + return new Builder(System.getenv()); + } + + public static Builder newBuilder(@Nonnull Map env) { + return new Builder(env); + } + + public static String getDefaultEndpoint() { + return resolveEndpoint(System.getenv()); + } + + public static String getDefaultServiceName() { + return resolveServiceName(System.getenv()); + } + + public static void configure( + @Nonnull WorkflowServiceStubsOptions.Builder serviceStubsOptions, + @Nonnull WorkflowClientOptions.Builder clientOptions, + @Nonnull WorkerFactoryOptions.Builder workerFactoryOptions, + @Nonnull Consumer addShutdownHook) { + configure( + serviceStubsOptions, clientOptions, workerFactoryOptions, addShutdownHook, builder -> {}); + } + + /** + * Configures metrics, tracing interceptors, and flushing for a Temporal worker. + * + *

By default this method creates an OpenTelemetry SDK with OTLP trace and metric exporters. If + * {@link Builder#setOpenTelemetry(OpenTelemetry)} is used, the provided instance is used instead + * and exporters are not created. + */ + public static void configure( + @Nonnull WorkflowServiceStubsOptions.Builder serviceStubsOptions, + @Nonnull WorkflowClientOptions.Builder clientOptions, + @Nonnull WorkerFactoryOptions.Builder workerFactoryOptions, + @Nonnull Consumer addShutdownHook, + @Nonnull Consumer configure) { + Builder builder = newBuilder(); + Objects.requireNonNull(configure, "configure").accept(builder); + builder.apply(serviceStubsOptions, clientOptions, workerFactoryOptions, addShutdownHook); + } + + /** + * Configures Temporal metrics with the default service name and reporting interval. + * + *

This helper installs the metrics scope and registers a hook that reports buffered Tally + * metrics before provider flushing. It does not configure tracing interceptors or an + * OpenTelemetry provider flush hook. + */ + public static void configureMetrics( + @Nonnull WorkflowServiceStubsOptions.Builder serviceStubsOptions, + @Nonnull Consumer addShutdownHook, + @Nonnull OpenTelemetry openTelemetry) { + configureMetrics( + serviceStubsOptions, + addShutdownHook, + openTelemetry, + getDefaultServiceName(), + DEFAULT_METRICS_REPORT_INTERVAL); + } + + /** + * Configures Temporal metrics with an application-owned OpenTelemetry provider. + * + *

This helper installs the metrics scope and registers a hook that reports buffered Tally + * metrics before provider flushing. It does not configure tracing interceptors or an + * OpenTelemetry provider flush hook. + */ + public static void configureMetrics( + @Nonnull WorkflowServiceStubsOptions.Builder serviceStubsOptions, + @Nonnull Consumer addShutdownHook, + @Nonnull OpenTelemetry openTelemetry, + @Nonnull String serviceName, + @Nonnull Duration reportInterval) { + Objects.requireNonNull(serviceStubsOptions, "serviceStubsOptions"); + Objects.requireNonNull(addShutdownHook, "addShutdownHook"); + Scope scope = createMetricsScope(openTelemetry, serviceName, reportInterval); + serviceStubsOptions.setMetricsScope(scope); + addShutdownHook.accept(new TallyScopeFlushHook(scope)); + } + + /** + * Configures Temporal tracing interceptors with an application-owned OpenTelemetry provider. + * + *

This helper only installs tracing interceptors. It does not configure metrics or register a + * flush hook. + */ + public static void configureTracing( + @Nonnull WorkflowClientOptions.Builder clientOptions, + @Nonnull WorkerFactoryOptions.Builder workerFactoryOptions, + @Nonnull OpenTelemetry openTelemetry) { + Objects.requireNonNull(clientOptions, "clientOptions"); + Objects.requireNonNull(workerFactoryOptions, "workerFactoryOptions"); + OpenTracingOptions tracingOptions = + createOpenTracingOptions(Objects.requireNonNull(openTelemetry, "openTelemetry")); + appendClientInterceptor(clientOptions, new OpenTracingClientInterceptor(tracingOptions)); + appendWorkerInterceptor(workerFactoryOptions, new OpenTracingWorkerInterceptor(tracingOptions)); + } + + /** + * Registers an OpenTelemetry force-flush hook. + * + *

This helper only registers the flush hook. It does not configure metrics or tracing. + */ + public static void configureFlushHook( + @Nonnull Consumer addShutdownHook, + @Nonnull OpenTelemetry openTelemetry, + @Nonnull Duration flushTimeout) { + Objects.requireNonNull(addShutdownHook, "addShutdownHook"); + addShutdownHook.accept( + new OpenTelemetryFlushHook( + Objects.requireNonNull(openTelemetry, "openTelemetry"), + requireNonNegative(flushTimeout, "flushTimeout"))); + } + + static String resolveEndpoint(Map env) { + String endpoint = nonEmptyEnv(env, OTEL_EXPORTER_OTLP_ENDPOINT); + return endpoint == null ? DEFAULT_OTLP_ENDPOINT : endpoint; + } + + static String resolveServiceName(Map env) { + String serviceName = nonEmptyEnv(env, OTEL_SERVICE_NAME); + return serviceName == null ? DEFAULT_SERVICE_NAME : serviceName; + } + + public static final class Builder { + private final Map env; + private OpenTelemetry openTelemetry; + private String endpoint; + private String serviceName; + private Duration metricsReportInterval = DEFAULT_METRICS_REPORT_INTERVAL; + private Duration flushTimeout = Duration.ofSeconds(10); + private Runnable flushHook; + private IdGenerator idGenerator; + private TelemetryFactory telemetryFactory = new DefaultTelemetryFactory(); + + private Builder(Map env) { + this.env = Objects.requireNonNull(env, "env"); + } + + /** + * Uses an application-owned OpenTelemetry instance instead of creating an SDK and exporters. + */ + public Builder setOpenTelemetry(@Nonnull OpenTelemetry openTelemetry) { + this.openTelemetry = Objects.requireNonNull(openTelemetry, "openTelemetry"); + return this; + } + + /** Sets the OTLP metric and trace exporter endpoint used by the default SDK setup. */ + public Builder setEndpoint(@Nonnull String endpoint) { + this.endpoint = Objects.requireNonNull(endpoint, "endpoint"); + return this; + } + + /** Sets the service name used by the default SDK resource and Temporal metrics reporter. */ + public Builder setServiceName(@Nonnull String serviceName) { + this.serviceName = Objects.requireNonNull(serviceName, "serviceName"); + return this; + } + + /** Sets the interval used by the Tally metrics scope and periodic metric reader. */ + public Builder setMetricsReportInterval(@Nonnull Duration metricsReportInterval) { + this.metricsReportInterval = requirePositive(metricsReportInterval, "metricsReportInterval"); + return this; + } + + /** Sets how long the OpenTelemetry flush hook waits for provider flushing. */ + public Builder setFlushTimeout(@Nonnull Duration flushTimeout) { + this.flushTimeout = requireNonNegative(flushTimeout, "flushTimeout"); + return this; + } + + /** Overrides the OpenTelemetry provider flush hook. */ + public Builder setFlushHook(@Nonnull Runnable flushHook) { + this.flushHook = Objects.requireNonNull(flushHook, "flushHook"); + return this; + } + + /** Sets the trace ID generator used by the default SDK setup. */ + public Builder setIdGenerator(@Nonnull IdGenerator idGenerator) { + this.idGenerator = Objects.requireNonNull(idGenerator, "idGenerator"); + return this; + } + + public String getEndpoint() { + return endpoint == null ? resolveEndpoint(env) : endpoint; + } + + public String getServiceName() { + return serviceName == null ? resolveServiceName(env) : serviceName; + } + + Builder setTelemetryFactory(TelemetryFactory telemetryFactory) { + this.telemetryFactory = Objects.requireNonNull(telemetryFactory, "telemetryFactory"); + return this; + } + + public OpenTelemetry createOpenTelemetry() { + return telemetryFactory.create( + getEndpoint(), getServiceName(), metricsReportInterval, flushTimeout, idGenerator); + } + + public void apply( + @Nonnull WorkflowServiceStubsOptions.Builder serviceStubsOptions, + @Nonnull WorkflowClientOptions.Builder clientOptions, + @Nonnull WorkerFactoryOptions.Builder workerFactoryOptions, + @Nonnull Consumer addShutdownHook) { + OpenTelemetry resolvedOpenTelemetry = + openTelemetry == null ? createOpenTelemetry() : openTelemetry; + configureMetrics( + serviceStubsOptions, + addShutdownHook, + resolvedOpenTelemetry, + getServiceName(), + metricsReportInterval); + configureTracing(clientOptions, workerFactoryOptions, resolvedOpenTelemetry); + if (flushHook == null) { + configureFlushHook(addShutdownHook, resolvedOpenTelemetry, flushTimeout); + } else { + Objects.requireNonNull(addShutdownHook, "addShutdownHook").accept(flushHook); + } + } + } + + interface TelemetryFactory { + OpenTelemetry create( + String endpoint, + String serviceName, + Duration metricsReportInterval, + Duration flushTimeout, + IdGenerator idGenerator); + } + + static final class DefaultTelemetryFactory implements TelemetryFactory { + @Override + public OpenTelemetry create( + String endpoint, + String serviceName, + Duration metricsReportInterval, + Duration flushTimeout, + IdGenerator idGenerator) { + Resource resource = + Resource.getDefault() + .merge(Resource.create(Attributes.of(SERVICE_NAME_ATTRIBUTE, serviceName))); + MetricExporter metricExporter = + OtlpGrpcMetricExporter.builder().setEndpoint(endpoint).build(); + SpanExporter spanExporter = OtlpGrpcSpanExporter.builder().setEndpoint(endpoint).build(); + + SdkMeterProvider meterProvider = + SdkMeterProvider.builder() + .setResource(resource) + .registerMetricReader( + PeriodicMetricReader.builder(metricExporter) + .setInterval(metricsReportInterval) + .build()) + .build(); + SdkTracerProviderBuilder tracerProviderBuilder = + SdkTracerProvider.builder().setResource(resource); + if (idGenerator != null) { + tracerProviderBuilder.setIdGenerator(idGenerator); + } + SdkTracerProvider tracerProvider = + tracerProviderBuilder + .addSpanProcessor( + BatchSpanProcessor.builder(spanExporter).setExporterTimeout(flushTimeout).build()) + .build(); + + return OpenTelemetrySdk.builder() + .setMeterProvider(meterProvider) + .setTracerProvider(tracerProvider) + .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance())) + .build(); + } + } + + private static String nonEmptyEnv(Map env, String name) { + if (env == null) { + return null; + } + String value = env.get(name); + return value == null || value.trim().isEmpty() ? null : value; + } + + static Scope createMetricsScope( + @Nonnull OpenTelemetry openTelemetry, + @Nonnull String serviceName, + @Nonnull Duration reportInterval) { + OpenTelemetryStatsReporter reporter = + new OpenTelemetryStatsReporter( + Objects.requireNonNull(openTelemetry, "openTelemetry"), + Objects.requireNonNull(serviceName, "serviceName")); + return new RootScopeBuilder() + .reporter(reporter) + .reportEvery( + com.uber.m3.util.Duration.ofMillis( + requirePositive(reportInterval, "reportInterval").toMillis())); + } + + static OpenTracingOptions createOpenTracingOptions(@Nonnull OpenTelemetry openTelemetry) { + return OpenTracingOptions.newBuilder() + .setTracer( + OpenTracingShim.createTracerShim( + Objects.requireNonNull(openTelemetry, "openTelemetry"))) + .build(); + } + + static void appendClientInterceptor( + WorkflowClientOptions.Builder options, WorkflowClientInterceptor interceptor) { + WorkflowClientOptions raw = options.build(); + WorkflowClientInterceptor[] existing = raw.getInterceptors(); + int existingLength = existing == null ? 0 : existing.length; + WorkflowClientInterceptor[] interceptors = + existingLength == 0 + ? new WorkflowClientInterceptor[1] + : Arrays.copyOf(existing, existingLength + 1); + interceptors[existingLength] = interceptor; + options.setInterceptors(interceptors); + } + + static void appendWorkerInterceptor( + WorkerFactoryOptions.Builder options, WorkerInterceptor interceptor) { + WorkerFactoryOptions raw = options.build(); + WorkerInterceptor[] existing = raw.getWorkerInterceptors(); + int existingLength = existing == null ? 0 : existing.length; + WorkerInterceptor[] interceptors = + existingLength == 0 + ? new WorkerInterceptor[1] + : Arrays.copyOf(existing, existingLength + 1); + interceptors[existingLength] = interceptor; + options.setWorkerInterceptors(interceptors); + } + + static Duration requirePositive(Duration value, String name) { + Objects.requireNonNull(value, name); + if (value.isZero() || value.isNegative()) { + throw new IllegalArgumentException(name + " must be positive"); + } + return value; + } + + static Duration requireNonNegative(Duration value, String name) { + Objects.requireNonNull(value, name); + if (value.isNegative()) { + throw new IllegalArgumentException(name + " must not be negative"); + } + return value; + } +} diff --git a/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/TallyScopeFlushHook.java b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/TallyScopeFlushHook.java new file mode 100644 index 0000000000..9f75394db0 --- /dev/null +++ b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/TallyScopeFlushHook.java @@ -0,0 +1,34 @@ +package io.temporal.opentelemetry; + +import com.uber.m3.tally.Scope; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.util.Objects; +import javax.annotation.Nonnull; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** Reports buffered Tally metrics without closing the scope. */ +public final class TallyScopeFlushHook implements Runnable { + private static final Logger log = LoggerFactory.getLogger(TallyScopeFlushHook.class); + + private final Scope scope; + + public TallyScopeFlushHook(@Nonnull Scope scope) { + this.scope = Objects.requireNonNull(scope, "scope"); + } + + @Override + public void run() { + try { + Method reportLoopIteration = scope.getClass().getDeclaredMethod("reportLoopIteration"); + reportLoopIteration.setAccessible(true); + reportLoopIteration.invoke(scope); + } catch (NoSuchMethodException + | IllegalAccessException + | InvocationTargetException + | RuntimeException e) { + log.warn("Tally scope flush failed scope={}", scope.getClass().getName(), e); + } + } +} diff --git a/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/TimedShutdownHook.java b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/TimedShutdownHook.java new file mode 100644 index 0000000000..0bc5a800b7 --- /dev/null +++ b/contrib/temporal-opentelemetry/src/main/java/io/temporal/opentelemetry/TimedShutdownHook.java @@ -0,0 +1,9 @@ +package io.temporal.opentelemetry; + +import java.time.Duration; +import javax.annotation.Nonnull; + +/** Shutdown hook that can use the caller's remaining cleanup timeout. */ +public interface TimedShutdownHook extends Runnable { + void run(@Nonnull Duration timeout); +} diff --git a/contrib/temporal-opentelemetry/src/test/java/io/temporal/opentelemetry/OpenTelemetryPluginTest.java b/contrib/temporal-opentelemetry/src/test/java/io/temporal/opentelemetry/OpenTelemetryPluginTest.java new file mode 100644 index 0000000000..2f63b88ae8 --- /dev/null +++ b/contrib/temporal-opentelemetry/src/test/java/io/temporal/opentelemetry/OpenTelemetryPluginTest.java @@ -0,0 +1,211 @@ +package io.temporal.opentelemetry; + +import static org.junit.Assert.*; + +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.api.metrics.MeterBuilder; +import io.opentelemetry.api.metrics.MeterProvider; +import io.opentelemetry.api.trace.Tracer; +import io.opentelemetry.api.trace.TracerProvider; +import io.opentelemetry.context.propagation.ContextPropagators; +import io.opentelemetry.sdk.common.CompletableResultCode; +import io.opentelemetry.sdk.trace.IdGenerator; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.WorkerFactoryOptions; +import java.io.Closeable; +import java.time.Duration; +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.Test; + +public class OpenTelemetryPluginTest { + @Test + public void defaultsResolveEndpointAndServiceName() { + assertEquals( + "http://localhost:4317", OpenTelemetryPlugin.newBuilder(new HashMap<>()).getEndpoint()); + assertEquals( + "temporal-worker", OpenTelemetryPlugin.newBuilder(new HashMap<>()).getServiceName()); + + Map env = new HashMap<>(); + env.put(OpenTelemetryWorker.OTEL_EXPORTER_OTLP_ENDPOINT, "http://collector:4317"); + env.put(OpenTelemetryWorker.OTEL_SERVICE_NAME, "service-name"); + + assertEquals("http://collector:4317", OpenTelemetryPlugin.newBuilder(env).getEndpoint()); + assertEquals("service-name", OpenTelemetryPlugin.newBuilder(env).getServiceName()); + } + + @Test + public void installsMetricsScopeAndTracingInterceptors() { + RecordingTelemetryFactory factory = new RecordingTelemetryFactory(); + OpenTelemetryPlugin plugin = + OpenTelemetryPlugin.newBuilder(new HashMap<>()).setTelemetryFactory(factory).build(); + Config config = new Config(); + + plugin.configureServiceStubs(config.serviceStubsOptions); + plugin.configureWorkflowClient(config.clientOptions); + plugin.configureWorkerFactory(config.workerFactoryOptions); + + assertEquals(1, factory.creates.get()); + assertNotNull(config.serviceStubsOptions.build().getMetricsScope()); + assertEquals(1, config.clientOptions.build().getInterceptors().length); + assertEquals(1, config.workerFactoryOptions.build().getWorkerInterceptors().length); + } + + @Test + public void customOpenTelemetryBypassesExporterCreation() { + RecordingTelemetryFactory factory = new RecordingTelemetryFactory(); + OpenTelemetryPlugin plugin = + OpenTelemetryPlugin.newBuilder(new HashMap<>()) + .setTelemetryFactory(factory) + .setOpenTelemetry(OpenTelemetry.noop()) + .build(); + Config config = new Config(); + + plugin.configureServiceStubs(config.serviceStubsOptions); + plugin.configureWorkflowClient(config.clientOptions); + plugin.configureWorkerFactory(config.workerFactoryOptions); + + assertEquals(0, factory.creates.get()); + assertNotNull(config.serviceStubsOptions.build().getMetricsScope()); + assertEquals(1, config.clientOptions.build().getInterceptors().length); + assertEquals(1, config.workerFactoryOptions.build().getWorkerInterceptors().length); + } + + @Test + public void flushHookForceFlushesProvidersWithoutClosingThem() { + CountingOpenTelemetry openTelemetry = new CountingOpenTelemetry(); + OpenTelemetryPlugin plugin = + OpenTelemetryPlugin.newBuilder(new HashMap<>()).setOpenTelemetry(openTelemetry).build(); + Config config = new Config(); + plugin.configureServiceStubs(config.serviceStubsOptions); + + plugin.newFlushHook().run(Duration.ofMillis(100)); + + assertEquals(1, openTelemetry.tracerProvider.flushes.get()); + assertEquals(1, openTelemetry.meterProvider.flushes.get()); + assertEquals(0, openTelemetry.tracerProvider.closes.get()); + assertEquals(0, openTelemetry.meterProvider.closes.get()); + } + + @Test + public void workerFactoryShutdownFlushCanBeDisabledForServerlessRuntimes() { + CountingOpenTelemetry openTelemetry = new CountingOpenTelemetry(); + OpenTelemetryPlugin plugin = + OpenTelemetryPlugin.newBuilder(new HashMap<>()) + .setOpenTelemetry(openTelemetry) + .setFlushOnWorkerFactoryShutdown(false) + .build(); + AtomicInteger shutdowns = new AtomicInteger(); + + plugin.shutdownWorkerFactory(null, factory -> shutdowns.incrementAndGet()); + + assertEquals(1, shutdowns.get()); + assertEquals(0, openTelemetry.tracerProvider.flushes.get()); + assertEquals(0, openTelemetry.meterProvider.flushes.get()); + } + + @Test + public void workerFactoryShutdownFlushesByDefault() { + CountingOpenTelemetry openTelemetry = new CountingOpenTelemetry(); + OpenTelemetryPlugin plugin = + OpenTelemetryPlugin.newBuilder(new HashMap<>()).setOpenTelemetry(openTelemetry).build(); + AtomicInteger shutdowns = new AtomicInteger(); + + plugin.shutdownWorkerFactory(null, factory -> shutdowns.incrementAndGet()); + + assertEquals(1, shutdowns.get()); + assertEquals(1, openTelemetry.tracerProvider.flushes.get()); + assertEquals(1, openTelemetry.meterProvider.flushes.get()); + } + + private static final class Config { + private final WorkflowServiceStubsOptions.Builder serviceStubsOptions = + WorkflowServiceStubsOptions.newBuilder(); + private final WorkflowClientOptions.Builder clientOptions = WorkflowClientOptions.newBuilder(); + private final WorkerFactoryOptions.Builder workerFactoryOptions = + WorkerFactoryOptions.newBuilder(); + } + + private static final class RecordingTelemetryFactory + implements OpenTelemetryWorker.TelemetryFactory { + private final AtomicInteger creates = new AtomicInteger(); + + @Override + public OpenTelemetry create( + String endpoint, + String serviceName, + Duration metricsReportInterval, + Duration flushTimeout, + IdGenerator idGenerator) { + creates.incrementAndGet(); + return OpenTelemetry.noop(); + } + } + + private static final class CountingOpenTelemetry implements OpenTelemetry { + private final CountingTracerProvider tracerProvider = new CountingTracerProvider(); + private final CountingMeterProvider meterProvider = new CountingMeterProvider(); + + @Override + public TracerProvider getTracerProvider() { + return tracerProvider; + } + + @Override + public MeterProvider getMeterProvider() { + return meterProvider; + } + + @Override + public ContextPropagators getPropagators() { + return ContextPropagators.noop(); + } + } + + public static final class CountingTracerProvider implements TracerProvider, Closeable { + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public Tracer get(String instrumentationName) { + return TracerProvider.noop().get(instrumentationName); + } + + @Override + public Tracer get(String instrumentationName, String instrumentationVersion) { + return TracerProvider.noop().get(instrumentationName, instrumentationVersion); + } + + public CompletableResultCode forceFlush() { + flushes.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } + + public static final class CountingMeterProvider implements MeterProvider, Closeable { + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public MeterBuilder meterBuilder(String instrumentationName) { + return MeterProvider.noop().meterBuilder(instrumentationName); + } + + public CompletableResultCode forceFlush() { + flushes.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } +} diff --git a/contrib/temporal-opentelemetry/src/test/java/io/temporal/opentelemetry/OpenTelemetryWorkerTest.java b/contrib/temporal-opentelemetry/src/test/java/io/temporal/opentelemetry/OpenTelemetryWorkerTest.java new file mode 100644 index 0000000000..f2e158989b --- /dev/null +++ b/contrib/temporal-opentelemetry/src/test/java/io/temporal/opentelemetry/OpenTelemetryWorkerTest.java @@ -0,0 +1,618 @@ +package io.temporal.opentelemetry; + +import static org.junit.Assert.*; + +import com.uber.m3.tally.Capabilities; +import com.uber.m3.tally.CapableOf; +import com.uber.m3.tally.RootScopeBuilder; +import com.uber.m3.tally.Scope; +import com.uber.m3.tally.StatsReporter; +import io.opentelemetry.api.OpenTelemetry; +import io.opentelemetry.api.metrics.MeterBuilder; +import io.opentelemetry.api.metrics.MeterProvider; +import io.opentelemetry.api.trace.Span; +import io.opentelemetry.api.trace.SpanContext; +import io.opentelemetry.api.trace.TraceFlags; +import io.opentelemetry.api.trace.TraceState; +import io.opentelemetry.api.trace.Tracer; +import io.opentelemetry.api.trace.TracerProvider; +import io.opentelemetry.context.propagation.ContextPropagators; +import io.opentelemetry.sdk.OpenTelemetrySdk; +import io.opentelemetry.sdk.common.CompletableResultCode; +import io.opentelemetry.sdk.metrics.Aggregation; +import io.opentelemetry.sdk.metrics.InstrumentType; +import io.opentelemetry.sdk.metrics.SdkMeterProvider; +import io.opentelemetry.sdk.metrics.data.AggregationTemporality; +import io.opentelemetry.sdk.metrics.export.CollectionRegistration; +import io.opentelemetry.sdk.metrics.export.DefaultAggregationSelector; +import io.opentelemetry.sdk.metrics.export.MetricReader; +import io.opentelemetry.sdk.trace.IdGenerator; +import io.opentelemetry.sdk.trace.ReadWriteSpan; +import io.opentelemetry.sdk.trace.ReadableSpan; +import io.opentelemetry.sdk.trace.SdkTracerProvider; +import io.opentelemetry.sdk.trace.SpanProcessor; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.WorkerFactoryOptions; +import java.io.Closeable; +import java.time.Duration; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import org.junit.Test; + +public class OpenTelemetryWorkerTest { + @Test + public void defaultsResolveEndpointAndServiceName() { + assertEquals( + "http://localhost:4317", OpenTelemetryWorker.newBuilder(new HashMap<>()).getEndpoint()); + assertEquals( + "temporal-worker", OpenTelemetryWorker.newBuilder(new HashMap<>()).getServiceName()); + + Map env = new HashMap<>(); + env.put(OpenTelemetryWorker.OTEL_EXPORTER_OTLP_ENDPOINT, "http://collector:4317"); + assertEquals("http://collector:4317", OpenTelemetryWorker.newBuilder(env).getEndpoint()); + assertEquals("temporal-worker", OpenTelemetryWorker.newBuilder(env).getServiceName()); + + env.put(OpenTelemetryWorker.OTEL_SERVICE_NAME, "explicit-service"); + assertEquals("explicit-service", OpenTelemetryWorker.newBuilder(env).getServiceName()); + } + + @Test + public void defaultFactoryCreatesExporterBackedSdkProviders() { + OpenTelemetry openTelemetry = + OpenTelemetryWorker.newBuilder(new HashMap<>()).createOpenTelemetry(); + + assertTrue(openTelemetry instanceof OpenTelemetrySdk); + + OpenTelemetrySdk sdk = (OpenTelemetrySdk) openTelemetry; + assertNotNull(sdk.getSdkTracerProvider()); + assertNotNull(sdk.getSdkMeterProvider()); + sdk.shutdown().join(1, TimeUnit.SECONDS); + } + + @Test + public void defaultFactoryConfiguresTraceContextPropagator() { + OpenTelemetry openTelemetry = + OpenTelemetryWorker.newBuilder(new HashMap<>()).createOpenTelemetry(); + + assertTrue(openTelemetry instanceof OpenTelemetrySdk); + + OpenTelemetrySdk sdk = (OpenTelemetrySdk) openTelemetry; + String traceId = "00000000000000000000000000000001"; + String spanId = "0000000000000002"; + Map carrier = new HashMap<>(); + sdk.getPropagators() + .getTextMapPropagator() + .inject( + io.opentelemetry.context.Context.root() + .with( + Span.wrap( + SpanContext.create( + traceId, spanId, TraceFlags.getSampled(), TraceState.getDefault()))), + carrier, + (map, key, value) -> map.put(key, value)); + + assertEquals("00-" + traceId + "-" + spanId + "-01", carrier.get("traceparent")); + sdk.shutdown().join(1, TimeUnit.SECONDS); + } + + @Test + public void exporterFactoryReceivesResolvedEndpointAndServiceName() { + Map env = new HashMap<>(); + env.put(OpenTelemetryWorker.OTEL_EXPORTER_OTLP_ENDPOINT, "http://collector:4317"); + env.put(OpenTelemetryWorker.OTEL_SERVICE_NAME, "service-name"); + RecordingTelemetryFactory factory = new RecordingTelemetryFactory(); + Config config = new Config(); + + OpenTelemetryWorker.newBuilder(env) + .setTelemetryFactory(factory) + .apply( + config.serviceStubsOptions, + config.clientOptions, + config.workerFactoryOptions, + config.shutdownHooks::add); + + assertEquals(1, factory.creates.get()); + assertEquals("http://collector:4317", factory.endpoint); + assertEquals("service-name", factory.serviceName); + } + + @Test + public void customEndpointAndServiceNameAreUsedByExporterFactory() { + RecordingTelemetryFactory factory = new RecordingTelemetryFactory(); + Config config = new Config(); + + OpenTelemetryWorker.newBuilder(new HashMap<>()) + .setTelemetryFactory(factory) + .setEndpoint("http://custom-collector:4317") + .setServiceName("custom-service") + .setMetricsReportInterval(Duration.ofSeconds(3)) + .setFlushTimeout(Duration.ofSeconds(4)) + .apply( + config.serviceStubsOptions, + config.clientOptions, + config.workerFactoryOptions, + config.shutdownHooks::add); + + assertEquals(1, factory.creates.get()); + assertEquals("http://custom-collector:4317", factory.endpoint); + assertEquals("custom-service", factory.serviceName); + assertEquals(Duration.ofSeconds(3), factory.metricsReportInterval); + assertEquals(Duration.ofSeconds(4), factory.flushTimeout); + } + + @Test + public void metricsScopeAndTracingInterceptorsAreInstalled() { + Config config = new Config(); + + OpenTelemetryWorker.configure( + config.serviceStubsOptions, + config.clientOptions, + config.workerFactoryOptions, + config.shutdownHooks::add, + builder -> builder.setOpenTelemetry(OpenTelemetry.noop()).setFlushHook(() -> {})); + + assertNotNull(config.serviceStubsOptions.build().getMetricsScope()); + assertEquals(1, config.clientOptions.build().getInterceptors().length); + assertEquals(1, config.workerFactoryOptions.build().getWorkerInterceptors().length); + } + + @Test + public void configureRegistersTallyFlushBeforeOpenTelemetryFlush() { + Config config = new Config(); + + OpenTelemetryWorker.configure( + config.serviceStubsOptions, + config.clientOptions, + config.workerFactoryOptions, + config.shutdownHooks::add, + builder -> builder.setOpenTelemetry(OpenTelemetry.noop())); + + assertEquals(2, config.shutdownHooks.size()); + assertTrue(config.shutdownHooks.get(0) instanceof TallyScopeFlushHook); + assertTrue(config.shutdownHooks.get(1) instanceof OpenTelemetryFlushHook); + } + + @Test + public void metricsOnlyInstallsScopeWithoutTracingInterceptors() { + Config config = new Config(); + + OpenTelemetryWorker.configureMetrics( + config.serviceStubsOptions, config.shutdownHooks::add, OpenTelemetry.noop()); + + assertNotNull(config.serviceStubsOptions.build().getMetricsScope()); + assertEquals(0, clientInterceptorCount(config.clientOptions)); + assertEquals(0, workerInterceptorCount(config.workerFactoryOptions)); + } + + @Test + public void tallyScopeFlushHookReportsBufferedMetricsWithoutClosingScope() throws Exception { + RecordingStatsReporter reporter = new RecordingStatsReporter(); + Scope scope = + new RootScopeBuilder().reporter(reporter).reportEvery(com.uber.m3.util.Duration.ofHours(1)); + try { + scope.counter("buffered-counter").inc(1); + + new TallyScopeFlushHook(scope).run(); + + assertTrue(reporter.counterReports.get() >= 1); + assertTrue(reporter.flushes.get() >= 1); + assertEquals(0, reporter.closes.get()); + } finally { + scope.close(); + } + } + + @Test + public void tracingOnlyInstallsInterceptorsWithoutMetricsScope() { + Config config = new Config(); + + OpenTelemetryWorker.configureTracing( + config.clientOptions, config.workerFactoryOptions, OpenTelemetry.noop()); + + assertNull(config.serviceStubsOptions.build().getMetricsScope()); + assertEquals(1, clientInterceptorCount(config.clientOptions)); + assertEquals(1, workerInterceptorCount(config.workerFactoryOptions)); + } + + @Test + public void flushHookUsesSmallerConfiguredAndCallerTimeout() { + FakeMonotonicClock clock = new FakeMonotonicClock(); + TimeoutRecordingOpenTelemetry openTelemetry = new TimeoutRecordingOpenTelemetry(); + + new OpenTelemetryFlushHook(openTelemetry, Duration.ofMillis(250), clock) + .run(Duration.ofSeconds(2)); + + assertEquals(250, openTelemetry.tracerProvider.joinTimeoutMillis.get()); + assertEquals(250, openTelemetry.meterProvider.joinTimeoutMillis.get()); + + clock = new FakeMonotonicClock(); + openTelemetry = new TimeoutRecordingOpenTelemetry(); + + new OpenTelemetryFlushHook(openTelemetry, Duration.ofSeconds(2), clock) + .run(Duration.ofMillis(125)); + + assertEquals(125, openTelemetry.tracerProvider.joinTimeoutMillis.get()); + assertEquals(125, openTelemetry.meterProvider.joinTimeoutMillis.get()); + } + + @Test + public void flushHookSpendsTimeoutOnceAcrossProviders() { + FakeMonotonicClock clock = new FakeMonotonicClock(); + TimeoutRecordingOpenTelemetry openTelemetry = + new TimeoutRecordingOpenTelemetry(clock, Duration.ofMillis(150)); + + new OpenTelemetryFlushHook(openTelemetry, Duration.ofMillis(250), clock) + .run(Duration.ofSeconds(2)); + + assertEquals(250, openTelemetry.tracerProvider.joinTimeoutMillis.get()); + assertEquals(100, openTelemetry.meterProvider.joinTimeoutMillis.get()); + } + + @Test + public void flushHookUnwrapsSdkProviders() { + RecordingSpanProcessor spanProcessor = new RecordingSpanProcessor(); + RecordingMetricReader metricReader = new RecordingMetricReader(); + OpenTelemetrySdk sdk = + OpenTelemetrySdk.builder() + .setTracerProvider(SdkTracerProvider.builder().addSpanProcessor(spanProcessor).build()) + .setMeterProvider(SdkMeterProvider.builder().registerMetricReader(metricReader).build()) + .build(); + try { + new OpenTelemetryFlushHook(sdk, Duration.ofSeconds(1), new FakeMonotonicClock()) + .run(Duration.ofSeconds(1)); + + assertEquals(1, spanProcessor.flushes.get()); + assertEquals(1, metricReader.flushes.get()); + } finally { + sdk.shutdown().join(1, TimeUnit.SECONDS); + } + } + + @Test + public void customOpenTelemetryBypassesExporterCreation() { + RecordingTelemetryFactory factory = new RecordingTelemetryFactory(); + Config config = new Config(); + + OpenTelemetryWorker.newBuilder(new HashMap<>()) + .setTelemetryFactory(factory) + .setOpenTelemetry(OpenTelemetry.noop()) + .apply( + config.serviceStubsOptions, + config.clientOptions, + config.workerFactoryOptions, + config.shutdownHooks::add); + + assertEquals(0, factory.creates.get()); + assertNotNull(config.serviceStubsOptions.build().getMetricsScope()); + } + + private int clientInterceptorCount(WorkflowClientOptions.Builder options) { + io.temporal.common.interceptors.WorkflowClientInterceptor[] interceptors = + options.build().getInterceptors(); + return interceptors == null ? 0 : interceptors.length; + } + + private int workerInterceptorCount(WorkerFactoryOptions.Builder options) { + io.temporal.common.interceptors.WorkerInterceptor[] interceptors = + options.build().getWorkerInterceptors(); + return interceptors == null ? 0 : interceptors.length; + } + + private static final class Config { + private final WorkflowServiceStubsOptions.Builder serviceStubsOptions = + WorkflowServiceStubsOptions.newBuilder(); + private final WorkflowClientOptions.Builder clientOptions = WorkflowClientOptions.newBuilder(); + private final WorkerFactoryOptions.Builder workerFactoryOptions = + WorkerFactoryOptions.newBuilder(); + private final List shutdownHooks = new ArrayList<>(); + } + + private static final class RecordingTelemetryFactory + implements OpenTelemetryWorker.TelemetryFactory { + private final AtomicInteger creates = new AtomicInteger(); + private String endpoint; + private String serviceName; + private Duration metricsReportInterval; + private Duration flushTimeout; + + @Override + public OpenTelemetry create( + String endpoint, + String serviceName, + Duration metricsReportInterval, + Duration flushTimeout, + IdGenerator idGenerator) { + creates.incrementAndGet(); + this.endpoint = endpoint; + this.serviceName = serviceName; + this.metricsReportInterval = metricsReportInterval; + this.flushTimeout = flushTimeout; + return OpenTelemetry.noop(); + } + } + + @SuppressWarnings("deprecation") + private static final class RecordingStatsReporter implements StatsReporter { + private final AtomicInteger counterReports = new AtomicInteger(); + private final AtomicInteger gaugeReports = new AtomicInteger(); + private final AtomicInteger timerReports = new AtomicInteger(); + private final AtomicInteger histogramReports = new AtomicInteger(); + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public void reportCounter(String name, Map tags, long value) { + counterReports.incrementAndGet(); + } + + @Override + public void reportGauge(String name, Map tags, double value) { + gaugeReports.incrementAndGet(); + } + + @Override + public void reportTimer( + String name, Map tags, com.uber.m3.util.Duration interval) { + timerReports.incrementAndGet(); + } + + @Override + public void reportHistogramValueSamples( + String name, + Map tags, + com.uber.m3.tally.Buckets buckets, + double bucketLowerBound, + double bucketUpperBound, + long samples) { + histogramReports.incrementAndGet(); + } + + @Override + public void reportHistogramDurationSamples( + String name, + Map tags, + com.uber.m3.tally.Buckets buckets, + com.uber.m3.util.Duration bucketLowerBound, + com.uber.m3.util.Duration bucketUpperBound, + long samples) { + histogramReports.incrementAndGet(); + } + + @Override + public Capabilities capabilities() { + return CapableOf.REPORTING_TAGGING; + } + + @Override + public void flush() { + flushes.incrementAndGet(); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } + + private static final class RecordingSpanProcessor implements SpanProcessor { + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger shutdowns = new AtomicInteger(); + + @Override + public void onStart(io.opentelemetry.context.Context parentContext, ReadWriteSpan span) {} + + @Override + public boolean isStartRequired() { + return false; + } + + @Override + public void onEnd(ReadableSpan span) {} + + @Override + public boolean isEndRequired() { + return false; + } + + @Override + public CompletableResultCode forceFlush() { + flushes.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public CompletableResultCode shutdown() { + shutdowns.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + } + + private static final class RecordingMetricReader implements MetricReader { + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger shutdowns = new AtomicInteger(); + + @Override + public void register(CollectionRegistration registration) {} + + @Override + public CompletableResultCode forceFlush() { + flushes.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public CompletableResultCode shutdown() { + shutdowns.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public AggregationTemporality getAggregationTemporality(InstrumentType instrumentType) { + return AggregationTemporality.CUMULATIVE; + } + + @Override + public Aggregation getDefaultAggregation(InstrumentType instrumentType) { + return DefaultAggregationSelector.getDefault().getDefaultAggregation(instrumentType); + } + } + + private static final class TimeoutRecordingOpenTelemetry implements OpenTelemetry { + private final TimeoutRecordingTracerProvider tracerProvider; + private final TimeoutRecordingMeterProvider meterProvider; + + private TimeoutRecordingOpenTelemetry() { + this(null, Duration.ZERO); + } + + private TimeoutRecordingOpenTelemetry(FakeMonotonicClock clock, Duration joinDuration) { + this.tracerProvider = new TimeoutRecordingTracerProvider(clock, joinDuration); + this.meterProvider = new TimeoutRecordingMeterProvider(clock, joinDuration); + } + + @Override + public TracerProvider getTracerProvider() { + return tracerProvider; + } + + @Override + public MeterProvider getMeterProvider() { + return meterProvider; + } + + @Override + public ContextPropagators getPropagators() { + return ContextPropagators.noop(); + } + } + + public static final class TimeoutRecordingTracerProvider implements TracerProvider { + private final AtomicLong joinTimeoutMillis = new AtomicLong(-1); + private final FakeMonotonicClock clock; + private final Duration joinDuration; + + private TimeoutRecordingTracerProvider(FakeMonotonicClock clock, Duration joinDuration) { + this.clock = clock; + this.joinDuration = joinDuration; + } + + @Override + public Tracer get(String instrumentationName) { + return TracerProvider.noop().get(instrumentationName); + } + + @Override + public Tracer get(String instrumentationName, String instrumentationVersion) { + return TracerProvider.noop().get(instrumentationName, instrumentationVersion); + } + + public TimeoutRecordingResult forceFlush() { + return new TimeoutRecordingResult(joinTimeoutMillis, clock, joinDuration); + } + } + + public static final class TimeoutRecordingMeterProvider implements MeterProvider { + private final AtomicLong joinTimeoutMillis = new AtomicLong(-1); + private final FakeMonotonicClock clock; + private final Duration joinDuration; + + private TimeoutRecordingMeterProvider(FakeMonotonicClock clock, Duration joinDuration) { + this.clock = clock; + this.joinDuration = joinDuration; + } + + @Override + public MeterBuilder meterBuilder(String instrumentationName) { + return MeterProvider.noop().meterBuilder(instrumentationName); + } + + public TimeoutRecordingResult forceFlush() { + return new TimeoutRecordingResult(joinTimeoutMillis, clock, joinDuration); + } + } + + public static final class TimeoutRecordingResult { + private final AtomicLong joinTimeoutMillis; + private final FakeMonotonicClock clock; + private final Duration joinDuration; + + private TimeoutRecordingResult( + AtomicLong joinTimeoutMillis, FakeMonotonicClock clock, Duration joinDuration) { + this.joinTimeoutMillis = joinTimeoutMillis; + this.clock = clock; + this.joinDuration = joinDuration; + } + + public TimeoutRecordingResult join(long timeout, TimeUnit unit) { + joinTimeoutMillis.set(unit.toMillis(timeout)); + if (clock != null) { + clock.advance(joinDuration); + } + return this; + } + } + + private static final class FakeMonotonicClock implements OpenTelemetryFlushHook.MonotonicClock { + private long nowNanos; + + @Override + public long nanoTime() { + return nowNanos; + } + + private void advance(Duration duration) { + nowNanos += duration.toNanos(); + } + } + + public static final class CountingTracerProvider implements TracerProvider, Closeable { + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public Tracer get(String instrumentationName) { + return TracerProvider.noop().get(instrumentationName); + } + + @Override + public Tracer get(String instrumentationName, String instrumentationVersion) { + return TracerProvider.noop().get(instrumentationName, instrumentationVersion); + } + + public CompletableResultCode forceFlush() { + flushes.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } + + public static final class CountingMeterProvider implements MeterProvider, Closeable { + private final AtomicInteger flushes = new AtomicInteger(); + private final AtomicInteger closes = new AtomicInteger(); + + @Override + public MeterBuilder meterBuilder(String instrumentationName) { + return MeterProvider.noop().meterBuilder(instrumentationName); + } + + public CompletableResultCode forceFlush() { + flushes.incrementAndGet(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public void close() { + closes.incrementAndGet(); + } + } +} diff --git a/settings.gradle b/settings.gradle index fe80370b0c..969ce7f93c 100644 --- a/settings.gradle +++ b/settings.gradle @@ -6,9 +6,13 @@ include 'temporal-testing' include 'temporal-test-server' include 'temporal-opentracing' project(':temporal-opentracing').projectDir = file('contrib/temporal-opentracing') +include 'temporal-opentelemetry' +project(':temporal-opentelemetry').projectDir = file('contrib/temporal-opentelemetry') include 'temporal-kotlin' include 'temporal-spring-ai' project(':temporal-spring-ai').projectDir = file('contrib/temporal-spring-ai') +include 'temporal-aws-lambda' +project(':temporal-aws-lambda').projectDir = file('contrib/temporal-aws-lambda') include 'temporal-spring-boot-autoconfigure' include 'temporal-spring-boot-starter' include 'temporal-remote-data-encoder' diff --git a/temporal-bom/build.gradle b/temporal-bom/build.gradle index e73d0d300e..3d9771704a 100644 --- a/temporal-bom/build.gradle +++ b/temporal-bom/build.gradle @@ -7,7 +7,9 @@ description = '''Temporal Java BOM''' dependencies { constraints { api project(':temporal-kotlin') + api project(':temporal-opentelemetry') api project(':temporal-opentracing') + api project(':temporal-aws-lambda') api project(':temporal-remote-data-encoder') api project(':temporal-sdk') api project(':temporal-serviceclient')