AWS Kinesis destination doesn't merge partial config updates

[leggetter]

Description

When updating an AWS Kinesis destination with a partial config (e.g., only streamName), the backend doesn't merge the update with existing config values. Instead, it returns the original values unchanged.

1 test is currently skipped in the SDK test suite due to this bug.

Current Behavior

When sending a PATCH request with partial config:

PATCH /api/v1/{tenant_id}/destinations/{id}
{
  "type": "aws_kinesis",
  "config": {
    "streamName": "updated-stream"
    // Missing region field
  }
}

Expected: Backend merges with existing config, returns updated streamName
Actual: Backend returns original streamName value unchanged

Evidence

Test Implementation: spec-sdk-tests/tests/destinations/aws-kinesis.test.ts lines 336-349

// TODO: Re-enable this test once backend properly handles partial config updates for AWS Kinesis
it.skip('should update destination config', async () => {
  const updated = await client.updateDestination(destinationId, {
    type: 'aws_kinesis',
    config: {
      streamName: 'updated-stream', // Only updating streamName
    },
  });

  expect(updated.config.streamName).to.equal('updated-stream'); // FAILS
});

Test Error:

AssertionError: expected 'my-stream' to equal 'updated-stream'
+ expected - actual

-my-stream
+updated-stream

Comparison with Other Destination Types

AWS S3 - WORKS CORRECTLY (aws-s3.test.ts lines 332-346):

it('should update destination config', async () => {
  const updated = await client.updateDestination(destinationId, {
    type: 'aws_s3',
    config: {
      bucket: 'updated-bucket', // Only updating bucket
    },
  });

  expect(updated.config.bucket).to.equal('updated-bucket'); // PASSES ✅
});

AWS SQS - WORKS CORRECTLY (aws-sqs.test.ts lines 248-262):

it('should update destination config', async () => {
  const updated = await client.updateDestination(destinationId, {
    type: 'aws_sqs',
    config: {
      queueUrl: 'https://sqs.us-west-2.amazonaws.com/123456789012/updated-queue',
    },
  });

  expect(updated.config.queueUrl).to.equal(
    'https://sqs.us-west-2.amazonaws.com/123456789012/updated-queue'
  ); // PASSES ✅
});

Impact

• Skipped Test: spec-sdk-tests/tests/destinations/aws-kinesis.test.ts line 336
• Users cannot update individual Kinesis config fields without providing all fields
• Inconsistent behavior compared to other AWS destination types

Steps to Reproduce

1. Create AWS Kinesis destination with streamName: 'my-stream' and region: 'us-east-1'
2. Update destination with partial config: { streamName: 'updated-stream' }
3. Observe that returned config still has streamName: 'my-stream'

Expected Behavior

Partial config updates should merge with existing config:

• Provided fields should be updated
• Omitted fields should retain their existing values
• Behavior should match AWS S3 and AWS SQS destinations

Additional Context

See detailed analysis in spec-sdk-tests/TEST_STATUS.md and spec-sdk-tests/GITHUB_ISSUES.md

[leggetter]

@alexluong - this is a potential bug identified as part of #521

[alexluong]

Hi @leggetter, thanks for reporting this issue. Super interesting and uncovered more issues. Short TLDR is I believe this is SDK issue caused by incorrect OpenAPI schema. I worked with Claude to investigate and put together this report with their help but I did a round of formatting and "sectioning" so that it's clear and straightforward, without LLM bluff. Please consider checking this out and let me know if this is clear from your end.

---

Summary

Partial configuration updates (PATCH requests) fail for destination types that have 2+ required config fields in the OpenAPI specification. The SDK's auto-generated validation schemas reject partial configs because required fields are missing, causing field name transformations (camelCase → snake_case) to be skipped. This results in the backend receiving camelCase field names instead of the expected snake_case, leading to config merge issues.

Affected Destination Types:

• AWS Kinesis (stream_name, region)
• AWS S3 (bucket, region)
• RabbitMQ (server_url, exchange)
• GCP Pub/Sub (project_id, topic)

Status: Confirmed via SDK tests and backend debug logging

---

Root Cause

OpenAPI Specification Issue

The DestinationUpdate schemas reference the full config schemas (e.g., AWSKinesisConfig) which define fields as required:

# docs/apis/openapi.yaml:1278
DestinationUpdateAWSKinesis:
  type: object
  properties:
    config:
      $ref: "#/components/schemas/AWSKinesisConfig" # stream_name/region required here, but PATCH means optional

# docs/apis/openapi.yaml:196
AWSKinesisConfig:
  type: object
  required: [stream_name, region]  # ❌ Required in schema
  properties:
    stream_name:
      type: string
    region:
      type: string

SDK Code Generation

Speakeasy generates TypeScript SDK schemas that enforce these requirements:

// sdks/outpost-typescript/src/models/components/awskinesisconfig.ts:60-64
export const AWSKinesisConfig$outboundSchema = z.object({
  streamName: z.string(),      // Required - no .optional()
  region: z.string(),          // Required - no .optional()
  endpoint: z.string().optional(),
  partitionKeyTemplate: z.string().optional(),
}).transform((v) => {
  return remap$(v, {
    streamName: "stream_name",
    partitionKeyTemplate: "partition_key_template",
  });
});

Request Flow Breakdown

When a partial config update is sent:

// Test sends:
await client.updateDestination(destinationId, {
  config: {
    streamName: 'updated-stream'  // Missing 'region'
  }
});

1. SDK Validation (Line 93 of destinationsUpdate.ts):

const parsed = safeParse(
  request,
  (value) => UpdateTenantDestinationRequest$outboundSchema.parse(value),
  "Input validation failed"
);

• The AWSKinesisConfig$outboundSchema expects both streamName AND region
• Validation FAILS because region is missing
• The zod union tries all destination types, all fail
• Object passes through untransformed

2. HTTP Request (Line 100):

const body = encodeJSON("body", payload.DestinationUpdate, { explode: true });

• Sends untransformed object: {"streamName": "updated-stream"}
• Should have sent: {"stream_name": "updated-stream"}

3. Backend Receives (destination_handlers.go:166):

updatedDestination.Config = maputil.MergeStringMaps(originalDestination.Config, input.Config)

• Original: {"stream_name": "my-stream", "region": "us-east-1"}
• Input: {"streamName": "updated-stream"} ❌ camelCase!
• Merged: {"stream_name": "my-stream", "streamName": "updated-stream", "region": "us-east-1"}

4. Result: Both keys exist in the config, causing undefined behavior

---

Evidence

Test Results

AWS Kinesis (FAILS):

✗ should update destination config

AssertionError: expected 'my-stream' to equal 'updated-stream'

Backend Log:

{
  "msg": "Config merge",
  "original": {"region":"us-east-1","stream_name":"my-stream"},
  "input": {"streamName":"updated-stream"},
  "merged": {"region":"us-east-1","streamName":"updated-stream","stream_name":"my-stream"}
}

GCP Pub/Sub (PASSES by accident):

Test only updates topic field, which doesn't require case transformation:

config: { topic: 'updated-topic' }  // 'topic' === 'topic' in both cases

When modified to update projectId instead, it fails identically to Kinesis.

Comparison with Working Destination Types

AWS SQS (WORKS):

• Only 1 required field: queue_url
• Test provides that field, validation passes
• Transformation happens correctly

Webhook (WORKS):

• Only 1 required field: url
• Test provides that field, validation passes
• Transformation happens correctly

---

Proposed Solution

✅ Option 1: Create Separate Update Schemas (Recommended)

Define separate schemas for PATCH operations where all fields are optional:

# New schemas
AWSKinesisConfigUpdate:
  type: object
  properties:
    stream_name:
      type: string
    region:
      type: string
    endpoint:
      type: string
    partition_key_template:
      type: string
  # NO required fields

# Use in DestinationUpdate
DestinationUpdateAWSKinesis:
  type: object
  properties:
    config:
      $ref: "#/components/schemas/AWSKinesisConfigUpdate"

Pros:

• Clear separation of concerns
• Explicit partial update support
• No ambiguity

Cons:

• Schema duplication
• More maintenance

Option 2: Use allOf with Required Override

DestinationUpdateAWSKinesis:
  type: object
  properties:
    config:
      allOf:
        - $ref: "#/components/schemas/AWSKinesisConfig"
        - required: []  # Override to make all fields optional

Pros:

• Less duplication
• DRY principle

Cons:

• OpenAPI 3.0 may not handle this correctly
• Speakeasy code generation may not respect the override

❌ Option 3: Backend Normalization (Not Recommended)

Add middleware to normalize camelCase → snake_case on the backend.

Pros:

• Works around SDK issue

Cons:

• Doesn't fix root cause
• Inconsistent with API contract
• May break other clients

---

Recommended Action Items

1. Update OpenAPI spec with separate *ConfigUpdate schemas for all destination types
2. Regenerate TypeScript SDK using updated spec
3. Unskip AWS Kinesis test and verify all partial update tests pass
4. Add tests for other affected destination types (S3, RabbitMQ, GCP)

---

Reference Information

Affected Code Locations

OpenAPI Specification:

• File: /docs/apis/openapi.yaml
• Lines: 1278-1301 (AWS Kinesis), 1207-1230 (AWS SQS), etc.
• Issue: DestinationUpdate schemas reference full config schemas with required fields

Generated SDK:

• File: /sdks/outpost-typescript/src/models/components/awskinesisconfig.ts
• Lines: 56-70 (outbound schema)
• Issue: Fields marked as required, no .optional() for partial updates

Tests:

• File: /spec-sdk-tests/tests/destinations/aws-kinesis.test.ts
• Line: 335 (skipped test)
• Status: Currently skipped with TODO comment

Related Files

• Issue tracker: AWS Kinesis destination doesn't merge partial config updates #524
• Test file: spec-sdk-tests/tests/destinations/aws-kinesis.test.ts:335
• OpenAPI spec: docs/apis/openapi.yaml:1278
• SDK generation: Speakeasy
• Backend handler: internal/services/api/destination_handlers.go:166

---

Additional Notes

This issue affects 4 out of 8 destination types. The remaining types (Webhook, AWS SQS, Azure Service Bus, Hookdeck) only have 1 required field each and are unaffected.

The issue was discovered when investigating why the AWS Kinesis partial config update test was skipped. Analysis revealed the underlying cause affects multiple destination types, though some tests pass by coincidence (e.g., GCP Pub/Sub test only updates a field that doesn't require transformation).

[alexluong]

@leggetter I'm closing this for now but please reopen if this is still an issue from your test suite!