Skip to content

Using multiple Kafka connectors from the same Spark session doesn't work#1436

Merged
SirOibaf merged 10 commits into
logicalclocks:branch-3.9from
SirOibaf:feat/multiple_projects_same_spark_sess
Aug 5, 2025
Merged

Using multiple Kafka connectors from the same Spark session doesn't work#1436
SirOibaf merged 10 commits into
logicalclocks:branch-3.9from
SirOibaf:feat/multiple_projects_same_spark_sess

Conversation

@SirOibaf
Copy link
Copy Markdown
Contributor

@SirOibaf SirOibaf commented Aug 5, 2025

This is caused by the addFile method not overwriting the content of the file if the file name matches.
This PR addresses the issue for Spark versions > 3.2 where the addFile method is not needed as the Kafka client authenticates using the certificates in PEM format rather than the JKS format.

This PR adds/fixes/changes...

  • please summarize your changes to the code
  • and make sure to include all changes to user-facing APIs

JIRA Issue: -

Priority for Review: -

Related PRs: -

How Has This Been Tested?

  • Unit Tests
  • Integration Tests
  • Manual Tests on VM

Checklist For The Assigned Reviewer:

- [ ] Checked if merge conflicts with master exist
- [ ] Checked if stylechecks for Java and Python pass
- [ ] Checked if all docstrings were added and/or updated appropriately
- [ ] Ran spellcheck on docstring
- [ ] Checked if guides & concepts need to be updated
- [ ] Checked if naming conventions for parameters and variables were followed
- [ ] Checked if private methods are properly declared and used
- [ ] Checked if hard-to-understand areas of code are commented
- [ ] Checked if tests are effective
- [ ] Built and deployed changes on dev VM and tested manually
- [x] (Checked if all type annotations were added and/or updated appropriately)

Fabio Buso added 8 commits August 5, 2025 16:42
This is caused by the addFile method not overwriting the content of the file
if the file name matches.
This PR addresses the issue for Spark versions > 3.2 where the addFile method
is not needed as the Kafka client authenticates using the certificates in PEM format
rather than the JKS format.
@SirOibaf SirOibaf merged commit b3591dc into logicalclocks:branch-3.9 Aug 5, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants