Skip to content

Commit 18daa67

Browse files
jupblbjupblb
committed
Merge scip-kotlin into scip-java
Merges the scip-kotlin repository (semanticdb-kotlinc compiler plugin) into scip-java so they live in a single repository. The kotlinc plugin is now built locally and embedded as a resource into the scip-java CLI distribution instead of being fetched from Maven at runtime. Source merge: - semanticdb-kotlinc/src/** — kotlinc plugin source (Kotlin 2.2.0) - semanticdb-kotlinc/minimized/src/** — fixtures + golden snapshots Build wiring (build.sbt + project/): - New semanticdbKotlinc project (KotlinPlugin + sbt-assembly fat-jar with com.intellij.** shading, mirroring the previous Gradle shadowJar setup). - New semanticdbKotlincMinimized project that compiles the Kotlin/Java fixtures with the locally-built plugin and exposes a kotlincSnapshots task that drives cli/runMain to (re)generate the golden snapshots. - V.kotlinVersion bumped 2.1.20 -> 2.2.0 to match the merged plugin. - V.semanticdbKotlin removed (no longer needed). - V.semanticdbKotlincProtobuf added (3.17.3) so the kotlinc module keeps its own protobuf codegen version without perturbing scip-java. - V.kotest, V.kctfork added for the kotlinc test suite. - cli now ships semanticdb-kotlinc.jar alongside the other embedded jars. - cli's Compile / run / fork is now true so ScipJava.main's System.exit cannot kill the surrounding sbt JVM during snapshot regeneration. - project/SemanticdbKotlincKeys.scala adds the kotlincSnapshots taskKey (separately named to avoid collision with the existing snapshots test project). - project/plugins.sbt picks up sbt-kotlin-plugin 3.1.6 and sbt-jupiter-interface 0.15.1. Runtime wiring — replace runtime Maven resolution with embedded jar: - Embedded.semanticdbKotlincJar(...) added. - ScipBuildTool.compileKotlinFiles now takes tmp: Path and uses the embedded jar instead of resolving com.sourcegraph:semanticdb-kotlinc via Coursier. Two debug println calls removed. - GradleBuildTool init script writes project.ext["semanticdbKotlincJar"] so the embedded jar is visible to the gradle plugin. - SemanticdbGradlePlugin reads semanticdbKotlincJar from extra properties instead of creating a detached configuration that resolves the artifact from Maven. Housekeeping: - .github/workflows/ci.yml gets a kotlin_plugin job that runs semanticdbKotlinc/test, semanticdbKotlincMinimized/kotlincSnapshots, and a snapshot drift check (mirrors scip-kotlin's old CI signal). - .gitignore: ignore semanticdb-kotlinc/META-INF/ (kctfork tests dump semanticdb output there) plus the standard project/target, project/project sbt directories. - tests/buildTools GradleBuildToolSuite kotlin plugin version bumped 2.1.20 -> 2.2.0 to match V.kotlinVersion. Deliberately NOT copied: - scip-kotlin's build.sbt, project/, snapshots-runner/, .github/, .gitignore, LICENSE, README.md, renovate.json — scip-java already has equivalents and snapshots-runner is replaced by direct cli/runMain invocation.
1 parent 3129891 commit 18daa67

52 files changed

Lines changed: 5429 additions & 34 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,31 @@ jobs:
115115
- run: du -h index.scip
116116
working-directory: examples/bazel-example
117117

118+
kotlin_plugin:
119+
runs-on: ubuntu-latest
120+
name: semanticdb-kotlinc
121+
steps:
122+
- uses: actions/checkout@v4
123+
124+
- uses: actions/setup-java@v4
125+
with:
126+
distribution: "temurin"
127+
cache: "sbt"
128+
java-version: 11
129+
130+
- uses: sbt/setup-sbt@v1
131+
132+
- name: semanticdb-kotlinc tests
133+
run: sbt semanticdbKotlinc/test
134+
135+
- name: Kotlin snapshots
136+
run: sbt semanticdbKotlincMinimized/kotlincSnapshots
137+
138+
- name: Check snapshot drift
139+
run: |
140+
git diff --exit-code \
141+
semanticdb-kotlinc/minimized/src/generatedSnapshots
142+
118143
check:
119144
runs-on: ubuntu-latest
120145
steps:

‎.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,3 +68,12 @@ semanticdb-gradle-plugin/gradle
6868
aspects/scip_java.bzl
6969

7070
tests/snapshots/META-INF/
71+
72+
# semanticdb-kotlinc kctfork-based tests run kotlinc with our plugin, which
73+
# writes META-INF/semanticdb/sources/Test.kt.semanticdb relative to the test
74+
# cwd.
75+
semanticdb-kotlinc/META-INF/
76+
77+
# Standard sbt project metadata directories.
78+
project/target/
79+
project/project/

‎build.sbt‎

Lines changed: 226 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import _root_.kotlin.Keys._
12
import sbtdocker.DockerfileBase
23
import scala.xml.{Node => XmlNode, NodeSeq => XmlNodeSeq, _}
34
import scala.xml.transform.{RewriteRule, RuleTransformer}
@@ -20,11 +21,16 @@ lazy val V =
2021
val scala3 = "3.3.3"
2122
val metals = "1.2.2"
2223
val scalameta = "4.9.3"
23-
val semanticdbKotlin = "0.5.0"
2424
val requests = "0.8.0"
2525
val minimalMillVersion = "0.10.0"
2626
val millScipVersion = "0.3.6"
27-
val kotlinVersion = "2.1.20"
27+
val kotlinVersion = "2.2.0"
28+
// semanticdb-kotlinc has its own (older) protobuf-java codegen pinned to
29+
// 3.17.3 to keep the wire format stable for the kotlinc plugin without
30+
// perturbing the rest of scip-java.
31+
val semanticdbKotlincProtobuf = "3.17.3"
32+
val kotest = "4.6.3"
33+
val kctfork = "0.7.1"
2834
}
2935

3036
inThisBuild(
@@ -132,7 +138,6 @@ lazy val gradlePlugin = project
132138
"sbtSourcegraphVersion" ->
133139
com.sourcegraph.sbtsourcegraph.BuildInfo.version,
134140
"semanticdbVersion" -> V.scalameta,
135-
"semanticdbKotlincVersion" -> V.semanticdbKotlin,
136141
"mtagsVersion" -> V.metals,
137142
"scala213" -> V.scala213,
138143
"scala3" -> V.scala3,
@@ -238,6 +243,10 @@ lazy val cli = project
238243
moduleName := "scip-java",
239244
(Compile / mainClass) := Some("com.sourcegraph.scip_java.ScipJava"),
240245
(run / baseDirectory) := (ThisBuild / baseDirectory).value,
246+
// ScipJava.main can call System.exit, so we always fork the JVM when
247+
// sbt invokes it directly (e.g. from the semanticdb-kotlinc snapshots
248+
// task) so it cannot kill the surrounding sbt process.
249+
Compile / run / fork := true,
241250
buildInfoKeys :=
242251
Seq[BuildInfoKey](
243252
version,
@@ -253,7 +262,6 @@ lazy val cli = project
253262
"sbtSourcegraphVersion" ->
254263
com.sourcegraph.sbtsourcegraph.BuildInfo.version,
255264
"semanticdbVersion" -> V.scalameta,
256-
"semanticdbKotlincVersion" -> V.semanticdbKotlin,
257265
"mtagsVersion" -> V.metals,
258266
"scala213" -> V.scala213,
259267
"scala3" -> V.scala3,
@@ -297,6 +305,10 @@ lazy val cli = project
297305
"semanticdb-agent.jar"
298306
)
299307
addJar((gradlePlugin / Compile / assembly).value, "gradle-plugin.jar")
308+
addJar(
309+
(semanticdbKotlinc / Compile / Keys.`package`).value,
310+
"semanticdb-kotlinc.jar"
311+
)
300312

301313
IO.copy(
302314
outs,
@@ -344,6 +356,216 @@ lazy val cli = project
344356
.enablePlugins(PackPlugin, DockerPlugin, BuildInfoPlugin)
345357
.dependsOn(scip)
346358

359+
import SemanticdbKotlincKeys._
360+
361+
// The semanticdb-kotlinc compiler plugin. Built as a fat-jar that is later
362+
// embedded into the scip-java CLI distribution (see cli's resourceGenerators)
363+
// so the runtime no longer needs to fetch a published semanticdb-kotlinc
364+
// artifact from Maven.
365+
lazy val semanticdbKotlinc = project
366+
.in(file("semanticdb-kotlinc"))
367+
.enablePlugins(KotlinPlugin)
368+
.settings(
369+
name := "semanticdb-kotlinc",
370+
moduleName := "semanticdb-kotlinc",
371+
description := "A kotlinc plugin to emit SemanticDB information",
372+
crossPaths := false,
373+
autoScalaLibrary := false,
374+
kotlinVersion := V.kotlinVersion,
375+
kotlincJvmTarget := "1.8",
376+
kotlincOptions ++= Seq("-Xinline-classes", "-Xcontext-parameters"),
377+
// sbt-kotlin-plugin defaults to adding `kotlin-scripting-compiler-embeddable`
378+
// (and its transitive kotlin-stdlib) as a regular dependency. Mark them
379+
// Provided — kotlinc supplies them at runtime, and we don't want them
380+
// bundled into the fat-jar.
381+
kotlinRuntimeProvided := true,
382+
// kotlin-stdlib is supplied by kotlinc at runtime — keep on compile
383+
// classpath via Provided so the assembled fat-jar does not bundle it.
384+
libraryDependencies +=
385+
"org.jetbrains.kotlin" % "kotlin-stdlib" % V.kotlinVersion % Provided,
386+
// protobuf java codegen — proto file lives at src/main/proto/...
387+
Compile / PB.protoSources :=
388+
Seq((Compile / sourceDirectory).value / "proto"),
389+
Compile / PB.targets :=
390+
Seq(
391+
PB.gens.java(V.semanticdbKotlincProtobuf) ->
392+
(Compile / sourceManaged).value
393+
),
394+
libraryDependencies +=
395+
"com.google.protobuf" % "protobuf-java" % V.semanticdbKotlincProtobuf,
396+
// kotlin-compiler-embeddable is supplied by kotlinc at runtime
397+
libraryDependencies +=
398+
"org.jetbrains.kotlin" % "kotlin-compiler-embeddable" % V.kotlinVersion %
399+
Provided,
400+
// ---- sbt-assembly fat-jar ---------------------------------------------
401+
// Mirrors scip-java's `fatjarPackageSettings`. Produces a shaded jar that
402+
// replaces the slim `packageBin` so `publishLocal` ships the shaded
403+
// artifact (the same artifact Gradle's shadowJar produced previously).
404+
assembly / assemblyShadeRules :=
405+
Seq(
406+
// Relocate any IntelliJ classes the same way kotlin-compiler-embeddable
407+
// does internally. Do NOT rename `com.sourcegraph.**` — the
408+
// META-INF/services files reference those FQNs.
409+
ShadeRule
410+
.rename("com.intellij.**" -> "org.jetbrains.kotlin.com.intellij.@1")
411+
.inAll
412+
),
413+
Compile / packageBin := assembly.value,
414+
// Strip every <dependency> from the POM — the fat-jar absorbs the
415+
// protobuf runtime, and the kotlin-* deps are Provided by kotlinc.
416+
pomPostProcess := { node =>
417+
new RuleTransformer(
418+
new RewriteRule {
419+
override def transform(n: XmlNode): XmlNodeSeq =
420+
if (n.label == "dependency")
421+
XmlNodeSeq.Empty
422+
else
423+
n
424+
}
425+
).transform(node).head
426+
},
427+
// tests
428+
libraryDependencies ++=
429+
Seq(
430+
"org.jetbrains.kotlin" % "kotlin-compiler-embeddable" %
431+
V.kotlinVersion % Test,
432+
"org.jetbrains.kotlin" % "kotlin-test" % V.kotlinVersion % Test,
433+
"org.jetbrains.kotlin" % "kotlin-test-junit5" % V.kotlinVersion % Test,
434+
"org.jetbrains.kotlin" % "kotlin-reflect" % V.kotlinVersion % Test,
435+
"io.kotest" % "kotest-assertions-core-jvm" % V.kotest % Test,
436+
"dev.zacsweers.kctfork" % "core" % V.kctfork % Test,
437+
"com.github.sbt.junit" % "jupiter-interface" %
438+
JupiterKeys.jupiterVersion.value % Test
439+
),
440+
Test / fork := true,
441+
Test / javaOptions += "-Xmx2g",
442+
// sbt-kotlin-plugin 3.1.6 inspects every jar on the kotlinc classpath and
443+
// moves any jar containing META-INF/services/org.jetbrains.kotlin.compiler.plugin.*
444+
// entries into the compiler-plugin classpath, removing it from the regular
445+
// classpath. kctfork ships such service files for its own internal use as a
446+
// KAPT/registrar shim, which makes its public API (com.tschuchort.compiletesting.*)
447+
// invisible to our test sources. Workaround: pre-extract kctfork to a
448+
// directory and add that directory to the test classpath — sbt-kotlin-plugin
449+
// only inspects .jar files, so directories pass through unmodified.
450+
Test / unmanagedJars += {
451+
val report = update.value
452+
val files = report.allFiles
453+
val jar = files
454+
.find(_.getName == s"core-${V.kctfork}.jar")
455+
.getOrElse(
456+
sys.error(s"kctfork core-${V.kctfork}.jar not found in update report")
457+
)
458+
val dir = target.value / s"kctfork-${V.kctfork}-extracted"
459+
val marker = dir / ".extracted"
460+
if (!marker.exists()) {
461+
IO.delete(dir)
462+
IO.unzip(jar, dir)
463+
IO.touch(marker)
464+
}
465+
Attributed.blank(dir)
466+
}
467+
)
468+
469+
// `semanticdbKotlincMinimized` mirrors the (still-present) Gradle build at
470+
// semanticdb-kotlinc/minimized/build.gradle.kts. It compiles a small set of
471+
// Kotlin and Java fixtures with the assembled `semanticdbKotlinc` plugin
472+
// attached to kotlinc/javac, producing *.semanticdb files under
473+
// target/semanticdb-targetroot/ which are then converted to SCIP and rendered
474+
// as the human-readable golden snapshots by the `snapshots` task.
475+
lazy val semanticdbKotlincMinimized = project
476+
.in(file("semanticdb-kotlinc/minimized"))
477+
.enablePlugins(KotlinPlugin)
478+
.settings(
479+
publish / skip := true,
480+
crossPaths := false,
481+
autoScalaLibrary := false,
482+
kotlinVersion := V.kotlinVersion,
483+
kotlincJvmTarget := "1.8",
484+
kotlinLib("stdlib"),
485+
// Force javac to fork. Two reasons:
486+
// 1. JDK 9+ strongly encapsulates jdk.compiler internals; semanticdb-javac
487+
// reflectively touches them and needs --add-exports flags. With a
488+
// forked javac we can pass `-J--add-exports=...` (mirrors scip-java).
489+
// 2. sbt's in-process javac receives `vf://` virtual-file URIs from the
490+
// MappedFileConverter, which semanticdb-javac cannot resolve via
491+
// java.nio.file.Path.of. Forked javac is invoked with absolute file
492+
// paths instead, so the plugin sees real paths.
493+
// Setting javaHome to Some(<current JVM home>) flips
494+
// ZincUtil.compilers/JavaTools.directOrFork from direct → fork.
495+
javaHome := Some(file(System.getProperty("java.home"))),
496+
Compile / javacOptions ++=
497+
Seq(
498+
"-J--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED",
499+
"-J--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED",
500+
"-J--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED",
501+
"-J--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED",
502+
"-J--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED"
503+
),
504+
// Attach the assembled kotlinc fat-jar to the compile classpath.
505+
// sbt-kotlin-plugin's AnalyzingKotlinCompiler partitions the classpath:
506+
// any jar containing META-INF/services/org.jetbrains.kotlin.compiler.plugin*
507+
// entries (which our fat-jar does, for both CommandLineProcessor and
508+
// CompilerPluginRegistrar) is moved into args.pluginClasspaths and removed
509+
// from the regular classpath. So no `-Xplugin=<path>` is needed and we
510+
// don't have to predict the assembled jar's filename. The .value reference
511+
// also gives us the right task ordering — assembly runs before compile.
512+
Compile / unmanagedJars +=
513+
Attributed.blank((semanticdbKotlinc / Compile / packageBin).value),
514+
// Wire the locally-built semanticdb-javac fat jar in place of fetching the
515+
// published `com.sourcegraph:semanticdb-javac` artifact at compile time.
516+
Compile / unmanagedJars +=
517+
Attributed.blank((javacPlugin / Compile / Keys.`package`).value),
518+
Compile / kotlincPluginOptions ++= {
519+
val srcRoot = (ThisBuild / baseDirectory).value.getAbsolutePath
520+
val tgtRoot = (target.value / "semanticdb-targetroot").getAbsolutePath
521+
Seq(
522+
s"plugin:semanticdb-kotlinc:sourceroot=$srcRoot",
523+
s"plugin:semanticdb-kotlinc:targetroot=$tgtRoot"
524+
)
525+
},
526+
// The semanticdb javac plugin parses its own argument string, so
527+
// `-Xplugin:semanticdb -sourceroot:<...> -targetroot:<...>` MUST be passed
528+
// as a single javac argument (matches the existing Gradle behavior).
529+
Compile / javacOptions += {
530+
val srcRoot = (ThisBuild / baseDirectory).value
531+
val tgtRoot = target.value / "semanticdb-targetroot"
532+
s"-Xplugin:semanticdb -sourceroot:${srcRoot.getAbsolutePath} " +
533+
s"-targetroot:${tgtRoot.getAbsolutePath}"
534+
},
535+
// ----- snapshots regeneration task -----
536+
// Invokes `com.sourcegraph.scip_java.ScipJava.main` twice in the cli JVM
537+
// (forked — ScipJava.main calls System.exit on failure). First pass
538+
// converts the *.semanticdb files under target/semanticdb-targetroot/
539+
// into an index.scip; second pass renders that index as the human-readable
540+
// golden snapshots.
541+
//
542+
// We use `kotlincSnapshots` (defined in project/SemanticdbKotlincKeys.scala)
543+
// instead of `snapshots` to avoid colliding with the existing top-level
544+
// `snapshots` test project.
545+
kotlincSnapshots :=
546+
Def
547+
.taskDyn {
548+
val srcRoot = (ThisBuild / baseDirectory).value.getAbsolutePath
549+
val tgtRoot = (target.value / "semanticdb-targetroot")
550+
.getAbsolutePath
551+
val snapDir =
552+
(baseDirectory.value / "src" / "generatedSnapshots" / "resources")
553+
.getAbsolutePath
554+
val scipOut = s"$tgtRoot/index.scip"
555+
val mainCls = "com.sourcegraph.scip_java.ScipJava"
556+
Def.sequential(
557+
Compile / compile,
558+
(cli / Compile / runMain).toTask(
559+
s" $mainCls index-semanticdb --no-emit-inverse-relationships --cwd $srcRoot --output $scipOut $tgtRoot"
560+
),
561+
(cli / Compile / runMain).toTask(
562+
s" $mainCls snapshot --cwd $srcRoot --output $snapDir $tgtRoot"
563+
)
564+
)
565+
}
566+
.value
567+
)
568+
347569
commands +=
348570
Command.command("nativeImageProfiled") { s =>
349571
val targetroot =

‎project/SemanticdbKotlincKeys.scala‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
import sbt._
2+
3+
// Task key for regenerating the SCIP/SemanticDB golden snapshots emitted by
4+
// the semanticdb-kotlinc compiler plugin over the Kotlin minimized fixtures.
5+
// Defined here (in the project/ build sources) so build.sbt can `import` it.
6+
//
7+
// We deliberately do NOT call this `snapshots` to avoid colliding with the
8+
// existing top-level `snapshots` test project (`lazy val snapshots = project`).
9+
object SemanticdbKotlincKeys {
10+
lazy val kotlincSnapshots = taskKey[Unit](
11+
"Run the SCIP snapshot generator over the semanticdb-kotlinc minimized project"
12+
)
13+
}

‎project/plugins.sbt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ addSbtPlugin("com.lightbend.sbt" % "sbt-java-formatter" % "0.6.1")
1111
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.4.3")
1212
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.15.0")
1313
addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.1")
14+
addSbtPlugin("org.jetbrains.scala" % "sbt-kotlin-plugin" % "3.1.6")
15+
addSbtPlugin("com.github.sbt.junit" % "sbt-jupiter-interface" % "0.15.1")
1416
// sbt-jdi-tools appears to fix an error related to this message:
1517
// [error] (plugin / Compile / compileIncremental) java.lang.NoClassDefFoundError: com/sun/tools/javac/code/Symbol
1618
addSbtPlugin("org.scala-debugger" % "sbt-jdi-tools" % "1.1.1")

‎scip-java/src/main/scala/com/sourcegraph/scip_java/Embedded.scala‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,11 @@ object Embedded {
2323

2424
def agentJar(tmpDir: Path): Path = copyFile(tmpDir, "semanticdb-agent.jar")
2525

26+
def semanticdbKotlincJar(tmpDir: Path): Path = copyFile(
27+
tmpDir,
28+
"semanticdb-kotlinc.jar"
29+
)
30+
2631
private def javacErrorpath(tmp: Path) = tmp.resolve("errorpath.txt")
2732

2833
def customJavac(

‎scip-java/src/main/scala/com/sourcegraph/scip_java/buildtools/GradleBuildTool.scala‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ class GradleBuildTool(index: IndexCommand) extends BuildTool("Gradle", index) {
105105
val agentpath = Embedded.agentJar(tmp)
106106
val pluginpath = Embedded.semanticdbJar(tmp)
107107
val gradlePluginPath = Embedded.gradlePluginJar(tmp)
108+
val semanticdbKotlincPath = Embedded.semanticdbKotlincJar(tmp)
108109
val dependenciesPath = targetroot.resolve("dependencies.txt")
109110
Files.deleteIfExists(dependenciesPath)
110111

@@ -123,6 +124,7 @@ class GradleBuildTool(index: IndexCommand) extends BuildTool("Gradle", index) {
123124
| project.ext["javacPluginJar"] = "$pluginpath"
124125
| project.ext["dependenciesOut"] = "$dependenciesPath"
125126
| project.ext["javacAgentPath"] = "$agentpath"
127+
| project.ext["semanticdbKotlincJar"] = "$semanticdbKotlincPath"
126128
| apply plugin: SemanticdbGradlePlugin
127129
| }
128130
""".stripMargin.trim

0 commit comments

Comments
 (0)