SOLR-17767: Add a Java Agent (SIP-24)

.github/workflows/bin-solr-test.yml

@@ -19,7 +19,7 @@ jobs:
     name: Run Solr Script Tests
 
     runs-on: ubuntu-latest
-    timeout-minutes: 40
+    timeout-minutes: 70
 
     steps:
     - name: Checkout code

AGENTS.md

@@ -37,6 +37,7 @@ While README.md and CONTRIBUTING.md are mainly written for humans, this file is
 - Use the project's custom `EnvUtils` to read system properties. It auto converts env.var SOLR_FOO_BAR to system property solr.foo.bar
 - Be careful to not add non-essential logging! If you add slf4j log calls, make sure to wrap debug/trace level calls in `logger.isXxxEnabled()` clause
 - Validate user input. For file paths, always call `myCoreContainer.assertPathAllowed(myPath)` before using
+- Do not use Fully Qualified Class Names (FQCNs) in code unless absolutely necessary; do use imports.
 
 ## Running Tests
 
@@ -63,7 +64,7 @@ While README.md and CONTRIBUTING.md are mainly written for humans, this file is
 - For major or breaking changes, add a prominent note in reference guide major-changes-in-solr-X.adoc
 - Always consider whether a reference-guide page needs updating due to the new/changed features. Target audience is end user
 - For changes to build system and other developer-focused changes, consider updating or adding docs in dev-docs/ folder
-- Keep all documentation including javadoc concise
+- Keep all documentation including javadoc concise and not overly verbose, do not add a comment if it states the obvious.
 - New classes should have some javadocs
 - Changes should not have code comments communicating the change, which are instead great comments to leave for code review / commentary
 

NOTICE.txt

@@ -652,3 +652,18 @@ The S3 Output Stream is based on ASL 2.0 reference implementations found at:
     https://github.com/confluentinc/kafka-connect-storage-cloud/blob/5.0.x/kafka-connect-s3/src/main/java/io/confluent/connect/s3/storage/S3OutputStream.java
 This files resides at:
     modules/s3-repository/src/java/org/apache/solr/s3/S3OutputStream.java
+
+=========================================================================
+==         java agent notice                                           ==
+=========================================================================
+
+This project contains source files in the "solr/agent-sm" module that were copied and adapted from the
+OpenSearch project (https://github.com/opensearch-project/OpenSearch), git tag "3.6.0", originally
+licensed under the Apache License 2.0.
+
+The following files were adapted from the interceptor classes in libs/agent-sm/agent/:
+  FileInterceptor.java, SocketChannelInterceptor.java, SystemExitInterceptor.java,
+  RuntimeHaltInterceptor.java, StackCallerClassChainExtractor.java
+
+The following files were adapted from the policy parser in libs/agent-sm/agent-policy/:
+  PolicyFileParser.java, PolicyToken.java, PolicyTokenStream.java

changelog/unreleased/SOLR-17767-java-agent.yml

@@ -0,0 +1,9 @@
+title: >
+  Introduce a Java agent (solr-agent-sm) that enforces file access, outbound network,
+  System.exit(), and process-spawn controls via ByteBuddy instrumentation.
+type: added
+authors:
+  - name: Jan Høydahl
+links:
+  - name: SOLR-17767
+    url: https://issues.apache.org/jira/browse/SOLR-17767

dev-docs/security-agent.adoc

@@ -0,0 +1,106 @@
+= Security Agent Developer Guide
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+This document is for Solr developers who need to work with or extend the `solr/agent-sm` security agent.
+
+== Module Overview
+
+`solr/agent-sm` is a standalone Gradle subproject that produces a Java agent JAR.
+The agent must be a standalone JAR (not a Solr module) because it is loaded by the JVM before Solr initializes.
+It intercepts bytecode using ByteBuddy and enforces a policy derived from JDK-style `.policy` files.
+
+Key classes:
+
+* `SolrAgentEntryPoint` — `premain()` and `agentmain()` entry points; wires all ByteBuddy interceptors
+* `PolicyLoader` — parses `.policy` files and performs variable substitution
+* `AgentPolicy` — immutable singleton holding the merged policy
+* `StackCallerClassChainExtractor` — `StackWalker`-based call chain analysis (virtual-thread safe)
+* `SecurityViolationLogger` — emits structured SLF4J log entries for violations
+* `ViolationMetricsReporter` — `LongAdder` counters per violation type; deferred registration with `SolrMetricManager`
+
+== Adding an Approved Exit Caller
+
+To allow a new class to call `System.exit()` or `Runtime.halt()`, add it to the approved callers list in `AgentPolicy`.
+The list is populated from the policy file using `java.lang.RuntimePermission "exitVM"` entries.
+Add an entry to `agent-security.policy`:
+
+[source,text]
+----
+grant codeBase "file:${solr.install.dir}/..." {
+  permission java.lang.RuntimePermission "exitVM";
+};
+----
+
+Alternatively, add the class name to the default approved list in `AgentPolicy` if it is always a trusted Solr internal class.
+
+== Adding an Approved Exec Caller
+
+The default exec allow-list is empty — `ProcessBuilder.start()` and `Runtime.exec()` are blocked for all callers.
+To permit a specific class, add a `java.lang.RuntimePermission "exec"` entry to the policy or the `AgentPolicy` defaults.
+Document the reason in a code comment.
+
+== Adding a Trusted Filesystem Scheme
+
+Paths on certain virtual filesystems (e.g., `jrt:/` for JDK resources) are excluded from file policy checks.
+To add a trusted scheme, extend the `trustedFileSystems` set in `AgentPolicy`.
+Trusted schemes bypass all file access checks.
+
+== Deferred Metrics Registration
+
+The agent starts before Solr and begins counting violations immediately via `ViolationMetricsReporter`.
+Once `CoreContainer` initializes, it calls `ViolationMetricsReporter.registerWithSolrMetrics()` via reflection (so `solr:core` has no compile-time dependency on `solr:agent-sm`).
+Counts accumulated before registration are preserved in the `LongAdder` instances and become visible in metrics after registration.
+
+If you change the `registerWithSolrMetrics` method signature, update the reflective call in `CoreContainer.java` to match.
+
+== Writing Tests Alongside the Agent
+
+Tests for `solr:agent-sm` extend `SolrTestCase` (not `SolrTestCaseJ4`).
+
+The agent-sm test suite runs with `SOLR_SECURITY_AGENT_MODE=enforce` so that blocking behaviour is tested directly.
+The broader Solr test suite runs in `warn` mode — violations are logged but do not fail tests.
+
+To run only the agent-sm tests:
+
+[source,bash]
+----
+./gradlew :solr:agent-sm:test
+----
+
+To reproduce a flaky test with a specific randomization seed:
+
+[source,bash]
+----
+./gradlew :solr:agent-sm:test --tests "org.apache.solr.security.agent.SolrAgentIntegrationTest" -Ptests.seed=HEXSEEDHERE
+----
+
+== Flipping the Broader Test Suite to Enforce Mode
+
+The broader Solr test suite currently runs in warn mode.
+Before flipping to enforce mode:
+
+1. Run the full suite in warn mode and collect all `SECURITY VIOLATION` log entries.
+2. Triage each violation — legitimate Solr operations need policy entries; plugin violations need plugin fixes or policy entries.
+3. Update `agent-security.policy` or test-specific extra-policy files as needed.
+
+== No Compile-Time Dependency from solr:core on solr:agent-sm
+
+`solr:core` must not declare a compile-time dependency on `solr:agent-sm`.
+The agent JAR is loaded by the JVM bootstrap classloader before application classloaders exist.
+All interaction from `solr:core` must go through reflection (`Class.forName` with bootstrap classloader).
+See `CoreContainer.java` for the pattern.

gradle/validation/rat-sources.gradle

@@ -92,6 +92,12 @@ allprojects {
           // Exclude github stuff (templates, workflows).
           exclude ".github"
 
+          // Exclude speckit planning artifacts and Claude Code tooling.
+          exclude "specs/**"
+          exclude ".claude/**"
+          exclude ".specify/**"
+          exclude "CLAUDE.md"
+
           exclude "dev-tools/scripts/cloud.sh"
           exclude "dev-tools/scripts/README.md"
           exclude "dev-tools/scripts/create_line_file_docs.py"
@@ -145,6 +151,11 @@ allprojects {
           exclude "src/test-files/**/*-snippet-*.xml"
           break
 
+        case ":solr:agent-sm":
+          // Test-programs are tiny standalone programs for BATS tests, not shipped artifacts.
+          exclude "src/test-programs/**"
+          break
+
         case ":solr:server":
           exclude "**/*.xml"
           exclude "**/*.sh"

settings.gradle

@@ -36,6 +36,7 @@ rootProject.name = "solr-root"
 includeBuild("build-tools/missing-doclet")
 
 include ":platform"
+include "solr:agent-sm"
 include "solr:api"
 include "solr:solrj"
 include "solr:solrj-jetty"

solr/agent-sm/build.gradle

@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: 'java-library'
+
+description = 'Apache Solr Security Agent'
+
+dependencies {
+  // ByteBuddy references SpotBugs annotations in its class files; needed to suppress -Xlint:classfile
+  // Not included in the release so exclude from license/checksum checks.
+  compileOnly libs.spotbugs.annotations
+
+  // !! DO NOT ADD SLF4J OR OTHER LOGGING JARS !!
+  // They are on Boot-Class-Path and initialise the logging binding before Log4j2 loads,
+  // permanently caching a NOP logger and preventing solr.log from being written.
+
+  // ByteBuddy for bytecode instrumentation - bundled into the fat agent JAR
+  implementation libs.bytebuddy
+  // byte-buddy-agent is the Java agent JAR itself; not directly imported in compiled classes
+  // but required at runtime for agent loading and bundled into the fat JAR.
+  implementation libs.bytebuddy.agent
+  permitUnusedDeclared libs.bytebuddy.agent
+
+
+  // Test dependencies - NOT bundled
+  testImplementation project(':solr:test-framework')
+  testImplementation project(':solr:core')
+  testImplementation libs.apache.lucene.testframework
+  testImplementation libs.junit.junit
+  testImplementation libs.carrotsearch.randomizedtesting.runner
+  testImplementation libs.hamcrest.hamcrest
+  // These are used indirectly via SolrTestCase (test-framework) and randomized runner base classes;
+  // the static analyzer does not see direct usage in agent-sm test sources.
+  permitTestUnusedDeclared project(':solr:core')
+  permitTestUnusedDeclared libs.apache.lucene.testframework
+  permitTestUnusedDeclared libs.carrotsearch.randomizedtesting.runner
+  permitTestUnusedDeclared libs.hamcrest.hamcrest
+}
+
+// Build a self-contained fat JAR so the agent can run before Solr's classpath is set up.
+// ByteBuddy is bundled; Solr classes are NOT included.
+//
+// Build a fat JAR so all agent classes (and ByteBuddy) are available on Boot-Class-Path before
+// the application main class loads. SLF4J is intentionally excluded from this JAR's dependencies:
+// placing SLF4J on Boot-Class-Path would initialise it before Log4j2 is loaded, permanently
+// caching a NOP logger and preventing solr.log from being created.
+jar {
+  manifest {
+    attributes(
+      'Premain-Class':           'org.apache.solr.security.agent.SolrAgentEntryPoint',
+      'Agent-Class':             'org.apache.solr.security.agent.SolrAgentEntryPoint',
+      'Can-Redefine-Classes':    'true',
+      'Can-Retransform-Classes': 'true',
+      'Boot-Class-Path':         archiveFileName
+    )
+  }
+
+  // Bundle runtime dependencies (ByteBuddy) into the fat agent JAR.
+  // Test dependencies are excluded because they are not on runtimeClasspath.
+  from {
+    configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
+  }
+  duplicatesStrategy = DuplicatesStrategy.EXCLUDE
+}
+
+// Copy the built agent JAR to server/lib/ext/ so startup scripts can find it at
+// ${SOLR_SERVER_DIR}/lib/ext/solr-agent-sm-*.jar, and so the dev distribution works.
+def agentJarTarget = file("${rootDir}/solr/server/lib/ext")
+
+task copyAgentJar(type: Copy, dependsOn: jar) {
+  description = 'Copies the agent fat JAR to solr/server/lib/ext/ for startup-script detection'
+  group = 'build'
+  from jar.outputs
+  into agentJarTarget
+  // Keep only the current version; remove stale versions on each build.
+  doFirst {
+    delete fileTree(agentJarTarget) { include 'solr-agent-sm-*.jar' }
+  }
+}
+
+// spotbugs-annotations is compileOnly and not included in the release.
+configurations.jarValidation {
+  exclude group: "com.github.spotbugs", module: "spotbugs-annotations"
+}
+
+assemble.dependsOn copyAgentJar
+
+// ---------------------------------------------------------------------------
+// Test programs — standalone Java programs compiled for BATS integration tests.
+// These have no Solr/ByteBuddy dependencies; pure JDK APIs only.
+// ---------------------------------------------------------------------------
+sourceSets {
+  testPrograms {
+    java.srcDirs = ['src/test-programs/java']
+  }
+}
+
+// These are tiny standalone programs with no annotation-processor needs.
+// Disabling annotation processing prevents the auto-created testProgramsAnnotationProcessor
+// configuration from pulling in error-prone and friends, which are not in the lock file.
+tasks.named('compileTestProgramsJava') {
+  options.annotationProcessorPath = files()
+}
+
+// Error-prone must also be disabled for compileTestProgramsJava: its Gradle plugin uses
+// tasks.withType(JavaCompile) and would inject -Xplugin:ErrorProne, but the
+// testProgramsAnnotationProcessor configuration has no errorprone JAR on it.
+plugins.withId(libs.plugins.ltgt.errorprone.get().pluginId) {
+  tasks.named('compileTestProgramsJava') {
+    options.errorprone.enabled = false
+  }
+}
+
+task testProgramsJar(type: Jar) {
+  description = 'JAR of standalone violation programs used by BATS integration tests'
+  group = 'build'
+  archiveClassifier = 'test-programs'
+  from sourceSets.testPrograms.output
+}
+