upgrade to docusaurus3 (#527)

Author: 317brian

README.md

@@ -25,15 +25,25 @@ If you need to update the contents of Upcoming Events, Featured Content or Lates
 
 If you only need to update the homepage widgets,  use the `--no-docs` flag (as shown in example 5) to build the site. This uses the existing docs in `latest` and only builds `latest`, which is what's used for the homepage and other non-doc pages.
 
-## Building the site for Druid 26 and later
+## Building the site for Druid
+
+> Note that different versions of Druid use different versions of Docusaurus due to upgrades
 
 By default, the repo always starts/builds latest if you run `npm start|build` or `yarn start|build`. This way, you can always view the site locally.
 
 The build scripts described in [Publish the site](#publish-the-site) handles building the versioned docs for when you're ready to do a release.
 
-To start, you'll need to install Docusaurus 2.  
+To start, you'll need to install Docusaurus 3.  
+
+### Docusaurus version
+
+Different Druid versions run different docusaurus versions due to upgrades. The versions are only somewhat compatible. The publishing process for Docusaurus 2 and 3 are the same. The scripts were upgraded behind the scenes, so the user-facing commands are still the same. Druid verisons before 26 (Docusaurus) 1 use a [different process](#building-the-site-before-druid-26).
+
+- Druid 32.0.1: Docusaurus 3
+- Druid 26-32.0.0 : Docusaurus 2 
+- Druid versions earlier than 26: Docusaurus 1
 
-### Install Docusaurus 2
+### Install Docusaurus 
 
 You need a supported version of node, such as the latest Node 18.
 

docs/32.0.0/api-reference/sql-ingestion-api.md

@@ -379,7 +379,8 @@ print(response.text)
 
 The response shows an example report for a query.
 
-<details><summary>View the response</summary>
+<details>
+<summary>View the response</summary>
 
 ```json
 {

docs/32.0.0/comparisons/druid-vs-spark.md

@@ -39,4 +39,4 @@ One typical setup seen in production is to process data in Spark, and load the p
 
 For more information about using Druid and Spark together, including benchmarks of the two systems, please see:
 
-<https://www.linkedin.com/pulse/combining-druid-spark-interactive-flexible-analytics-scale-butani>
+https://www.linkedin.com/pulse/combining-druid-spark-interactive-flexible-analytics-scale-butani

docs/32.0.0/configuration/extensions.md

@@ -54,7 +54,6 @@ Core extensions are maintained by Druid committers.
 |druid-orc-extensions|Support for data in Apache ORC data format.|[link](../development/extensions-core/orc.md)|
 |druid-parquet-extensions|Support for data in Apache Parquet data format. Requires druid-avro-extensions to be loaded.|[link](../development/extensions-core/parquet.md)|
 |druid-protobuf-extensions| Support for data in Protobuf data format.|[link](../development/extensions-core/protobuf.md)|
-|druid-ranger-security|Support for access control through Apache Ranger.|[link](../development/extensions-core/druid-ranger-security.md)|
 |druid-s3-extensions|Interfacing with data in Amazon S3, and using S3 as deep storage.|[link](../development/extensions-core/s3.md)|
 |druid-ec2-extensions|Interfacing with AWS EC2 for autoscaling middle managers|UNDOCUMENTED|
 |druid-aws-rds-extensions|Support for AWS token based access to AWS RDS DB Cluster.|[link](../development/extensions-core/druid-aws-rds.md)|
@@ -64,6 +63,7 @@ Core extensions are maintained by Druid committers.
 |simple-client-sslcontext|Simple SSLContext provider module to be used by Druid's internal HttpClient when talking to other Druid processes over HTTPS.|[link](../development/extensions-core/simple-client-sslcontext.md)|
 |druid-pac4j|OpenID Connect authentication for druid processes.|[link](../development/extensions-core/druid-pac4j.md)|
 |druid-kubernetes-extensions|Druid cluster deployment on Kubernetes without Zookeeper.|[link](../development/extensions-core/kubernetes.md)|
+|druid-kubernetes-overlord-extensions|Support for launching tasks in k8s without Middle Managers|[link](../development/extensions-core/k8s-jobs.md)|
 
 ## Community extensions
 
@@ -101,10 +101,10 @@ All of these community extensions can be downloaded using [pull-deps](../operati
 |druid-momentsketch|Support for approximate quantile queries using the [momentsketch](https://github.com/stanford-futuredata/momentsketch) library|[link](../development/extensions-contrib/momentsketch-quantiles.md)|
 |druid-tdigestsketch|Support for approximate sketch aggregators based on [T-Digest](https://github.com/tdunning/t-digest)|[link](../development/extensions-contrib/tdigestsketch-quantiles.md)|
 |gce-extensions|GCE Extensions|[link](../development/extensions-contrib/gce-extensions.md)|
-|prometheus-emitter|Exposes [Druid metrics](../operations/metrics.md) for Prometheus server collection (<https://prometheus.io/>)|[link](../development/extensions-contrib/prometheus.md)|
-|druid-kubernetes-overlord-extensions|Support for launching tasks in k8s without Middle Managers|[link](../development/extensions-contrib/k8s-jobs.md)|
+|prometheus-emitter|Exposes [Druid metrics](../operations/metrics.md) for Prometheus server collection (https://prometheus.io/)|[link](../development/extensions-contrib/prometheus.md)|
 |druid-spectator-histogram|Support for efficient approximate percentile queries|[link](../development/extensions-contrib/spectator-histogram.md)|
 |druid-rabbit-indexing-service|Support for creating and managing [RabbitMQ](https://www.rabbitmq.com/) indexing tasks|[link](../development/extensions-contrib/rabbit-stream-ingestion.md)|
+|druid-ranger-security|Support for access control through Apache Ranger.|[link](../development/extensions-contrib/druid-ranger-security.md)|
 
 ## Promoting community extensions to core extensions
 

docs/32.0.0/configuration/index.md

@@ -403,7 +403,7 @@ Metric monitoring is an essential part of Druid operations. The following monito
 |`org.apache.druid.server.metrics.SegmentStatsMonitor` | **EXPERIMENTAL** Reports statistics about segments on Historical services. Available only on Historical services. Not to be used when lazy loading is configured.|
 |`org.apache.druid.server.metrics.QueryCountStatsMonitor`|Reports how many queries have been successful/failed/interrupted.|
 |`org.apache.druid.server.metrics.SubqueryCountStatsMonitor`|Reports how many subqueries have been materialized as rows or bytes and various other statistics related to the subquery execution|
-|`org.apache.druid.server.emitter.HttpEmittingMonitor`|Reports internal metrics of `http` or `parametrized` emitter (see below). Must not be used with another emitter type. See the description of the metrics here: <https://github.com/apache/druid/pull/4973>.|
+|`org.apache.druid.server.emitter.HttpEmittingMonitor`|Reports internal metrics of `http` or `parametrized` emitter (see below). Must not be used with another emitter type. See the description of the metrics here: https://github.com/apache/druid/pull/4973.|
 |`org.apache.druid.server.metrics.TaskCountStatsMonitor`|Reports how many ingestion tasks are currently running/pending/waiting and also the number of successful/failed tasks per emission period.|
 |`org.apache.druid.server.metrics.TaskSlotCountStatsMonitor`|Reports metrics about task slot usage per emission period.|
 |`org.apache.druid.server.metrics.WorkerTaskCountStatsMonitor`|Reports how many ingestion tasks are currently running/pending/waiting, the number of successful/failed tasks, and metrics about task slot usage for the reporting worker, per emission period. Only supported by Middle Manager node types.|
@@ -882,6 +882,7 @@ These Coordinator static configurations can be defined in the `coordinator/runti
 |`druid.coordinator.kill.ignoreDurationToRetain`|A way to override `druid.coordinator.kill.durationToRetain` and tell the coordinator that you do not care about the end date of unused segment intervals when it comes to killing them. If true, the coordinator considers all unused segments as eligible to be killed.|false|
 |`druid.coordinator.kill.bufferPeriod`|The amount of time that a segment must be unused before it is able to be permanently removed from metadata and deep storage. This can serve as a buffer period to prevent data loss if data ends up being needed after being marked unused.|`P30D`|
 |`druid.coordinator.kill.maxSegments`|The number of unused segments to kill per kill task. This number must be greater than 0. This only applies when `druid.coordinator.kill.on=true`.|100|
+|`druid.coordinator.kill.maxInterval`|The largest interval, as an [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations), of segments to delete per kill task. Set to zero, e.g. `PT0S`, for unlimited. This only applies when `druid.coordinator.kill.on=true`.|`P30D`|
 |`druid.coordinator.balancer.strategy`|The [balancing strategy](../design/coordinator.md#balancing-segments-in-a-tier) used by the Coordinator to distribute segments among the Historical servers in a tier. The `cost` strategy distributes segments by minimizing a cost function, `diskNormalized` weights these costs with the disk usage ratios of the servers and `random` distributes segments randomly.|`cost`|
 |`druid.coordinator.loadqueuepeon.http.repeatDelay`|The start and repeat delay (in milliseconds) for the load queue peon, which manages the load/drop queue of segments for any server.|1 minute|
 |`druid.coordinator.loadqueuepeon.http.batchSize`|Number of segment load/drop requests to batch in one HTTP request. Note that it must be smaller than `druid.segmentCache.numLoadingThreads` config on Historical service.|1|
@@ -1181,7 +1182,8 @@ The following table shows the dynamic configuration properties for the Overlord.
 
 The following is an example of an Overlord dynamic config:
 
-<details><summary>Click to view the example</summary>
+<details>
+<summary>Click to view the example</summary>
 
 ```json
 {
@@ -2112,8 +2114,8 @@ The `druid.query.default.context.{query_context_key}` runtime property prefix ap
 
 The precedence chain for query context values is as follows:
 
-hard-coded default value in Druid code <- runtime property not prefixed with `druid.query.default.context`
-<- runtime property prefixed with `druid.query.default.context` <- context parameter in the query
+hard-coded default value in Druid code `<-` runtime property not prefixed with `druid.query.default.context`
+`<-` runtime property prefixed with `druid.query.default.context` `<-` context parameter in the query
 
 Note that not all query context key has a runtime property not prefixed with `druid.query.default.context` that can
 override the hard-coded default value. For example, `maxQueuedBytes` has `druid.broker.http.maxQueuedBytes`

docs/32.0.0/design/architecture.md

@@ -107,7 +107,7 @@ forking separate JVM processes per-task, the Indexer runs tasks as individual th
 
 The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system and to better enable resource sharing across tasks, which can help streaming ingestion. The Indexer is currently designated [experimental](../development/experimental.md).
 
-Typically, you would deploy one of the following: MiddleManagers, [MiddleManager-less ingestion using Kubernetes](../development/extensions-contrib/k8s-jobs.md), or Indexers. You wouldn't deploy more than one of these options.
+Typically, you would deploy one of the following: MiddleManagers, [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md), or Indexers. You wouldn't deploy more than one of these options.
 
 ## Colocation of services
 

docs/32.0.0/design/indexer.md

@@ -24,7 +24,7 @@ sidebar_label: "Indexer"
   -->
 
 :::info
- The Indexer is an optional and experimental feature. If you're primarily performing batch ingestion, we recommend you use either the MiddleManager and Peon task execution system or [MiddleManager-less ingestion using Kubernetes](../development/extensions-contrib/k8s-jobs.md). If you're primarily doing streaming ingestion, you may want to try either [MiddleManager-less ingestion using Kubernetes](../development/extensions-contrib/k8s-jobs.md) or the Indexer service.
+ The Indexer is an optional and experimental feature. If you're primarily performing batch ingestion, we recommend you use either the MiddleManager and Peon task execution system or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md). If you're primarily doing streaming ingestion, you may want to try either [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md) or the Indexer service.
 :::
 
 The Apache Druid Indexer service is an alternative to the Middle Manager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process.

docs/32.0.0/development/docs-contribute.md

@@ -34,7 +34,7 @@ Druid docs contributors:
 Druid docs contributors can open an issue about documentation, or contribute a change with a pull request (PR).
 
 The open source Druid docs are located here:
-<https://druid.apache.org/docs/latest/design/index.html>
+https://druid.apache.org/docs/latest/design/index.html
 
 If you need to update a Druid doc, locate and update the doc in the Druid repo following the instructions below.
 

…extensions-core/druid-ranger-security.md …ensions-contrib/druid-ranger-security.mddocs/32.0.0/development/extensions-core/druid-ranger-security.md renamed to docs/32.0.0/development/extensions-contrib/druid-ranger-security.md docs/32.0.0/development/extensions-core/druid-ranger-security.md renamed to docs/32.0.0/development/extensions-contrib/druid-ranger-security.md

@@ -149,27 +149,3 @@ Similar to quantilesFromTDigestSketch except it takes in a single fraction for c
 |name|A String for the output (result) name of the calculation.|yes|
 |field|A field reference pointing to the field aggregated/combined T-Digest sketch.|yes|
 |fraction|Decimal value between 0 and 1|yes|
-
-### SQL functions
-
-Once you load the T-Digest extension, you can use the following SQL functions.
-
-#### TDIGEST_GENERATE_SKETCH
-
-Builds a T-Digest sketch on values produced by an expression.
-Compression parameter (default value 100) determines the accuracy and size of the sketch.
-Higher compression provides higher accuracy but requires more storage space.
-
-* **Syntax**: `TDIGEST_GENERATE_SKETCH(expr, [compression])`
-* **Default**: Empty Base64-encoded T-Digest sketch string
-* **Function type**: [Aggregation](../../querying/sql-aggregations.md)
-
-#### TDIGEST_QUANTILE
-
-Builds a T-Digest sketch on values produced by an expression and returns the value for the quantile.
-Compression parameter (default value 100) determines the accuracy and size of the sketch.
-Higher compression provides higher accuracy but requires more storage space.
-
-* **Syntax**: `TDIGEST_QUANTILE(expr, quantileFraction, [compression])`
-* **Default**: `Double.NaN`
-* **Function type**: [Aggregation](../../querying/sql-aggregations.md)