Merge pull request #499 from IBM/issue-317

Issue #317 - Composite Search

Author: prb112

docs/src/pages/Conformance.md

@@ -7,12 +7,12 @@ permalink: /conformance/
 ---
 
 # Conformance to the HL7 FHIR Specification
-The IBM FHIR Server aims to be a conformant implementation of the HL7 FHIR specification, version 4.0.0 (R4). However, the FHIR specification is very broad and not all implementations are expected to implement every feature. We prioritize performance and configurability over spec coverage.
+The IBM FHIR Server aims to be a conformant implementation of the HL7 FHIR specification, version 4.0.1 (R4). However, the FHIR specification is very broad and not all implementations are expected to implement every feature. We prioritize performance and configurability over spec coverage.
 
 ## Capability statement
 The HL7 FHIR specification defines [an interaction](https://www.hl7.org/fhir/R4/http.html#capabilities) for retrieving a machine-readable description of the server's capabilities via the `[base]/metadata` endpoint. The IBM FHIR Server implements this interaction and generates a `CapabilityStatement` resource based on the current server configuration. While the `CapabilityStatement` resource is ideal for certain uses, this markdown document provides a human-readable summary of important details, with a special focus on limitations of the current implementation and deviations from the specification.
 
-The IBM FHIR Server supports only version 4.0.0 of the specification and presently has no support for the MIME-type parameter `fhirVersion`.
+The IBM FHIR Server supports only version 4.0.1 of the specification and presently has no support for the MIME-type parameter `fhirVersion`.
 
 ## FHIR HTTP API
 The HL7 FHIR specification is more than just a data format. It defines an [HTTP API](https://www.hl7.org/fhir/R4/http.html) for creating, reading, updating, deleting, and searching over FHIR resources. The IBM FHIR Server implements almost the full API for every resource defined in the specification, with the following exceptions:
@@ -24,17 +24,21 @@ The IBM FHIR Server implements a linear versioning scheme for resources and full
 ### General parameters
 The `_format` parameter is supported and provides a useful mechanism for requesting a specific format (`XML` or `JSON`) in requests made from a browser. In the absence of either an `Accept` header or a `_format` query parameter, the server defaults to `application/fhir+json`.
 
-The `_pretty` parameter is also supported. 
+The `_pretty` parameter is also supported.
 
 The `_summary` and `_elements` parameters are supported on the search interaction as documented.
 
 ## Search
-The IBM FHIR Server supports search parameters of type `Number`, `Date/DateTime`, `String`, `Token`, `Reference`, `Quantity`, and `URI`.
-
-Search parameters of type [Composite](https://www.hl7.org/fhir/R4/search.html#composite) and [Special](https://www.hl7.org/fhir/R4/search.html#special) are not currently supported.
-
-For all other types, the IBM FHIR Server supports the parameters defined in the
-specification and allows for the configuration of additional ones.
+The IBM FHIR Server supports all search parameter types defined in the specification:
+* `Number`
+* `Date/DateTime`
+* `String`
+* `Token`
+* `Reference`
+* `Composite`
+* `Quantity`
+* `URI`
+* `Special` (Location-near)
 
 ### Search parameters
 Search parameters defined in the specification can be found by browsing the R4 FHIR specification by resource type. For example, to find the search parameters for the Patient resource, navigate to https://www.hl7.org/fhir/R4/patient.html and scroll to the Search Parameters section near the end of the page.
@@ -46,10 +50,14 @@ In addition, the following search parameters are supported on all resources:
 * `_profile`
 * `_security`
 * `_source`
+* `_type`
 
-The `_text`, `_content`, `_list`, `_has`, `_type`, `_query`, and `_filter` parameters are not supported at this time.
+These parameters can be used while searching any single resource type or while searching across resource types (whole system search).
+The `_type` parameter is special in that it is only applicable for whole system search.
 
-Finally, the specification defines a set of <q>Search result parameters</q> for controlling the search behavior. The IBM FHIR Server supports the following:
+The `_text`, `_content`, `_list`, `_has`, `_query`, and `_filter` parameters are not supported at this time.
+
+Finally, the specification defines a set of "Search result parameters" for controlling the search behavior. The IBM FHIR Server supports the following:
 * `_sort`
 * `_count`
 * `_include`
@@ -71,24 +79,24 @@ For information on how to specify custom search parameters, see [FHIRSearchConfi
 ### Search modifiers
 FHIR search modifiers are described at https://www.hl7.org/fhir/R4/search.html#modifiers and vary by search parameter type. The IBM FHIR Server implements a subset of the spec-defined search modifiers that is defined in the following table:
 
-|FHIR Search Parameter Type|Supported Modifiers|"Default" search behavior when no Modifier is present|
+|FHIR Search Parameter Type|Supported Modifiers|"Default" search behavior when no Modifier or Prefix is present|
 |--------------------------|-------------------|-----------------------------------------------------|
-|String                 |`:exact`,`:contains`,`:missing` |Performs a "starts with" search that is case-insensitive and accent-insensitive|
-|Reference              |`:[type]`,`:missing`            |Performs an exact match search|
-|URI                    |`:below`,`:above`,`:missing`    |Performs a "starts with" search (issue #273), component based search for the path in the URL (issue #448)|
-|Token                  |`:below`,`:not`,`:missing`      |Performs an exact match search|
-|Number                 |`:missing`                      |Honors prefix if present, otherwise performs an exact match search|
-|Date                   |`:missing`                      |Honors prefix if present, otherwise performs an exact match search|
-|Quantity               |`:missing`                      |Honors prefix if present, otherwise performs an exact match search|
+|String                    |`:exact`,`:contains`,`:missing` |"starts with" search that is case-insensitive and accent-insensitive|
+|Reference                 |`:[type]`,`:missing`            |exact match search|
+|URI                       |`:below`,`:above`,`:missing`    |exact match search|
+|Token                     |`:below`,`:not`,`:missing`      |exact match search|
+|Number                    |`:missing`                      |exact match search|
+|Date                      |`:missing`                      |exact match search|
+|Quantity                  |`:missing`                      |implicit range search (see http://hl7.org/fhir/R4/search.html#quantity)|
+|Composite                 |`:missing`                      |processes each parameter component according to its type|
+|Special (near)            | none                           |searches a bounding area according to the value of the `fhirServer/search/useBoundingRadius` property|
 
-Note that the default IBM FHIR Server behavior for URI search parameters differs from the behavior defined at https://www.hl7.org/fhir/R4/search.html#uri.
+Due to performance implications, the `:exact` modifier should be used for String searches where possible.
 
 At present, modifiers cannot be used with chained parameters. For example, a search with query string like `subject:Basic.date:missing` will result in an `OperationOutcome` explaining that the search parameter could not be processed.
 
 The `:text` modifier is not supported in this version of the FHIR server and use of this modifier will results in an HTTP 400 error with an `OperationOutcome` that describes the failure.
 
-Due to performance implications, the `:exact` modifier should be used for String searches where possible.
-
 ### Search prefixes
 FHIR search prefixes are described at https://www.hl7.org/fhir/R4/search.html#prefix.
 
@@ -147,20 +155,24 @@ For search parameters of type token that are defined on data fields of type `Con
 Searching string values via a token search parameter is not currently supported.
 
 ### Searching on Number
-For fields of type `decimal`, the IBM FHIR Server does not compute an implicit range for search. Search query values must match to the same precision, or greater precision, as the value in the resource.
+For fields of type `decimal`, the IBM FHIR Server computes an implicit range when the query parameter value has a prefix of `eq` (the default), `ne`, or `ap`. The computed range is based on the number of significant figures passed in the query string and further information can be found at https://www.hl7.org/fhir/R4/search.html#number.
+For searches with the `ap` prefix, we use the range `[implicitLowerBound - searchQueryValue * .1, implicitUpperBound + searchQueryValue * .1)` to ensure that the `ap` range is broader than the implicit range of `eq`.
 
 ### Searching on Quantity
-Quantity elements are not indexed unless they include either a valid `system` **and** `code` for their unit **or** a human-readable `unit` field. Quantities that don't include a `value` element are also skipped.
+Quantity elements are not indexed unless they include either a valid `system` **and** `code` for their unit **or** a human-readable `unit` field.
+If a Quantity element contains both a coded unit **and** a display unit, then both will be indexed. Quantities that don't include a `value` element are also skipped.
+
+The FHIR server does not perform any unit conversion or unit manipulation at this time. Quantity values should be searched using the same unit `code` that is included in the original resource.
 
-The FHIR server does not perform any unit conversion or unit manipulation at this time. Quantity values **must** be searched using the same unit `code` that is included in the original resource. If, and only if, a coded unit is not present on a resource, then the FHIR server indexes the human-readable `unit` field, which can then be searched by omitting the `system` in the search query.
+Similar to Numeric searches, the FHIR Server computes an implicit range for search query values with no range prefix (e.g. `eq`, `ne`, `ap`) based on the number of significant figures passed in the query string.
+For searches with the `ap` prefix, we use the range `[implicitLowerBound - searchQueryValue * .1, implicitUpperBound + searchQueryValue * .1)` to ensure that the `ap` range is broader than the implicit range of `eq`.
 
 The IBM FHIR Server does not consider the `Quantity.comparator` field as part of search processing at this time.
-Similar to Numeric searches, the FHIR Server does not compute an implicit range for quantity values...the precise number given in the resource is required for retrieval via standard search (i.e. searches with either no prefix or the `eq` prefix).
 
 ### Searching on URI
-URI searches on the IBM FHIR Server are case-insensitive, similar to the default behavior of searching on string values.
+URI searches on the IBM FHIR Server are case-sensitive with "exact-match" semantics. The `above` and `below` prefixes can be used to perform path-based matching that is based on the `/` delimiter.
 
-## HL7 FHIR R4 (v4.0.0) errata
+## HL7 FHIR R4 (v4.0.1) errata
 We add information here as we find issues with the artifacts provided with this version of the specification.
 
-FHIR® is the registered trademark of HL7 and is used with the permission of HL7.
+FHIR® is the registered trademark of HL7 and is used with the permission of HL7.

fhir-database-utils/src/main/java/com/ibm/fhir/database/utils/api/IDatabaseAdapter.java

@@ -11,6 +11,7 @@
 import java.util.function.Supplier;
 
 import com.ibm.fhir.database.utils.model.ColumnBase;
+import com.ibm.fhir.database.utils.model.IdentityDef;
 import com.ibm.fhir.database.utils.model.PrimaryKeyDef;
 import com.ibm.fhir.database.utils.model.Privilege;
 import com.ibm.fhir.database.utils.model.Table;
@@ -71,10 +72,11 @@ public interface IDatabaseAdapter {
      * @param tenantColumnName optional column name to enable multi-tenancy
      * @param columns
      * @param primaryKey
+     * @param identity
      * @param tablespaceName
      */
     public void createTable(String schemaName, String name, String tenantColumnName, List<ColumnBase> columns,
-            PrimaryKeyDef primaryKey, String tablespaceName);
+            PrimaryKeyDef primaryKey, IdentityDef identity, String tablespaceName);
 
     /**
      * Create ROW type used for passing values to stored procedures e.g.:

fhir-database-utils/src/main/java/com/ibm/fhir/database/utils/common/CommonDatabaseAdapter.java

@@ -27,6 +27,7 @@
 import com.ibm.fhir.database.utils.api.TenantStatus;
 import com.ibm.fhir.database.utils.api.UndefinedNameException;
 import com.ibm.fhir.database.utils.model.ColumnBase;
+import com.ibm.fhir.database.utils.model.IdentityDef;
 import com.ibm.fhir.database.utils.model.PrimaryKeyDef;
 import com.ibm.fhir.database.utils.model.Privilege;
 import com.ibm.fhir.database.utils.model.Tenant;
@@ -82,10 +83,8 @@ public IDatabaseTranslator getTranslator() {
 
     /**
      * Build the list of columns in the create table statement
-     * @param columns
-     * @return
      */
-    protected String buildColumns(List<ColumnBase> columns) {
+    protected String buildColumns(List<ColumnBase> columns, IdentityDef identity) {
         StringBuilder result = new StringBuilder();
         for (ColumnBase column: columns) {
             if (result.length() > 0) {
@@ -95,7 +94,10 @@ protected String buildColumns(List<ColumnBase> columns) {
             result.append(column.getName());
             result.append(" ");
             result.append(column.getTypeInfo(this));
-            if (!column.isNullable()) {
+            if (identity != null && column.getName().equals(identity.getColumnName())) {
+                result.append(" GENERATED " + identity.getGenerated() + " AS IDENTITY");
+            } // AS IDENTITY implies NOT NULL so this can be and else if
+            else if (!column.isNullable()) {
                 result.append(" NOT NULL");
             }
         }
@@ -114,12 +116,12 @@ protected String buildColumns(List<ColumnBase> columns) {
      * @param tablespaceName
      * @return
      */
-    protected String buildCreateTableStatement(String schema, String name, List<ColumnBase> columns, PrimaryKeyDef pkDef, String tablespaceName) {
+    protected String buildCreateTableStatement(String schema, String name, List<ColumnBase> columns, PrimaryKeyDef pkDef, IdentityDef identity, String tablespaceName) {
         StringBuilder result = new StringBuilder();
         result.append("CREATE TABLE ");
         result.append(getQualifiedName(schema, name));
-        result.append("(");
-        result.append(buildColumns(columns));
+        result.append('(');
+        result.append(buildColumns(columns, identity));
 
         // Add the primary key definition after the columns
         if (pkDef != null) {
@@ -136,9 +138,9 @@ protected String buildCreateTableStatement(String schema, String name, List<Colu
             }
 
             result.append(cols);
-            result.append(")");
+            result.append(')');
         }
-        result.append(")");
+        result.append(')');
 
         if (tablespaceName != null) {
             DataDefinitionUtil.assertValidName(tablespaceName);

fhir-database-utils/src/main/java/com/ibm/fhir/database/utils/db2/Db2Adapter.java

@@ -28,6 +28,7 @@
 import com.ibm.fhir.database.utils.common.CommonDatabaseAdapter;
 import com.ibm.fhir.database.utils.common.DataDefinitionUtil;
 import com.ibm.fhir.database.utils.model.ColumnBase;
+import com.ibm.fhir.database.utils.model.IdentityDef;
 import com.ibm.fhir.database.utils.model.IntColumn;
 import com.ibm.fhir.database.utils.model.PrimaryKeyDef;
 import com.ibm.fhir.database.utils.model.Table;
@@ -54,7 +55,7 @@ public Db2Adapter(IConnectionProvider cp) {
 
     @Override
     public void createTable(String schemaName, String name, String tenantColumnName, List<ColumnBase> columns, PrimaryKeyDef primaryKey,
-            String tablespaceName) {
+            IdentityDef identity, String tablespaceName) {
 
         // With DB2 we can implement support for multi-tenancy, which we do by injecting a MT_ID column
         // to the definition and partitioning on that column
@@ -78,7 +79,7 @@ public void createTable(String schemaName, String name, String tenantColumnName,
         // Now append all the actual columns we want in the table
         cols.addAll(columns);
 
-        String ddl = buildCreateTableStatement(schemaName, name, cols, primaryKey, tablespaceName);
+        String ddl = buildCreateTableStatement(schemaName, name, cols, primaryKey, identity, tablespaceName);
 
         // Our multi-tenant tables are range-partitioned as part of our data isolation strategy
         // We reserve partition 0. Real tenant partitions start at 1...

fhir-database-utils/src/main/java/com/ibm/fhir/database/utils/derby/DerbyAdapter.java

@@ -18,6 +18,7 @@
 import com.ibm.fhir.database.utils.common.CommonDatabaseAdapter;
 import com.ibm.fhir.database.utils.common.DataDefinitionUtil;
 import com.ibm.fhir.database.utils.model.ColumnBase;
+import com.ibm.fhir.database.utils.model.IdentityDef;
 import com.ibm.fhir.database.utils.model.PrimaryKeyDef;
 import com.ibm.fhir.database.utils.model.Table;
 
@@ -59,15 +60,15 @@ public void warnOnce(MessageKey messageKey, String msg) {
 
     @Override
     public void createTable(String schemaName, String name, String tenantColumnName, List<ColumnBase> columns, PrimaryKeyDef primaryKey,
-            String tablespaceName) {
+            IdentityDef identity, String tablespaceName) {
 
         // Derby doesn't support partitioning, so we ignore tenantColumnName
         if (tenantColumnName != null) {
             warnOnce(MessageKey.MULTITENANCY, "Derby does support not multi-tenancy: " + name);
         }
 
         // We also ignore tablespace for Derby
-        String ddl = buildCreateTableStatement(schemaName, name, columns, primaryKey, null);
+        String ddl = buildCreateTableStatement(schemaName, name, columns, primaryKey, identity, null);
 
 
         runStatement(ddl);

fhir-database-utils/src/main/java/com/ibm/fhir/database/utils/model/Generated.java

@@ -0,0 +1,27 @@
+/*
+ * (C) Copyright IBM Corp. 2019
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package com.ibm.fhir.database.utils.model;
+
+/**
+ * When to generate a value for an identity column.
+ */
+public enum Generated {
+    ALWAYS, BY_DEFAULT;
+    
+    /**
+     * @return the SQL keyword or phrase for this Generated option
+     */
+    @Override
+    public String toString() {
+        switch (this) {
+        case BY_DEFAULT:
+            return "BY DEFAULT";
+        default:
+            return this.name();
+        }
+    }
+}

fhir-database-utils/src/main/java/com/ibm/fhir/database/utils/model/IdentityDef.java

@@ -0,0 +1,38 @@
+/*
+ * (C) Copyright IBM Corp. 2019
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package com.ibm.fhir.database.utils.model;
+
+/**
+ * Represents the definition of a primary key constraint on a table
+ */
+public class IdentityDef {
+
+    // The name of the column to be the identity
+    public final String column;
+
+    // The list of columns comprising the primary key
+    public final Generated generated;
+
+    public IdentityDef(String columnName, Generated generatedPref) {
+        this.column = columnName;
+        this.generated = generatedPref;
+    }
+
+    /**
+     * Getter for the name of the identity column
+     */
+    public String getColumnName() {
+        return this.column;
+    }
+
+    /**
+     * Getter for the generation preference
+     */
+    public Generated getGenerated() {
+        return this.generated;
+    }
+}

fhir-database-utils/src/main/java/com/ibm/fhir/database/utils/model/IndexDef.java

@@ -67,10 +67,10 @@ public void apply(String schemaName, String tableName, String tenantColumnName,
             target.createUniqueIndex(schemaName, tableName, indexName, tenantColumnName, indexColumns, includeColumns);
         }
         else if (unique) {
-            target.createUniqueIndex(schemaName, tableName, indexName, tenantColumnName, indexColumns);            
+            target.createUniqueIndex(schemaName, tableName, indexName, tenantColumnName, indexColumns);
         }
         else {
-            target.createIndex(schemaName, tableName, indexName, tenantColumnName, indexColumns);            
+            target.createIndex(schemaName, tableName, indexName, tenantColumnName, indexColumns);
         }
     }
 }