Skip to content

Commit 8ad2b85

Browse files
tomkralidistomkralidis
committed
update distributed search functionality
1 parent a959bca commit 8ad2b85

64 files changed

Lines changed: 700 additions & 136 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎default-sample.yml‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# Authors: Tom Kralidis <tomkralidis@gmail.com>
44
# Angelos Tzotsos <tzotsos@gmail.com>
55
#
6-
# Copyright (c) 2024 Tom Kralidis
6+
# Copyright (c) 2026 Tom Kralidis
77
# Copyright (c) 2024 Angelos Tzotsos
88
#
99
# Permission is hereby granted, free of charge, to any person
@@ -55,7 +55,20 @@ profiles:
5555
- apiso
5656

5757
federatedcatalogues:
58-
- http://catalog.data.gov/csw
58+
- id: arctic-sdi-csw
59+
type: CSW
60+
title: Arctic SDI
61+
url: https://catalogue.arctic-sdi.org/csw
62+
- id: pycsw-cite-demo
63+
type: OARec
64+
title: pycsw OGC CITE demo and Reference Implementation
65+
uresultsrl: https://demo.pycsw.org/cite
66+
- id: fedcat03
67+
type: STAC-API
68+
title: Copernicus Data Space Ecosystem (CDSE) asset-level STAC catalogue
69+
url: https://stac.dataspace.copernicus.eu/v1
70+
collections:
71+
- daymet-annual-pr
5972

6073
manager:
6174
transactions: false

‎docker/compose/pycsw.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# Authors: Tom Kralidis <tomkralidis@gmail.com>
44
# Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
55
#
6-
# Copyright (c) 2024 Tom Kralidis
6+
# Copyright (c) 2026 Tom Kralidis
77
# Copyright (c) 2017 Ricardo Garcia Silva
88
#
99
# Permission is hereby granted, free of charge, to any person
@@ -52,7 +52,10 @@ profiles:
5252
- apiso
5353

5454
federatedcatalogues:
55-
- http://catalog.data.gov/csw
55+
- id: fedcat01
56+
type: CSW
57+
title: Arctic SDI
58+
url: https://catalogue.arctic-sdi.org/csw
5659

5760
manager:
5861
transactions: false

‎docker/helm/values.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,10 @@ pycsw:
4848
profiles:
4949
- apiso
5050
# federatedcatalogues:
51-
# - http://catalog.data.gov/csw
51+
# - id: fedcat01
52+
# type: CSW
53+
# title: Arctic SDI
54+
# url: https://catalogue.arctic-sdi.org/csw
5255
manager:
5356
transactions: "false"
5457
allowed_ips:

‎docker/kubernetes/pycsw-configmap.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ data:
77
# Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
88
# Angelos Tzotsos <tzotsos@gmail.com>
99
#
10-
# Copyright (c) 2024 Tom Kralidis
10+
# Copyright (c) 2026 Tom Kralidis
1111
# Copyright (c) 2017 Ricardo Garcia Silva
1212
# Copyright (c) 2024 Angelos Tzotsos
1313
#
@@ -57,7 +57,10 @@ data:
5757
- apiso
5858
5959
federatedcatalogues:
60-
- http://catalog.data.gov/csw
60+
- id: fedcat01
61+
type: CSW
62+
title: Arctic SDI
63+
url: https://catalogue.arctic-sdi.org/csw
6164
6265
manager:
6366
transactions: false

‎docker/pycsw.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# Authors: Tom Kralidis <tomkralidis@gmail.com>
44
# Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
55
#
6-
# Copyright (c) 2024 Tom Kralidis
6+
# Copyright (c) 2026 Tom Kralidis
77
# Copyright (c) 2017 Ricardo Garcia Silva
88
#
99
# Permission is hereby granted, free of charge, to any person
@@ -52,7 +52,10 @@ profiles:
5252
- apiso
5353

5454
federatedcatalogues:
55-
- http://catalog.data.gov/csw
55+
- id: fedcat01
56+
type: CSW
57+
title: Arctic SDI
58+
url: https://catalogue.arctic-sdi.org/csw
5659

5760
manager:
5861
transactions: false

‎docs/configuration.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ pycsw's runtime configuration is defined by ``default.yml``. pycsw ships with a
1616
- **level**: the logging level (see https://docs.python.org/library/logging.html#logging-levels)
1717
- **logfile**: the full file path to the logfile
1818
- **ogc_schemas_base**: base URL of OGC XML schemas tree file structure (default is http://schemas.opengis.net)
19-
- **federatedcatalogues**: comma delimited list of CSW endpoints to be used for distributed searching, if requested by the client (see :ref:`distributedsearching`)
19+
- **federatedcatalogues**: arrray of distributed catalogue endpoints to be used for distributed searching, if requested by the client (see :ref:`distributedsearching`)
2020
- **pretty_print**: whether to pretty print the output (``true`` or ``false``). Default is ``false``
2121
- **gzip_compresslevel**: gzip compression level, lowest is ``1``, highest is ``9``. Default is off. **NOTE**: if gzip compression is already enabled via your web server, do not enable this directive (or else the server will try to compress the response twice, resulting in degraded performance)
2222
- **domainquerytype**: for GetDomain operations, how to output domain values. Accepted values are ``list`` and ``range`` (min/max). Default is ``list``

‎docs/distributedsearching.rst‎

Lines changed: 53 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,18 @@ in the CSW-all configuration:
5757
.. code-block:: yaml
5858
5959
federatedcatalogues:
60-
- http://localhost/pycsw/csw.py?config=CSW-1.yml
61-
- http://localhost/pycsw/csw.py?config=CSW-2.yml
62-
- http://localhost/pycsw/csw.py?config=CSW-3.yml
60+
- id: fedcat01
61+
type: CSW
62+
title: Federated catalogue 1
63+
url: http://localhost/pycsw/csw.py?config=CSW-1.yml
64+
- id: fedcat02
65+
type: CSW
66+
title: Federated catalogue 2
67+
url: http://localhost/pycsw/csw.py?config=CSW-2.yml
68+
- id: fedcat03
69+
type: CSW
70+
title: Federated catalogue 3
71+
url: http://localhost/pycsw/csw.py?config=CSW-3.yml
6372
6473
At which point a CSW client request to CSW-all with ``distributedsearch=TRUE``, while specifying an optional ``hopCount``. Query network topology:
6574

@@ -88,7 +97,7 @@ A very important facet of distributed search is as per Annex B of OGC:CSW 2.0.2.
8897
OGC API - Records
8998
-----------------
9099

91-
Experimental support for distibuted searching is available in pycsw's OGC API - Records support to allow for searching remote services. The implementation uses the same approach as described above, operating in OGC API - Records mode.
100+
Experimental support for distibuted searching is available in pycsw's OGC API - Records support to allow for searching remote services. The implementation uses the same approach as described above, operating in OGC API - Records mode as per `OGC API - Records - Part 4: Federated Search`_ (draft).
92101

93102
.. note::
94103

@@ -97,9 +106,46 @@ Experimental support for distibuted searching is available in pycsw's OGC API -
97106
.. code-block:: yaml
98107
99108
federatedcatalogues:
100-
- https://example.org/collections/collection1
101-
- https://example.org/collections/collection2
109+
- id: fedcat01
110+
type: OARec
111+
title: Federated catalogue 1
112+
url: https://example.org/collections/collection1
113+
- id: fedcat02
114+
type: OARec
115+
title: Federated catalogue 2
116+
url: https://example.org/collections/collection2
102117
103118
With the above configured, a distributed search can be invoked as follows:
104119

105-
http://localhost/collections/metadata:main/items?distributed=true
120+
http://localhost/collections/metadata:main/items?distributedSearch=true
121+
122+
STAC API
123+
--------
124+
125+
Experimental support for distibuted searching is available in pycsw's STAC API support to allow for searching remote services. The implementation uses the same approach as described above.
126+
127+
.. note::
128+
129+
The ``federatedcatalogues`` directives must point to a STAC API endpoint.
130+
131+
.. code-block:: yaml
132+
133+
federatedcatalogues:
134+
- id: fedcat03
135+
type: STAC-API
136+
title: Copernicus Data Space Ecosystem (CDSE) asset-level STAC catalogue
137+
url: https://stac.dataspace.copernicus.eu/v1
138+
collections:
139+
- daymet-annual-pr
140+
141+
142+
.. note::
143+
144+
To constrain STAC API distributed search to specific collections, define one to many in the `collections` (array) directive.
145+
146+
147+
With the above configured, a distributed search can be invoked as follows:
148+
149+
http://localhost/stac/search?distributedSearch=true
150+
151+
.. _`OGC API - Records - Part 4: Federated Search`: https://github.com/opengeospatial/ogcapi-records/blob/master/extensions/federated-search/document.adoc

‎docs/introduction.rst‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ Features
99
========
1010

1111
- implements `OGC API - Records - Part 1: Core`_
12+
- implements `OGC API - Records - Part 2: Facets`_
13+
- implements `OGC API - Records - Part 3: Create, Replace, Update, Delete, Harvest`_
14+
- implements `OGC API - Records - Part 4: Federated Search`_
1215
- implements `OGC API - Features - Part 3: Filtering`_
1316
- implements `STAC API`_
1417
- implements `Common Query Language (CQL2)`_
@@ -235,7 +238,10 @@ Paging
235238

236239
- resumptionToken
237240

238-
.. _`OGC API - Records - Part 1: Core`: https://ogcapi.ogc.org/records
241+
.. _`OGC API - Records - Part 1: Core`: https://docs.ogc.org/is/20-004r1/20-004r1.html
242+
.. _`OGC API - Records - Part 2: Facets`: https://docs.ogc.org/DRAFTS/25-013.html
243+
.. _`OGC API - Records - Part 3: Create, Replace, Update, Delete, Harvest`: https://docs.ogc.org/DRAFTS/25-015.html
244+
.. _`OGC API - Records - Part 4: Federated Search`: https://github.com/opengeospatial/ogcapi-records/blob/master/extensions/federated-search/document.adoc
239245
.. _`OGC API - Features - Part 3: Filtering`: http://docs.ogc.org/DRAFTS/19-079.html
240246
.. _`Common Query Language (CQL2)`: https://docs.ogc.org/DRAFTS/21-065.html
241247
.. _`OGC CSW`: https://www.ogc.org/standards/cat

‎pycsw/core/admin.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# Authors: Tom Kralidis <tomkralidis@gmail.com>
55
# Angelos Tzotsos <tzotsos@gmail.com>
66
#
7-
# Copyright (c) 2024 Tom Kralidis
7+
# Copyright (c) 2026 Tom Kralidis
88
# Copyright (c) 2015 Angelos Tzotsos
99
#
1010
# Permission is hereby granted, free of charge, to any person
@@ -632,7 +632,9 @@ def cli_migrate_config(ctx, config, verbosity):
632632
elif name == 'profiles':
633633
dict_[name] = value.split(',')
634634
elif name == 'federatedcatalogues':
635-
dict_[name] = value.split(',')
635+
dict_[name] = []
636+
for count, fc in enumerate(value.split(',')):
637+
dict_[name].append({'id': f'fedcat{count}', 'url': fc})
636638
else:
637639
dict_['server'][name] = get_typed_value(value)
638640

‎pycsw/ogc/api/oapi.py‎

Lines changed: 60 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
#
33
# Authors: Tom Kralidis <tomkralidis@gmail.com>
44
#
5-
# Copyright (c) 2024 Tom Kralidis
5+
# Copyright (c) 2026 Tom Kralidis
66
#
77
# Permission is hereby granted, free of charge, to any person
88
# obtaining a copy of this software and associated documentation
@@ -199,10 +199,10 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
199199
'style': 'form',
200200
'explode': False
201201
}
202-
oapi['components']['parameters']['distributed'] = {
203-
'name': 'distributed',
202+
oapi['components']['parameters']['distributedSearch'] = {
203+
'name': 'distributedSearch',
204204
'in': 'query',
205-
'description': 'Whether to invoke distributed mode',
205+
'description': 'Whether to invoke distributed search',
206206
'schema': {
207207
'type': 'boolean',
208208
'default': False
@@ -390,6 +390,60 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
390390
oapi['paths']['/collections/{collectionId}/queryables'] = path2
391391
oapi['components']['parameters']['collectionId']['default'] = 'metadata:main' # noqa
392392

393+
path = {
394+
'get': {
395+
'tags': ['Federated catalogs'],
396+
'summary': 'Federated catalogs page',
397+
'description': 'Federated catalogs page',
398+
'operationId': 'getFederatedCatalogs',
399+
'parameters': [
400+
{'$ref': '#/components/parameters/collectionId'},
401+
{'$ref': '#/components/parameters/f'}
402+
],
403+
'responses': {
404+
'200': {
405+
'$ref': '#/components/responses/FederatedCatalogs'
406+
},
407+
'500': {
408+
'$ref': '#/components/responses/ServerError'
409+
}
410+
}
411+
}
412+
}
413+
414+
oapi['paths']['/collections/{collectionId}/federatedCatalogs'] = path
415+
416+
path = {
417+
'get': {
418+
'tags': ['Federated catalogs'],
419+
'summary': 'Federated catalogs page',
420+
'description': 'Federated catalogs page',
421+
'operationId': 'getFederatedCatalog',
422+
'parameters': [
423+
{'$ref': '#/components/parameters/collectionId'},
424+
{'name': 'catalogId',
425+
'in': 'path',
426+
'description': 'catalog ID',
427+
'required': True,
428+
'schema': {
429+
'type': 'string'
430+
}
431+
},
432+
{'$ref': '#/components/parameters/f'}
433+
],
434+
'responses': {
435+
'200': {
436+
'$ref': '#/components/responses/FederatedCatalog'
437+
},
438+
'500': {
439+
'$ref': '#/components/responses/ServerError'
440+
}
441+
}
442+
}
443+
}
444+
445+
oapi['paths']['/collections/{collectionId}/federatedCatalogs/{catalogId}'] = path
446+
393447
path = {
394448
'get': {
395449
'tags': ['metadata'],
@@ -411,7 +465,7 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
411465
{'$ref': '#/components/parameters/f'},
412466
{'$ref': '#/components/parameters/offset'},
413467
{'$ref': '#/components/parameters/facets'},
414-
{'$ref': '#/components/parameters/distributed'},
468+
{'$ref': '#/components/parameters/distributedSearch'},
415469
{'$ref': '#/components/parameters/vendorSpecificParameters'}
416470
],
417471
'responses': {
@@ -488,7 +542,7 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
488542
'parameters': [
489543
{'$ref': '#/components/parameters/collectionId'},
490544
{'$ref': '#/components/parameters/recordId'},
491-
{'$ref': '#/components/parameters/distributed'},
545+
{'$ref': '#/components/parameters/distributedSearch'},
492546
f
493547
],
494548
'responses': {

0 commit comments

Comments
 (0)