Skip to content

Commit 58fb99c

Browse files
tomkralidistomkralidis
committed
update distributed search functionality
1 parent dbbd7e2 commit 58fb99c

62 files changed

Lines changed: 572 additions & 130 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎default-sample.yml‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# Authors: Tom Kralidis <tomkralidis@gmail.com>
44
# Angelos Tzotsos <tzotsos@gmail.com>
55
#
6-
# Copyright (c) 2024 Tom Kralidis
6+
# Copyright (c) 2026 Tom Kralidis
77
# Copyright (c) 2024 Angelos Tzotsos
88
#
99
# Permission is hereby granted, free of charge, to any person
@@ -55,7 +55,14 @@ profiles:
5555
- apiso
5656

5757
federatedcatalogues:
58-
- http://catalog.data.gov/csw
58+
- id: arctic-sdi-csw
59+
type: CSW
60+
title: Arctic SDI
61+
url: https://catalogue.arctic-sdi.org/csw
62+
- id: pycsw-cite-demo
63+
type: OARec
64+
title: pycsw OGC CITE demo and Reference Implementation
65+
url: https://demo.pycsw.org/cite
5966

6067
manager:
6168
transactions: false

‎docker/compose/pycsw.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# Authors: Tom Kralidis <tomkralidis@gmail.com>
44
# Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
55
#
6-
# Copyright (c) 2024 Tom Kralidis
6+
# Copyright (c) 2026 Tom Kralidis
77
# Copyright (c) 2017 Ricardo Garcia Silva
88
#
99
# Permission is hereby granted, free of charge, to any person
@@ -52,7 +52,10 @@ profiles:
5252
- apiso
5353

5454
federatedcatalogues:
55-
- http://catalog.data.gov/csw
55+
- id: fedcat01
56+
type: CSW
57+
title: Arctic SDI
58+
url: https://catalogue.arctic-sdi.org/csw
5659

5760
manager:
5861
transactions: false

‎docker/helm/values.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,10 @@ pycsw:
4848
profiles:
4949
- apiso
5050
# federatedcatalogues:
51-
# - http://catalog.data.gov/csw
51+
# - id: fedcat01
52+
# type: CSW
53+
# title: Arctic SDI
54+
# url: https://catalogue.arctic-sdi.org/csw
5255
manager:
5356
transactions: "false"
5457
allowed_ips:

‎docker/kubernetes/pycsw-configmap.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ data:
77
# Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
88
# Angelos Tzotsos <tzotsos@gmail.com>
99
#
10-
# Copyright (c) 2024 Tom Kralidis
10+
# Copyright (c) 2026 Tom Kralidis
1111
# Copyright (c) 2017 Ricardo Garcia Silva
1212
# Copyright (c) 2024 Angelos Tzotsos
1313
#
@@ -57,7 +57,10 @@ data:
5757
- apiso
5858
5959
federatedcatalogues:
60-
- http://catalog.data.gov/csw
60+
- id: fedcat01
61+
type: CSW
62+
title: Arctic SDI
63+
url: https://catalogue.arctic-sdi.org/csw
6164
6265
manager:
6366
transactions: false

‎docker/pycsw.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# Authors: Tom Kralidis <tomkralidis@gmail.com>
44
# Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
55
#
6-
# Copyright (c) 2024 Tom Kralidis
6+
# Copyright (c) 2026 Tom Kralidis
77
# Copyright (c) 2017 Ricardo Garcia Silva
88
#
99
# Permission is hereby granted, free of charge, to any person
@@ -52,7 +52,10 @@ profiles:
5252
- apiso
5353

5454
federatedcatalogues:
55-
- http://catalog.data.gov/csw
55+
- id: fedcat01
56+
type: CSW
57+
title: Arctic SDI
58+
url: https://catalogue.arctic-sdi.org/csw
5659

5760
manager:
5861
transactions: false

‎docs/configuration.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ pycsw's runtime configuration is defined by ``default.yml``. pycsw ships with a
1616
- **level**: the logging level (see https://docs.python.org/library/logging.html#logging-levels)
1717
- **logfile**: the full file path to the logfile
1818
- **ogc_schemas_base**: base URL of OGC XML schemas tree file structure (default is http://schemas.opengis.net)
19-
- **federatedcatalogues**: comma delimited list of CSW endpoints to be used for distributed searching, if requested by the client (see :ref:`distributedsearching`)
19+
- **federatedcatalogues**: arrray of distributed catalogue endpoints to be used for distributed searching, if requested by the client (see :ref:`distributedsearching`)
2020
- **pretty_print**: whether to pretty print the output (``true`` or ``false``). Default is ``false``
2121
- **gzip_compresslevel**: gzip compression level, lowest is ``1``, highest is ``9``. Default is off. **NOTE**: if gzip compression is already enabled via your web server, do not enable this directive (or else the server will try to compress the response twice, resulting in degraded performance)
2222
- **domainquerytype**: for GetDomain operations, how to output domain values. Accepted values are ``list`` and ``range`` (min/max). Default is ``list``

‎docs/distributedsearching.rst‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,18 @@ in the CSW-all configuration:
5757
.. code-block:: yaml
5858
5959
federatedcatalogues:
60-
- http://localhost/pycsw/csw.py?config=CSW-1.yml
61-
- http://localhost/pycsw/csw.py?config=CSW-2.yml
62-
- http://localhost/pycsw/csw.py?config=CSW-3.yml
60+
- id: fedcat01
61+
type: CSW
62+
title: Federated catalogue 1
63+
url: http://localhost/pycsw/csw.py?config=CSW-1.yml
64+
- id: fedcat02
65+
type: CSW
66+
title: Federated catalogue 2
67+
url: http://localhost/pycsw/csw.py?config=CSW-2.yml
68+
- id: fedcat03
69+
type: CSW
70+
title: Federated catalogue 3
71+
url: http://localhost/pycsw/csw.py?config=CSW-3.yml
6372
6473
At which point a CSW client request to CSW-all with ``distributedsearch=TRUE``, while specifying an optional ``hopCount``. Query network topology:
6574

@@ -97,9 +106,15 @@ Experimental support for distibuted searching is available in pycsw's OGC API -
97106
.. code-block:: yaml
98107
99108
federatedcatalogues:
100-
- https://example.org/collections/collection1
101-
- https://example.org/collections/collection2
109+
- id: fedcat01
110+
type: OARec
111+
title: Federated catalogue 1
112+
url: https://example.org/collections/collection1
113+
- id: fedcat02
114+
type: OARec
115+
title: Federated catalogue 2
116+
url: https://example.org/collections/collection2
102117
103118
With the above configured, a distributed search can be invoked as follows:
104119

105-
http://localhost/collections/metadata:main/items?distributed=true
120+
http://localhost/collections/metadata:main/items?distributedSearch=true

‎pycsw/core/admin.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# Authors: Tom Kralidis <tomkralidis@gmail.com>
55
# Angelos Tzotsos <tzotsos@gmail.com>
66
#
7-
# Copyright (c) 2024 Tom Kralidis
7+
# Copyright (c) 2026 Tom Kralidis
88
# Copyright (c) 2015 Angelos Tzotsos
99
#
1010
# Permission is hereby granted, free of charge, to any person
@@ -632,7 +632,9 @@ def cli_migrate_config(ctx, config, verbosity):
632632
elif name == 'profiles':
633633
dict_[name] = value.split(',')
634634
elif name == 'federatedcatalogues':
635-
dict_[name] = value.split(',')
635+
dict_[name] = []
636+
for count, fc in enumerate(value.split(',')):
637+
dict_[name].append({'id': f'fedcat{count}', 'url': fc})
636638
else:
637639
dict_['server'][name] = get_typed_value(value)
638640

‎pycsw/ogc/api/oapi.py‎

Lines changed: 60 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
#
33
# Authors: Tom Kralidis <tomkralidis@gmail.com>
44
#
5-
# Copyright (c) 2024 Tom Kralidis
5+
# Copyright (c) 2026 Tom Kralidis
66
#
77
# Permission is hereby granted, free of charge, to any person
88
# obtaining a copy of this software and associated documentation
@@ -199,10 +199,10 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
199199
'style': 'form',
200200
'explode': False
201201
}
202-
oapi['components']['parameters']['distributed'] = {
203-
'name': 'distributed',
202+
oapi['components']['parameters']['distributedSearch'] = {
203+
'name': 'distributedSearch',
204204
'in': 'query',
205-
'description': 'Whether to invoke distributed mode',
205+
'description': 'Whether to invoke distributed search',
206206
'schema': {
207207
'type': 'boolean',
208208
'default': False
@@ -390,6 +390,60 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
390390
oapi['paths']['/collections/{collectionId}/queryables'] = path2
391391
oapi['components']['parameters']['collectionId']['default'] = 'metadata:main' # noqa
392392

393+
path = {
394+
'get': {
395+
'tags': ['Federated catalogs'],
396+
'summary': 'Federated catalogs page',
397+
'description': 'Federated catalogs page',
398+
'operationId': 'getFederatedCatalogs',
399+
'parameters': [
400+
{'$ref': '#/components/parameters/collectionId'},
401+
{'$ref': '#/components/parameters/f'}
402+
],
403+
'responses': {
404+
'200': {
405+
'$ref': '#/components/responses/FederatedCatalogs'
406+
},
407+
'500': {
408+
'$ref': '#/components/responses/ServerError'
409+
}
410+
}
411+
}
412+
}
413+
414+
oapi['paths']['/collections/{collectionId}/federatedCatalogs'] = path
415+
416+
path = {
417+
'get': {
418+
'tags': ['Federated catalogs'],
419+
'summary': 'Federated catalogs page',
420+
'description': 'Federated catalogs page',
421+
'operationId': 'getFederatedCatalog',
422+
'parameters': [
423+
{'$ref': '#/components/parameters/collectionId'},
424+
{'name': 'catalogId',
425+
'in': 'path',
426+
'description': 'catalog ID',
427+
'required': True,
428+
'schema': {
429+
'type': 'string'
430+
}
431+
},
432+
{'$ref': '#/components/parameters/f'}
433+
],
434+
'responses': {
435+
'200': {
436+
'$ref': '#/components/responses/FederatedCatalog'
437+
},
438+
'500': {
439+
'$ref': '#/components/responses/ServerError'
440+
}
441+
}
442+
}
443+
}
444+
445+
oapi['paths']['/collections/{collectionId}/federatedCatalogs/{catalogId}'] = path
446+
393447
path = {
394448
'get': {
395449
'tags': ['metadata'],
@@ -411,7 +465,7 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
411465
{'$ref': '#/components/parameters/f'},
412466
{'$ref': '#/components/parameters/offset'},
413467
{'$ref': '#/components/parameters/facets'},
414-
{'$ref': '#/components/parameters/distributed'},
468+
{'$ref': '#/components/parameters/distributedSearch'},
415469
{'$ref': '#/components/parameters/vendorSpecificParameters'}
416470
],
417471
'responses': {
@@ -488,7 +542,7 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
488542
'parameters': [
489543
{'$ref': '#/components/parameters/collectionId'},
490544
{'$ref': '#/components/parameters/recordId'},
491-
{'$ref': '#/components/parameters/distributed'},
545+
{'$ref': '#/components/parameters/distributedSearch'},
492546
f
493547
],
494548
'responses': {

0 commit comments

Comments
 (0)