Update vtiles cache utilities cleanup and MV refresh improvements (#716)

* Update config and paths to deploy services with docker compose  (#696)

* Fix docker tiler volume for staging and production

* Fix taginfo data processor image

* Update readme

* update

* Update configs

* Update deploy scripts

* Update scripts

* Display template

* Remove tilerdb adn imposm from base

* Add network tiler-db

* Restart taginfo web after data has been proceed

* Update overpass docker image

* Update cgimap, teginfo and overpass config (#698)

* Fix cgimap attribution setup

* Deploy in staging

* Update taginfo cronjob to process db files

* Update taginfo script

* Fix endpoint

* Update osm-seed version

* Update overpass version

* Update overpass docker image

* Adding non-administrative boundaries in vector tiles (#684)

* Store all the objects that have boundary=*

* Remove unused imposm config

* Add role and add index for osm_admin_relation_members

* Script optimization for admin

* Add query to create non admin boundaries views

* Add config  to display non_admin in tegola

* Add extra columns for no-bounduary areas

* Add religion, timezone, postal_code, etc attribute in no_admin layers

* Rename source layer to non_admin

* Rename provider from osm -> ohm (#700)

* Rename provider from osm to ohm

* Update traefik to redirect osm provider  to ohm tiler provider

* Better to call ohm_non_admin

* Rename osm to ohm  vtiles source name (#701)

* Rename provider from osm to ohm

* Update traefik to redirect osm provider  to ohm tiler provider

* Better to call ohm_non_admin

* Rename ohm_non_admin to ohm_other_boundaries

* Release overpass-turbo v2.1.11 (#704)

* Release overpass-turbo v2.1.11

* No need to change the bucket name

---------

Co-authored-by: Rub21 <rub2106@gmail.com>

* Vtiles improvements - communication_lines,   transport_lines and route_lines (#703)

* Include street and communication layers in vtiles

* Update imposm config to import data to routes and communication multilines

* update sql scripts

* Update imposm configs

* Update tiler compose

* update comunication

* script to reimport

* update to reimport tables

* Update

* Create a Mview for transport lines

* Update communication mview

* Add highway in routes_line layer

* Update columns for communication

* Update communication props

* Display communication from zoom 10 on

* Do not returm empty values for communitacion_lines layer

* Add relation attribute

* Display the way itseff too.

* Merge results in only one feature in case the attributes are the same

* Update branch naem to generate docker images

* Change yarn -> pnpm overpass-turbo and Docker images for Tiler (#705)

* Fixed pnpm

* Fixed communication - layer name

* Update docker images and volume for prod deployment

* testing vite upgrade

* Release overpass-turbo v2.1.12

* Clean cache in different times (#710)

* Clean cache in different times

* Fix comments

* Generate docker images

* Update images for tiler cahe

* Fix tiler cache paths to clean (#712)

* Fix path to clean

* Update path to remove from s3

* Ongoing translations: ohm-website

* Fix script that refresh Material views - Imposm (#714)

* Update script to refresh mviews

* Update docker image for imposm

* Update docker image

* Update script to refresh views

* Update tiler imposm docker image (#715)

* Update

* Update imposm docker image

* Update script to refresh material views in tiler (#717)

* Update

* Update imposm docker image

* Update branches name to deploy the docker image

---------

Co-authored-by: Eric Theise <erictheise@gmail.com>

Author: Rub21

hetzner/tiler/README.md

@@ -33,7 +33,7 @@ docker compose -f hetzner/tiler/tiler.base.yml -f hetzner/tiler/tiler.production
 # docker compose -f hetzner/tiler/tiler.base.yml -f hetzner/tiler/tiler.production.yml up tiler_sqs_cleaner -d --force-recreate
 # docker compose -f hetzner/tiler/tiler.base.yml -f hetzner/tiler/tiler.production.yml up tile_global_seeding -d --force-recreate
 # docker compose -f hetzner/tiler/tiler.base.yml -f hetzner/tiler/tiler.production.yml up tile_coverage_seeding -d --force-recreate
-# docker compose -f hetzner/tiler/tiler.base.yml -f hetzner/tiler/tiler.production.yml run tiler_s3_cleaner tiler-cache-cleaner clean_by_prefix
+# docker compose -f hetzner/tiler/tiler.base.yml -f hetzner/tiler/tiler.production.yml run tiler_s3_cleaner
 # docker compose -f hetzner/tiler/tiler.base.yml -f hetzner/tiler/tiler.production.yml up tiler_monitor -d --force-recreate 
 ```
 

hetzner/tiler/tiler.production.yml

@@ -5,8 +5,8 @@ services:
     volumes: !overwrite
       - tiler_pgdata:/var/lib/postgresql/data
       - ./config/postgresql.production.conf:/etc/postgresql/postgresql.conf
-    # ports:
-    #   - "54329:5432"
+    ports:
+      - "54329:5432"
     env_file:
       - .env.tiler
     mem_limit: 55G
@@ -16,7 +16,7 @@ services:
 
   tiler_imposm:
     container_name: tiler_imposm
-    image: ghcr.io/openhistoricalmap/tiler-imposm:0.0.1-0.dev.git.3294.h12c71f95
+    image: ghcr.io/openhistoricalmap/tiler-imposm:0.0.1-0.dev.git.3309.h529b5183
     volumes:
       - tiler_imposm_data:/mnt/data
     env_file:
@@ -47,7 +47,7 @@ services:
 
   tiler_sqs_cleaner:
     container_name: tiler_sqs_cleaner
-    image: ghcr.io/openhistoricalmap/tiler-cache:0.0.1-0.dev.git.3305.hda810a70
+    image: ghcr.io/openhistoricalmap/tiler-cache:0.0.1-0.dev.git.3290.h63549fc1
     environment:
       - PORT=8000
     env_file:
@@ -70,7 +70,7 @@ services:
 
   tiler_s3_cleaner:
     container_name: tiler_s3_cleaner
-    image: ghcr.io/openhistoricalmap/tiler-cache:0.0.1-0.dev.git.3305.hda810a70
+    image: ghcr.io/openhistoricalmap/tiler-cache:0.0.1-0.dev.git.3290.h63549fc1
     # image: tiler-cache:latest
     # build:
     #   context: ../../images/tiler-cache

images/tiler-imposm/queries/ohm_mviews/communication.sql

@@ -8,8 +8,8 @@
 -- Unified view: merge lines + multilines into a single source
 -- TODO, Do we need to add name_lcale columns?
 -- ============================================================================
-DROP VIEW IF EXISTS mv_communication_z16_20 CASCADE;
-CREATE VIEW mv_communication_z16_20 AS
+DROP MATERIALIZED VIEW IF EXISTS mv_communication_z16_20 CASCADE;
+CREATE MATERIALIZED VIEW mv_communication_z16_20 AS
     -- Ways (simple linestrings)
     SELECT
         id,
@@ -81,6 +81,10 @@ CREATE VIEW mv_communication_z16_20 AS
     WHERE geometry IS NOT NULL
       AND ST_GeometryType(geometry) IN ('ST_LineString', 'ST_MultiLineString');
 
+-- Unique index required for REFRESH MATERIALIZED VIEW CONCURRENTLY
+CREATE UNIQUE INDEX idx_mv_communication_z16_20_unique ON mv_communication_z16_20 (id, osm_id, source_type);
+-- Spatial index for geometry queries
+CREATE INDEX idx_mv_communication_z16_20_geometry ON mv_communication_z16_20 USING GIST (geometry);
 
 -- ============================================================================
 -- Zoom 14-15: Slight simplification (5m tolerance)

images/tiler-imposm/scripts/refresh_mviews.sh

@@ -39,13 +39,20 @@ function refresh_mviews_group() {
             log_message "[$group_name] Refreshing $mview..."
             local error_output
             # Disable statement_timeout for long-running refresh operations (0 = no limit)
-            error_output=$(psql "$PG_CONNECTION" -c "SET statement_timeout = 0; REFRESH MATERIALIZED VIEW CONCURRENTLY $mview;" 2>&1)
-            local exit_code=$?
+            local exit_code=0
+            local start_time=$SECONDS
+            error_output=$(psql "$PG_CONNECTION" -v ON_ERROR_STOP=1 -c "SET statement_timeout = 0" -c "REFRESH MATERIALIZED VIEW CONCURRENTLY $mview;" 2>&1) || exit_code=$?
+            local elapsed=$((SECONDS - start_time))
             if [ $exit_code -eq 0 ]; then
-                log_message "[$group_name] ✅ Successfully refreshed $mview."
+                log_message "[$group_name] ✅ Successfully refreshed $mview. Time: ${elapsed}s"
             else
                 log_message "[$group_name] ❌ ERROR refreshing $mview! Exit code: $exit_code"
                 log_message "[$group_name] ❌ Error details: $error_output"
+                # If connection failed, skip remaining views and wait before retrying
+                if echo "$error_output" | grep -qi "connection\|could not connect\|server closed\|SSL"; then
+                    log_message "[$group_name] ⚠️ Connection error detected. Waiting ${sleep_interval}s before retrying all views..."
+                    break
+                fi
             fi
         done
         sleep "$sleep_interval"
@@ -142,7 +149,8 @@ others_views=(
 communication_views=(
     # lines
     mv_communication_z16_20
-    mv_communication_z14_15
+    mv_communication_z13_15
+    mv_communication_z10_12
 )
 
 places_views=(
@@ -255,4 +263,4 @@ refresh_mviews_group "BUILDINGS" 180 "${buildings_views[@]}" &
 refresh_mviews_group "ROUTES" 180 "${routes_views[@]}" &
 
 ## This group high demand, so we refresh every 1 hour
-refresh_mviews_group "NO_ADMIN_BOUNDARIES" 36000 "${no_admin_boundaries_views[@]}" &
+refresh_mviews_group "NO_ADMIN_BOUNDARIES" 36000 "${no_admin_boundaries_views[@]}" &

images/web/Dockerfile

@@ -24,7 +24,7 @@ RUN apt-get update && \
     && apt-get clean && rm -rf /var/lib/apt/lists/*
 
 # Download OHM Website using gitsha, faster than cloning
-ENV OPENHISTORICALMAP_WEBSITE_GITSHA=658bab0b9357cae3ac0d30f4f6a4693a16d36d4e
+ENV OPENHISTORICALMAP_WEBSITE_GITSHA=a53c1384ce78e7ce3c41e410e9ec61315f0afa93
 ENV OHM_WEBSITE_URL=https://github.com/OpenHistoricalMap/ohm-website/archive/${OPENHISTORICALMAP_WEBSITE_GITSHA}.zip
 RUN rm -rf $workdir/* && curl -fsSL $OHM_WEBSITE_URL -o /tmp/ohm-website.zip && \
     unzip /tmp/ohm-website.zip -d /tmp && \