Sync Schools and Roles to Salesforce on Create and Update (#677)

## Status

- Closes
RaspberryPiFoundation/experience-cs#1677

## What's changed?

### Infrastructure
- Adds a `salesforce_connect` database configuration in `database.yml`
to connect to the `rpf-heroku-connect` Heroku Connect datastore
(read/write via a separate PostgreSQL connection, with `database_tasks:
false` so Rails migrations leave it alone).
- Adds a `salesforce_connect` Docker service (image:
`ghcr.io/raspberrypifoundation/heroku-connect`) to `docker-compose.yml`
for local development, with a named volume and a health check.
- Adds the `SALESFORCE_CONNECT_*` environment variables to
`.env.example` and `docker-compose.yml`.
- Updates CI (`.github/workflows/ci.yml`) to spin up the
`heroku-connect` service container, pass the Salesforce connection env
vars, and add `packages: read` permission so the private image can be
pulled.

### Models
- `Salesforce::Base` — abstract Active Record base class that routes all
reads and writes to the `salesforce_connect` database.
- `Salesforce::School` — maps to the `salesforce.editor__c` table (PK:
`editoruuid__c`).
- `Salesforce::Role` — maps to the
`salesforce.contact_editor_affiliation__c` table (PK:
`affiliation_id__c`).
- `Salesforce::Contact` — maps to the `salesforce.contact` table (PK:
`pi_accounts_unique_id__c`).

### Jobs
- `Salesforce::SalesforceSyncJob` — base job class that:
- Checks the `SALESFORCE_ENABLED` env var and discards the job (no
retry) if it is not `true`.
- Retries on `SalesforceRecordNotFound` with polynomial backoff, up to
10 attempts.
- Truncates string values to respect Salesforce column limits (appending
`…`).
  - Enqueues onto the new `salesforce_sync` queue.
- `Salesforce::SchoolSyncJob` — upserts a `School` record into
`salesforce.editor__c`, mapping all address, status, and metadata
fields. Defaults `userorigin__c` to `'for_education'` when blank.
- `Salesforce::RoleSyncJob` — upserts non-student `Role` records into
`salesforce.contact_editor_affiliation__c`. Skips student roles
entirely.
- `Salesforce::ContactSyncJob` — looks up the Salesforce Contact by the
school creator's user ID and syncs the `creator_agree_to_ux_contact`
flag. Raises `SalesforceRecordNotFound` (triggering retry) if no Contact
exists yet.

### Model callbacks
- `School` — `after_commit` on create/update enqueues both
`SchoolSyncJob` and `ContactSyncJob`.
- `Role` — `after_commit` on create/update enqueues `RoleSyncJob`.

### GoodJob queue configuration
- Adds the `salesforce_sync:10` queue to allow up to 10 concurrent
Salesforce sync workers.

### Rake tasks
- `salesforce_sync:school` — bulk-enqueues `SchoolSyncJob` for every
School (for backfilling).
- `salesforce_sync:role` — bulk-enqueues `RoleSyncJob` for every Role
(for backfilling).
- `salesforce_sync:contact` — bulk-enqueues `ContactSyncJob` for every
School (for backfilling the UX contact flag).

### Tests
- Full RSpec coverage for `SchoolSyncJob`, `RoleSyncJob`, and
`ContactSyncJob`, including field mapping, skipping/discarding
behaviour, and error cases.
- Model specs for `School` and `Role` verify that the correct jobs are
enqueued after create/update.

## Points for consideration

- **Security**: The `salesforce_connect` connection credentials are
injected via environment variables. No secrets are committed to the
repo.
- **Performance**: Each `after_commit` callback enqueues a background
job (GoodJob), so there is no synchronous overhead on the request.
Concurrency is capped per-record to avoid TOCTOU races.
- **Salesforce Contact availability**: `ContactSyncJob` requires the
Salesforce Contact record to already exist (keyed by `creator_id`). If
it does not exist yet, the job retries up to 10 times with polynomial
backoff.

## How to Test Locally

To test these changes locally, you can:

1. Log in as a user that has no school.
2. Go to For Education > Create your school account
3. Complete the sign-up form

Observe the GoodJob dashboard - you should see new `SchoolSyncJob` and
`RoleSyncJob` jobs in the `salesforce_sync` queue. In the Rails Console,
you can also inspect the number of `Salesforce::School` and
`Salesforce::Role` objects that have been created - you should see them
increase when you add a new school.

## Steps to perform after deploying to production

After deploying, run the rake backfill tasks to sync existing data:
```
rails salesforce_sync:school
rails salesforce_sync:role
rails salesforce_sync:contact
```

Author: fspeirs

.env.example

@@ -55,3 +55,10 @@ GOOGLE_CLIENT_SECRET=changeme
 # E2E tests can supply this to enable test utility endpoints
 RESEED_API_KEY=changeme
 
+# Salesforce Connect
+SALESFORCE_ENABLED=true
+SALESFORCE_CONNECT_HOST=salesforce_connect
+SALESFORCE_CONNECT_PORT=5432
+SALESFORCE_CONNECT_DB=salesforce_development
+SALESFORCE_CONNECT_PASSWORD=password
+SALESFORCE_CONNECT_USER=postgres

.github/workflows/ci.yml

@@ -43,6 +43,7 @@ jobs:
       contents: read
       issues: write
       pull-requests: write
+      packages: read
     env:
       RAILS_ENV: test
       POSTGRES_DB: choco_cake_test
@@ -56,6 +57,11 @@ jobs:
       ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY: deterministic-key
       ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT: derivation-salt
       EDITOR_ENCRYPTION_KEY: a1b2c3d4e5f67890123456789abcdef0123456789abcdef0123456789abcdef0
+      SALESFORCE_CONNECT_HOST: 127.0.0.1
+      SALESFORCE_CONNECT_PORT: '4101'
+      SALESFORCE_CONNECT_USER: postgres
+      SALESFORCE_CONNECT_PASSWORD: password
+      SALESFORCE_CONNECT_DB: salesforce_test
     services:
       postgres:
         image: postgres:12
@@ -74,6 +80,22 @@ jobs:
         image: redis:6.2-alpine
         ports:
           - 6379:6379
+      salesforce_connect:
+        image: 'ghcr.io/raspberrypifoundation/heroku-connect'
+        credentials:
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+        env:
+          POSTGRES_DB: salesforce_test
+          POSTGRES_PASSWORD: password
+          POSTGRES_USER: postgres
+        ports:
+          - 4101:5432
+        options: >-
+          --health-cmd="pg_isready -h 127.0.0.1 -U postgres"
+          --health-interval=5s
+          --health-timeout=5s
+          --health-retries=5
 
     steps:
       - uses: actions/checkout@v4

app/jobs/salesforce/contact_sync_job.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module Salesforce
+  class ContactSyncJob < SalesforceSyncJob
+    MODEL_CLASS = Salesforce::Contact
+
+    def perform(school_id:)
+      school = ::School.find(school_id)
+
+      sf_contact = Salesforce::Contact.find_by(pi_accounts_unique_id__c: school.creator_id)
+      raise SalesforceRecordNotFound, "Contact not found for creator_id: #{school.creator_id}" unless sf_contact
+
+      sf_contact.editoragreetouxcontact__c = school.creator_agree_to_ux_contact
+      sf_contact.save!
+    end
+
+    private
+
+    def concurrency_key_id = arguments.first.with_indifferent_access[:school_id]
+  end
+end

app/jobs/salesforce/role_sync_job.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+module Salesforce
+  class RoleSyncJob < SalesforceSyncJob
+    MODEL_CLASS = Salesforce::Role
+
+    FIELD_MAPPINGS = {
+      affiliation_id__c: :id,
+      contact__r__pi_accounts_unique_id__c: :user_id,
+      editor__r__editoruuid__c: :school_id,
+      roletype__c: :role,
+      createdat__c: :created_at,
+      updatedat__c: :updated_at
+    }.freeze
+
+    def perform(role_id:)
+      role = ::Role.find(role_id)
+
+      return if role.student?
+
+      sf_role = Salesforce::Role.find_or_initialize_by(affiliation_id__c: role_id)
+      sf_role.attributes = sf_role_attributes(role:)
+      sf_role.save!
+    end
+
+    private
+
+    def sf_role_attributes(role:)
+      mapped_attributes(role:).to_h do |sf_field, value|
+        value = truncate_value(sf_field:, value:) if value.is_a?(String)
+
+        [sf_field, value]
+      end
+    end
+
+    def mapped_attributes(role:)
+      FIELD_MAPPINGS.transform_values do |role_field|
+        role.send(role_field)
+      end
+    end
+
+    def concurrency_key_id = arguments.first.with_indifferent_access[:role_id]
+  end
+end

app/jobs/salesforce/salesforce_sync_job.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+module Salesforce
+  class SalesforceSyncJob < ApplicationJob
+    include GoodJob::ActiveJobExtensions::Concurrency
+
+    # Serialise concurrent performs for the same record (same class + record ID)
+    # to prevent TOCTOU races on find_or_initialize_by + save!, while allowing
+    # jobs for different records to run fully in parallel.
+    good_job_control_concurrency_with(
+      perform_limit: 1,
+      key: -> { "#{self.class.name}/#{concurrency_key_id}" }
+    )
+
+    class SalesforceRecordNotFound < StandardError
+    end
+
+    class SkipBecauseSalesforceIsDisabled < StandardError
+    end
+
+    discard_on SkipBecauseSalesforceIsDisabled
+
+    retry_on SalesforceRecordNotFound, wait: :polynomially_longer, attempts: 10
+
+    queue_as :salesforce_sync
+
+    before_perform do |_job|
+      salesforce_enabled = FeatureFlags.salesforce_sync?
+      raise SkipBecauseSalesforceIsDisabled, 'SALESFORCE_ENABLED is not true.' unless salesforce_enabled
+    end
+
+    def perform(*)
+      raise NotImplementedError, 'Subclasses must implement perform'
+    end
+
+    private
+
+    def concurrency_key_id
+      raise NotImplementedError, "#{self.class.name} must implement concurrency_key_id"
+    end
+
+    def truncate_value(sf_field:, value:)
+      column = self.class::MODEL_CLASS.column_for_attribute(sf_field)
+      return value if column.limit.nil?
+
+      value.truncate(column.limit, omission: '…')
+    end
+  end
+end

app/jobs/salesforce/school_sync_job.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+module Salesforce
+  class SchoolSyncJob < SalesforceSyncJob
+    MODEL_CLASS = Salesforce::School
+
+    FIELD_MAPPINGS = {
+      editoruuid__c: :id,
+      name: :name,
+      editorreference__c: :reference,
+      addressline1__c: :address_line_1,
+      addressline2__c: :address_line_2,
+      editormunicipality__c: :municipality,
+      editoradministrativearea__c: :administrative_area,
+      postcode__c: :postal_code,
+      countrycode__c: :country_code,
+      verifiedat__c: :verified_at,
+      createdat__c: :created_at,
+      updatedat__c: :updated_at,
+      rejectedat__c: :rejected_at,
+      website__c: :website,
+      userorigin__c: :user_origin,
+      districtnamesupplied__c: :district_name,
+      ncesid__c: :district_nces_id,
+      schoolrollnumber__c: :school_roll_number
+    }.freeze
+
+    def perform(school_id:)
+      school = ::School.find(school_id)
+
+      sf_school = Salesforce::School.find_or_initialize_by(editoruuid__c: school_id)
+      sf_school.attributes = sf_school_attributes(school:)
+      sf_school.save!
+    end
+
+    private
+
+    def sf_school_attributes(school:)
+      mapped_attributes(school:).to_h do |sf_field, value|
+        value = 'for_education' if sf_field == :userorigin__c && value.nil?
+        value = truncate_value(sf_field:, value:) if value.is_a?(String)
+
+        [sf_field, value]
+      end
+    end
+
+    def mapped_attributes(school:)
+      FIELD_MAPPINGS.transform_values do |school_field|
+        school.send(school_field)
+      end
+    end
+
+    def concurrency_key_id = arguments.first.with_indifferent_access[:school_id]
+  end
+end

app/models/role.rb

@@ -16,6 +16,8 @@ class Role < ApplicationRecord
     }
   )
 
+  after_commit :do_salesforce_sync, on: %i[create update], if: -> { FeatureFlags.salesforce_sync? && !student? }
+
   private
 
   def students_cannot_have_additional_roles
@@ -38,4 +40,8 @@ def users_can_only_have_roles_in_one_school
 
     errors.add(:base, 'Cannot create role as this user already has a role in a different school')
   end
+
+  def do_salesforce_sync
+    Salesforce::RoleSyncJob.perform_later(role_id: id)
+  end
 end

app/models/salesforce/base.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Salesforce
+  class Base < ApplicationRecord
+    self.abstract_class = true
+
+    connects_to database: { writing: :salesforce_connect }
+  end
+end

app/models/salesforce/contact.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module Salesforce
+  class Contact < Salesforce::Base
+    self.table_name = 'salesforce.contact'
+    self.primary_key = :pi_accounts_unique_id__c
+  end
+end

app/models/salesforce/role.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module Salesforce
+  class Role < Salesforce::Base
+    self.table_name = 'salesforce.contact_editor_affiliation__c'
+    self.primary_key = :affiliation_id__c
+  end
+end