Version-by-version upgrades

Note: For Docker, follow the SQL/version requirements below and the Docker upgrade procedure. Rebuild images and configure Compose jobs instead of installing host packages, cron jobs or systemd units.

Native Git upgrade, step by step

This procedure applies to an existing /opt/lebontag-v2 installation using systemd services under lbt-integration. For a legacy PM2/cron installation, also read the transition below before starting. Docker installations follow their dedicated procedure.

A new installation uses sql/lebontag_v2.sql; an upgrade uses only outstanding migrations, in the order listed below. Never import the full schema into an existing database.

1. Check the repository and installed version

Run these blocks in the same Bash terminal. set -euo pipefail exits this maintenance shell if a command fails; fix the error before resuming. Record the backup path when created: after closing the terminal, set LBT_UPDATE_BACKUP to that path again to resume.

bash
set -euo pipefail
cd /opt/lebontag-v2
sudo git status --short
sudo git branch --show-current
sudo git log -1 --format='%h %s'
sudo git remote -v

Identify applied database migrations separately. The code version does not prove the schema version. If git status shows local changes, review and preserve them before upgrading; do not use reset --hard, clean or an automatic stash. Personal env files must remain untracked. Check the branch and its upstream: currently fix/validation-integration during development, then the chosen release branch. Do not implicitly switch branches during an upgrade.

2. Suspend schedules and wait for running jobs

Plan a maintenance window and prevent concurrent manual launches. Also check cron or other schedulers. This block records currently active LeBonTag timers, stops them and waits for jobs to finish; it does not kill an append or segmentation. If waiting takes longer than expected, inspect logs in a second terminal. Do not reboot during maintenance: timers remain enabled at boot.

LBT_UPDATE_BACKUP="/opt/lebontag-v2/osm/backups/update-$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -o root -g root -m 0700 "$LBT_UPDATE_BACKUP"
sudo git rev-parse HEAD | sudo tee "$LBT_UPDATE_BACKUP/commit-before.txt" >/dev/null
sudo systemctl list-units --type=timer --state=active --plain --no-legend 'lbt-*.timer' | awk '{print $1}' | sudo tee "$LBT_UPDATE_BACKUP/active-timers.txt" >/dev/null
sudo systemctl list-timers --all 'lbt-*'
while IFS= read -r lbt_timer; do
  sudo systemctl stop "$lbt_timer"
done < <(sudo cat "$LBT_UPDATE_BACKUP/active-timers.txt")
while true; do
  lbt_running_jobs=$(sudo systemctl list-units --type=service --state=active,activating,deactivating,reloading --plain --no-legend lbt-pipeline.service lbt-integration.service lbt-segmentation.service lbt-cleaning.service lbt-email-digests.service)
  if [ -z "$lbt_running_jobs" ]; then
    break
  fi
  printf '%s\n' "$lbt_running_jobs"
  sleep 5
done
sudo systemctl stop lbt-web.service

3. Back up before changing anything

Replace DB_HOST, DB_PORT, DB_USER, DB_NAME with the actual database connection. The password is prompted for, not written in the command. Use a pg_dump compatible with the server major version and a role allowed to back up the whole database. All data writers must be stopped, including those outside these services.

sudo -H pg_dump --host=DB_HOST --port=DB_PORT --username=DB_USER --dbname=DB_NAME --password --format=custom --file="$LBT_UPDATE_BACKUP/database.dump"
sudo pg_restore --list "$LBT_UPDATE_BACKUP/database.dump" >/dev/null
sudo tar --acls --xattrs --exclude='./osm/backups' --exclude='./node_modules' --exclude='./.next' --exclude='./.venv' -czf "$LBT_UPDATE_BACKUP/application.tar.gz" -C /opt/lebontag-v2 .
sudo git bundle create "$LBT_UPDATE_BACKUP/repository.bundle" --all
printf 'Backup / sauvegarde : %s\n' "$LBT_UPDATE_BACKUP"

The backup includes all database schemas, sources, settings, styles, data and replication states. Reinstallable dependencies and the build are excluded from the archive. PBFs and reports may make it large: provision enough disk space. The directory remains root-private because it contains secrets. Reading the dump catalogue is not a restoration test. If a PostgreSQL role must be recreated on another server, separately preserve the necessary information.

4. Retrieve sources

sudo git pull --ff-only
sudo git log -1 --format='%h %s'
sudo git diff --stat "$(sudo cat "$LBT_UPDATE_BACKUP/commit-before.txt")" HEAD

--ff-only rejects divergence instead of merging on the server. Resolve divergence before continuing. Read the newly downloaded version-specific instructions below and check dependency and environment changes.

5. Install dependencies and outstanding migrations

sudo -H env PATH=/usr/local/bin:/usr/bin:/bin HUSKY=0 /usr/local/bin/npm ci
sudo -H /opt/lebontag-v2/.venv/bin/python3 -m pip install -r /opt/lebontag-v2/python/requirements.txt

Add new settings to .env.production.local using the example without overwriting existing values. Preserve secrets, the explicitly authorized target and job settings. .env.local remains generated by the application. Install or upgrade system tools only when the target version requires it.

SQL example only for a 2.7.4 database that has not already received this migration; adapt the connection as in step 3. For multiple migrations, execute each file separately in order and stop at the first error. Skip this block if no SQL change is needed.

sudo -H psql --host=DB_HOST --port=5432 --username=DB_USER --dbname=DB_NAME --password -v ON_ERROR_STOP=1 -f /opt/lebontag-v2/sql/update_2.7.4_to_2.8.0.sql

6. Check and rebuild web

sudo -H env PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/npm run prepare:env
sudo -H env PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/npm run check
sudo -H env PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/npm run build

Keep web stopped throughout npm ci and the build. Do not restart if checks or compilation fail. Do not change source files during compilation.

7. Reapply permissions after building

The lbt-integration account and acl package must already exist. Code, the venv and dependencies remain root-administered; only the rebuilt cache receives a recursive ownership change. Confirm that the listed data directories belong to this installation.

sudo install -d -o lbt-integration -g lbt-integration -m 0700 /opt/lebontag-v2/.next/cache /opt/lebontag-v2/log /opt/lebontag-v2/osm/replication /opt/lebontag-v2/osm/changeset /opt/lebontag-v2/osm/osc_update /opt/lebontag-v2/osm/integration/runs
sudo chown -R --no-dereference lbt-integration:lbt-integration /opt/lebontag-v2/.next/cache
sudo setfacl -m u:lbt-integration:--x /opt /opt/lebontag-v2 /opt/lebontag-v2/osm /opt/lebontag-v2/osm/integration /opt/lebontag-v2/osm/style
sudo setfacl -R -m u:lbt-integration:r-X /opt/lebontag-v2/python /opt/lebontag-v2/scripts /opt/lebontag-v2/.venv /opt/lebontag-v2/node_modules /opt/lebontag-v2/.next /opt/lebontag-v2/public
sudo setfacl -m u:lbt-integration:r-- /opt/lebontag-v2/package.json /opt/lebontag-v2/next.config.ts /opt/lebontag-v2/tsconfig.json /opt/lebontag-v2/.env.local /opt/lebontag-v2/osm/style/default.style
sudo chmod 0600 /opt/lebontag-v2/.env.production.local
sudo setfacl -m u:lbt-integration:r-- /opt/lebontag-v2/.env.production.local
sudo -u lbt-integration test -r /opt/lebontag-v2/.env.production.local
sudo -u lbt-integration test -w /opt/lebontag-v2/.next/cache
sudo -u lbt-integration /opt/lebontag-v2/.venv/bin/python3 --version
sudo -u lbt-integration /usr/local/bin/node --version
sudo -u lbt-integration /usr/local/bin/osm2pgsql --version

Existing data directories must already belong to the service account. For historical root-owned files, follow the non-root transition below, limited to affected data. Never run chown -R on the whole project or use chmod 777. Do not grant the service access to private backups.

8. Reload units, start web and check

Units installed as links to services/ use the updated files after daemon-reload. If copied into /etc/systemd/system, compare copies to sources, back up customizations and install the updated versions before continuing. Inspect drop-ins: they may override shipped settings. Install new units according to version-specific instructions.

sudo systemctl daemon-reload
sudo systemctl cat lbt-web.service lbt-pipeline.service lbt-integration.service
sudo systemctl show lbt-web.service -p User -p Group -p FragmentPath -p DropInPaths
sudo systemctl start lbt-web.service
sudo systemctl status lbt-web.service --no-pager
sudo journalctl -u lbt-web.service -n 80 --no-pager
curl --fail --silent --show-error --output /dev/null http://127.0.0.1:3000/fr

Then check the public Nginx/HTTPS URL, sign-in, administration and a map. The local HTTP check does not cover these functions.

Run an explicit integration check regardless of the service's usual mode. This command uses the dedicated account permissions but not the systemd sandbox; check the next scheduled run logs to verify that sandbox too.

lbt_check_status=0
sudo -u lbt-integration env PYTHONDONTWRITEBYTECODE=1 /opt/lebontag-v2/.venv/bin/python3 /opt/lebontag-v2/python/lbt-run-update.py --env-file /opt/lebontag-v2/.env.production.local --output-root /opt/lebontag-v2/osm/integration/runs --style /opt/lebontag-v2/osm/style/default.style --mode check || lbt_check_status=$?
printf 'Integration check exit code : %s\n' "$lbt_check_status"
if [ "$lbt_check_status" -ne 0 ] && [ "$lbt_check_status" -ne 2 ]; then
  printf 'Integration check failed / contrôle en échec\n' >&2
  exit "$lbt_check_status"
fi

Exit code 2 may mean business blockers or a partial batch: review the report and administration before resuming. Resolve uncertain states (needs_review) before resuming writes. Check mode does not append, but writes its journal and may prepare a batch. Do not start the integration service merely to check availability: its normal mode may be execute and invoke segmentation.

9. Resume previously active timers

After successful checks, resume only the recorded timers. This preserves optional module choices and does not activate email or segmentation timers that were previously stopped. A Persistent=true timer may immediately run a job missed during maintenance.

while IFS= read -r lbt_timer; do
  sudo systemctl start "$lbt_timer"
done < <(sudo cat "$LBT_UPDATE_BACKUP/active-timers.txt")
sudo systemctl list-timers --all 'lbt-*'
sudo systemctl --failed --no-pager

Review logs and administration tracking after the first runs. Explicitly enable any newly required timer after configuring it. If a step failed, keep jobs suspended and follow the consistent restoration procedure at the end of this page; reverting only the commit is insufficient after a SQL migration.

Script order

Transition	SQL file
2.1.2 → 2.1.3	update_2.1.2_to_2.1.3.sql
2.1.3 → 2.2.0	update_2.1.3_to_2.2.0.sql
2.2.0 → 2.4.8	update_2.2.0_to_2.4.8.sql
2.4.8 → 2.5.2	update_2.4.8_to_2.5.2.sql
2.6.2 → 2.6.3	update_2.6.2_to_2.6.3.sql
2.6.4 → 2.6.5	update_2.6.4_to_2.6.5.sql
2.6.5 → 2.6.6	update_2.6.5_to_2.6.6.sql
2.6.8 → 2.6.9	update_2.6.8_to_2.6.9.sql
2.6.9 → 2.7.2	update_2.6.9_to_2.7.2.sql
2.7.2 → 2.7.3	update_2.7.2_to_2.7.3.sql
2.7.3 → 2.7.4	update_2.7.3_to_2.7.4.sql
2.7.4 → 2.8.0	update_2.7.4_to_2.8.0.sql

The repository does not provide a script for every intermediate version. Start at changes not already present; do not rerun scripts because an intermediate version number is absent. Inspect old or locally modified schemas before migration. Scripts do not automatically detect the installed version.

2.1.2 → 2.2.0: legacy quality rules

The 2.1.3 → 2.2.0 transition replaces legacy quality rules with the new model and initial rules. Back up custom rules for recreation. Existing quality acknowledgements stop the migration before changes: their reset must be an explicit decision, because old rule IDs cannot be reassigned to new rules. The migration restores the rule foreign key.

2.2.0 → 2.4.8: OSM replication

Install osm2pgsql 2.2.0, OSM tools and Python dependencies as described in Installation. Configure matching PBF, state and minute-diff source using Configuration, then reload the reference. Replace legacy Overpass retrieval with lbt-pipeline.service and its timer. Never start replication from sequence zero.

2.6.2 → 2.6.9: interface and SQL functions

npm ci installs Recharts and JSZip from the lockfile. Apply the relevant SQL files in order for statistics, exports and partial validation. The 2.6.4 → 2.6.5 transition recalculates footprint links and may take time on large databases.

2.6.9 → 2.7.2: in-app notifications

Apply the SQL and rebuild the application; Mon territoire notifications require no additional service.

2.7.2 → 2.7.3: email notifications

npm ci installs the project's SMTP dependency. Set AUTH_URL to the public application address. Generate distinct secrets for LBT_MAIL_ENCRYPTION_KEY and LBT_EMAIL_DIGEST_SECRET and retain them securely; changing the encryption key would make the encrypted SMTP configuration unreadable.

openssl rand -hex 32
openssl rand -hex 32

Configure the worker in .env.production.local: web and worker share LBT_EMAIL_DIGEST_SECRET, an empty LBT_DIGEST_WORKER_URL uses AUTH_URL, and LBT_DIGEST_BATCH_SIZE=20. Protect the file as described in Deployment. Test SMTP before enabling sending. Adapt the absolute Node path in the service and remove duplicate email schedules.

# Check the Node path in the unit before enabling it.
/usr/local/bin/node --version
sudo systemctl link /opt/lebontag-v2/services/lbt-email-digests.service /opt/lebontag-v2/services/lbt-email-digests.timer
sudo systemctl daemon-reload
sudo systemctl enable --now lbt-email-digests.timer

2.7.3 → 2.7.4: user preferences

Apply the SQL and rebuild the application. Database user preferences complement administrator defaults.

2.7.4 → 2.8.0: integration, segmentation and cleanup

When upgrading to version 2.8.0, we strongly recommend completely reinstalling the application in a new directory. The database can be retained after the update and the reboot procedure.

• Apply update_2.7.4_to_2.8.0.sql once to a 2.7.4 database. It includes contexts, integration journals, typed identities, functions, cursor and segmentation journal/indexes. SQL files 001 through 011 are no longer installation steps.
• If the migration reports ambiguous identities, it rolls back. Do not guess types. After backup and resolution of active batches, follow the reload procedure below: Reboot explicitly deletes diffs and decisions before reloading. The migration never runs this destructive cleanup automatically.
• Inspect partially migrated databases before executing anything: this migration is not repeatable. Do not remove guards to force it.
• Enable LBT_CAPTURE_RELATION_CONTEXT=1 in the pipeline environment before ingestion.
• Remove legacy integration and cleanup cron entries, then install the services in Production. Delete cron/lbt-cleaning.sh and any remaining lbt-update-db-by-osc.py / .sh on the server. Keep node/way/relation ingestion scripts.
• Segmentation is optional. If derived tables do not exist, the migration does not create them: initial loading creates them and their indexes. Segment IDs are not stable.

Configuration consolidation in 2.8.0

For a new installation, .env.production.local.example and .env.development.local.example provide the required settings. A server running a production build uses .env.production.local. Docker adds only .env.db; load its Compose options with docker compose --env-file .env.db .... .env.local remains generated public metadata only.

For an installation using the previous separate files:

1. Stop timers and wait for writers to finish. Back up configurations privately.
2. Move integration-runner.conf settings into .env.production.local. Check its DB_* against the old integration.env; do not silently change databases or roles.
3. Put target.json content into LBT_INTEGRATION_TARGET, single-line JSON in single quotes, and remove LBT_INTEGRATION_TARGET_CONFIG from active environments. Simultaneously defining both targets is rejected.
4. Copy only worker URL and batch size from .env.digests.local. The email secret must already match .env.production.local; resolve any discrepancy before enabling. An empty worker URL uses AUTH_URL.
5. Install updated units and apply deployment permissions. The segmentation hook is now in lbt-integration.service. After backing up the previous drop-in, remove only the supplied old hook:

sudo rm -f /etc/systemd/system/lbt-integration.service.d/segmentation.conf
sudo systemctl daemon-reload
sudo systemctl cat lbt-integration.service lbt-segmentation.service lbt-email-digests.service

Verify exactly one segmentation ExecStartPost and no old configuration paths. Preserve unrelated local drop-ins. Source folders services/integration/, services/lbt-integration.service.d/ and .env.digests.local.example are no longer needed; .service and .timer units in services/ remain required.

6. In Docker, recreate affected containers and restart Ofelia. Move any options from an old Compose .env into .env.db, preserving the project name and volume.
7. After a successful check against the intended target, archive and remove old personal integration.env, integration-runner.conf, target.json and .env.digests.local files. They are not deleted automatically. Python commands still accept legacy --target-config for transition, but the normal procedure no longer depends on it.

This consolidation needs no additional SQL and changes no batches or business states. Dependency and context blockers still require review; never bypass their checks.

Transition to non-root services

This concerns native installation; Docker retains its own procedure. The Linux account remains lbt-integration; this transition creates no PostgreSQL role and requires no additional SQL.

1. Stop installed timers (pipeline, integration, segmentation, cleanup, emails), wait for their services to finish, then stop web before replacing files. Disable equivalent legacy cron jobs. Do not kill an active append.
2. Back up the entire database, code, configuration, style, replication files and reports. PostgreSQL backups must include osm2pgsql middle tables and journals, not just geometries.
3. Install updated sources and units. Back up old unit copies in /etc/systemd/system and replace them with links to the source files, reviewing local customizations. Inspect drop-ins with systemctl cat: no remaining User=root must override the new units.
4. Apply Deployment permissions after rebuilding and any reload. Previously root-owned data needs an ownership change limited to the listed data directories, including sequence.txt and log subdirectories. Never recursively change ownership of code, the venv or the whole repository.
5. If web used root-managed PM2, identify and stop only LeBonTag before activating lbt-web.service:

sudo -H env PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/pm2 list
# Adapt the name to the actual process:
sudo -H env PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/pm2 delete LeBonTagv2
sudo -H env PATH=/usr/local/bin:/usr/bin:/bin /usr/local/bin/pm2 save --force

Use the old installation's PM2/Node paths if different. Do not uninstall PM2 or disable pm2-root if it manages other applications. With another supervisor, disable only the old LeBonTag definition; just one server must listen on port 3000.

6. Link units and run sudo systemctl daemon-reload. Check each User and Group using systemctl show NAME.service -p User -p Group. Start web and verify login and administration log access.
7. Check the pipeline, integration in check mode, segmentation and cleanup as described in Deployment. The first cleanup applies actual retention: verify its values. Test optional emails, then enable timers, preserving target and business-blocker checks.

Daily cleanup installation belongs in installation.md. Upgrade-specific work removes its old schedule and fixes historical cache ownership. Remove cron/lbt-cleaning.sh and obsolete integration scripts after switching, but retain node/way/relation ingestion scripts.

Reloading an old database and rollback (reboot)

A reference reload is recommended for old databases without reliable typed identities. After backup and resolution of active batches, read Reboot: this explicit cleanup deletes diffs, validations and quality acknowledgements while preserving accounts and settings. It may precede identity migration if ambiguity prevents it. Never guess types or remove SQL guards to force migration.

Apply outstanding migrations in order, reload the PBF and initialize both replication states following Configuration. Archive old replication files to avoid mixing initializations. Never transfer a prepared batch from another database or merely change its target.

To roll back, stop every writer and restore a consistent whole: complete database, code, style and matching replication files/states. Verify restoration into a separate database. Repeat checks before re-enabling timers.

Protected map feeds

The 2.7.4 → 2.8.0 migration and fresh-install schema include the map_auth columns on tile, WMS and WMTS tables. Install the updated npm lockfile dependencies and rebuild the web application. Configure LBT_MAP_ENCRYPTION_KEY only when using protected feeds, following Background catalogue. No worker or systemd unit changes are required.

Additional migration: validation map appearance

New installations and the current update_2.7.4_to_2.8.0.sql include the appearance setting. If that upgrade was already applied before this feature was added, apply only the following idempotent supplement before starting the updated web application (replace connection values):

sudo -H psql --host=DB_HOST --port=5432 --username=DB_USER --dbname=DB_NAME --password -v ON_ERROR_STOP=1 -f /opt/lebontag-v2/sql/validation_appearance.sql

It adds the nullable user preference and the action administration default without changing saved choices. No new npm package, environment variable or background service is required. Rebuild and restart the web service using the update procedure above.

Reference-tag and automatic-validation correction

This delivery does not change the SQL schema. Deploy the Python scripts together with the new lbt_reference.py and lbt_review_db.py modules using the upgrade procedure above. Python dependencies and pipeline service commands remain unchanged.

New diffs use primitive reference tags. Existing sheets are not all repaired automatically, and potentially excessive historical automatic approvals are not revoked: historical automatic approvals cannot be distinguished reliably from explicit approvals.

During maintenance, after backup and suspension of jobs and user writes, first run this read-only diagnostic:

cd /opt/lebontag-v2
sudo -u lbt-integration .venv/bin/python3 python/lbt-repair-reference-tags.py \
  --env-file /opt/lebontag-v2/.env.production.local

Only tovalidate sheets without any manual decision qualify, and only when the old version matches the reference primitive (or a creation is absent from the reference). New tags come from the exact typed/versioned OSC root. Sheets with decisions or unproven references appear in skipped. Inspect the output before applying:

sudo -u lbt-integration .venv/bin/python3 python/lbt-repair-reference-tags.py \
  --env-file /opt/lebontag-v2/.env.production.local --apply

Statuses, decisions, geometries and OSC fragments stay unchanged. Only old/new tags and their differences are recalculated in one transaction. The tool locks affected tables; failures do not publish partial repairs.

Reprepare existing prepared batches with the new policy (lbt-stage-update.py prepare --replace-prepared …, using the usual connection options), then run integration in check mode before execute. Never automatically cancel or replay a needs_review or integrated batch. Skipped sheets and historical automatic approvals need targeted review; a full reload is an alternative only with backups and the initial-loading procedure.