Forum:Announcements/The technical update blog
A hopefully frequently updated blog (target: monthly) on what's happening behind the scenes on the OpenGeofiction servers.
6 Sep 2026
Six weeks on from the last update, and the summer slow down didn't quite materialise. The big one has been the elevation process, which has been completely reworked. There's also a rebuilt main public tile server and topo layer, the usual bot nonsense, and the slow shuffle toward the main server refresh continues.
Elevation process
See also: OpenGeofiction:Topo layer
The elevation process has been torn out and replaced. Gone are the last of the legacy Thilo terrain machinery; the interpolator is now isofill, a small C tool we've released on the GitHub org which does banded, memory-budgeted fills against GDAL, and contours are produced without phyghtmap any more.
The process itself now runs nightly from a timer on the utility server, fetches its own source data (backed up to S3), publishes the source squares, in .osm.xz, for mappers to work from, and publishes an index of zones and squares to the wiki via MultiMaps as each run goes. Four zones whose squares had been lost were recovered along the way, with care taken not to overwrite anything which came back.
The published DEM is compressed float data now (LERC/ZSTD) rather than the legacy tiff zip, and a new zero-line check reports how far the published sea level sits from the drawn shoreline. The tile expiry side was rewritten to match, plus a guard so the renderd cache can no longer quietly fill its filesystem.
Tile servers and the topo layer
tiles04 has been rebuilt on Debian 13. It serves two styles: ogf-carto and ttopo, the OGF fork of the Tracestrack Topo style (also new on the GitHub org). ttopo has graduated from "trial" to being the site's topographic base layer proper - it's offered as layer V on the main map. The fork's missing layers and labels were restored along the way (roads, water, POIs and addresses, placenames and admin, junctions from z16), the relief ladder was enabled with the hillshade ramp. Wiki map pages can show the new layer too, thanks to a matching update to our MultiMaps extension.
tiles05 gained an optional "6ma" style - the same map, but with its data held six months behind the API, via a lagged replication.
Bots, crawlers and the nightly review
The nightly review of the main servers trundles on, and assists in dealing with the crap we get thrown at us... It earned its keep on 2 September when the wiki took a connection flood followed by a pagination crawl; the resulting anti-flood hardening is now documented.
robots.txt grew a sitemap reference - helps out the good bots.
And the API rules gained an allowlist generator for known editors' IPs - prevent blocking our known mappers!
Coastline process
Way back in March 2022 I opened an issue for one of the peculiar coastline bugs which throws our coastline process. Fast-forward four years and I've also submitted a PR to fix the issue. The OGF coastline process has been using the fix for the last three weeks.
The twice hourly coastline process continues, and now also publishes historic shapefiles.
Moving toward the main server refresh
The osm-website fork continues to track upstream, with iD updated to 2.42.x as the upstream releases came in. The export work flagged in the last update has started to land: "Build export download links from locale keys" was merged upstream in early August.
Brothie and other bits
User patrol has been improved again: violations are now split into pre- and post-notification categories with "comply-aware" classification, notified-user tracking survives renames, and there's a review of users whom human mappers have already contacted (over a four-week window). The revert queue scripts gained 30-second timeouts on their API calls and no longer fail silently on connection errors. And the internal technical documentation (docs-internal) is now synced out to the admin wiki automatically, so future updates may be slightly less overdue.
/wangi (talk) 18:44, 6 September 2026 (UTC)
26 Jul 2026
A bit of a summer slow down, but things still happening. Minor improvements have been made to the automated Brothie link template replacement, new user patrol and revert queue processing.
Moving toward main server refresh
The PR which makes the GPS traces functionality (which we do not use) optional was successfully merged to the upstream osm-website code.
Also looking at changes to improve the export functionality.
Territories
The script which checks and simplifies the admin territories has been fixed for an occasional bug (which impacted on TA130∈⊾ƨ and TA250∈⊾ƨ only), and then completely rewritten. Previously the script was still based on the legacy Thilo one, but it's now completely rewritten, removing all dependencies and optimising it to run 8x faster, using much less memory. Ages ago, the resource required for this script was the main reason to move it off the main api server and to a dedicated util server. No plan to reverse that setup, but it no-longer would drive such a choice.
Secondary servers
The server which provides the tile04, tiles05, Overpass, Spyglass, Brothie and automated util functionality suffered from unplanned down-time. This was due to a power-saving bug in the network card / driver - if the network link was lost it was not able to successfully recover when the network came back. This has been fixed with disabling power-management on the device, along with a poke script to ensure the network is recovered.
/wangi (talk) 16:28, 26 July 2026 (UTC)
17 Jun 2026
Nightly server monitoring
A busy spell... As with many internet sites these days the maintenance required to keep on top of bots, crawlers and AI LLM massively distributed scrapers is relentless. Previously the approach was "mitigate, run, wait for the site to be hammered into the ground, re-mitigate". Now there's a proactive process behind the scenes. Nightly a script pulls the previous days logs from the four main impact servers (API, overpass, wiki, data). These are then analysed for patterns (IPs, ranges, queries, user agents), RewriteRules compared against documented rules, and structured report with recommendations created. That report is delivered automatically on the OGF Admin Channel Discord.
This is also giving visibility into the number of user's who are getting "false positive" impacted by the anti-bot rules.
Brothie, the admin bot
Brothie is OpenGefiction's own LLM powered bot. It's running on a Hermes Agent install, normally using a DeepSeek v4-flash model. Most of what it runs is actually plain old - non AI - Python scripts, but a nightly review does run using AI to pick up on trends and problems.
User patrol
The user patrol page now has automated assistance. Twice hourly new user edits are checked to ensure they are mapping in the right areas, with notifications sent out when required.
Revert queue
Bulk reverts used to be done manually using osm-revert-scripts on the util server. Brothie now monitors OpenGeofiction:Revert queue and processes the requested reverts - much more straightforward!
Link templates
Brothie is also monitoring all wiki page edits, replacing bare URL links with appropriate link templates. This may appear "a bit retentive", but it opens up future changes to the URLs used. That simplifies future changes and also opens up another possible way to mitigate scraping attacks.
Overpass
The Overpass server was refreshed, updating it to run on Debian 13. To effect this the backup overpass server was completely rebuilt and then switched in place of the primary one. So, no down time.
Moving toward main server refresh
Starting some small steps toward the long overdue refresh of both the main OpenGeofiction server (we call it the API server) and the version of openstreetmap-website & associated tools which run on it:
- Submitted a PR which makes the GPS traces functionality (which we do not use) optional
- Working on a script (git)bin/applyLocaleSubs.py which replaces all OSM-related terminology with OGF. An alternative would have been to change OSM's locale setup to use tvars for lots of the stuff, but I "read the room" on the likelihood of that being well received!
- Investigating creating files to populate iD with OGF, rather than OSM, relevant presets
Documentation for all this is being maintained on our OGF-private Forgejo git server.