Wikimedia Maps/Tile generation report

Group:	Reading Infrastructure
Start:	2018-10-22
Backlog:	#maps
Lead:	Mateus Santos

Introduction

The map infrastructure store vector-tiles in a Cassandra DB and can store data from zoom levels 0 to 15 where each zoom level has 2^zoom tiles. Tilerator generates 22 tiles per second in average for each CPU available. Generating the whole planet can take more than a month for 24 CPU's.

Tile generation Benchmark per zoom level for 24 CPU's
Zoom	Total tiles	Estimated Time to finish
		Hours	Days
0	1	0.00	0.00
1	4	0.00	0.00
2	16	0.00	0.00
3	64	0.00	0.00
4	256	0.00	0.00
5	1024	0.00	0.00
6	4096	0.00	0.00
7	16384	0.01	0.00
8	65536	0.03	0.00
9	262144	0.14	0.01
10	1048576	0.55	0.02
11	4194304	2.21	0.09
12	16777216	8.83	0.37
13	67108864	35.31	1.47
14	268435456	141.22	5.88
15	1073741824	564.89	23.54
Total	1431655765	753.19	31.38

Debian upgrade to stretch

Tracked in Phabricator
Task T205462

Because of the Debian upgrade to stretch, OSM data needed to be reloaded and all tiles needed to be generated from scratch. The tile generation started at 10/22/2018 and it's an ongoing process until now (last update was 12/19/2018). After some delay on tile generation we started keeping manual log through time regarding the tile generation health using a node tool that collect metrics generated by Tilerator, which can be seem in the following table:

Tile generation health monitoring

Metrics for tile generation (last update 12/19/2018)
processed	tilegenok	tilenodata	processedRate	missing	okTilesRate	totalTiles	timestamp
329605850	18235862	311369988	23.02%	1102049915	5.53%	1431655765	1543273715182
329631323	18236329	311394994	23.02%	1102024442	5.53%	1431655765	1543273762771
329656677	18236516	311420161	23.03%	1101999088	5.53%	1431655765	1543273810907
329678926	18236752	311442174	23.03%	1101976839	5.53%	1431655765	1543273851202
329839665	18238151	311601514	23.04%	1101816100	5.53%	1431655765	1543274169998
330184897	18244086	311940811	23.06%	1101470868	5.53%	1431655765	1543274814486
330242080	18244458	311997622	23.07%	1101413685	5.52%	1431655765	1543274924304
357986666	18878807	339107859	25.01%	1073669099	5.27%	1431655765	1543327057836
358478689	18890753	339587936	25.04%	1073177076	5.27%	1431655765	1543327999390
358481072	18890909	339590163	25.04%	1073174693	5.27%	1431655765	1543328003357
359466359	18915835	340550524	25.11%	1072189406	5.26%	1431655765	1543329887495
359468695	18915835	340552860	25.11%	1072187070	5.26%	1431655765	1543329890136
359501841	18916801	340585040	25.11%	1072153924	5.26%	1431655765	1543329957355
360883733	18960382	341923351	25.21%	1070772032	5.25%	1431655765	1543332566057
361105061	18968522	342136539	25.22%	1070550704	5.25%	1431655765	1543332988692
369736755	19204832	350531923	25.83%	1061919010	5.19%	1431655765	1543349355705
369754943	19205421	350549522	25.83%	1061900822	5.19%	1431655765	1543349393845
371169554	19214191	351955363	25.93%	1060486211	5.18%	1431655765	1543352016372
404768668	19856922	384911746	28.27%	1026887097	4.91%	1431655765	1543414812441
445075588	21248972	423826616	31.09%	986580177	4.77%	1431655765	1543490292663
448271613	21337248	426934365	31.31%	983384152	4.76%	1431655765	1543496265671
448305075	21338033	426967042	31.31%	983350690	4.76%	1431655765	1543496330227
448361068	21339359	427021709	31.32%	983294697	4.76%	1431655765	1543496440993
448376008	21339996	427036012	31.32%	983279757	4.76%	1431655765	1543496464526
448376958	21339996	427036962	31.32%	983278807	4.76%	1431655765	1543496466399
448444000	21342452	427101548	31.32%	983211765	4.76%	1431655765	1543496592815
448454022	21342935	427111087	31.32%	983201743	4.76%	1431655765	1543496611603
448455063	21342935	427112128	31.32%	983200702	4.76%	1431655765	1543496613191
448456352	21342935	427113417	31.32%	983199413	4.76%	1431655765	1543496616785
448463340	21343345	427119995	31.32%	983192425	4.76%	1431655765	1543496632324
448488739	21344399	427144340	31.33%	983167026	4.76%	1431655765	1543496680542
448493339	21344827	427148512	31.33%	983162426	4.76%	1431655765	1543496686367
454149805	21521204	432628601	31.72%	977505960	4.74%	1431655765	1543507258034
454419148	21531201	432887947	31.74%	977236617	4.74%	1431655765	1543507751739
459882332	21665626	438216706	32.12%	971773433	4.71%	1431655765	1543517792995
495010058	22871478	472138580	34.58%	936645707	4.62%	1431655765	1543582997617
495015752	22871535	472144217	34.58%	936640013	4.62%	1431655765	1543583005146
500146062	23023677	477122385	34.93%	931509703	4.60%	1431655765	1543592543127
501753227	23090414	478662813	35.05%	929902538	4.60%	1431655765	1543595574348
643485413	27850894	615634519	44.95%	788170352	4.33%	1431655765	1543858114418
643557080	27851243	615705837	44.95%	788098685	4.33%	1431655765	1543858251078
643575347	27851292	615724055	44.95%	788080418	4.33%	1431655765	1543858283434
643588132	27851365	615736767	44.95%	788067633	4.33%	1431655765	1543858306499
647378317	27923230	619455087	45.22%	784277448	4.31%	1431655765	1543865399600
647390779	27923944	619466835	45.22%	784264986	4.31%	1431655765	1543865417667
554720434	23601894	531118540	38.75%	876935331	4.25%	1431655765	1543935910238
554724936	23601894	531123042	38.75%	876930829	4.25%	1431655765	1543935916982
571188609	24069424	547119185	39.90%	860467156	4.21%	1431655765	1544026116760
586166907	24103862	562063045	40.94%	845488858	4.11%	1431655765	1544053956011
568689760	22641417	546048343	39.72%	862966005	3.98%	1431655765	1544156936945
589400156	22071213	567328943	41.17%	842255609	3.74%	1431655765	1544221066989
589418129	22071231	567346898	41.17%	842237636	3.74%	1431655765	1544221094608
637406144	21566271	615839873	44.52%	794249621	3.38%	1431655765	1544394201838
637408830	21566273	615842557	44.52%	794246935	3.38%	1431655765	1544394205758
637411640	21566273	615845367	44.52%	794244125	3.38%	1431655765	1544394209308
680100749	22261714	657839035	47.50%	751555016	3.27%	1431655765	1544472254313
717843703	22810362	695033341	50.14%	713812062	3.18%	1431655765	1544541762797
718145412	22816663	695328749	50.16%	713510353	3.18%	1431655765	1544542312249
756847246	23204744	733642502	52.87%	674808519	3.07%	1431655765	1544641669818
773737425	23618129	750119296	54.04%	657918340	3.05%	1431655765	1544710360518
819218612	23969150	795249462	57.22%	612437153	2.93%	1431655765	1544794097109
820425720	23984577	796441143	57.31%	611230045	2.92%	1431655765	1544796310087
826459063	24036728	802422335	57.73%	605196702	2.91%	1431655765	1544807325635
863417905	24334279	839083626	60.31%	568237860	2.82%	1431655765	1544875191371
962442186	26073481	936368705	67.23%	469213579	2.71%	1431655765	1545057598335
1056702638	27869655	1028832983	73.81%	374953127	2.64%	1431655765	1545228533232
1057790348	27879600	1029910748	73.89%	373865417	2.64%	1431655765	1545230495655
1078537988	28093576	1050444412	75.34%	353117777	2.60%	1431655765	1545268190817
1078577797	28094882	1050482915	75.34%	353077968	2.60%	1431655765	1545268269267
1078605334	28095272	1050510062	75.34%	353050431	2.60%	1431655765	1545268313809
1078621315	28095300	1050526015	75.34%	353034450	2.60%	1431655765	1545268340157
1078662686	28095300	1050567386	75.34%	352993079	2.60%	1431655765	1545268411926
1078675899	28095300	1050580599	75.34%	352979866	2.60%	1431655765	1545268439748
1078681693	28095300	1050586393	75.35%	352974072	2.60%	1431655765	1545268451792
1114218777	28670702	1085548075	77.83%	317436988	2.57%	1431655765	1545332461909
1117927315	28776224	1089151091	78.09%	313728450	2.57%	1431655765	1545339308889
1146342225	29491813	1116850412	80.07%	285313540	2.57%	1431655765	1545391152361
1150046974	29624996	1120421978	80.33%	281608791	2.58%	1431655765	1545397953049
1150162021	29625299	1120536722	80.34%	281493744	2.58%	1431655765	1545398162508
1150170358	29625299	1120545059	80.34%	281485407	2.58%	1431655765	1545398177374
1212590800	30434496	1182156304	84.70%	219064965	2.51%	1431655765	1545512936226
1212608207	30434600	1182173607	84.70%	219047558	2.51%	1431655765	1545512969903
1714812975	37884031	1676928944	119.78%	0	2.21%	1431655765	1546432643215
1715522034	37884184	1677637850	119.83%	0	2.21%	1431655765	1546433933383
1957220075	38965688	1918254387	136.71%	0	1.99%	1431655765	1546869993652
1374582898	34267596	1340315302	96.01%	57072867	2.49%	1431655765	1546961480404
1377981578	34267596	1343713982	96.25%	53674187	2.49%	1431655765	1546967794888
1416730878	34267675	1382463203	98.96%	14924887	2.42%	1431655765	1547040742682
1419676228	34267736	1385408492	99.16%	11979537	2.41%	1431655765	1547046408039
1340948139	37151623	1303796516	93.66%	90707626	2.77%	1431655765	1547133254764
1342254006	38076370	1304177636	93.76%	89401759	2.84%	1431655765	1547138839505

It is possible to see that Tilerator processed more tiles than expected and also, after some jobs failing with message "TTL exceeded" the number of tiles processed dropped and Tilerator rerun the jobs. It's not known if the work is committed and tiles were stored in Cassandra, even though maps1004 increased its size of generated tiles compared with maps1001 (old master). The remaining jobs were stopped at 2019-09-01 15:23 freeing CPU to enable the OSM sync replication script.

General Updates

Tile processing reached near 100% (01-08-2019)

After a long run of tile generation jobs, and some difficulties to reach an end, the team decided that would be reasonable to continue with the migration plan without waiting for the full tile processing. Analyzing the collected data we figured out that only a small part of the tiles are stored and we might have enough tiles stored.

Also, it was found a failure in the strategy to regenerate the full planet tiles. It was documented a different process that is faster if you have a set of tiles pre-generated before to use as a basis to generate only the specific indexes. The next time that full planet tile regeneration is needed, we should consider this approach which would generate tiles according to the following benchmark:

Tile generation Benchmark per zoom level for 24 CPU's
Zoom	Total tiles	Estimated Time to finish
		Hours	Days
0	0	0.00	0.00
1	4	0.00	0.00
2	16	0.00	0.00
3	57	0.00	0.00
4	218	0.00	0.00
5	846	0.00	0.00
6	3301	0.00	0.00
7	13024	0.01	0.00
8	51509	0.03	0.00
9	202145	0.11	0.00
10	159340	0.08	0.00
11	526282	0.28	0.01
12	1822626	0.96	0.04
13	6099772	3.21	0.13
14	19000727	10.00	0.42
15	61208604	32.20	1.34
Total		36.87	1.54

Is important to notice that after the tile regeneration we had an increase of tiles on maps1004. For example, z15 increased the most with 61,208,604 tiles, that is 12,749,948 more tiles than our current production (~26% increase).

Useful Commands

The following Tilerator command was used to schedule and queue tile generation jobs:

sudo -u tileratorui node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js --config /etc/tileratorui/config.yaml \
-j.zoom 0 \
-j.fromZoom 0 \
-j.beforeZoom 16 \
-j.generatorId gen \
-j.storageId v4 \
-j.deleteEmpty true \
-j.keepJob true

Problems during tile generation

Tile generation for zoom levels 14 and 15 failed. Tilerator tries to run jobs with too much tiles and got the error TTL exceeded.
- Solution: restart jobs using Tilerator UI

What we have learned

A few things that was learned during this process are worth to mention:

Generating an entire zoom level is painful and slow and we should avoid at all costs for zoom levels greater than 13
This work is an incentive for a refactor of the available documentation
There isn't yet a reliable tool for Cassandra data copy, which would save us a lot of time too