Toolserver:OpenStreetMap server/Setup notes
This page was moved from the Toolserver wiki.
Toolserver has been replaced by Toolforge. As such, the instructions here may no longer work, but may still be of historical interest.
Please help by updating examples, links, template links, etc. If a page is still relevant, move it to a normal title and leave a redirect.
- Comment
- Cassini is no longer in use and its successor ptolemy is setup differently. Haggi 19:04, 12 August 2011 (UTC)
These are notes I (Ævar Arnfjörð Bjarmason) am keeping when setting up Cassini.
Software
[edit]This is the software running on Cassini
From apt
[edit]Non-standard software installed from the package system:
- aptitude install screen pv bzip2
- aptitude install subversion
- aptitude build-dep mapnik
- aptitude install python-cairo-dev libcairomm-1.0-dev # for mapnik
- aptitude install iotop
- aptitude install unzip # To get a stupd CPAN package that used a .zip file instead of gz/bzip2
- aptitude install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev proj # for osm2pgsql
- aptitude install postgresql-8.3-postgis
- aptitude install apache2
- aptitude build-dep libapache2-mod-python # to get apache/apr dev headers for mod_tile
- aptitude install strace
- aptitude install gdb
- aptitude install git-core
- aptitude install ttf-unifont # for i18n support in rendering
- aptitude install gdal-bin # for e.g. hillshading
- aptitude install libgdal1-dev # for building Mapnik with gdal support
From CPAN
[edit]Things installed from the CPAN. I'm using CPANPLUS (cpanplus) to install things. If this listing is incomplete /home/ravar/.cpanplus/install-logs/ should be the full listing.
- WWW::Mechanize
- HTML::TableParser::Grid
- YAML::Syck
- App::Ack
From source
[edit]Directories with stuff built from source, in /usr/local/src/
mapnik
[edit]ravar@cassini:/usr/local/src$ svn co http://svn.mapnik.org/trunk mapnik
Make it build with gdal and other goodies:
ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py INPUT_PLUGINS='all'
Build it:
ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py configure ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py ravar@cassini:/usr/local/src/mapnik$ sudo python scons/scons.py install
By default mapnik can't find its .so:
ravar@cassini:/usr/local/src/mapnik$ python
Python 2.5.2 (r252:60911, Jan 4 2009, 21:59:32)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mapnik
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/site-packages/mapnik/__init__.py", line 54, in <module>
from _mapnik import *
ImportError: libmapnik.so.0.6: cannot open shared object file: No such file or directory
>>>
You need to add /usr/local/lib64 to /etc/ld.so.conf.d/x86_64-linux-gnu.conf
ravar@cassini:~$ diff -ru /etc/ld.so.conf.d/x86_64-linux-gnu.conf~ /etc/ld.so.conf.d/x86_64-linux-gnu.conf --- /etc/ld.so.conf.d/x86_64-linux-gnu.conf~ 2009-07-21 14:54:36.000000000 +0000 +++ /etc/ld.so.conf.d/x86_64-linux-gnu.conf 2009-07-21 14:54:36.000000000 +0000 @@ -1,3 +1,4 @@ # Multiarch support /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu +/usr/local/lib64
Then run:
sudo ldconfig
unifonts
[edit]aptitude install ttf-unifont
Make renderd / mapnik load it:
sudo ln -s /usr/share/fonts/truetype/unifont/unifont.ttf /usr/local/lib64/mapnik/fonts/unifont.ttf
osm
[edit]We're using various stuff from the OSM SVN, copied with:
ravar@cassini:/usr/local/src$ svn co http://svn.openstreetmap.org/ osm
osm2pgsql
[edit]ravar@cassini:/usr/local/src/osm/applications/utils/export/osm2pgsql$ make ravar@cassini:/usr/local/src/osm/applications/utils/export/osm2pgsql$ sudo make install
osm-tools
[edit]These are our tools for setting up or own rendering/stuff hosted in MediaWiki SVN:
ravar@cassini:/usr/local/src$ svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/osm-tools
cgi-bin/export
[edit]sudo aptitude install python-yaml sudo ln -s /usr/local/src/osm-tools/cgi-bin/export /usr/lib/cgi-bin/export sudo ln -s /usr/local/src/osm-tools/cgi-bin/export-config.yml /sql/etc/export-config.yml
renderd
[edit]Create the renderd.conf:
# perl create-renderd-conf.pl --languages /sql/misc-data/wikipedia-languages.yml > /etc/renderd.conf
Create the directories renderd emits to:
avar@cassini:/sql/mod_tile$ mkdir $( perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl | perl -MYAML::Syck -E 'say join q[ ], sort keys %{ LoadFile(*STDIN) }' )
Data
[edit]Planet.osm
[edit]We have a copy of the Planet.osm file in /sql/planet.osm/
TODO: Keep it automatically up-to-date: http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#--read-change-interval_.28--rci.29
PostGIS
[edit]PostgreSQL has been setup to store its data on the /sql partition:
ravar@cassini:/sql$ sudo ack -a /sql /etc/postgresql /etc/postgresql/8.3/main/postgresql.conf 38:data_directory = '/sql/postgresql/8.3/main' # use data in another directory
And for allowing password auth from localhost:
ravar@cassini:~$ sudo tail -n9 /etc/postgresql/8.3/main/pg_hba.conf # "local" is for Unix domain socket connections only #local all all ident sameuser # IPv4 local connections: #host all all 127.0.0.1/32 md5 # IPv6 local connections: #host all all ::1/128 md5 local all all trust host all all 127.0.0.1/32 trust
Imported databases
[edit]At first I wasn't using --slim, which is less usable, see: http://lists.openstreetmap.org/pipermail/talk/2009-July/038786.html
TODO: Re-import with intarray support to support OSC updating: see [1]
With i18n support
[edit]createdb -E UTF8 -O gis gis createlang plpgsql gis psql -d gis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql echo "ALTER TABLE geometry_columns OWNER TO gis; ALTER TABLE spatial_ref_sys OWNER TO gis;" | psql -d gis
Create language list & style:
perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl > /sql/misc-data/languages.yml perl /usr/local/src/osm-tools/osm2pgsql-style/create-wikimedia-style.pl --style /usr/local/src/osm/applications/utils/export/osm2pgsql/default.style --languages /sql/misc-data/languages.yml > /sql/planet.osm/wikimedia.style
Import the planet:
osm2pgsql --slim -d gis -U avar -W -S ./wikimedia.style planet-090715.osm
world_boundaries
[edit]As documented on the OSM wiki:
avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/world_boundaries-spherical.tgz avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/processed_p.zip avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/shoreline_300.tar.bz2
avar@cassini:/sql/world_boundaries/unpacked$ unzip ../packed/processed_p.zip Archive: ../packed/processed_p.zip inflating: coastlines/processed_p.dbf inflating: coastlines/processed_p.index inflating: coastlines/processed_p.shp inflating: coastlines/processed_p.shx avar@cassini:/sql/world_boundaries/unpacked$ tar xjvf ../packed/shoreline_300.tar.bz2 shoreline_300.dbf shoreline_300.index shoreline_300.shp shoreline_300.shx avar@cassini:/sql/world_boundaries/unpacked$ tar xzvf ../packed/world_boundaries-spherical.tgz world_boundaries/ world_boundaries/places.shx world_boundaries/world_boundaries_m.index world_boundaries/world_bnd_m.shx world_boundaries/builtup_area.shx world_boundaries/world_bnd_m.dbf world_boundaries/builtup_area.prj world_boundaries/places.shp world_boundaries/world_boundaries_m.shx world_boundaries/world_boundaries_m.shp world_boundaries/places.dbf world_boundaries/places.prj world_boundaries/builtup_area.dbf world_boundaries/world_bnd_m.shp world_boundaries/world_bnd_m.prj world_boundaries/world_boundaries_m.dbf world_boundaries/builtup_area.shp world_boundaries/world_boundaries_m.prj world_boundaries/world_bnd_m.index world_boundaries/builtup_area.index
avar@cassini:/sql/world_boundaries/unpacked$ mv coastlines/* world_boundaries/* . avar@cassini:/sql/world_boundaries/unpacked$ rmdir coastlines world_boundaries
avar@cassini:/sql/world_boundaries/unpacked$ ls -l total 534M -rw-r--r-- 1 avar users 3.4M 2007-03-10 10:05 builtup_area.dbf -rw-r--r-- 1 avar users 273K 2007-03-10 10:05 builtup_area.index -rw-r--r-- 1 avar users 513 2007-03-10 10:05 builtup_area.prj -rw-r--r-- 1 avar users 13M 2007-03-10 10:05 builtup_area.shp -rw-r--r-- 1 avar users 285K 2007-03-10 10:05 builtup_area.shx -rw-r--r-- 1 avar users 16K 2007-03-10 10:07 places.dbf -rw-r--r-- 1 avar users 333 2007-03-10 10:07 places.prj -rw-r--r-- 1 avar users 7.0K 2007-03-10 10:07 places.shp -rw-r--r-- 1 avar users 2.1K 2007-03-10 10:07 places.shx -rw-r--r-- 1 avar users 4.3M 2008-08-10 20:48 processed_p.dbf -rw-r--r-- 1 avar users 2.0M 2008-08-10 21:59 processed_p.index -rw-r--r-- 1 avar users 357M 2008-08-10 20:48 processed_p.shp -rw-r--r-- 1 avar users 3.4M 2008-08-10 20:48 processed_p.shx -rw-r--r-- 1 avar users 3.0M 2009-07-10 04:01 shoreline_300.dbf -rw-r--r-- 1 avar users 1.3M 2009-07-10 04:01 shoreline_300.index -rw-r--r-- 1 avar users 86M 2009-07-10 04:01 shoreline_300.shp -rw-r--r-- 1 avar users 2.0M 2009-07-10 04:01 shoreline_300.shx -rw-r--r-- 1 avar users 4.0M 2007-03-09 11:31 world_bnd_m.dbf -rw-r--r-- 1 avar users 246K 2007-09-09 19:28 world_bnd_m.index -rw-r--r-- 1 avar users 355 2007-03-09 11:31 world_bnd_m.prj -rw-r--r-- 1 avar users 47M 2007-03-09 11:32 world_bnd_m.shp -rw-r--r-- 1 avar users 244K 2007-03-09 11:32 world_bnd_m.shx -rw-r--r-- 1 avar users 659K 2008-03-30 21:42 world_boundaries_m.dbf -rw-r--r-- 1 avar users 85K 2008-03-30 21:48 world_boundaries_m.index -rw-r--r-- 1 avar users 355 2008-03-30 21:51 world_boundaries_m.prj -rw-r--r-- 1 avar users 6.4M 2008-03-30 21:42 world_boundaries_m.shp -rw-r--r-- 1 avar users 30K 2008-03-30 21:42 world_boundaries_m.shx
/var/www/tile-browse
[edit]Create a HTML page to browse each of our 279 languages:
avar@cassini:/var/www/tile-browse$ for lang in $(perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl | perl -MYAML::Syck -E 'say join q[ ], sort keys %{ LoadFile(*STDIN) }'); do cp .browse-template.html browse-$lang.html && lang=$lang perl -pi -e 's[(?<=osm-like/)(\w+)][$ENV{lang}]' browse-$lang.html; done
Category:OpenStreetMap --Cseki 04:22, 19 November 2009 (UTC)