User:BDavis (WMF)/Projects/Logstash
Appearance
Notes on Logstash in Labs
Setup server
[edit]- Added firewall hole for ports 80 and 9200.
- Turned on shared home dirs and project storage
- Created x-large instance named logstash
- ssh logstash.pmtpa.wmflabs
- sudo puppetd --test --verbose
- sudo reboot
- Added puppet classes:
- role::labsnfs::client
- webserver::php5
- ssh logstash.pmtpa.wmflabs
- sudo puppetd --test --verbose
- sudo reboot
- ssh logstash.pmtpa.wmflabs
- verified that nfs client mounted /home and /data/project
- created /data/project/logstash to hold project files
- wget https://download.elasticsearch.org/logstash/logstash/logstash-1.2.2-flatjar.jar
- sudo apt-get install openjdk-7-jdk
Simple test of logstash
[edit]- created simple testing config for logstash:
input { stdin { } } output { stdout { codec => rubydebug } elasticsearch { embedded => true } }
- java -jar logstash-1.2.2-flatjar.jar agent -f logstash-simple.conf
- download kibana3 and install in /data/project/logstash/kibana
- edit /data/project/logstash/kibana/config.js
elasticsearch: "http://"+window.location.hostname,
- make /data/project/logstash/kibana/app/dashboards/default.json a symlink to logstash.json
- created apache reverse proxy conf in /data/project/logstash/apache:
<VirtualHost *:80> ServerAdmin webmaster@localhost DocumentRoot /data/project/logstash/kibana <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory /data/project/logstash/kibana> Order allow,deny allow from all </Directory> ErrorLog ${APACHE_LOG_DIR}/error.log LogLevel warn CustomLog ${APACHE_LOG_DIR}/access.log combined <IfModule mod_proxy.c> ProxyRequests Off <Proxy http://127.0.0.1:9200> ProxySet connectiontimeout=5 timeout=90 </Proxy> # Proxy for _aliases and .*/_search <LocationMatch "^/(_nodes|_aliases|_search|.*/_search|_mapping|.*/_mapping)$"> ProxyPassMatch http://127.0.0.1:9200/$1 ProxyPassReverse http://127.0.0.1:9200/$1 </LocationMatch> # Proxy for kibana-int/{dashboard,temp} stuff (if you don't want auth on # /, then you will want these to be protected) <LocationMatch "^/(kibana-int/dashboard/|kibana-int/temp)(.*)$"> ProxyPassMatch http://127.0.0.1:9200/$1$2 ProxyPassReverse http://127.0.0.1:9200/$1$2 </LocationMatch> </IfModule> </VirtualHost>
- sudo a2enmod proxy
- sudo a2enmod proxy_http
- sudo service apache restart
- cd /etc/apache2/sites-available
- sudo ln -s /data/project/logstash/apache/proxy
- sudo service apache2 reload
- browse to http://logstash.instance-proxy.wmflabs.org/ and see kibana load!