GeoIp Plugin

• When installed, the plugin will detect more accurately the country of your visitors using the MaxMind database.
• The existing reports for country and continent will be replaced by the more accurate GeoIp report in your Piwik interface.
• Piwik will also detect the city of your visitors. You can see all cities by clicking on the country row in the table.
• You can apply the GeoIp data to all your previous visits and re-generate your reports, see below.

Requirements

You need at least Piwik 0.2.33 to run the GeoIp plugin.

How to install?

• Download Piwik GeoIp Plugin
• Unzip the plugin and copy the extracted directory "GeoIp" in the directory piwik/plugins/
• Configure the MaxMind database:
  • Either download MaxMind GeoLite City or GeoIP City database from http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz Uncompress this file and copy it in plugins/GeoIP/libs/GeoLiteCity.dat
  • For more performance, it is recommanded to install the module apache mod_geoip. Configure your Apache according to: http://www.maxmind.com/app/mod_geoip
• Enable GeoIP plugin in Piwik. Go to Piwik > Settings > Plugins > Geoip, click Enable

How to apply the GeoIp to all your previous visits to fix your previous reports

You can apply GeoIP precise localization on your past data:

• run geoipUpdateRows.php script, from command line (or from the browser, but risk of timing out if your database is large)
• wait a few minutes until all your visitors data from the piwik_log_visit table have been processed.
• you now need to force Piwik to re-process all the past reports: drop all piwik_archive_* tables in your Piwik database. Piwik will automatically create them and re-process the reports.
• You should see Visitors countries and cities in your Visitors>Location report.

Authors

• Mikael Letang
• Maciej Zawadziski
• Matthieu Aubry
• Anthon Pang

Changelog

• version 0.7: cleaned up code, compatible with Piwik 0.2.33, improved documentation
• version 0.8: fixed bug in geoIpUpdateRows.php (thanks looztra)
• version 0.9: all fields now NULLable to make sure that insert still work fine after plugin is disabled
• version 0.10: compatibility with 0.2.35 after API change in the AddWidget function
• version 0.11: fixing bug with country detection
• version 0.12: updated for auto loader
• version 0.13: Compatible with Piwik 0.6
• version 0.14:
  • Fixed geoipUpdateRows script which was failing with errors. It also now checks that logged in user is Super User, and that the GeoIP plugin is enabled.
  • Fixed memory leak when using visitor generator.
  • Fixed .htaccess was blocking users from executing the script with the browser.
  • Added .htaccess to allow to execute the script directly from the browser.
  • Fixed Sparkline distinct countries
• version 0.15:
  • Updated for Piwik 1.3 (refactoring for upgrade.php)
  • No match for country now set to 'xx' (unknown)
• version 0.16:
  • fix bootstrap in geoipUpdateRows.php and clean-up package
  • geoipUpdateRows.php - MYSQLI compatibility
  • default to disk-based search; load into memory if multiple tracking requests
• version 0.17:
  • fixes from peterb & mruge
• version 0.18: compatible with Piwik 1.6

Feedback

Please leave a comment if you have any feedback, suggestion, or bug report. Keywords: third-party-plugin

In Piwik 1.8 we released the great new feature to import access logs and generate statistics.

The V1 release works very well (it was tracked in #703), but there are ideas to improve it. This ticket is a placeholder of all ideas and discussions related to the Log Analytics feature!

New features

• Track non-bot activity only. When --enable-bots is not specified, it would be a nice improvement if we:

  • exclude visits with more than 150 actions per visitorID to block crawlers (detected at the python level by counting requests for that IP in the queue)
  • exclude visits that do not have User Agent or beyond the very basic ones used by all bots
  • exclude all requests when one of the first ones is for /robots.txt -- if we see a robots.txt in the middle we could stop tracking subsequent requests
  • check that /index.php?minimize_js=file.js is counted as a static file since it ends in .js

  After that bots & crawlers detection would be much better.

• Support Accept-Language header and forward to piwik via the &lang= parameter. That might also be useful to some users who need to use this data in a custom plugin.

• we could make it easy to delete logs for one day so to reimport one log file

• This would be a new option to the python script. It would reuse the code from the Log Delete feature, but would only delete one day. The python script would call the CoreAdmin API for example, deleting this single day for a given website. This would allow to easily re-import data that didn't work the first time or was bogus.

• Detect when log-lines are re-imported and only import them once.

  • Implementation: add new table piwik_log_lines (hash_tracking_request, day ))
  • In Piwik Tracker, before looping on the bulk requests, SELECT all the log lines that have already been processed on this day (WHERE hash_tracking_request IN (a,b,c,d) AND day=?) & Skip these requests from import
  • After bulk requests are processed in piwik.php process, INSERT in bulk (hash, day)

• By default this feature would be enabled only for "Log import" script,

  • via a parameter that we know is the log import (&li=1 /import_logs=1)
  • but may be later useful to all users of Tracking API for general deduping service.

PERFORMANCE'

How to debug performance? First of all, you can run the script with --dry-run to see how many log lines per second are parsed. It typically should be between 2,000 and 5,000. When you don't do a dry run, it will insert new pageviews and visits calling Piwik API.

Other tickets

• #3867 cannot resume with line number reported by skip for ncsa_extended log format
• #4045 autodetection hangs on a weird formatted line

phpMyVisites supports ClickHeat (http://www.labsmedia.com/clickheat/) and Piwik could do the same.

The DEV version is available on http://www.labsmedia.com/clickheat/piwik/piwik-clickheat.tar.gz This is DEV only and may not work with latest trunk.

If you're interested to contribute please contact hello at piwik.org and yvan DOT taviaud at m4am.net Keywords: third-party-plugin

BotTracker Plugin

* When installed, the plugin will detect configured bots & webspiders and exclude them from the visitor-log
* All bot-hits are counted and for every bot the last visit is logged
* Some well-known bots are pre-configured

How to install?

* Download Piwik BotTracker Plugin
* Unzip the plugin and copy the extracted directory "BotTracker" in the directory piwik/plugins/
* Configure the Bot-List by editing the MySQL-Table *piwik_bot_db*.

Author

* Thomas Fasselt

Any help is welcome

Changelog

* version 0.10: first public version

Feedback

Please leave a comment if you have any feedback, suggestion, or bug report.

Keywords: bot, spider, search, engine, agent, third-party-plugin

Urchin Alternative: Import your server logs in Piwik, the Free web analytics platform!

See blog post Piwik alternative to Urchin for more information.

Piwik is the Urchin alternative but also Webalyzer and AWStats alternative: with a Python script, you can now import webserver logs (apache, iis, and more) in Piwik, instead of using the javascript tracking.

Description A Python script available in piwik/misc/log-analytics/ will parse server logs efficiently and automatically call the Piwik Tracking API to inject the visits/pageviews/downloads in Piwik.

How to install / how to use

• Requires Piwik >= 1.7.2-rc2. Download the latest version from http://builds.piwik.org/?C=N;O=D
• Requires at least Python 2.6
• Requires one or many server log files, typically called access.log in Apache for example. These log files will be imported into Piwik.
• You can also create a "test website" in Piwik to import all data into, rather than importing into your existing websites. Then, use the command --idsite=X to force all info from the log files to be imported into this idsite
• You can use --dry-run command to have a test run and make sure you will not track data or create new websites

SEE FOLLOW UP TICKET #3163

How you can help?

• please use the script and report your feedback and bugs here
• if you are a hacker yourself, please review the code and consider submitting performance optimization, or improvements.
• If you are a webhost or web agency and wish to offer Piwik to hundreds of your customers, please contact us
• review the doc at Server log analytics

Tasks to do before final release

• Test, test and test
• Setup on demo.piwik.org in a new website
• Check all code review feedback managed
• Review Import Logs in Piwik doc page.
• decomission apache2piwik (update blog post)

Feature requests for V2 or later

SEE FOLLOW UP TICKET #3163

In #2268 we have implemented a Referrer spam list, initially seeded with the worst of all spammers: semalt.

However it turns out there are thousands of other spammers that attack Piwik users with their lame websites. It will be hard to keep track of them all and write the list in the config file here.

List of spammers:

• see the current list of referrer spammer in Piwik global.ini.php file here

What is our best way to move forward?

Since upgrading to 2.10 the archive blobs and archive numbers tables have tripled in size (blob tables usually between 30-40MB, now they are 140-500MB).

The archiving process is set in cron to run every hour and nothing else has changed in the system beyond upgrading from 2.9 to 2.10 (no drastic change in amount of visitors, or to the structure of the site).

The tables are reduced in size a bit by running core:run-scheduled-tasks --force, but they're still quite a bit bigger than they used to be.

More background info in the following thread: http://forum.piwik.org/read.php?2,123852

Running Piwik on: Red Hat 5 apache 2.4 PHP 5.4.35 MySQL 5.0.45

Awesome feature: Analytics Previous / next pages

For each page URL, this plugin reports the top X

• previous pages
• following pages
• outlinks
• downloads
• campaigns
• search engine keywords
• external websites

The information is given in a popover which can be launched via a table action on the pages report.

We need to do an update of JQuery and JQueryUI.

JQuery 1.7.2 -> 1.9.1 JQueryUI 1.8.22 -> 1.10.2

As the last update was quite a long time ago, there might be some code changes required...

In order to show analytics data as an overlay, we have to set up a cross-domain communication protocol. The aim of this ticket is to complete a simple version of the feature that provides extensible mechanisms for the communication. Based on this, we can add more things to it.

Page Overlay displays a block around each link showing the % of clicks on this link. the user can still browse the website. On the left a sidebar displays the key statistics for the currently viewed page. It also easily allows the user to open the Page Transitions report and the Row evolution historical report for this particular page.

See #3530 List of Page Overlay Improvements for more info and the current state of the art!

UPDATE! Fund this project now to help us finish the work and release the feature!

It takes 2 minutes: http://crowdfunding.piwik.org/analytics-maps-world-country-city-region/

Right now the world map only shows which country the visits are coming from. It would be really great / useful if we could narrow it down to what state the visitor came from or even which city. Like in the Google analytics.

It'll be a great feature for marketing so we know where most of the visitors are!

We would like to also move the technology from Flash to full open source stack, and have the map displayed in SVG or Vector.

Once the Maps show cities and region, we definitely have to show the maps in the actual Country report: full awesomeness! see #1821

Once implemented we should remove SWFobject lib: #3666

As show in the following Screenshots the difference in times recorded

Time zone is accurate for values saved in serverTimePretty fields for visits and actions.

As of now, we only have "Anonymize previously tracked raw data" settings under Privacy > Anonymize data which is applicable to each website (measurable).

I think it would be a good idea to make all of the 'Anonymize Data', 'Regularly delete old raw data', 'Delete old aggregated report data' and 'Anonymize previously tracked raw data' settings website specific, as well as the option that sets a cookie to prevent your own visit from being tracked.

Matomo provides the following Ecommerce > Products reports:

• Product Name
• Product SKU
• Product Category

In these reports, when hovering on a row, the Segmented Visits log icon is not displayed: (only the row evolution icon displays):

Instead, we expected that the Segmented Visits log icon displays.

The icon displays correctly in the "Sales" reports eg:

Showing this icon in the "Products" reports will be very valuable to Ecommerce operators who can then easily view the visits where customers purchased the product that is being clicked.

Hello All, not an issue more more an enhancement. Would it be able to integrate a link to the above mentioned URL with the IP of a visit so to be able to lookup more info about the visitors IP.

thank you

A client of mine use the geolocation feature with the Nginx module. Only depending on the consent, the location of the visitor is determined by the web server.

Now it is the case that the consent for the location is only obtained after the first page impression. Consequently, in Matomo the visit has already been created and the location for a country is determined based on the browser language. Unfortunately, however, Matomo does not update the location in the visit after the visit has started. Although the Nginx module now delivers the concrete location and Matomo can read the server variables, the location is not updated in the visit anymore.

Is there a (future) possibility to update/override the location during the visit?

Description:

Suppose we have two extra log tables in a plugin: log_thing_event and log_thing. log_thing_event joins log_visit on idvisit, while log_thing joins log_thing_event on idlogthing.

If we try to use Segment to generate SQL that queries log_thing_event, it will automatically join on log_visit to get segmentation to work. BUT if we try to use it to generate SQL that queries log_thing, it will not be able to make segmentation work, even though it can be joined to log_visit through log_thing_event.

This PR fixes this case. When looking for joins, the existing code tries to join a table to one of the previously seen tables in the $from array. It will look to see if it is directly joinable to log_visit, otherwise it will look in the getWaysToJoinOtherTables() of the table to join. This doesn't work in our case where $from = ['log_thing', 'log_thing_event'], because 'log_thing_event' won't be in the getWaysToJoinOtherTables() of 'log_thing'. Instead, log_thing will be in the getWaysToJoinOtherTables() of log_thing_event. So this adds the reverse of the existing check to make this type of query work.

The use case here is that log_thing will have far less rows than log_thing_event and querying over it first instead of log_thing_event can in some cases be far more efficient.

Review

• [ ] Functional review done
• [ ] Potential edge cases thought about (behavior of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
• [ ] Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
• [ ] Security review done
• [ ] Wording review done
• [ ] Code review done
• [ ] Tests were added if useful/possible
• [ ] Reviewed for breaking changes
• [ ] Developer changelog updated if needed
• [ ] Documentation added if needed
• [ ] Existing documentation updated if needed