GeoIp Plugin

• When installed, the plugin will detect more accurately the country of your visitors using the MaxMind database.
• The existing reports for country and continent will be replaced by the more accurate GeoIp report in your Piwik interface.
• Piwik will also detect the city of your visitors. You can see all cities by clicking on the country row in the table.
• You can apply the GeoIp data to all your previous visits and re-generate your reports, see below.

Requirements

You need at least Piwik 0.2.33 to run the GeoIp plugin.

How to install?

• Download Piwik GeoIp Plugin
• Unzip the plugin and copy the extracted directory "GeoIp" in the directory piwik/plugins/
• Configure the MaxMind database:
  • Either download MaxMind GeoLite City or GeoIP City database from http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz Uncompress this file and copy it in plugins/GeoIP/libs/GeoLiteCity.dat
  • For more performance, it is recommanded to install the module apache mod_geoip. Configure your Apache according to: http://www.maxmind.com/app/mod_geoip
• Enable GeoIP plugin in Piwik. Go to Piwik > Settings > Plugins > Geoip, click Enable

How to apply the GeoIp to all your previous visits to fix your previous reports

You can apply GeoIP precise localization on your past data:

• run geoipUpdateRows.php script, from command line (or from the browser, but risk of timing out if your database is large)
• wait a few minutes until all your visitors data from the piwik_log_visit table have been processed.
• you now need to force Piwik to re-process all the past reports: drop all piwik_archive_* tables in your Piwik database. Piwik will automatically create them and re-process the reports.
• You should see Visitors countries and cities in your Visitors>Location report.

Authors

• Mikael Letang
• Maciej Zawadziski
• Matthieu Aubry
• Anthon Pang

Changelog

• version 0.7: cleaned up code, compatible with Piwik 0.2.33, improved documentation
• version 0.8: fixed bug in geoIpUpdateRows.php (thanks looztra)
• version 0.9: all fields now NULLable to make sure that insert still work fine after plugin is disabled
• version 0.10: compatibility with 0.2.35 after API change in the AddWidget function
• version 0.11: fixing bug with country detection
• version 0.12: updated for auto loader
• version 0.13: Compatible with Piwik 0.6
• version 0.14:
  • Fixed geoipUpdateRows script which was failing with errors. It also now checks that logged in user is Super User, and that the GeoIP plugin is enabled.
  • Fixed memory leak when using visitor generator.
  • Fixed .htaccess was blocking users from executing the script with the browser.
  • Added .htaccess to allow to execute the script directly from the browser.
  • Fixed Sparkline distinct countries
• version 0.15:
  • Updated for Piwik 1.3 (refactoring for upgrade.php)
  • No match for country now set to 'xx' (unknown)
• version 0.16:
  • fix bootstrap in geoipUpdateRows.php and clean-up package
  • geoipUpdateRows.php - MYSQLI compatibility
  • default to disk-based search; load into memory if multiple tracking requests
• version 0.17:
  • fixes from peterb & mruge
• version 0.18: compatible with Piwik 1.6

Feedback

Please leave a comment if you have any feedback, suggestion, or bug report. Keywords: third-party-plugin

In Piwik 1.8 we released the great new feature to import access logs and generate statistics.

The V1 release works very well (it was tracked in #703), but there are ideas to improve it. This ticket is a placeholder of all ideas and discussions related to the Log Analytics feature!

New features

• Track non-bot activity only. When --enable-bots is not specified, it would be a nice improvement if we:

  • exclude visits with more than 150 actions per visitorID to block crawlers (detected at the python level by counting requests for that IP in the queue)
  • exclude visits that do not have User Agent or beyond the very basic ones used by all bots
  • exclude all requests when one of the first ones is for /robots.txt -- if we see a robots.txt in the middle we could stop tracking subsequent requests
  • check that /index.php?minimize_js=file.js is counted as a static file since it ends in .js

  After that bots & crawlers detection would be much better.

• Support Accept-Language header and forward to piwik via the &lang= parameter. That might also be useful to some users who need to use this data in a custom plugin.

• we could make it easy to delete logs for one day so to reimport one log file

• This would be a new option to the python script. It would reuse the code from the Log Delete feature, but would only delete one day. The python script would call the CoreAdmin API for example, deleting this single day for a given website. This would allow to easily re-import data that didn't work the first time or was bogus.

• Detect when log-lines are re-imported and only import them once.

  • Implementation: add new table piwik_log_lines (hash_tracking_request, day ))
  • In Piwik Tracker, before looping on the bulk requests, SELECT all the log lines that have already been processed on this day (WHERE hash_tracking_request IN (a,b,c,d) AND day=?) & Skip these requests from import
  • After bulk requests are processed in piwik.php process, INSERT in bulk (hash, day)

• By default this feature would be enabled only for "Log import" script,

  • via a parameter that we know is the log import (&li=1 /import_logs=1)
  • but may be later useful to all users of Tracking API for general deduping service.

PERFORMANCE'

How to debug performance? First of all, you can run the script with --dry-run to see how many log lines per second are parsed. It typically should be between 2,000 and 5,000. When you don't do a dry run, it will insert new pageviews and visits calling Piwik API.

Other tickets

• #3867 cannot resume with line number reported by skip for ncsa_extended log format
• #4045 autodetection hangs on a weird formatted line

phpMyVisites supports ClickHeat (http://www.labsmedia.com/clickheat/) and Piwik could do the same.

The DEV version is available on http://www.labsmedia.com/clickheat/piwik/piwik-clickheat.tar.gz This is DEV only and may not work with latest trunk.

If you're interested to contribute please contact hello at piwik.org and yvan DOT taviaud at m4am.net Keywords: third-party-plugin

BotTracker Plugin

* When installed, the plugin will detect configured bots & webspiders and exclude them from the visitor-log
* All bot-hits are counted and for every bot the last visit is logged
* Some well-known bots are pre-configured

How to install?

* Download Piwik BotTracker Plugin
* Unzip the plugin and copy the extracted directory "BotTracker" in the directory piwik/plugins/
* Configure the Bot-List by editing the MySQL-Table *piwik_bot_db*.

Author

* Thomas Fasselt

Any help is welcome

Changelog

* version 0.10: first public version

Feedback

Please leave a comment if you have any feedback, suggestion, or bug report.

Keywords: bot, spider, search, engine, agent, third-party-plugin

Urchin Alternative: Import your server logs in Piwik, the Free web analytics platform!

See blog post Piwik alternative to Urchin for more information.

Piwik is the Urchin alternative but also Webalyzer and AWStats alternative: with a Python script, you can now import webserver logs (apache, iis, and more) in Piwik, instead of using the javascript tracking.

Description A Python script available in piwik/misc/log-analytics/ will parse server logs efficiently and automatically call the Piwik Tracking API to inject the visits/pageviews/downloads in Piwik.

How to install / how to use

• Requires Piwik >= 1.7.2-rc2. Download the latest version from http://builds.piwik.org/?C=N;O=D
• Requires at least Python 2.6
• Requires one or many server log files, typically called access.log in Apache for example. These log files will be imported into Piwik.
• You can also create a "test website" in Piwik to import all data into, rather than importing into your existing websites. Then, use the command --idsite=X to force all info from the log files to be imported into this idsite
• You can use --dry-run command to have a test run and make sure you will not track data or create new websites

SEE FOLLOW UP TICKET #3163

How you can help?

• please use the script and report your feedback and bugs here
• if you are a hacker yourself, please review the code and consider submitting performance optimization, or improvements.
• If you are a webhost or web agency and wish to offer Piwik to hundreds of your customers, please contact us
• review the doc at Server log analytics

Tasks to do before final release

• Test, test and test
• Setup on demo.piwik.org in a new website
• Check all code review feedback managed
• Review Import Logs in Piwik doc page.
• decomission apache2piwik (update blog post)

Feature requests for V2 or later

SEE FOLLOW UP TICKET #3163

In #2268 we have implemented a Referrer spam list, initially seeded with the worst of all spammers: semalt.

However it turns out there are thousands of other spammers that attack Piwik users with their lame websites. It will be hard to keep track of them all and write the list in the config file here.

List of spammers:

• see the current list of referrer spammer in Piwik global.ini.php file here

What is our best way to move forward?

Since upgrading to 2.10 the archive blobs and archive numbers tables have tripled in size (blob tables usually between 30-40MB, now they are 140-500MB).

The archiving process is set in cron to run every hour and nothing else has changed in the system beyond upgrading from 2.9 to 2.10 (no drastic change in amount of visitors, or to the structure of the site).

The tables are reduced in size a bit by running core:run-scheduled-tasks --force, but they're still quite a bit bigger than they used to be.

More background info in the following thread: http://forum.piwik.org/read.php?2,123852

Running Piwik on: Red Hat 5 apache 2.4 PHP 5.4.35 MySQL 5.0.45

Awesome feature: Analytics Previous / next pages

For each page URL, this plugin reports the top X

• previous pages
• following pages
• outlinks
• downloads
• campaigns
• search engine keywords
• external websites

The information is given in a popover which can be launched via a table action on the pages report.

We need to do an update of JQuery and JQueryUI.

JQuery 1.7.2 -> 1.9.1 JQueryUI 1.8.22 -> 1.10.2

As the last update was quite a long time ago, there might be some code changes required...

In order to show analytics data as an overlay, we have to set up a cross-domain communication protocol. The aim of this ticket is to complete a simple version of the feature that provides extensible mechanisms for the communication. Based on this, we can add more things to it.

Page Overlay displays a block around each link showing the % of clicks on this link. the user can still browse the website. On the left a sidebar displays the key statistics for the currently viewed page. It also easily allows the user to open the Page Transitions report and the Row evolution historical report for this particular page.

See #3530 List of Page Overlay Improvements for more info and the current state of the art!

UPDATE! Fund this project now to help us finish the work and release the feature!

It takes 2 minutes: http://crowdfunding.piwik.org/analytics-maps-world-country-city-region/

Right now the world map only shows which country the visits are coming from. It would be really great / useful if we could narrow it down to what state the visitor came from or even which city. Like in the Google analytics.

It'll be a great feature for marketing so we know where most of the visitors are!

We would like to also move the technology from Flash to full open source stack, and have the map displayed in SVG or Vector.

Once the Maps show cities and region, we definitely have to show the maps in the actual Country report: full awesomeness! see #1821

Once implemented we should remove SWFobject lib: #3666

After changing from php version 7.4.33 to 8.1.13, matomo users with "Superuser access" right can not login any more For all other user's with lower rights, login still works.

Behavior

A matomo error banner is displayed to the user:

            "Error: no website was found in this Matomo installation.
            Check the table '$siteTableName' in your database, it should contain your Matomo websites."

In the code at Plugin/Controller.php line 956 Piwik::hasUserSuperUserAccess() block is entered and throws the exception.

public function redirectToIndex($moduleToRedirect, $actionToRedirect, $websiteId = null, $defaultPeriod = null,
                                    $defaultDate = null, $parameters = array())
....
        if (Piwik::hasUserSuperUserAccess()) {
            $siteTableName = Common::prefixTable('site');
            $message = "Error: no website was found in this Matomo installation.
                        <br />Check the table '$siteTableName' in your database, it should contain your Matomo websites.";

            $ex = new NoWebsiteFoundException($message);
            $ex->setIsHtmlMessage();

            throw $ex;
        }

Expected Behavior

Piwik::hasUserSuperUserAccess() should not return true in this context. Why does it work with php7.4?

Possible Solution

I skipped the Piwik::hasUserSuperUserAccess() section -> login works.

My Environment

• Matomo Version: 4.13.0
• PHP Version: 8.1.13
• Server Operating System: Linux/Ubuntu 5.4.0-135-generic
• Webserver: nginx 1.22.1
• Browser: Firefox 108.0.1

Summary

Would it be possible to have sortable columns in the GDPR tools. I have found many case scenarios where It would be ideal to be able to sort the columns specifically the Visitor IP address one.

More details here - https://forum.matomo.org/t/gdpr-tools-sortable-columns/48053

Your Environment

• Matomo Version: 4.13.0
• PHP Version: 8.0.26
• Server Operating System:
• Additionally installed plugins:

Expected Behavior

Installling Matomo via composer should ship the TagManager plugin or notify about next step to get the complete installation

Current Behavior

When creating a new Matomo instance via composer create-project matomo/matomo the directory ./plugins/TagManager is empty.

total 8.0K
drwxrwxr-x  2 ubuntu ubuntu 4.0K Dec 27 10:31 .
drwxrwxr-x 95 ubuntu ubuntu 4.0K Dec 27 10:31 ..

Environment

Distro:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian

PHP:

PHP 8.1.11 (cli) (built: Sep 29 2022 22:29:14) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.11, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.11, Copyright (c), by Zend Technologies

Composer:

Composer version 2.4.2 2022-09-14 16:11:15

With DevicesDetection installed and making a request for any reports, thousands of requests are made for cached items for DevicesDetection even when it's not required. We are using Redis for cache. Could be that this a general "thing", not just for DevicesDetection. And could also be expected behaviour, that I don't know.

Expected Behavior

Cached items should only be asked for when needed.

Current Behavior

Requests are done for cached items not needed.

Steps to Reproduce (for Bugs)

1. Activate DevicesDetection
2. Do a request for a report not displaying any info about devices.
3. With a network sniffer, you could see requests done for DevicesDetection cached items.

Your Environment

• Matomo Version: 4.12.3
• PHP Version: 8..0.25
• Server Operating System: Linux Alpine

Description:

Thomas noticed that the Date->isToday() method was producing inconsistent results when dealing with multiple time zones. This is an effort to improve that accuracy.

Review

• [ ] Functional review done
• [ ] Potential edge cases thought about (behavior of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
• [ ] Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
• [ ] Security review done
• [ ] Wording review done
• [ ] Code review done
• [ ] Tests were added if useful/possible
• [ ] Reviewed for breaking changes
• [ ] Developer changelog updated if needed
• [ ] Documentation added if needed
• [ ] Existing documentation updated if needed