@Thomas--F opened this Issue on May 2nd 2011 Contributor

BotTracker Plugin

* When installed, the plugin will detect configured bots & webspiders and exclude them from the visitor-log
* All bot-hits are counted and for every bot the last visit is logged
* Some well-known bots are pre-configured

How to install?

* Download Piwik BotTracker Plugin
* Unzip the plugin and copy the extracted directory "BotTracker" in the directory piwik/plugins/
* Configure the Bot-List by editing the MySQL-Table *piwik_bot_db*.

Author

* Thomas Fasselt

Any help is welcome

Changelog

* version 0.10: first public version

Feedback

Please leave a comment if you have any feedback, suggestion, or bug report.

Keywords: bot, spider, search, engine, agent, third-party-plugin

@anonymous-piwik-user commented on January 13th 2013

Attachment: Hungarian translation
hu.php

@Thomas--F commented on September 19th 2013 Contributor

Attachment: v 0.31: fixed static calls in API
BotTracker.2.zip

@hpvd commented on October 26th 2013
@Thomas--F commented on December 30th 2013 Contributor

Attachment: v0.32 - last version for Piwik 1.x
BotTracker_alt.zip

@hpvd commented on December 31st 2013
@hpvd commented on December 31st 2013
@Thomas--F commented on January 7th 2014 Contributor

Attachment: v 0.43: adding extra Statistics
BotTracker.zip

@Thomas--F commented on May 2nd 2011 Contributor

The plugins comes with a widget to show the data from the bot_db-table.
I've testet the plugin, but there maybe still bugs in it. I don't suggest to use it in a productive enviroment.

ToDos:

  • Create Widget to configure the bot_db (add new bots, change keyword, (de-)activate bots, etc.)
  • testing! testing! testing!

Oh, and there is one big limitation:
Most bots don't use JavaScript. So if you use only the Piwik-JS-API (default), you don't get any results.
I use the PHP-API, so every bot hits my visitor-log. That's why I wrote this plugin.

@anonymous-piwik-user commented on May 2nd 2011

Nice I exactly look for this module,but after copying this module I get this error.

Unable to load plugin 'BotTracker' because '/home/**/public_html/plugins/BotTracker/BotTracker.php' couldn't be found. You can manually uninstall the plugin by removing the line Plugins[] = BotTracker from the Piwik config file.

The file is there! ?? Any Ideas

@Thomas--F commented on May 3rd 2011 Contributor

Hmmm.... sounds strange. I installed the plugin about 30 times during development and test.

First check, if the folder looks exact as shown in the message (e.g. "BotTracker" or "bottracker") and then check the folder and file permissions. Is the read-access restricted?

@anonymous-piwik-user commented on May 3rd 2011

We are using piwik 1.4 and getting getting the following error when activating the plugin

Fatal error: Call to undefined method Piwik_BotTracker::LogToFile() in example.com/analytics/piwik/plugins/BotTracker/BotTracker.php on line 41

@anonymous-piwik-user commented on May 3rd 2011

Replying to jekko:

We are using piwik 1.4 and getting getting the following error when activating the plugin

Fatal error: Call to undefined method Piwik_BotTracker::logToFile() in example.com/analytics/piwik/plugins/BotTracker/BotTracker.php on line 41

@Thomas--F commented on May 3rd 2011 Contributor

Hi jekko,

try the new version (v0.12).
I wrote the LogToFile-function for some debug-logging and in v0.10 i deleted the function but not all of the calls.

btw: I tested the plugin with Piwik 1.3 and 1.4

@anonymous-piwik-user commented on May 3rd 2011

thomas its working, ty

@Thomas--F commented on May 5th 2011 Contributor

Changelog

* version 0.15: better Widget & new entry in visitor-menue "Bot Tracker"
@Thomas--F commented on May 5th 2011 Contributor

Changelog

  • version 0.16:
    • Widget shows only active bots
    • BotTracker-Menue shows active-status as icon
@anonymous-piwik-user commented on May 8th 2011

Hello,

I'm running piwik 1.4 with BotTracker 0.16.
My Piwik Installation monitors multiple sites.

Plugin has been installed and activated. Widged has been added. Sites contain the Java Script code.
I can't see that any bot access gets counted. After some days I used google webmaster tools trying to force a "access like a bot" access. Still no count.

Question:
Is there any alternate method to simulate an access and verify the installation ?
Should the bots which allow JS be catched by the Plugin ?
Might the problem be caused by that I'm tracking multiple sites on the installation ?

Any hint very much appreciated.

Best regards, sun

@anonymous-piwik-user commented on May 8th 2011

did you read this?

Oh, and there is one big limitation: Most bots don't use JavaScript. So if you use only the Piwik-JS-API (default), you don't get any results. I use the PHP-API, so every bot hits my visitor-log. That's why I wrote this plugin.

@Thomas--F commented on May 8th 2011 Contributor

Hi sun,

first of all: the plugin is not able to track multi-sites. But I will put that on my todo-list.
The results are currently the sum of all sites.

To test the plugin I use Firefox with a plugin called "User Agend R G".

But remeber: The plugin will only catch non-JS-Vots if you use the PHP-Tracking-API!
Most Bots don't use JS, so don't expect much results when you only use the standard-tracking-code!

@Thomas--F commented on May 9th 2011 Contributor

Dev-Status:
I am currently trying to improve the configuration of the plugin:

  • enable/disable a bot by clicking on the icon
  • open a sub-table with a click on the bot-name. In this sub-table you change the bot-name and/or the keyword. There should also be a delete-button
  • add-new-bot-button at the bottom of the table

In addition to that I will change the database so the plugin can track multiple sites seperately.


I'm a professional programmer for more than 16 years now, but mainly on the mainframe. In the last years I also code some java-applications but PHP is only a hobby. So don't expect a fast solution here. I try to learn the API by looking into the examples, the source and doing some try-and-error-debugging.

If someone will jump in and offers help... the door is wide open! Just leave me a message.

And the last point:
I try to track even this site by using the Image-Tracking. [[Image(http://piwik.rwk-kempen-krefeld.de/piwik.php?idsite=2&rec=1)]] Is this going to work...?

@Thomas--F commented on May 10th 2011 Contributor

Changelog

* version 0.18: Multi-Site

I had add a column to the database, so if you install the new version, it will drop the old database and create a new one. Then the script will insert the bot-list for all sites where the user is admin.
To run the install-script you have to follow these steps:

  • deactivate BotTracker
  • log out (I'm not sure about that)
  • remove the row
    PluginsInstalled[] = "BotTracker"
    from the file
    /piwik/config/config.ini.php
  • log in and activate BotTracker
@anonymous-piwik-user commented on May 11th 2011

Hi Thomas,

I looked further into my issue not getting any count or timestamp.
First of all. I updated to 0.18.
Second, please forgive me, but I havn't any glue about PHP.

Having that said, I modified BotTracker.php providing me some more log data at function checkBot.

[2011/05/10 21:42:05] user Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
[2011/05/10 21:42:05] SiteID:3
[2011/05/10 21:42:05] Row:43
[2011/05/10 21:42:05] Row[botId]:4

I can see, that the bot access gets catched and that the proper row gets detected (43). GoogleBot matches in my database ID 43. Also I can see, that the code execution enters the if statement where I print the value of $row['botId'] which by above example is 4. So the query updating the database gets performed. But, rather than updating row 43, he is updating row 4, which is the wrong site as well as the wrong bot.

Shouldn't the if statement at the checkBot statement more be like:

        if ($row > 0 ){
            $query = "UPDATE `".Piwik_Common::prefixTable('bot_db')."` 
                      SET botCount = botCount + 1
                        , botLastVisit = CURRENT_TIMESTAMP()
                      WHERE botId = ".$row." ";

            Piwik_Query($query);

            $exclude =& $notification->getNotificationObject();
            $exclude = true;
        }

I replaced $row['botId'] with just $row. At least on my installation he updates now the correct row.

Could you please verify and let me know your comment.

Best regards,

sun

@Thomas--F commented on May 11th 2011 Contributor

Hi sun,

I get the variable $row from the function Piwik_FetchOne:

        $row = Piwik_FetchOne("SELECT botId FROM ".Piwik_Common::prefixTable('bot_db')."
                               WHERE botActive = 1 
                               AND   idSite = ".$idSite."
                               AND   LOCATE(botKeyword,'".$ua."') >0
                           LIMIT 1");

This function returns an labled array with all selected columns. In your case it should be
array( 'botId' => 43)

Because of this I'm surprised, that you get an positive result when you only use $row.

How do you print the log-data?
What PHP-Version do you use?

Best regards,
Thomas

@anonymous-piwik-user commented on May 11th 2011

Hi Thomas,

it's PHP 5.2.12 and MYSQL 5.1

The 2 lines

     Piwik_BotTracker::logToFile('Row:'.$row);
     Piwik_BotTracker::logToFile('Row[botId]:'.$row['botId']);

just before the if statement generate

[2011/05/11 17:16:11] Row:43
[2011/05/11 17:16:11] Row[botId]:4

which results into using your code that row 4 gets updated and using my code, row 43 gets updated.

sun

@anonymous-piwik-user commented on May 11th 2011

One more.
Above was a googlebot access.
With MSNBOT I get

[2011/05/11 17:36:13] Row:41
[2011/05/11 17:36:13] Row[botId]:4

So with my code it counts for Row 41, which is MSN in my database.
With your's, it also ends up in row 4

sun

@Thomas--F commented on May 11th 2011 Contributor

Hi sun,

I found the error.

The function Piwik_FetchOne did not return an array, it returns the value.
Because "botId" was not defined, the array-access returns only the first char:
4 instead of 41

I will fix an test it. Thank you.

@anonymous-piwik-user commented on May 11th 2011

Yep. That works for me as well. Thanks!

@robocoder commented on May 12th 2011 Contributor

Thomas: I'm glad to see you're continuing to develop this oft-requested feature.

A couple of comments:

@Thomas--F commented on May 12th 2011 Contributor

Hi vipsoft,

thanks for the tips. As you can see I already updated the plugin.

Will the update-scripts run automaticly when you update the plugin? Will all scripts run in the right order if I have skipped a version?

And I have some questions concerning the usage of Trac:

  • How can I change the description of the ticket? I want to add all the changelog-entries.
  • How can I delete the old version "BotTracker.2.zip"? I forgot once to check the box "delete existing file".
@Thomas--F commented on May 12th 2011 Contributor

Oh, and by the way:
Can you delete the last row (last point) in comment:13?

@Thomas--F commented on May 14th 2011 Contributor

Changelog 0.21

  • Change format of botCount from varchar to INT
  • Add new Bot "Exabot"
  • Add Update-Script for v0.18 in case someone has to update BotTracker from an very old version

I have written and tested 2 update-scripts: v0.18 and v0.21
It's a great feature of piwik. Thanks a lot to vipsoft for the hint.

@robocoder commented on May 15th 2011 Contributor

In checkBot():

  • why does the UPDATE not include idSite in the WHERE clause?
  • have you compared the performance of the SELECT vs using the tracker cache? (I suspect the LOCATE() equates to a table scan.)
@Thomas--F commented on May 15th 2011 Contributor

The UPDATE uses only the botId because it's the unique index of the table. So if I found a qualified row while using the idSite, I can update the table with just using the botId.

Do you have a description of the tracker cache and how to use it? The table is very small and my sites doesn't get very much hits, so performance tuning is not my top priority. But I will compare both ways if I can implement the cache.

@robocoder commented on May 15th 2011 Contributor

The tracker cache are files in tmp/cache/tracker that are automatically loaded with each tracker request. You can see this being used by SitesManager.php in recordWebSiteDataInCache().

Please also take a look at Matt's ideas in #653

@Thomas--F commented on May 15th 2011 Contributor

There are some points to think about:

  • Change from bot-hits to bot-visits?
    This means, if a bot hits the tracker, the botCount is only updated, if the last hit was 30 minutes ago.
  • Log every visit in an other(new) table to get a real history?
    This has to be, if I want to implement any of the "Additional features", Matt describes in his comment. My first intention was to block bots from my visitor-log. But I can understand, that these features contain very useful informations.
  • Enable the tracker cache?
    The main point to think about ist: What shoud I cache to improve performance? Maybe it's a good idea to store the last 5 user agents (or the hash of it) and the botId (zero if it wasn't a bot). So if a visitor walks through the page, the script has not to seach the whole table for every track-call.

I wish to make some of these features flexible (e.g. how much user agents should be stored or how much time between 2 hits for logging).
Should I use global variables in BotTracker.php or are there any plans for a config-dialog for plugins?

@Thomas--F commented on May 20th 2011 Contributor

Changelog 0.22

  • implementation of an config-menu
    In "Settings" you find a new sub-menu "BotTracker" where you can change bot-names and bot-keywords, (de-)activate entries and add new entries. The config-dialog works for multi-site-installations.

I've tested a lot, but I'm sure there are bugs left. Please report anything, that doesn't work as designed.

@Thomas--F commented on May 23rd 2011 Contributor

To solve some of Matt's "aditional features" I have to generate a new table that logs every hit of a bot.
What do you think, what information should be stored in that table?

  • Timestamp
  • IP-adress (masked if the plugin is used)
  • User-Agent
  • page viewed

Is there more to think about?

@anonymous-piwik-user commented on May 24th 2011

Got an error message regarding

duplicate "idsite" column name

when installing on a fresh 1.4 install.

Fix: remove the 0.18.php file from the updates folder (was trying to alter table and add the "idsite" column when said column was already created during installation process).

@Thomas--F commented on June 3rd 2011 Contributor

@nslyv

I could not reproduce your error. I reinstalled Piwik on a fresh server and tried to install BotTracker several times without any issues.

But the last two weeks were full of work so I hadn't much time to continue on BotTracker. I hope the next weeks will be better....

@anonymous-piwik-user commented on June 14th 2011

Hello, Thomas!

I'm lost.
I downloaded the plugin, unzipped and uploaded to server. There is no install file, and I don't know where to look for the database.
Any pointers would be appreciated.
Best regards

@Thomas--F commented on June 14th 2011 Contributor

Hi marketeer,

first you have to upload the plugin to the folder
www.(myServer).xyz/(myPiwikFolder)/plugins/BotTracker

Then login as Super-Admin and klick on "Settings" and then on the "Plugins"-tab.
There you shold see now an entry "BotTracker" with status "Inactive" an the possible action "Activate"....

@anonymous-piwik-user commented on June 15th 2011

Replying to ThomasF:

Hi marketeer,

first you have to upload the plugin to the folder
www.(myServer).xyz/(myPiwikFolder)/plugins/BotTracker

Then login as Super-Admin and klick on "Settings" and then on the "Plugins"-tab.
There you shold see now an entry "BotTracker" with status "Inactive" an the possible action "Activate"....

Hello, Thomas!

Thank you very much for your prompt reaction.

Yes, I followed the instructions, but after activating the plug-in I received this error message: "SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'idsite'".

another result is that I can no longer login to my piwik installation.

Is this because I have multiple sites listed?

Do I need to re-install?

@Thomas--F commented on June 15th 2011 Contributor

Hi marketeer,

that's strange. It's the same error that nslyv already postet. But I wasn't able to reproduce it.

As a walkaround just delete the file 0.18.php in the folder update

@Thomas--F commented on June 16th 2011 Contributor

As reaction on the "Duplicate column"-error I deletet the 0.18.php-update-script.

@anonymous-piwik-user commented on June 16th 2011

Replying to ThomasF:

As reaction on the "Duplicate column"-error I deletet the 0.18.php-update-script.

Hello, Thomas!

Thank you for your attention to this matter.

Yes, deleting 0.18.php puts me back on track.

Now I need to figure out the api-php for a joomla installation (into template or module).

Best regards

@anonymous-piwik-user commented on June 16th 2011

Hi, Thomas!

My logfiles show several IPs that access my website at a rate of about 2500 hits per hour each! I would like to track these using your plug-in and the ip-adresses. Can this be done, configured?

@Thomas--F commented on June 16th 2011 Contributor

Hi marketeer,

the plugin works only on keywords in the User Agent not on IPs. In your server-access-log you should see the User Agent of these visitors. If they are specific enough you can use the plugin to count their hits.

For a full track (logging each visit in a seperate table) you have to wait for the next version I'm currently working on.

For using the PHP-API in Joomla just copy the "PiwikTracker.php" to your template-folder and enter the following code in the index.php of your template:
(I put it in the "footer_r")

<!-- Piwik -->
<?php 
// -- Piwik Tracking API init --
require_once "PiwikTracker.php";
$pageTitle = $this->getTitle();

$piwikTracker = new PiwikTracker( $idSite = 1 );
PiwikTracker::$URL = 'http://yourwebsite.org/piwik/';

$piwikTracker->setTokenAuth = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
$piwikTracker->setUrl( $url = JURI::current());
$piwikTracker->setIp( $_SERVER['REMOTE_ADDR'] );
// Sends Tracker request via http
$piwikTracker->doTrackPageView($pageTitle);

?>
<!-- End Piwik Tracking Code -->
@anonymous-piwik-user commented on July 7th 2011

ThomasF, you write that your tracker is intended for use with the PHP-API. I'm trying to track visits/bots with the help of Piwik's GIF image. However, loading the GIF image with your plugin activated, I get the following error output when loading the GIF file:

<b>Warning</b>: fopen(tmp/logs/log.txt) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: No such file or directory in <b>/public_html/piwik/plugins/BotTracker/BotTracker.php</b> on line <b>157</b>
<b>Warning</b>: fwrite(): supplied argument is not a valid stream resource in <b>/public_html/piwik/plugins/BotTracker/BotTracker.php</b> on line <b>166</b>
<b>Warning</b>: fclose(): supplied argument is not a valid stream resource in <b>/public_html/piwik/plugins/BotTracker/BotTracker.php</b> on line <b>168</b>

Any way around this?

@Thomas--F commented on July 7th 2011 Contributor

Hi opensourcer,

I used a log-file to track all user-agents and find new bots. I haven't planed to include this to the public version so I removed the function in v.23. This new version should work with the GIF-image.

But I found another problem I couldn't fix so far:
The pie-chart is showing the wrong entries. If I use the bar-chart, everything is ok. All values and descriptors are corret, but when I switch to the pie-chart, the biggest entry shows something like "bot x 1%(2 hits)" instead of "bot y 90% (345 hits)".
Can anyone confirm the problem?

@robocoder commented on July 7th 2011 Contributor

The pie chart problem is a Piwik bug. It's fixed in trunk.

@anonymous-piwik-user commented on July 20th 2011

I get always an error if I vist the Bot Tracker configuration page. The main url which are created during the installtion process shows me correct the bots, but if I change to a newly created Domain I get this error

There is an error. Please report the message and full backtrace in the Piwik forums (please do a Search first as it might have been reported already!).

Notice: Undefined index: bot in /home/piwik/public_html/tmp/templates_c/%%EC^EC3^EC318862%%config.tpl.php on line 76

Backtrace -->
<a href='/0'>#0</a> Piwik_ErrorHandler(...) called at [/home/piwik/public_html/tmp/templates_c/%%EC^EC3^EC318862%%config.tpl.php:76]<a href='/1'>#1</a> include(...) called at [/home/piwik/public_html/libs/Smarty/Smarty.class.php:1263]<a href='/2'>#2</a> Smarty->fetch(...) called at [/home/piwik/public_html/core/View.php:138]<a href='/3'>#3</a> Piwik_View->render(...) called at [/home/piwik/public_html/plugins/BotTracker/Controller.php:83]<a href='/4'>#4</a> Piwik_BotTracker_Controller->config(...) called at [/home/piwik/public_html/plugins/BotTracker/Controller.php:95]<a href='/5'>#5</a> Piwik_BotTracker_Controller->config_reload(...) called at [:]<a href='/6'>#6</a> call_user_func_array(...) called at [/home/piwik/public_html/core/FrontController.php:136]<a href='/7'>#7</a> Piwik_FrontController->dispatch(...) called at [/home/piwik/public_html/index.php:53]
@Thomas--F commented on July 20th 2011 Contributor

Hi bronco,

I can reproduce the error and I'm trying to fix it. It seems to be a problem in the config-template.

@anonymous-piwik-user commented on July 20th 2011

thx for your absolutely fast reply!!

@anonymous-piwik-user commented on July 20th 2011

The error is gone,but now i can't see any standart bot entries. The bot table output for the added domains are empty. Maybe you have an idea?

@Thomas--F commented on July 20th 2011 Contributor

Hi bronco,
that's correct. There isn't (or at least I haven't found it) a hook, where I can insert the default-values after creating a new website.
Here is the hooks-list:
[http://piwik.org/docs/plugins/hooks/]

If you have access to PHPMyAdmin, I can give you a SQL-Statement to insert the missing rows. Otherwise you can only add all rows manual.

@anonymous-piwik-user commented on July 20th 2011

Yes, Thomas please give me the statement and I think you are right. I also didn't found any hook and this should really be solved b Piwik. Not only for your module I think this is a globaly disadvantage.Anyway your module is great and I love it!

@Thomas--F commented on July 20th 2011 Contributor

Hi bronco,
I'm glad you like the plugin.
Maybe I try to insert a button to "Reset DB to default" for a single website.

Here is the SQL-statement. Just replace the xxx by the site-id.

INSERT INTO `piwik_bot_db` 
(idsite,botName, botActive, botKeyword, botCount, botLastVisit)
VALUES (xxx,'MSN Search'              ,1,'MSNBOT'              ,0,'0000-00-00 00:00:00')
     , (xxx,'Bingbot'                 ,1,'bingbot'             ,0,'0000-00-00 00:00:00')
     , (xxx,'GoogleBot'               ,1,'GoogleBot'           ,0,'0000-00-00 00:00:00')
     , (xxx,'Google Instant'          ,1,'Google Web Preview'  ,0,'0000-00-00 00:00:00')
     , (xxx,'Media Partners GoogleBot',1,'Mediapartners-Google',0,'0000-00-00 00:00:00')
     , (xxx,'Baiduspider'             ,1,'BaiDuSpider'         ,0,'0000-00-00 00:00:00')
     , (xxx,'Ezooms'                  ,1,'Ezooms'              ,0,'0000-00-00 00:00:00')
     , (xxx,'YahooSeeker'             ,1,'YahooSeeker'         ,0,'0000-00-00 00:00:00')
     , (xxx,'Yahoo! Slurp'            ,1,'Yahoo! Slurp'        ,0,'0000-00-00 00:00:00')
     , (xxx,'Altavista 1'             ,1,'AltaVista'           ,0,'0000-00-00 00:00:00')
     , (xxx,'Altavista 2'             ,1,'AVSearch'            ,0,'0000-00-00 00:00:00')
     , (xxx,'Altavista 3'             ,1,'Mercator'            ,0,'0000-00-00 00:00:00')
     , (xxx,'Altavista 4'             ,1,'Scooter'             ,0,'0000-00-00 00:00:00')
     , (xxx,'Infoseek 1'              ,1,'InfoSeek'            ,0,'0000-00-00 00:00:00')
     , (xxx,'Infoseek 2'              ,1,'Ultraseek'           ,0,'0000-00-00 00:00:00')
     , (xxx,'Lycos'                   ,1,'Lycos'               ,0,'0000-00-00 00:00:00')
     , (xxx,'Wget'                    ,1,'Wget'                ,0,'0000-00-00 00:00:00')
     , (xxx,'Yandex'                  ,1,'YandexBot'           ,0,'0000-00-00 00:00:00')
     , (xxx,'Yandex ?'                ,1,'Java/1.4.1_04'       ,0,'0000-00-00 00:00:00')
     , (xxx,'SiteBot'                 ,1,'SiteBot'             ,0,'0000-00-00 00:00:00')
     , (xxx,'Exabot'                  ,1,'Exabot'              ,0,'0000-00-00 00:00:00')
@anonymous-piwik-user commented on July 20th 2011

What about prepare db for this domain button or do you want the standart bots pre install buton? I like the idea with the button.It's user friendly until piwik offers an hook for this task.

But anyway thx for your module and I will also make a post in the drupal piwik forum to extend their module to track also bots. I am absolutely sure that this is realy a missing feature in piwik and you fill it up.

@Thomas--F commented on July 20th 2011 Contributor

Now I added the "Insert Default-Bots"-Button that will insert the 21 default-Bots if they are not already in the DB (check for BotName and SiteID).
I also added a "delete"-Button to every row so you can simply delete a row with one click.
Report any errors here. Thank you.

@anonymous-piwik-user commented on July 21st 2011

Fat! Man this small feature ist great. I added a domain and everything works as expected! Thx again. Okay, next Idea how is it to add a Tracker code generator ?? I know endless wishes and I realy dont need it for me because I use your example in comment#40 but I think some people are lazy and they will get a little bit confused how to use your bot tracker. This is just an idea because of your example and is it really nessesary to use the tokenauth in the request? If someone needs the drupal snippet I am willing to share it here also.

Greetings Bronco

@Thomas--F commented on July 21st 2011 Contributor

I don't think it's possible to generate the Tracker code for the PHP Tracking API. It's too specific for every webpage. An example is a good start but you have to modify it depending on the values you use in your own page.
For the tokenauth:
On my webpage I saw most of the hits reported with the IP of my own webserver. So I fixed it by setting the IP explicit to the tracker. But the IP can only be set, if you set the tokenauth.
Try to delete the 2 rows

  • $piwikTracker->setTokenAuth = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
  • $piwikTracker->setIp( $_SERVER('REMOTE_ADDR'));
    Maybe you don't need it.
@Thomas--F commented on July 21st 2011 Contributor

And one additional note:
I haven't had much time to develop the plugin during the last weeks an it doesn't look good in the near future.
I will support the plugin - that out of the question. And I will fix every bug as fast as I can.
But the next big release (tracking of all hits in a second table and maybe implementing the tracker cache) has to wait. I'm sorry.

@anonymous-piwik-user commented on July 22nd 2011

Thank you for your explanation of the tokenauth and the ip.I had a similar problem this will solve it.

@anonymous-piwik-user commented on August 27th 2011

This plugin was throwing PHP Notice errors when my site was visited by a bot that did not send a user agent. Therefore, I modified line 127 of BotTracker.php as follows:

$ua = isset($_SERVER[? $_SERVER'HTTP_USER_AGENT') : '';

@Thomas--F commented on August 27th 2011 Contributor

Thanks for the tip. That's something I never expected....

@anonymous-piwik-user commented on September 18th 2012

Is this really the way it suppose to work? By the what could mean error fatal error access to undeclared static property ?

@Thomas--F commented on September 18th 2012 Contributor

Replying to klindonbamost1984:

Is this really the way it suppose to work? By the what could mean error fatal error access to undeclared static property ?

What do you mean by that? Identifying a bot by it's user agent?

When do you get the error message? Can you post the exact error-text?

@anonymous-piwik-user commented on November 5th 2012

Hi Thomas,

thank you for the Plugin! I just installed it and it seem to work.
But I also got some errors. I am using Piwik 1.9.

On the visitors->BotTracker Site I klicked on Row Evolution and got the following error message:
(*** = my edit)

Requested report BotTracker.getBotTrackerView for Website id=13 not found in the list of available reports.

Backtrace:
<a href='/0'>#0</a> ***************/piwik/plugins/API/API.php(1217): Piwik_API_API->getRowEvolutionMetaData('13', 'day', '2012-10-06,2012...', 'BotTracker', 'getBotTrackerVi...', false, false)
<a href='/1'>#1</a> ***************/piwik/plugins/API/API.php(1206): Piwik_API_API->getSingleRowEvolution('13', 'day', '2012-10-06,2012...', 'BotTracker', 'getBotTrackerVi...', '%20', false, false, false, true)
<a href='/2'>#2</a> [internal function]: Piwik_API_API->getRowEvolution('13', 'day', '2012-10-06,2012...', 'BotTracker', 'getBotTrackerVi...', '%20', false, false, false, false, true, true)
<a href='/3'>#3</a> ***************/piwik/core/API/Proxy.php(190): call_user_func_array(Array, Array)
<a href='/4'>#4</a> ***************/piwik/core/API/Request.php(128): Piwik_API_Proxy->call('Piwik_API_API', 'getRowEvolution', Array)
<a href='/5'>#5</a> ***************/piwik/plugins/CoreHome/DataTableRowAction/RowEvolution.php(160): Piwik_API_Request->process()
<a href='/6'>#6</a> ***************/piwik/plugins/CoreHome/DataTableRowAction/RowEvolution.php(97): Piwik_CoreHome_DataTableRowAction_RowEvolution->loadEvolutionReport()
<a href='/7'>#7</a> ***************/piwik/plugins/CoreHome/Controller.php(179): Piwik_CoreHome_DataTableRowAction_RowEvolution->__construct(13, Object(Piwik_Date), NULL)
<a href='/8'>#8</a> ***************/piwik/plugins/CoreHome/Controller.php(138): Piwik_CoreHome_Controller->makeRowEvolution(false)
<a href='/9'>#9</a> [internal function]: Piwik_CoreHome_Controller->getRowEvolutionPopover()
<a href='/10'>#10</a> ***************/piwik/core/FrontController.php(138): call_user_func_array(Array, Array)
<a href='/11'>#11</a> ***************/piwik/index.php(53): Piwik_FrontController->dispatch()
<a href='/12'>#12</a> {main}

I also get a second error on the configuratione site:

Notice: Undefined index: usingOldGeoIPPlugin in *************/piwik/tmp/templates_c/%%38^383^383EA889%%header.tpl.php on line 108 

Backtrace -->

 <a href='/0'>#0</a> Piwik_ErrorHandler(...) called at [*************/piwik/tmp/templates_c/%%38^383^383EA889%%header.tpl.php:108]
 <a href='/1'>#1</a> include(...) called at [*************/piwik/libs/Smarty/Smarty.class.php:1869]
 <a href='/2'>#2</a> Smarty->_smarty_include(...) called at [*************/piwik/tmp/templates_c/%%3A^3AF^3AF5ACE5%%config.tpl.php:8]
 <a href='/3'>#3</a> include(...) called at [*************/piwik/libs/Smarty/Smarty.class.php:1263]
 <a href='/4'>#4</a> Smarty->fetch(...) called at [*************/piwik/core/View.php:133]
 <a href='/5'>#5</a> Piwik_View->render(...) called at [*************/piwik/plugins/BotTracker/Controller.php:83]
 <a href='/6'>#6</a> Piwik_BotTracker_Controller->config(...) called at [:]
 <a href='/7'>#7</a> call_user_func_array(...) called at [*************/piwik/core/FrontController.php:138]
 <a href='/8'>#8</a> Piwik_FrontController->dispatch(...) called at [*************/piwik/index.php:53]

Could you give me a hint how to prevent these errors?

Greetings Felix

@Thomas--F commented on November 9th 2012 Contributor

Hi Felix,

first I apologize for my late answer. I was on a technical conference in Berlin this week and I was not able to search the error.

Now to the error-messages:
I followed your descrition and could reproduce the error on the row evolution. If you try this on the right side (turning the diagram into a table and click on row evolution) you will get another error "Parameter label not set."

But I couldn't get the other error on the configuration side. Do you still use the old GeoIP-Plugin? What exactly did you do to trigger the error?

I will try to fix the first part during the weekend and then I will further search for the other one.

@Thomas--F commented on November 10th 2012 Contributor

I had to disable the RowEvolution-Feature for the BotTracker-Plugin.

The purpose of this plugin was to hide the Bots from my statistics. The only thing that is tracked is the last visit-timestamp and the total hits by one bot (better: by a UserAgend wit this keyword).
So there is no data about "hits during a time-window" and no way to produce a "row evolution chart".
The easiest way was to disable the row evolution, so no one will get this error anymore.

@anonymous-piwik-user commented on November 12th 2012

Hi Thomas,

thank you for the update, now the row evolution feature is disabled.

I solved the second issue:
The second error occured when I opend the settings - BotTracker tab. Without doing any action.
But it seems, as if the error doesn't come from your Plugin.
This change worked for me:
[http://dev.piwik.org/trac/changeset/7300]

Just two more little notes:

  • the error bkonia mentioned (where you set the $ua userAgent) happend also in my case. Maybe you could set $ua to an empty string, when there is no USER_AGENT given.
  • The settings - BotTracker tab is not marked as active when you open it.

Greetings Felix

@Thomas--F commented on November 15th 2012 Contributor

Hi Felix,
thanks for the hint. I fixed both issues in version 0.27

Replying to felix:

Just two more little notes:

  • the error bkonia mentioned (where you set the $ua userAgent) happend also in my case. Maybe you could set $ua to an empty string, when there is no USER_AGENT given.
  • The settings - BotTracker tab is not marked as active when you open it.
@anonymous-piwik-user commented on January 13th 2013

Using Piwik API to track visits (by setting the parameter ua), BotTracker not record properly Bot.
I solved this way:

Original code in BotTracker.php file:

    public function checkBot($notification)
    {
        $ua = $_SERVER['HTTP_USER_AGENT'];
        if (empty($ua)){
            $ua = "      ";
        }

Modified code:

    public function checkBot($notification)
    {
        $ua = $_REQUEST['ua'];
        if (empty($ua)){
            $ua = $_SERVER['HTTP_USER_AGENT'];
        }
        if (empty($ua)){
            $ua = "      ";
        }

If it can be useful;)

@anonymous-piwik-user commented on January 13th 2013

Thanks for this great plugin. I've made a Hungarian translation as one of my clients required it. You can use the localization file as attached above if you wish.

@Thomas--F commented on January 13th 2013 Contributor

@fratac:
I testet your idea and it works fine. So I implemented it in v 0.28

@ciastko:
I put your language-file into the zip.

Thanks to both of you.

@anonymous-piwik-user commented on January 26th 2013

Hi,

BotTracker just recently installed under Piwik 1.10.1 and I get this error at the Setting Page:

There is an error. Please report the message (Piwik 1.10.1) and full backtrace in the Piwik forums (please do a Search first as it might have been reported already!).

Strict Notice: Non-static method Piwik_BotTracker_API::getAllBotDataForConfig() should not be called statically, assuming $this from incompatible context in /var/www/html/piwik-aws/plugins/BotTracker/Controller.php on line 73 

Backtrace -->

<a href='/0'>#0</a> Piwik_ErrorHandler(...) called at [/var/www/html/piwik-aws/plugins/BotTracker/Controller.php:73]
<a href='/1'>#1</a> Piwik_BotTracker_Controller->config(...) called at [:]
<a href='/2'>#2</a> call_user_func_array(...) called at [/var/www/html/piwik-aws/core/FrontController.php:138]
<a href='/3'>#3</a> Piwik_FrontController->dispatch(...) called at [/var/www/html/piwik-aws/index.php:53]

Help anyone?

Thanks

Juan

@Thomas--F commented on January 26th 2013 Contributor

Hi Juan,

I also use Piwik 1.10.1, but I cannot reproduce the error.

But you can try something:
Open the file /piwik/plugins/BotTracker/API.php
and modify the row 40:
static function getAllBotDataForConfig($idsite)

Please give us a feedback if the error is gone now.

@anonymous-piwik-user commented on January 26th 2013

Thomas,

With that change the error has disappeared. Perfect!

I'm using import_logs.py --enable-bots to import my logs files so I've deleted the old data and tried to import my logs again but the BotTracker statistics are still empty.
Giving a look to the httpd/error_log I've noticed this:

[Sat Jan 26 13:44:17 2013] [error] [client 127.0.0.1] PHP Notice:  Undefined index: idsite in /var/www/html/piwik-aws/plugins/BotTracker/BotTracker.php on line 132
[Sat Jan 26 13:44:17 2013] [error] [client 127.0.0.1] PHP Notice:  Undefined index: idsite in /var/www/html/piwik-aws/plugins/BotTracker/BotTracker.php on line 132
[Sat Jan 26 13:44:17 2013] [error] [client 127.0.0.1] PHP Notice:  Undefined index: idsite in /var/www/html/piwik-aws/plugins/BotTracker/BotTracker.php on line 132

Thanks

Juan

@Thomas--F commented on January 26th 2013 Contributor

You have to say the script, for which site you will import the logs.

Try this:
import_logs.py --enable-bots --idsite=1

@anonymous-piwik-user commented on January 26th 2013

Thomas,
I know. Sorry for not mentioning it before. This is the complete command I'm using:

python /var/www/html/piwik-aws/misc/log-analytics/import_logs.py --url=http://127.0.0.1/piwik-aws --idsite=1 --enable-http-errors --enable-http-redirects --enable-static --enable-bots --recorder-max-payload-size=600 --add-sites-new-hosts --show-progress --recorders=60 access.log
@Thomas--F commented on January 26th 2013 Contributor

Hi Juan,

I don't know how the log-import works. Normally, the site-ID comes as parameter, so I can get it by $_REQUEST.

You can try to set the ID hard in the source. If it works, you have to restore the old version after the import.

Here's the workaround:
File: /piwik/plugins/BotTracker/BotTracker.php

Line 132:
$idSite = 1;

@anonymous-piwik-user commented on March 5th 2013

Hi, sorry I'm not a developer so I've got as far and uploading the plugin!

I've looked for MySQL-Table piwik_bot_db and it's not there (not surprising), so I guess we have to create that ourselves but how do we populate it? Are we importing logs from elsewhere or are we manually typing UAs into the DB or something else?

And is there any other step to activate the plugin so it works in the piwik dash?

Sorry to be a pain, but I promise if you make a readme or something for mere mortals like myself then I'll make a video tut for installation!

@Thomas--F commented on March 5th 2013 Contributor

Installing a plugin in Piwik is very easy.
[http://piwik.org/faq/plugins/#faq_21]

Just unzip the file and put it in a subfolder piwik/plugins/BotTracker
Then Login as administrator and activate it in "Settings-> Plugins"

Done!

You don't have to create tables. Everything is done automatically by the piwik install routines.

@anonymous-piwik-user commented on March 6th 2013

Thank you for your help!

@anonymous-piwik-user commented on March 22nd 2013

I get the following error when going to the BorTracker settings page.

There is an error. Please report the message (Piwik 1.11.1) and full backtrace in the Piwik forums (please do a Search first as it might have been reported already!).

Notice: Undefined index: topMenu in /home/puncaro1/public_html/stats/tmp/templates_c/%%25^25A^25A8A371%%top_bar_top_menu.tpl.php on line 6

Backtrace -->
<a href='/0'>#0</a> Piwik_ErrorHandler(...) called at [/stats/tmp/templates_c/%%25^25A^25A8A371%%top_bar_top_menu.tpl.php:6]
...
<a href='/9'>#9</a> Piwik_View->render(...) called at [/stats/plugins/BotTracker/Controller.php:86]
<a href='/10'>#10</a> Piwik_BotTracker_Controller->config(...) called at [:]

You need to add the following line in Controller.php, after line 85, to fix it:

$view->topMenu = Piwik_GetTopMenu();

Thanks,
Alex

@Thomas--F commented on March 22nd 2013 Contributor

Hi Alex,

thanks for your help. I fixed and tested your suggestion.

@anonymous-piwik-user commented on May 3rd 2013

Please help me. What code needs to be inserted for PHP that the plug-in considered all search robots and displayed on what page they were.

@anonymous-piwik-user commented on May 3rd 2013

when using JS doesn't consider robot Yandex, and it isn't visible on what of pages google bot came. It was visible just it.

@anonymous-piwik-user commented on May 3rd 2013

There is an error. Please report the message (Piwik 1.11.1) and full backtrace in the Piwik forums (please do a Search first as it might have been reported already!).

Strict Notice: Non-static method Piwik_BotTracker_API::insertBot() should not be called statically, assuming $this from incompatible context in /home////public_html/piwik/plugins/BotTracker/Controller.php on line 154

Backtrace -->

#0 Piwik_ErrorHandler(...) called at [/home////public_html/piwik/plugins/BotTracker/Controller.php:154]
#1 Piwik_BotTracker_Controller->saveConfig(...) called at [:]
#2 call_user_func_array(...) called at [/home////public_html/piwik/core/FrontController.php:136]
#3 Piwik_FrontController->dispatch(...) called at [/home////public_html/piwik/index.php:52]

@Thomas--F commented on May 3rd 2013 Contributor

Hi solnce,

first of all:
BotTracker is not designed to track pages, it only tracks the total amount of hits and the last timestamp, when a special bot cam along. It also avoids the bot-hits in the normal results.
The description of the PHP-Api can be found here:
http://piwik.org/docs/tracking-api/
Section: Piwik Tracking API (Advanced users)

Now to the Error:
What did you do when the error occurs?
Can you reproduce it?
What PHP-Version do you use?

@anonymous-piwik-user commented on May 4th 2013

Hi ThomasF! This mistake occurs when I add a new line in the list of search bots. by pressing the button to keep. php version 5.4. this error each time at preservation. And thus the new line is added

@Thomas--F commented on May 4th 2013 Contributor

Hi solnce,

I cannot reproduce the error. Maybe some PHP-settings....

Try this:

Open File plugins/BotTracker/API.php
change line 92, 102 and 111 from

function ....

to
static function ....

Then try again to insert a bot.

@anonymous-piwik-user commented on May 5th 2013

Hi ThomasF!
it helped. now works without a mistake. thanks! :-)

@anonymous-piwik-user commented on August 3rd 2013

Hi,

I'm a little problem on : /index.php?module=BotTracker&action=config&idSite=1&period=day&date=yesterday

There is an error. Please report the message (Piwik 1.12) and full backtrace in the Piwik forums (please do a Search first as it might have been reported already!).

Notice: Undefined index: phpVersion in /web/www/piwik/tmp/templates_c/%%38^383^383EA889%%header.tpl.php on line 127

Backtrace -->

<a href='/0'>#0</a> Piwik_ErrorHandler(...) called at [/web/www/piwik/tmp/templates_c/%%38^383^383EA889%%header.tpl.php:127]
<a href='/1'>#1</a> include(...) called at [/web/www/piwik/libs/Smarty/Smarty.class.php:1869]
<a href='/2'>#2</a> Smarty->_smarty_include(...) called at [/web/www/piwik/tmp/templates_c/%%5E^5EC^5EC5A660%%config.tpl.php:8]
<a href='/3'>#3</a> include(...) called at [/web/www/piwik/libs/Smarty/Smarty.class.php:1263]
<a href='/4'>#4</a> Smarty->fetch(...) called at [/web/www/piwik/core/View.php:122]
<a href='/5'>#5</a> Piwik_View->render(...) called at [/web/www/piwik/plugins/BotTracker/Controller.php:88]
<a href='/6'>#6</a> Piwik_BotTracker_Controller->config(...) called at [:]
<a href='/7'>#7</a> call_user_func_array(...) called at [/web/www/piwik/core/FrontController.php:125]
<a href='/8'>#8</a> Piwik_FrontController->dispatch(...) called at [/web/www/piwik/index.php:47]
@Thomas--F commented on August 3rd 2013 Contributor

Hi Zefling,

I found the error and fixed it. Just install the new version 0.30

@anonymous-piwik-user commented on September 17th 2013

Just tried your plugin v .30 and have hit a error.

There is an error. Please report the message (Piwik 1.12) and full backtrace in the Piwik forums (please do a Search first as it might have been reported already!).

Strict Notice: Non-static method Piwik_BotTracker_API::getAllBotDataForConfig() should not be called statically, assuming $this from incompatible context in /srv/http/MySite.tld/plugins/BotTracker/Controller.php on line 73

Backtrace -->

<a href='/0'>#0</a> Piwik_ErrorHandler(...) called at [/srv/http/MySite.tld/plugins/BotTracker/Controller.php:73]
<a href='/1'>#1</a> Piwik_BotTracker_Controller->config(...) called at [:]
<a href='/2'>#2</a> call_user_func_array(...) called at [/srv/http/MySite.tld/core/FrontController.php:125]
<a href='/3'>#3</a> Piwik_FrontController->dispatch(...) called at [/srv/http/MySite.tld/index.php:47]

I am running Arch Linux and PHP version

PHP 5.5.3 (cli) (built: Aug 23 2013 08:41:45) 
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2013 Zend Technologies

Also why is Exabot listed three times with bot IDs of 21, 295, 310. Also why does the numbering go from 21 to 295 in the default setup?

@Thomas--F commented on September 19th 2013 Contributor

Hi vwyoda,

I fixed the static calls. Try version 0.31

How many sites do you have in your Piwik-installation? You can insert the default-bots for every webpage and there is no check if you input a bot 2 times for one page.
But you can delete all entries in the table and use the "insert default bots" to make a clean restart.

@anonymous-piwik-user commented on September 19th 2013

Thanks ThomasF I will give it a shot and post back the results.

I have 30 something in the install that I tried the plugin on. I did not input anything it was the default setup as it came once I clicked activate. I noticed there were duplicate bots listed and the numbering was off. I will try and get you a screen shot to show. I have never previously used your plugin so just seemed weird to see that.

@anonymous-piwik-user commented on September 20th 2013

The issue seems to have been solved.

I am still seeing once the plugin is installed that it is with the defaults and not inserting anything to any site yet, that Exabot shows three times. I attached a picture to show. Should I just delete the extras, or is there some reason that there is three on the default?

Thanks for the hard work on this!
[http://tinypic.com/r/w720bm/5]

@Thomas--F commented on September 20th 2013 Contributor

The Exabot-problem confuses me.

All bots are inserted with the current site-id.
During first install, all 21 lines were inserted once per site-id (file "BotTreakcer.php" line 63-89), in the configuration all bots are inserted for the selected site-id (file "Controller.php" line 200-231).
The bot-ID is generated by MySQL because the column is AUTO_INCREMENT.

If you still get duplicates just delete them. Only one row will be affected when the bot visits your page.

@anonymous-piwik-user commented on October 6th 2013

here the spanish translation

<?php
/**
 * Piwik - Open source web analytics
 *
 * <a class='mention' href='https://github.com/link'>@link</a> http://piwik.org
 * <a class='mention' href='https://github.com/license'>@license</a> http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
 *
 * <a class='mention' href='https://github.com/category'>@category</a> Piwik_Plugins
 * <a class='mention' href='https://github.com/package'>@package</a> Piwik_BotTracker
 */

$translations = array(
    'BotTrackerPlugin_PluginDescription' => 'Detecta bots & spiders y las contabiliza sin agregarlas al contador de visitas.',
    'BotTracker_Widgets' => 'BotTracker',
    'BotTracker_DisplayWidget' => 'Mostar BotTracker',
    'BotName' => 'Bot',
    'BotCount' => 'Visitas',
    'BotKeyword' => 'Keyword',
    'BotLastVisit' => 'Visto',
    'BotActive' => 'Activo',
    'BotTracker_Config' => 'Configuraci&oacute;n del BotTracker',
    'save_changes' => 'Guardar',
    'true' => 'S&iacute;',
    'false' => 'No',
    'Error_empty' => ' esta vacio!',
    'Message_deleted' => ' fu&eacute; eliminado!',
    'Message_bot_inserted' => ' Bots a&ntilde;adidos',
    'insert_db' => 'Insertar Bots por defecto',
    'delete' => 'eliminar'
);

I have problems with the import_logs.py script. It show this error:
Notice: Undefined index: idsite in /var/www/piwik/plugins/BotTracker/BotTracker.php on line 132
GIF89a!,D;

it's a bug?

@Thomas--F commented on October 6th 2013 Contributor

Hi platix,

thanks for your translation. I added the file so the zip-archive.

There is a workaround for the problem with the import-script. Just look at comment 74.

@hpvd commented on October 26th 2013

Hi Thomas

just updated my testing environment to Piwik 2.0b2.
Your great BotTracker Plugin was deactivated because it seems not to be compatible to this new upcoming piwik version.
An updated of your plugin to the latest version doesn't solve it...

please see attachement

@Thomas--F commented on October 26th 2013 Contributor

Hi hpvd,

I installed Piwik 2.0b2 to my local XAMPP and there seems to be a lot of changes for the plugins. Sadly there is no documentation about a plugin-migration.
Until end of the year I have not much time to spend for this plugin, so I won't be able to migrate it to Piwik 2.0 soon. If there's someone who wants to migrate it - go ahead and try it, I will be thankful. Otherwise I will try it next year.

@hpvd commented on November 13th 2013

I've opened a ticket for doc for Plugin migration #4246

result:

"We will not write documentation to migrate from 1.X to 2.0

Instead we will provide new documentation that will explain the new 2.0 way of building plugins (since a lot has changed)

stay tuned on http://developer.piwik.org/ and already fixed #4266 "

@Thomas--F commented on November 13th 2013 Contributor

Hi hpvd,

thanks for the infos.
I already made some progress, but it's hard to do it just with the more or less working samples. I think I'm on 30-40%, but I haven't much time to do it....

@anonymous-piwik-user commented on December 12th 2013

Hi!
I'm using this plugin with Piwik 1.12 and import_logs.py (called from the command line), but its statistics keep staying empty... If I uncomment the LogToFile function I just get a lot of

"SiteID:1 user Agent: Python-urllib/2.7"

Am I missing something?
Thanks in advance!

@Thomas--F commented on December 12th 2013 Contributor

Replying to joril:

Hi!
I'm using this plugin with Piwik 1.12 and import_logs.py ...

"SiteID:1 user Agent: Python-urllib/2.7"

I think the plugin doesn't work with the Python-Import-Script.
The plugin detects Bots on the base of the used user agent. It seems the Python-script send's always "Python-urllib/2.7" as user agent, so the plugin cannot detect any bot.

@hpvd commented on December 27th 2013

just got the piece of information that excluding bots from statistic is a core function of piwik:
#4441

wasn't this one of the great functions of this plugin?
How does these things work together?
Who excludes what?
can I be sure the statistics are correct?

@hpvd commented on December 27th 2013

btw: hopefully there will be a piwik 2.0 compatible version of this great always installed plugin?

@Thomas--F commented on December 27th 2013 Contributor

Hi hpvd,
this excluding was already in the old piwik. But there were only a few bots, it was not easy to add new bots and after every piwik update, all added bots were lost again.
This was the poin where I startet do write this plugin.

If the chronological order is not changed in piwik 2 the plugin works fine.
But I will test it, when the plugin is rewritten for piwik 2. I'm still working on it, but it's not done yet.

@hpvd commented on December 27th 2013

@ThomasF
Many thanks for your explanation and all the work with this plugin!

@Thomas--F commented on December 29th 2013 Contributor

There are good and bad news:

I spend about 6 hours in the last two days to rewrite the plugin to piwik 2.0.
I have finished the whole config so far, so I'm able to add the default bots and update, insert or delete a single entry.

But while I'm looking for the hook-entry fr the exclude-part I found the following entry in the website-settings:
"Global list of user agents to exclude"

This funktion replaced this plugin nearly entirely. The only part that isn't availble are the stats. If you can live without it, there's no reason to install my little plugin anymore.

Before I will go on and work some more hours in this code, I have to ask you, the users:
Do you need the plugin for Piwik 2 or can I burn the code and move on to other projects?

@hpvd commented on December 29th 2013

Hi ThomasF

ooh -just a short answer (ok not that short..): excluding bots from the stats is only one part of the business - the stats if, when and which bots are visiting your sites are more than interesting for serious webmarketing of your site!

hmm don't know what piwik core programmers are planning regarding this topic

-maybe one should think of integrating your code directly into piwik? Not only for the stat but also for good usability?

-on the other hand there is plenty of room for extension the functions of your plugin futher:
one idea for interesting additional stats would be not only show the last visits of robots but the development of the visits of each robot over time - and in comparison to each other.
The more often they deeply crawl your site the better your work in fields of seo/ advertising always new great content...
(- because before they can give you a good rank or an even better rank than before they ALWAYS have to know what your site is advertising for the users and what has changed so crawling is a good early indicator..)
Maybe one could treat robots some how as fake visitors, to use the available stats of the visitor log/ profile?

=> only having advanced functions like that one could notice on time if something went wrong or everything is on the track...

futher thinking this could be coupled with upcoming custom email alerts, mover and shaker feature and many more

@Thomas--F commented on December 30th 2013 Contributor

OK, today I had another 3-hour-session with my plugin and I'm nearly finished.

The UI works as the old version (Widget, BotTracker-View and Config) and ss far as my tests are going all User-Agents are detected.

I will do some more tests but if anyone wants to be a beta-tester, I will attach the new plugin.
This version only runs on Piwik2 so don't try it on 1.x!

It starts with version 0.40.

@hpvd commented on December 31st 2013

Many thanks for your work!
just tried to install it by the following steps:

the old version 0.32 was installed since piwik 2.0 was not a fresh install but an upgrade from piwik 1.12
Piwik 2.0 has deactivated this old version of the plugin since it was not compatible and give the incompatible message.
I used the uninstall option within the new marketplace for this version
After that I copied the new version to the plugin folder of piwik
This new version was now shown in the pluginmanager and one could click "activate"
After this the following error message appearers:

"There was a problem installing the plugin BotTracker: SQLSTATE[Base table or view already exists: 1050 Table 'piwik_bot_db' already exists If this plugin has already been installed, and if you want to hide this message</b>, you must add the following line under the PluginsInstalled entry in your config/config.ini.php file: PluginsInstalled[] = BotTracker"
please see attachment
using piwik 2.03b3, php latest from 5.3

@hpvd commented on December 31st 2013

simply using the "go to piwik" link below this message, you could see in the plugin manager that this plugin is still "Inactive"

@Thomas--F commented on December 31st 2013 Contributor

Hi hpvd,

I modified the install-routine. First I try go create the new table. If it exists, the installer ends. Otherwise it inserts the default bots for all websites.

@hpvd commented on December 31st 2013

wow that was fast. Now doing the same again I got the following message:
"Well done! You have successfully activated BotTracker."
:-)

@hpvd commented on December 31st 2013

looks great so far!

at this time its installed in a testing environment so I can not tell if it works well. Will be upgrade piwik on the other site to 2.0 as soon as the bugfix 2.03 is available...

Just two comments on the very first view:

  • would be nice to see the number of available default bots e.g. simply within the button "Add default bots (xx)" or if you want not only to show the number but also which ones e.g. with a (i) with hover beside the button
  • would be nice to see how many bots are set up for one site since the IDs of the bots do not start with 1 if you have a multi site piwik setup (please see attachment)
@hpvd commented on December 31st 2013

=> do you make your great plugin available within the market?

@hpvd commented on January 2nd 2014

maybe this ticket regarding periodic bots would also fit within this plugin
#3556

@Thomas--F commented on January 6th 2014 Contributor

I've implemented you suggestions from comment 113:

There is a number behind the Insert-Bot-Button and above the table is a short information about the number of defines bots for the current site.

Currently I'm experimenting with an additional table with "extended stats" where every bot hit is stored. I'm not sure what to put in that table and how to manage the data, but let's do this step for step.
I think, the most interesting informations about a bot visit is Who/Where/When so the new table will store the BotID, timestamp and the visited page, maybe the complete useragent, so you can see if your administration was correct.

Any suggestions to this plan?

@hpvd commented on January 6th 2014

Hi ThomasF

many thanks for your continuous work!

here is a small collection of ideas what would be so interesting about the bot visits:

1) Details of the last visit:

when was the last visit?
how many sites were visited?
which sites were visited?

2) Details of the visits within the last month:

how often did the bot visite the website e.g. within the last month?
how many sites were visited within the last month?
how many different sites were visited within the last month?
which sites were visited?

3) how does the points above compare to the month before?
or to the avarage of the last 12 month?
how does these counts develop over time?

4) overview sites:

which sites were visit most often
-from a special bot and
-from all bots?

and the other way around:
which sites were visit most rarest?

5) overview bots:

which was the most often visiting bot?
which was the most different sites crawling bot?
and the other way around...

@hpvd commented on January 6th 2014

what would be also interesting is

6) a number in % of how many of the visitors
-of a site
-a part (branch) of a site
-the complete website
are bots.

On some sites/branches of a website this amount is not that small and causes a considerable portion of the traffic.

Beside to determine the added server load,
one small example where this may become even more interesting:
if you are using certain services for onsite search you have an extra server or external service where you have to pay per visit.
so if you know 80% of your searching visitors are bots you could decide to block this part of your website for bots - or only for one special e.g. chinese bot which may causes 40% of the search traffic but from whom you will never ever benefit from...

@Thomas--F commented on January 7th 2014 Contributor

Thanks for your suggestions.

I've added now the extra-statistics. You can turn them on for every single bot on every site. But there is currently no widget or page, that shows the extra data. It's stores in a table called _piwik_bot_dbstat.
But most of your reports can created by a single SQL-statement. I will try to add some more info-screens but first I have to test this new function.
The "new" BotTracker for Piwik2 is still BETA so please test it if you can but don't try it in production....

@Thomas--F commented on January 7th 2014 Contributor

BTW:
I'm planing to move the source to Github and maybe I put it into the Piwik-Marketplace.

But I'm not very talented in languages. I can only maintain the german and english language-files.
So are there any volunteers for the ES and HU-file? (or any other language)

@anonymous-piwik-user commented on January 8th 2014

Hi ThomasF, thanks for your great work!
If you move your project to GitHub I'd Gladly help you.
I can help to translate it into Italian language and for the development of the plugin

@Thomas--F commented on January 8th 2014 Contributor

I uploaded the current version (0.43) to a Github-repository:
https://github.com/Thomas--F/BotTracker

@hpvd commented on January 14th 2014

since piwik core adds more and more bots on its own "do not track"-list,
a feature for handling this is needed.
One idea would be to read this list, copy it, clear it and put the bots from there to the bot tracker plugin list.

Or are these bots from the piwik core do not tracklist already automatically manageable with this plugin (because there is only one list used by core and by this extension)?

@Thomas--F commented on January 14th 2014 Contributor

As far as I tested, the BotTracker-plugin is called before the core-"do not track"-list.
So it doesn't matter, how many entries they add to the core-list.

The only effect is:
If you didn't add the bot to the plugin but it is on the core-list, you didn't see the visit at all.

@hpvd commented on January 14th 2014

many thanks for clarifying this!!

maybe it make sense use the effort of the core developers by reusing their list within this plugin as a second list for setting up default bots from - what do you think?

one way to implement this would be a second button:
"ad bots from piwik standard bot ignore list for tracking"
ok it will be a very wide button ;-)

the already available button for setting defaults may be renamed to
"ad default bots from plugin internal list"
ok it's very long again...

@hpvd commented on February 1st 2014

maybe this is interesting?
#1877

@Thomas--F commented on February 1st 2014 Contributor

Interesting, but nothig for this plugin.

The new hook is for detecting a visitor, that comes via search engine to your site. It has nothing to do with bots and crawlers.

@mattab commented on February 3rd 2014 Member

@Thomas, it would be great if you could maybe publish your plugin on the Piwik Marketplace, when you get a chance. Instructions can be found at: http://developer.piwik.org/guides/distributing-your-plugin#adding-your-plugin-to-the-marketplace

If the plugin BotTracker is working well, then it would be a nice thing to put it on the marketplace as many more users would see it and use it :)

@mattab commented on February 6th 2014 Member

Along with the Piwik 2.0 release and new design for Piwik, we have also launched the official Plugins Marketplace to let any developer share their work to the thousands of Piwik users worldwide.

Maybe you'd like to publish your plugin there?

In any case, keep up the good work and we hope you enjoy Piwik 2!

@anonymous-piwik-user commented on February 19th 2014

Hello,
I'm relatively new to Piwik, and having trouble installing this plugin. I posted for help in the General Question board, but was told to ask for help here instead.

Apparently I've at least partially installed it correctly, because it does show up in my list of plugins, and I've activated it. But no new widgets appear or anything new or about search engine bots/spiders. This ticket does say some bots have been preconfigured, but I certainly can't find any stats. And I do have plenty of search engine visits (according to my SMF data).

I don't understand the 3rd step for installing:
"Configure the Bot-List by editing the MySQL-Table piwik_bot_db."

I cannot find any such table in the piwik db (using phpMyAdmin, which is part of my server). Am I supposed to create the table? If so, can you please tell me how to do that?

Or otherwise, how can I make this plugin work?

Thank you very much :-)

Edit
PS - This is on version 2.02

@Thomas--F commented on February 20th 2014 Contributor

Hi brynn,

first some general infos:
You can find and download the newest version for this plugin on GitHub:
https://github.com/Thomas--F/BotTracker

But since version 0.43 on this page there were only some additions on the language-files.

I plan to migrate the plugin to the Piwik-Marketplace, so it will be easier to find and install.

Now to your problem:
The part "Configure the Bot-List by editing the MySQL-Table piwik_bot_db" is no longer needet, because there is an configuration-panel in the Piwik-Settings to do this.
But if you cannot see the tables piwik_bot_db and piwik_bot_db_stat, the install-routine hasn't run properly.
Did Piwik shows some error-messages during the activation of the plugin?

Try to deactivate an deinstall the plugin and then make a clean new install.

For the Updates:
I just updated my less-priotriy-piwik (only with some private web-pages) from 1.12 to 2.03.
Piwik didn't run the 0.43-Update-Script so I had to do this manual:

ALTER TABLE `piwik_bot_db'`
ADD `extra_stats` BOOLEAN NOT NULL DEFAULT FALSE
CREATE TABLE IF NOT EXISTS `piwik_bot_db_stat`
(
 `botId` INTEGER(10) UNSIGNED NOT NULL,
 `idsite` INTEGER(10) UNSIGNED NOT NULL,
 `page` VARCHAR(100) NOT NULL,
 `visit_timestamp` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
 `useragent` VARCHAR(100) NOT NULL,

 PRIMARY KEY(`botId`)
)  DEFAULT CHARSET=utf8

But then everything works fine.

@anonymous-piwik-user commented on February 20th 2014

Ok yes, that's what I downloaded. I did not see any error messages.

I deactivated, uninstalled, downloaded fresh, reinstalled, and reactivated. I did get a success message ("Well done! You have successfully activated BotTracker-master.")

But I still don't see any piwik_bot_db and piwik_bot_db_stat tables through PMA.

Unfortunately, I don't understand what you mean by running 0.43-Update-Script. Would you please tell me how to do that?

Thanks for your help!

@Thomas--F commented on February 20th 2014 Contributor

The update-script is only important for those, who migrate Piwik from 1.x to 2.x
It's in the subfolder "Updates"

But I think, I found your problem.
Remove the Plugin (deactivate and deinstall) and then rename the folder from "BotTracker-master" to "BotTracker" and try it again.

GitHub names the download-file "BotTracker-master" because it's the master-branch of the source. The plugin is names BotTracker and the folder must have exact the same name.

@anonymous-piwik-user commented on February 21st 2014

Oh yeah, I noticed that "master" part. At first I thought I had downloaded the wrong file.

Anyway, I renamed it like you said, and it worked! The new widget is all up and running now.

Thank you very much :-)

@mattab commented on February 21st 2014 Member

a quick note; if you "Upload the plugin via ZIP" (you can find this link in Settings > Marketplace > Upload a ZIP plugin") then the directory would have been put with the right name automatically.

@anonymous-piwik-user commented on February 21st 2014

Oh ok, that's good to know.

I have another question now. 2 or 3 spiders have visited while I've been watching the past 30-45 minutes, but they don't show up in the widget. Such as google and baidu, which I guess are both pre-configured. Does this plugin only show the bots/spiders under certain circumstances?

@anonymous-piwik-user commented on February 21st 2014

Yeah, over a 2 hour period, Google visited twice and Baidu 3 times. But the new BotTracker widget (installed 2 hours before the 1st search engine spider visit) doesn't show any visits.

Maybe I'm misunderstanding something?

There's also another discrepancy, which you probably can't help with. But maybe worth a mention, in case I'm misunderstanding, or it represents a larger problem.

SMF could be reporting 8 guests, 1 user, 2 spiders. But the Real Time Visitor Count stays at zero. And I realize that SMF reports "in the last 15 minutes" and Piwik Real Time reports every 3 minutes. But I refreshed SMF to get it's latest, and then immediately switched to Piwik tab and refreshed it. So I would think that's pretty close -- enough to catch at least a couple of visitors.

I went through that routine several times over the last 4 hours, but the SMF visitors is always higher than Piwik. And Piwik most of the time is on zero, while SMF almost always is reporting visitors. So maybe I just don't understand something?

Thanks again :-)

@Thomas--F commented on February 21st 2014 Contributor

Woohoo - this ticket is definitely growing too big g
Maybe I should use some forum.

Hi Brynn,
how do you integrate Piwik in your website? Do you use the PHP-API or the image/javascript-tag?

Many webcrawler, spider and bots don't load the images in a page and most of them don't execute JacaScript. So you cannot track them if you don't use the PHP-API.

@Thomas--F commented on February 21st 2014 Contributor

From now on, the current version can be downloaded in

GitHub
https://github.com/Thomas--F/BotTracker

or the
Piwik Marketplace
http://plugins.piwik.org/BotTracker

@anonymous-piwik-user commented on February 21st 2014

Oh, wait a minute....either more controls have become available, or I did not see them initially.... Now there's an update, that I'm pretty sure wasn't there before. And also, there's Settings > BotTracker Configuration, which I don't think was there before, but maybe I just missed it. However, it appears to be already set up anyway.

I use both js and image tracking. (Although that seems to essentially double my visit count, so I may think about changing that. However, I'm not even clear what php-api is. I guess I can find something in the user manual about it. Although it will probably be too complicated for me to figure out. But I'll read up on it anyway.

Maybe it would be a good idea to include a note with the BotTracker plugin, that it doesn't work with js/image tracking? Or I don't know, maybe most people who use Piwik would already know that much. But like I said, I don't even know what api-php is yet.

Ok, well thanks for your help anyway. If I can figure out the api-php, and it's something I can use, maybe I will. Otherwise, I guess I just won't have info about the spiders from Piwik.

@hpvd commented on February 21st 2014

update function has some problems at the moment
see #4703

@hpvd commented on April 14th 2014

there seem to be a problem wit bottracker and piwik 2.2rc1
please see #4988

@anonymous-piwik-user commented on July 2nd 2014

Just install latest version of BotTracker v.0.49 from 22. Jun. 2014.
During installation this plug-in (or something in PIWIK) messed up my "config.ini.php". It modified "password " field in "[database]" section. I always keep backup of important file so restore was quick. I suggest that "config.ini.php" is treated as "global.ini.php" and backup is performed before any plug-in installation.

  • centos 6.5
  • piwik 2.4.0
  • php 5.4.1.7
@Thomas--F commented on July 3rd 2014 Contributor

This looks weird. I don't think the plugin is responible for this unwanted update.
The only thing, PIWIK should update in the config.ini is putting BotTracker into the parts [plugins_installed and [blugins_tracker].
These changes are made by PIWIK-core-functions.

As you suggest, there could be a problem in the backup-routine durng install or update.

@mattab commented on July 4th 2014 Member

Plugin is not responsible for this bug, see: #5409

@dandv commented on May 22nd 2015 Contributor

Wanted to report a typo in the description of this plugin:

Landed here from the list of plugins in Piwik:

image

  1. Should that link be updated?
  2. Is there a better place to report this issue?
@sgiehl commented on May 22nd 2015 Member

@dandv see https://github.com/Thomas--F/BotTracker
The link is defined in the plugin.json of the plugin. So it would need to be update there.

@dandv commented on May 22nd 2015 Contributor

Thanks @sgiehl. Created PRs there.

This Issue was closed on July 4th 2014
Powered by GitHub Issue Mirror