@sgiehl opened this Pull Request on January 7th 2022 Member

Description:

In the past the CronArchiving tests sometimes failed with the error:

Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2019-12-10&format=json&segment=actions%3E%3D2&trigger=archivephp. Response was '{"result":"error","message":"SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'archive_numeric_2019_12' for key 'PRIMARY' 
<a href='/0'>#0</a> \/home\/travis\/build\/matomo-org\/matomo\/libs\/Zend\/Db\/Statement.php(300): Zend_Db_Statement_Pdo-&gt;_execute(Array) 
<a href='/1'>#1</a> \/home\/travis\/build\/matomo-org\/matomo\/libs\/Zend\/Db\/Adapter\/Abstract.php(479): Zend_Db_Statement-&gt;execute(Array) 
<a href='/2'>#2</a> \/home\/travis\/build\/matomo-org\/matomo\/libs\/Zend\/Db\/Adapter\/Pdo\/Abstract.php(238): Zend_Db_Adapter_Abstract-&gt;query('INSERT INTO `se...', Array) 
<a href='/3'>#3</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/Db\/Adapter\/Pdo\/Mysql.php(310): Zend_Db_Adapter_Pdo_Abstract-&gt;query('INSERT INTO `se...', Array) 
<a href='/4'>#4</a> \/home\/travis\/build\/matomo-org\/matomo\/libs\/Zend\/Db\/Adapter\/Abstract.php(575): Piwik\\Db\\Adapter\\Pdo\\Mysql-&gt;query('INSERT INTO `se...', Array) 
<a href='/5'>#5</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/Sequence.php(67): Zend_Db_Adapter_Abstract-&gt;insert('sequence', Array) 
<a href='/6'>#6</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/DataAccess\/Model.php(515): Piwik\\Sequence-&gt;create() 
<a href='/7'>#7</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/DataAccess\/ArchiveWriter.php(195): Piwik\\DataAccess\\Model-&gt;allocateNewArchiveId('archive_numeric...') 
<a href='/8'>#8</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/DataAccess\/ArchiveWriter.php(157): Piwik\\DataAccess\\ArchiveWriter-&gt;allocateNewArchiveId() 
<a href='/9'>#9</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(68): Piwik\\DataAccess\\ArchiveWriter-&gt;initNewArchive() 
<a href='/10'>#10</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(273): Piwik\\ArchiveProcessor\\PluginsArchiver-&gt;__construct(Object(Piwik\\ArchiveProcessor\\Parameters)) 
<a href='/11'>#11</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(176): Piwik\\ArchiveProcessor\\Loader-&gt;prepareAllPluginsArchive(false, false) 
<a href='/12'>#12</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(159): Piwik\\ArchiveProcessor\\Loader-&gt;insertArchiveData(false, false) 
<a href='/13'>#13</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(99): Piwik\\ArchiveProcessor\\Loader-&gt;prepareArchiveImpl(false) 
<a href='/14'>#14</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/Context.php(75): Piwik\\ArchiveProcessor\\Loader-&gt;Piwik\\ArchiveProcessor\\{closure}() 
<a href='/15'>#15</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(103): Piwik\\Context::changeIdSite(1, Object(Closure)) 
<a href='/16'>#16</a> \/home\/travis\/build\/matomo-org\/matomo\/plugins\/CoreAdminHome\/API.php(278): Piwik\\ArchiveProcessor\\Loader-&gt;prepareArchive(false) 
<a href='/17'>#17</a> [internal function]: Piwik\\Plugins\\CoreAdminHome\\API-&gt;archiveReports('1', Object(Piwik\\Period\\Day), '2019-12-10', 'actions%3E%3D2', false, false) 
<a href='/18'>#18</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/API\/Proxy.php(244): call_user_func_array(Array, Array) 
<a href='/19'>#19</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/Context.php(28): Piwik\\API\\Proxy-&gt;Piwik\\API\\{closure}() 
<a href='/20'>#20</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/API\/Proxy.php(335): Piwik\\Context::executeWithQueryParameters(Array, Object(Closure)) 
<a href='/21'>#21</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/API\/Request.php(266): Piwik\\API\\Proxy-&gt;call('\\\\Piwik\\\\Plugins\\\\...', 'archiveReports', Array) 
<a href='/22'>#22</a> \/home\/travis\/build\/matomo-org\/matomo\/plugins\/API\/Controller.php(46): Piwik\\API\\Request-&gt;process() 
<a href='/23'>#23</a> [internal function]: Piwik\\Plugins\\API\\Controller-&gt;index() 
<a href='/24'>#24</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/FrontController.php(619): call_user_func_array(Array, Array) 
<a href='/25'>#25</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/FrontController.php(168): Piwik\\FrontController-&gt;doDispatch('API', false, Array) 
<a href='/26'>#26</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/dispatch.php(32): Piwik\\FrontController-&gt;dispatch() 
<a href='/27'>#27</a> \/home\/travis\/build\/matomo-org\/matomo\/index.php(25): require_once('\/home\/travis\/bu...') 
<a href='/28'>#28</a> \/home\/travis\/build\/matomo-org\/matomo\/tests\/PHPUnit\/proxy\/index.php(15): include('\/home\/travis\/bu...') 
<a href='/29'>#29</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/CliMulti\/RequestCommand.php(79): require_once('\/home\/travis\/bu...') 
<a href='/30'>#30</a> \/home\/travis\/build\/matomo-org\/matomo\/vendor\/symfony\/console\/Symfony\/Component\/Console\/Command\/Command.php(257): Piwik\\CliMulti\\RequestCommand-&gt;execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) 
<a href='/31'>#31</a> \/home\/travis\/build\/matomo-org\/matomo\/vendor\/symfony\/console\/Symfony\/Component\/Console\/Application.php(874): Symfony\\Component\\Console\\Command\\Command-&gt;run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) 
<a href='/32'>#32</a> \/home\/travis\/build\/matomo-org\/matomo\/vendor\/symfony\/console\/Symfony\/Component\/Console\/Application.php(195): Symfony\\Component\\Console\\Application-&gt;doRunCommand(Object(Piwik\\CliMulti\\RequestCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) <a href='/33'>#33</a> [internal function]: Symfony\\Component\\Console\\Application-&gt;doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) 
<a href='/34'>#34</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/Console.php(135): call_user_func(Array, Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) 
<a href='/35'>#35</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/Access.php(670): Piwik\\Console-&gt;Piwik\\{closure}() 
<a href='/36'>#36</a> \/home\/travis\/build\/matomo-org\/matomo\/core\/Console.php(136): P ... ild\/matomo-org\/matomo\/console(32): Symfony\\Component\\Console\\Application->run()\n<a href='/41'>#41</a> {main}"}'

See e.g. https://app.travis-ci.com/github/matomo-org/matomo/jobs/554673498

Looking at the code it seems the reason might be a rase condition, where either the first call to getNextId fails for a temporary reason or where two archiving requests try to create a sequence simultaneously.
Ignoring the duplicate entry error should be safe in that case, as it means the record already exists and the following getNextId should work as expected nevertheless.

Review

@tsteur commented on January 10th 2022 Member

I'm not sure if this could maybe hide some error? If it's only in tests then it may indicate that archive tables may be deleted but the sequence table is not emptied/truncated (or not deleted and recreated). I actually had such a problem recently when I tested something manually.

Is it reproducible in the tests? If so might be good to check if we can rather fix the root cause maybe

@sgiehl commented on January 12th 2022 Member

I'm not able to reproduce that locally. But it happens frequently on travis. Looking at the code I can't see how the changes could hide any errors. It only catches one specific error, that should only happen for that special race condition (even though it might happen unlikely).
This fix might also be related to #8444 and #18088

This Pull Request was closed on January 13th 2022
Powered by GitHub Issue Mirror