Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ignore duplicate entry error when creating a sequence #18590

Merged
merged 1 commit into from Jan 13, 2022
Merged

Ignore duplicate entry error when creating a sequence #18590

merged 1 commit into from Jan 13, 2022

Conversation

sgiehl
Copy link
Member

@sgiehl sgiehl commented Jan 7, 2022

Description:

In the past the CronArchiving tests sometimes failed with the error:

Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2019-12-10&format=json&segment=actions%3E%3D2&trigger=archivephp. Response was '{"result":"error","message":"SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'archive_numeric_2019_12' for key 'PRIMARY' 
#0 \/home\/travis\/build\/matomo-org\/matomo\/libs\/Zend\/Db\/Statement.php(300): Zend_Db_Statement_Pdo->_execute(Array) 
#1 \/home\/travis\/build\/matomo-org\/matomo\/libs\/Zend\/Db\/Adapter\/Abstract.php(479): Zend_Db_Statement->execute(Array) 
#2 \/home\/travis\/build\/matomo-org\/matomo\/libs\/Zend\/Db\/Adapter\/Pdo\/Abstract.php(238): Zend_Db_Adapter_Abstract->query('INSERT INTO `se...', Array) 
#3 \/home\/travis\/build\/matomo-org\/matomo\/core\/Db\/Adapter\/Pdo\/Mysql.php(310): Zend_Db_Adapter_Pdo_Abstract->query('INSERT INTO `se...', Array) 
#4 \/home\/travis\/build\/matomo-org\/matomo\/libs\/Zend\/Db\/Adapter\/Abstract.php(575): Piwik\\Db\\Adapter\\Pdo\\Mysql->query('INSERT INTO `se...', Array) 
#5 \/home\/travis\/build\/matomo-org\/matomo\/core\/Sequence.php(67): Zend_Db_Adapter_Abstract->insert('sequence', Array) 
#6 \/home\/travis\/build\/matomo-org\/matomo\/core\/DataAccess\/Model.php(515): Piwik\\Sequence->create() 
#7 \/home\/travis\/build\/matomo-org\/matomo\/core\/DataAccess\/ArchiveWriter.php(195): Piwik\\DataAccess\\Model->allocateNewArchiveId('archive_numeric...') 
#8 \/home\/travis\/build\/matomo-org\/matomo\/core\/DataAccess\/ArchiveWriter.php(157): Piwik\\DataAccess\\ArchiveWriter->allocateNewArchiveId() 
#9 \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(68): Piwik\\DataAccess\\ArchiveWriter->initNewArchive() 
#10 \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(273): Piwik\\ArchiveProcessor\\PluginsArchiver->__construct(Object(Piwik\\ArchiveProcessor\\Parameters)) 
#11 \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(176): Piwik\\ArchiveProcessor\\Loader->prepareAllPluginsArchive(false, false) 
#12 \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(159): Piwik\\ArchiveProcessor\\Loader->insertArchiveData(false, false) 
#13 \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(99): Piwik\\ArchiveProcessor\\Loader->prepareArchiveImpl(false) 
#14 \/home\/travis\/build\/matomo-org\/matomo\/core\/Context.php(75): Piwik\\ArchiveProcessor\\Loader->Piwik\\ArchiveProcessor\\{closure}() 
#15 \/home\/travis\/build\/matomo-org\/matomo\/core\/ArchiveProcessor\/Loader.php(103): Piwik\\Context::changeIdSite(1, Object(Closure)) 
#16 \/home\/travis\/build\/matomo-org\/matomo\/plugins\/CoreAdminHome\/API.php(278): Piwik\\ArchiveProcessor\\Loader->prepareArchive(false) 
#17 [internal function]: Piwik\\Plugins\\CoreAdminHome\\API->archiveReports('1', Object(Piwik\\Period\\Day), '2019-12-10', 'actions%3E%3D2', false, false) 
#18 \/home\/travis\/build\/matomo-org\/matomo\/core\/API\/Proxy.php(244): call_user_func_array(Array, Array) 
#19 \/home\/travis\/build\/matomo-org\/matomo\/core\/Context.php(28): Piwik\\API\\Proxy->Piwik\\API\\{closure}() 
#20 \/home\/travis\/build\/matomo-org\/matomo\/core\/API\/Proxy.php(335): Piwik\\Context::executeWithQueryParameters(Array, Object(Closure)) 
#21 \/home\/travis\/build\/matomo-org\/matomo\/core\/API\/Request.php(266): Piwik\\API\\Proxy->call('\\\\Piwik\\\\Plugins\\\\...', 'archiveReports', Array) 
#22 \/home\/travis\/build\/matomo-org\/matomo\/plugins\/API\/Controller.php(46): Piwik\\API\\Request->process() 
#23 [internal function]: Piwik\\Plugins\\API\\Controller->index() 
#24 \/home\/travis\/build\/matomo-org\/matomo\/core\/FrontController.php(619): call_user_func_array(Array, Array) 
#25 \/home\/travis\/build\/matomo-org\/matomo\/core\/FrontController.php(168): Piwik\\FrontController->doDispatch('API', false, Array) 
#26 \/home\/travis\/build\/matomo-org\/matomo\/core\/dispatch.php(32): Piwik\\FrontController->dispatch() 
#27 \/home\/travis\/build\/matomo-org\/matomo\/index.php(25): require_once('\/home\/travis\/bu...') 
#28 \/home\/travis\/build\/matomo-org\/matomo\/tests\/PHPUnit\/proxy\/index.php(15): include('\/home\/travis\/bu...') 
#29 \/home\/travis\/build\/matomo-org\/matomo\/core\/CliMulti\/RequestCommand.php(79): require_once('\/home\/travis\/bu...') 
#30 \/home\/travis\/build\/matomo-org\/matomo\/vendor\/symfony\/console\/Symfony\/Component\/Console\/Command\/Command.php(257): Piwik\\CliMulti\\RequestCommand->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) 
#31 \/home\/travis\/build\/matomo-org\/matomo\/vendor\/symfony\/console\/Symfony\/Component\/Console\/Application.php(874): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) 
#32 \/home\/travis\/build\/matomo-org\/matomo\/vendor\/symfony\/console\/Symfony\/Component\/Console\/Application.php(195): Symfony\\Component\\Console\\Application->doRunCommand(Object(Piwik\\CliMulti\\RequestCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) #33 [internal function]: Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) 
#34 \/home\/travis\/build\/matomo-org\/matomo\/core\/Console.php(135): call_user_func(Array, Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput)) 
#35 \/home\/travis\/build\/matomo-org\/matomo\/core\/Access.php(670): Piwik\\Console->Piwik\\{closure}() 
#36 \/home\/travis\/build\/matomo-org\/matomo\/core\/Console.php(136): P ... ild\/matomo-org\/matomo\/console(32): Symfony\\Component\\Console\\Application->run()\n#41 {main}"}'

See e.g. https://app.travis-ci.com/github/matomo-org/matomo/jobs/554673498

Looking at the code it seems the reason might be a rase condition, where either the first call to getNextId fails for a temporary reason or where two archiving requests try to create a sequence simultaneously.
Ignoring the duplicate entry error should be safe in that case, as it means the record already exists and the following getNextId should work as expected nevertheless.

Review

@sgiehl sgiehl added Bug For errors / faults / flaws / inconsistencies etc. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Needs Review PRs that need a code review labels Jan 7, 2022
@sgiehl sgiehl added this to the 4.7.0 milestone Jan 7, 2022
@tsteur
Copy link
Member

tsteur commented Jan 10, 2022

I'm not sure if this could maybe hide some error? If it's only in tests then it may indicate that archive tables may be deleted but the sequence table is not emptied/truncated (or not deleted and recreated). I actually had such a problem recently when I tested something manually.

Is it reproducible in the tests? If so might be good to check if we can rather fix the root cause maybe

@sgiehl
Copy link
Member Author

sgiehl commented Jan 12, 2022

I'm not able to reproduce that locally. But it happens frequently on travis. Looking at the code I can't see how the changes could hide any errors. It only catches one specific error, that should only happen for that special race condition (even though it might happen unlikely).
This fix might also be related to #8444 and #18088

@sgiehl sgiehl merged commit 296de5d into 4.x-dev Jan 13, 2022
@sgiehl sgiehl deleted the fixedgecase branch January 13, 2022 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tsteur tsteur tsteur approved these changes

Assignees
No one assigned
Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Needs Review PRs that need a code review
Projects
None yet
Milestone
4.7.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@sgiehl @tsteur