Memory exhausted (?) during Archive (no replies)

Hello,

I am running v2.4.1 on CentOS 6.5 (fully updated) under nginx 1.6.1 with php-fpm, PHP 5.3.3-27 and mysql 5.5.34.

It is a VM under KVM. The VM has 6GB (virtual) RAM and 4 (virtual) CPUs.

I have been having this issue for about a month now: Archiving is successful for day and week but it always terminates during month/year archiving with a memory exhaustion error.

Please let me know if this may be caused by some bug (e.g. memory leak) in piwik archive code or I am doing something wrong.

Is it impossible to run piwik archiving successfully on such a system (which is by no means inefficient for every other task and its web server can serve thousands of requests per second)?

Here is a typical log:

# /usr/bin/php /var/webs/wwwpiwik/www/console core:archive --url=http://wstat1.noa.gr
INFO CoreConsole[2014-08-09 18:58:41] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 18:58:41] [9a9a2] INIT
INFO CoreConsole[2014-08-09 18:58:41] [9a9a2] Piwik is installed at: http://wstat1.noa.gr/index.php
INFO CoreConsole[2014-08-09 18:58:41] [9a9a2] Running Piwik 2.4.1 as Super User
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] NOTES
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] - Reports for today will be processed at most every 21600 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] - Archiving was last executed without error 28 days 9 hours ago
INFO CoreConsole[2014-08-09 18:58:48] [9a9a2] - Will process 1 websites with new visits since 28 days 9 hours , IDs: 1
INFO CoreConsole[2014-08-09 18:58:48] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 18:58:48] [9a9a2] START
INFO CoreConsole[2014-08-09 18:58:48] [9a9a2] Starting Piwik reports archiving...
INFO CoreConsole[2014-08-09 18:59:07] [9a9a2] Archived website id = 1, period = day, 14780 visits in last last2 days, 1492 visits today, Time elapsed: 18.157s

INFO CoreConsole[2014-08-09 20:20:34] [9a9a2] Archived website id = 1, period = week, 1706036 visits in last last30 weeks, 64467 visits this week, Time elapsed: 4887.189s
ERROR CoreConsole[2014-08-09 20:49:26] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
ERROR CoreConsole[2014-08-09 20:49:26] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 20:49:27] [9a9a2] Archived website id = 1, period = month, 0 visits in last last30 months, 0 visits this month, Time elapsed: 1732.712s
ERROR CoreConsole[2014-08-09 21:20:27] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
ERROR CoreConsole[2014-08-09 21:20:27] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 21:20:27] [9a9a2] Archived website id = 1, period = year, 0 visits in last last7 years, 0 visits this year, Time elapsed: 1860.777s
INFO CoreConsole[2014-08-09 21:20:28] [9a9a2] Archived website id = 1, 4 API requests, Time elapsed: 8498.919s [1/1 done]
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Done archiving!
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] SUMMARY
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Total visits for today across archived websites: 1492
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Archived today's reports for 1 websites
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Archived week/month/year for 1 websites
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Skipped 0 websites day archiving: existing daily reports are less than 21600 seconds old
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Total API requests: 4
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] done: 1/1 100%, 1492 vtoday, 1 wtoday, 1 wperiods, 4 req, 8501091 ms, 2 errors.
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Time elapsed: 8501.091s
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] SCHEDULED TASKS
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Starting Scheduled tasks...
ERROR CoreConsole[2014-08-09 21:20:44] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-09 21:20:44] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.

INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] done
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] SUMMARY OF ERRORS
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-09 21:20:46] [9a9a2] 3 total errors during this script execution, please investigate and try and fix these errors.
ERROR CoreConsole[2014-08-09 21:20:46] [9a9a2] 3 total errors during this script execution, please investigate and try and fix these errors.

More technical details follow:

I have configured in php.ini a memory_limit which is already high for this system:
memory_limit = 5120M
Here is the memory footprint under normal operation (when piwik Archiving is NOT running):

# free -m
             total       used       free     shared    buffers     cached
Mem:          5850       3226       2624          0        499       1058
-/+ buffers/cache:       1667       4182
Swap:         3023          0       3023

During piwik Archive processing (week), "top" shows:

# ps axuwww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root     16753  7.6  0.6 418116 38032 pts/0    S+   21:58   1:08 /usr/bin/php /var/webs/wwwpiwik/www/console core:archive --url=http://wstat1.noa.gr
root     17213 96.7 38.2 2690984 2289900 pts/0 R+   21:59  14:01 /usr/bin/php -q /var/webs/wwwpiwik/www/console climulti:request --piwik-domain=wstat1.noa.gr module=API&method=API.get&idSite=1&period=week&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp&pid=0207c2ff9f9c0e77d44c8898be105e2572332c0bc7463180829bfed4da5cd6b92d3d8eff87a0595240329d6bc8cf865884af0
...
root      1183  0.0  0.0 108300  1584 ?        S    14:35   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql     1644 35.5 16.0 2317780 964340 ?      Sl   14:35 167:30 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
...

I noticed that during all this period, memory usage of process 17213 was steady at 38.2% and things run smoothly. Yet, after the line:

INFO CoreConsole[2014-08-09 20:20:34] [9a9a2] Archived website id = 1, period = week, 1706036 visits in last last30 weeks, 64467 visits this week, Time elapsed: 4887.189s

is printed, then for a while mysql takes over for a while, and php seems a bit idle (memory and cpu usage drops). After that, php starts again, and memory consumption grows quickly; swapping takes place (both for php and mysql) and not much later swap size reaches as high as ~1.7GB:

             total       used       free     shared    buffers     cached
Mem:          5850       5721        129          0          9         18
-/+ buffers/cache:       5693        157
Swap:         3023       1687       1336

and piwik Archive stops month archiving:

ERROR CoreConsole[2014-08-09 20:49:26] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 20:49:27] [9a9a2] Archived website id = 1, period = month, 0 visits in last last30 months, 0 visits this month, Time elapsed: 1732.712s

Swap size after that drops to ~1GB.

Then the same happens for the year archiving procedure. Swap size rises even more:

top - 00:16:31 up  9:41,  3 users,  load average: 3.08, 2.85, 2.60
Tasks: 155 total,   1 running, 154 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.6%us, 10.1%sy,  0.0%ni, 34.6%id, 47.1%wa,  0.0%hi,  0.1%si,  0.5%st
Mem:   5990960k total,  5873464k used,   117496k free,    11060k buffers
Swap:  3096568k total,  2183080k used,   913488k free,    41876k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
27384 root      20   0 5953m 4.3g 2128 D  1.0 75.4  22:28.26 1.1g php
 1644 mysql     20   0 2327m 719m 3824 S  1.0 12.3 168:59.59 493m mysqld
 1866 nginx     20   0  438m 9780 2968 S  0.0  0.2   0:09.75  19m php-fpm
16753 root      20   0  409m 6672 1896 S  6.6  0.1  10:11.09  19m php

and then this phase terminates with the same error.

Please advise!

Thanks,
Nick