I run a medium-sized website that gets about 50k hits each week, but Piwik has *really* been struggling lately. I've been a devoted Piwik user for years, I've filed bugs, and even worked on documentation, but I'm getting really fed up. I'm tired of dealing with what should be a reliable part of my instructure so I'm hoping for some help before admitting defeat and jumping ship.
Some background:
1. Our server has LOTS of power and I can't believe how much Piwik slows it down. The server has 24 cores, 128GB of RAM and SSD drives mounted in a RAID array. There is NO reason Piwik should struggle on this machine, but I regularly see Piwik archiving chugging along at 99% of a CPU. Something is very wrong about that, since 50k hits/week is not really very many. I regularly do data work on this server to the tune of changing a million records -- that's no big deal...why is Piwik's 10k/day a big deal?
2. I've read the FAQ about setting up archives, and the FAQ about tweaking MySQL for performance.
I've tweaked my settings so that:
- PHP's memory_limit = -1,
- MySQL's wait_timeout = 20600 (about six hours) and so
- MySQL's max_packet = 128.
Six hours and unlimited memory should be enough for just about anything related to tracking users. I can' timagine a MySQL packet bigger than 128MB.
3. The FAQ about setting up cron jobs suggests an hourly cron job, but since it takes ages to run and chugs along at 99% CPU, I changed my cron jobs to run only once per day at five minutes past midnight:
5 * * * * /usr/bin/php5 /var/www/piwik/console core:archive --url=https://www.my-website.com/piwik/ > /dev/null
4. In Piwik's settings webpage I've set the archiving to trigger no more than every 18000 seconds and I've disabled the web-based archiving.
I believe these are all of the recommended settings for making Piwik work...and yet...I got zero results in my archive for last week. I could go and delete the MySQL tables for last week, as I did for the week prior and then re-run the archives, watching to make sure it works, but this all feels very wrong.
Questions:
- I don't ever want to see zero results again. I didn't get an error email from cron this time. What do I need to do to make Piwik work reliably as we scale to 100k hits/week?
- The amount of processing Piwik does is unbelievable when it runs its archive. Is this expected behavior? It's only 50k people per week, less than 10k/day. That's **nothing**, right? Why should Piwik struggle with that on a server like this? What can we do to make it do less work? It seems a bit unbelievable that I can't have hourly results without pegging a CPU at 99%, but that's where I'm at right now.
Really appreciate any help. It's the community that's gotten me this far.
Mike
Some background:
1. Our server has LOTS of power and I can't believe how much Piwik slows it down. The server has 24 cores, 128GB of RAM and SSD drives mounted in a RAID array. There is NO reason Piwik should struggle on this machine, but I regularly see Piwik archiving chugging along at 99% of a CPU. Something is very wrong about that, since 50k hits/week is not really very many. I regularly do data work on this server to the tune of changing a million records -- that's no big deal...why is Piwik's 10k/day a big deal?
2. I've read the FAQ about setting up archives, and the FAQ about tweaking MySQL for performance.
I've tweaked my settings so that:
- PHP's memory_limit = -1,
- MySQL's wait_timeout = 20600 (about six hours) and so
- MySQL's max_packet = 128.
Six hours and unlimited memory should be enough for just about anything related to tracking users. I can' timagine a MySQL packet bigger than 128MB.
3. The FAQ about setting up cron jobs suggests an hourly cron job, but since it takes ages to run and chugs along at 99% CPU, I changed my cron jobs to run only once per day at five minutes past midnight:
5 * * * * /usr/bin/php5 /var/www/piwik/console core:archive --url=https://www.my-website.com/piwik/ > /dev/null
4. In Piwik's settings webpage I've set the archiving to trigger no more than every 18000 seconds and I've disabled the web-based archiving.
I believe these are all of the recommended settings for making Piwik work...and yet...I got zero results in my archive for last week. I could go and delete the MySQL tables for last week, as I did for the week prior and then re-run the archives, watching to make sure it works, but this all feels very wrong.
Questions:
- I don't ever want to see zero results again. I didn't get an error email from cron this time. What do I need to do to make Piwik work reliably as we scale to 100k hits/week?
- The amount of processing Piwik does is unbelievable when it runs its archive. Is this expected behavior? It's only 50k people per week, less than 10k/day. That's **nothing**, right? Why should Piwik struggle with that on a server like this? What can we do to make it do less work? It seems a bit unbelievable that I can't have hourly results without pegging a CPU at 99%, but that's where I'm at right now.
Really appreciate any help. It's the community that's gotten me this far.
Mike
