Quantcast
Channel: Piwik Forums - Support & Bugs
Viewing all articles
Browse latest Browse all 4243

Import very slow on Solaris 10 (no replies)

$
0
0
Hi,

I'm a new user trying to see if piwik will work for our organization.

We will be using log analytics to import apache logs.

The import is very slow. It seems to parse at an ok speed, but the updates to the database are infrequent and slow. I have tried many tuning options in mysql and they have zero impact. They are being loaded, as I see them in the log files.

At this point, I think this is an issue with either piwik or the http call to piwik. Any help would be much appreciated. We are close to throwing in the towel.

Thank you
-Kyle

-----------------------------------
Versions:
Box Sun T3
RAM 64G
Disk Hitachi SAN
OS Solaris 10
piwik 2.14.0
python 2.7.8
php 5.6.9
apache 2.4
mysql 5.6.25

-----------------------------------

Example import output:

2015-07-14 17:21:06,132: [DEBUG] Accepted hostnames: all
2015-07-14 17:21:06,141: [DEBUG] Piwik URL is: http://qaswtreports.ourdomain.com/piwik
2015-07-14 17:21:06,142: [DEBUG] No token-auth specified
2015-07-14 17:21:06,142: [DEBUG] Using credentials: (login = importuser, password = xxxxxxxxxxxxxxxxxxxxxxxxxxxx)
2015-07-14 17:21:07,288: [DEBUG] Authentication token token_auth is: xxxxxxxxxxxxxxxxxxxxxxxxxx
2015-07-14 17:21:07,289: [DEBUG] Resolver: static
2015-07-14 17:21:08,403: [DEBUG] Launched recorder
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2015-07-14 17:21:08,405: [DEBUG] Launched recorder
2015-07-14 17:21:08,407: [DEBUG] Launched recorder
2015-07-14 17:21:08,408: [DEBUG] Launched recorder
Parsing log /apache_logs/01r-rp1/qaoneplace.ehr.com_access_2015.07.11...
820 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1708 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2573 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 25 records/sec (avg), 427 records/sec (current)
3200 lines parsed, 427 lines recorded, 24 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 23 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 21 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 20 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 19 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 19 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 18 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 17 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 16 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 16 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 427 lines recorded, 15 records/sec (avg), 0 records/sec (current)
3780 lines parsed, 800 lines recorded, 28 records/sec (avg), 373 records/sec (current)
4000 lines parsed, 800 lines recorded, 27 records/sec (avg), 0 records/sec (current)
4000 lines parsed, 805 lines recorded, 26 records/sec (avg), 5 records/sec (current)
4000 lines parsed, 805 lines recorded, 25 records/sec (avg), 0 records/sec (current)

-----------------------------------

-----------------------------------

Output from phpinfo

PHP Version 5.6.9
System SunOS eapwr-na2-01r 5.10 Generic_150400-18 sun4v
Build Date Jul 1 2015 16:33:04
Configure Command './configure' '--prefix=/apps/eap/piwik_qa/php_qa' '--with-config-file-path=/apps/eap/piwik_qa/php_qa/lib' '--with-apxs2=/apps/eap/piwik_qa/apache2_swr1_qa/bin/apxs' '--with-ldap' '--with-mysql' '--with-mysqli' '--with-pdo-mysql' '--enable-sockets' '--enable-mysqlnd' '--with-zlib' '--enable-mbstring' '--enable-mbregex' '--with-gd' '--disable-zend-signals' '--disable-static' 'CC=cc' 'CFLAGS=-m64 '-fast' '-xO3' '-m64' '-L/lib' '-R/lib' '-R/usr/lib/64' '-L/usr/lib/64'' 'LIBS= '-lintl'' 'CPPFLAGS=-D_LARGEFILE_SOURCE '-D_FILE_OFFSET_BITS=64'' 'CXX=cc '-fast' '-xO3' '-m64'' 'CXXFLAGS=-L/usr/lib/64'
Server API Apache 2.0 Handler
Virtual Directory Support disabled
Configuration File (php.ini) Path /apps/eap/piwik_qa/php_qa/lib
Loaded Configuration File /usr/local/lib/php.ini
Scan this dir for additional .ini files (none)
Additional .ini files parsed (none)
PHP API 20131106
PHP Extension 20131226
Zend Extension 220131226
Zend Extension Build API220131226,NTS
PHP Extension Build API20131226,NTS
Debug Build no
Thread Safety disabled
Zend Signal Handling disabled
Zend Memory Manager enabled
Zend Multibyte Support provided by mbstring
IPv6 Support enabled
DTrace Support disabled
Registered PHP Streams compress.zlib, php, file, glob, data, http, ftp, phar
Registered Stream Socket Transports tcp, udp, unix, udg
Registered Stream Filters zlib.*, convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, dechunk
Zend logo This program makes use of the Zend Scripting Language Engine:
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
Configuration
apache2handler
Apache Version Unknown
Apache API Version 20120211
Server Administrator you@example.com
Hostname:Port qaswtreports.ourdomain.com:80
User/Group webservd(80)/80
Max Requests Per Child: 0 - Keep Alive: on - Max Per Connection: 100
Timeouts Connection: 600 - Keep-Alive: 5
Virtual Server No
Server Root /apps/eap/piwik_qa/apache2_swr1_qa
Loaded Modules core mod_so http_core prefork mod_authn_core mod_authz_host mod_authz_core mod_access_compat mod_include mod_deflate mod_log_config mod_env mod_mime_magic mod_expires mod_headers mod_usertrack mod_setenvif mod_proxy mod_proxy_http mod_unixd mod_mime mod_status mod_info mod_suexec mod_negotiation mod_dir mod_userdir mod_alias mod_rewrite mod_php5
Directive Local Value Master Value
engine 1 1
last_modified 0 0
xbithack 0 0
Apache Environment
Variable Value
HTTP_HOST qaswtreports.ourdomain.com
HTTP_USER_AGENT Mozilla/5.0 (Windows NT 6.1; rv:39.0) Gecko/20100101 Firefox/39.0
HTTP_ACCEPT text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP_ACCEPT_LANGUAGE en-US,en;q=0.5
HTTP_COOKIE piwik_lang=language%3DczoyOiJlbiI7%3A_%3Dba7e2dca770db47ee7d458fe2cb0215165cf7919; piwik_auth=login%3Dczo2OiJ3ZWJlbmciOw%3D%3D%3Atoken_auth%3DczozMjoiNmVmZTIyY2JhZjNiZGM5NmFmMDU5NWY4YTI0YmM4YWEiOw%3D%3D%3A_%3D815fd1223965e5c5c1e6c6c24d0456993ab9ad41; PIWIK_SESSID=t2c1e0g8pmu5jue7piuerv2484
HTTP_CONNECTION keep-alive
HTTP_X_RBT_OPTIMIZED_BY phi1-officewa01 (RiOS 6.5.6a) SC
PATH /apps/eap/piwik_qa/apache2_swr1_qa/bin:/usr/sbin:/usr/bin
LD_LIBRARY_PATH /apps/eap/piwik_qa/apache2_swr1_qa/lib
SERVER_SIGNATURE no value
SERVER_SOFTWARE Unknown
SERVER_NAME qaswtreports.ourdomain.com
SERVER_ADDR 10.244.180.23
SERVER_PORT 80
REMOTE_ADDR 10.129.131.138
DOCUMENT_ROOT /apps/eap/piwik_qa/apache2_swr1_qa/htdocs
REQUEST_SCHEME http
CONTEXT_PREFIX /piwik
CONTEXT_DOCUMENT_ROOT /apps/eap/piwik_qa/piwik_qa
SERVER_ADMIN you@example.com
SCRIPT_FILENAME /apps/eap/piwik_qa/piwik_qa/phpinfo.php
REMOTE_PORT 62137
GATEWAY_INTERFACE CGI/1.1
SERVER_PROTOCOL HTTP/1.1
REQUEST_METHOD GET
QUERY_STRING no value
REQUEST_URI /piwik/phpinfo.php
SCRIPT_NAME /piwik/phpinfo.php
HTTP Headers Information
HTTP Request Headers
HTTP Request GET /piwik/phpinfo.php HTTP/1.1
Host qaswtreports.ourdomain.com
User-Agent Mozilla/5.0 (Windows NT 6.1; rv:39.0) Gecko/20100101 Firefox/39.0
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-US,en;q=0.5
Cookie piwik_lang=language%3DczoyOiJlbiI7%3A_%3Dba7e2dca770db47ee7d458fe2cb0215165cf7919; piwik_auth=login%3Dczo2OiJ3ZWJlbmciOw%3D%3D%3Atoken_auth%3DczozMjoiNmVmZTIyY2JhZjNiZGM5NmFmMDU5NWY4YTI0YmM4YWEiOw%3D%3D%3A_%3D815fd1223965e5c5c1e6c6c24d0456993ab9ad41; PIWIK_SESSID=t2c1e0g8pmu5jue7piuerv2484
Connection keep-alive
X-RBT-Optimized-By phi1-officewa01 (RiOS 6.5.6a) SC
HTTP Response Headers
X-Powered-By PHP/5.6.9
Core
PHP Version 5.6.9
Directive Local Value Master Value
allow_url_fopen On On
allow_url_include Off Off
always_populate_raw_post_data -1 -1
arg_separator.input & &
arg_separator.output & &
asp_tags Off Off
auto_append_file no value no value
auto_globals_jit On On
auto_prepend_file no value no value
browscap no value no value
default_charset UTF-8 UTF-8
default_mimetype text/html text/html
disable_classes no value no value
disable_functions no value no value
display_errors On On
display_startup_errors On On
doc_root no value no value
docref_ext no value no value
docref_root no value no value
enable_dl Off Off
enable_post_data_reading On On
error_append_string no value no value
error_log /var/log/php_errors.log /var/log/php_errors.log
error_prepend_string no value no value
error_reporting 32767 32767
exit_on_timeout Off Off
expose_php On On
extension_dir /apps/eap/piwik_qa/php_qa/lib/php/extensions/no-debug-non-zts-20131226 /apps/eap/piwik_qa/php_qa/lib/php/extensions/no-debug-non-zts-20131226
file_uploads On On
highlight.comment #FF8000 #FF8000
highlight.default #0000BB #0000BB
highlight.html #000000 #000000
highlight.keyword #007700 #007700
highlight.string #DD0000 #DD0000
html_errors On On
ignore_repeated_errors Off Off
ignore_repeated_source Off Off
ignore_user_abort Off Off
implicit_flush Off Off
include_path .:/apps/eap/piwik_qa/php_qa/lib/php .:/apps/eap/piwik_qa/php_qa/lib/php
input_encoding no value no value
internal_encoding no value no value
log_errors On On
log_errors_max_len 1024 1024
mail.add_x_header On On
mail.force_extra_parameters no value no value
mail.log no value no value
max_execution_time 30 30
max_file_uploads 20 20
max_input_nesting_level 64 64
max_input_time 60 60
max_input_vars 1000 1000
memory_limit 512M 512M
open_basedir no value no value
output_buffering 4096 4096
output_encoding no value no value
output_handler no value no value
post_max_size 8M 8M
precision 14 14
realpath_cache_size 16K 16K
realpath_cache_ttl 120 120
register_argc_argv Off Off
report_memleaks On On
report_zend_debug On On
request_order GP GP
sendmail_from no value no value
sendmail_path /usr/sbin/sendmail -t -i /usr/sbin/sendmail -t -i
serialize_precision 17 17
short_open_tag Off Off
SMTP localhost localhost
smtp_port 25 25
sql.safe_mode Off Off
sys_temp_dir no value no value
track_errors On On
unserialize_callback_func no value no value
upload_max_filesize 2M 2M
upload_tmp_dir no value no value
user_dir no value no value
user_ini.cache_ttl 300 300
user_ini.filename .user.ini .user.ini
variables_order GPCS GPCS
xmlrpc_error_number 0 0
xmlrpc_errors Off Off
zend.detect_unicode On On
zend.enable_gc On On
zend.multibyte Off Off
zend.script_encoding no value no value
ctype
ctype functions enabled
date

**** there is more from phpinfo --- but the forums wont let me post it. Let me know if it will help

Viewing all articles
Browse latest Browse all 4243

Trending Articles