Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance Issue: Slow Page Load Times (500-700ms) on ARM64 Devices #5385

Closed
codenamek83 opened this issue Jan 3, 2025 · 5 comments
Closed
Labels

Comments

@codenamek83
Copy link

codenamek83 commented Jan 3, 2025

Describe the Bug

I have deployed BookStack as a Docker container on a Raspberry Pi 4 with 8 GB of RAM, using the Docker image built by LinuxServer for both BookStack and MariaDB. The server is hosted locally behind a reverse proxy. I have consistently observed slow page load times ranging from 500 to 700 ms. This issue seems to be specific to the architecture, as other platforms perform significantly better.

The load time is slow when loading the page.

Screenshot from 2025-01-02 22-42-22

The load time is normal when loading the kb-icon.png.

Screenshot from 2025-01-02 22-42-31

To isolate the problem, I created a test environment without a reverse proxy server. The Raspberry Pi and VM used for testing are hosted locally on my LAN. All tests were performed using http://ip:port.

  1. I deployed a new BookStack stack (LinuxServer images) on an AMD64 VM with 2 GB of RAM and 1 vCPU using a Docker Compose file. The page load time was normal, and no slowness was observed.

  2. I deployed a new BookStack stack (LinuxServer images) on an AMD64 VM with 2 GB of RAM and 1 vCPU using a Docker Compose file. I copied the data and database from my primary node to test with the actual data. The page load time was normal, and no slowness was observed.

  3. I deployed a new BookStack stack (Solidnerd image for Bookstack) on an AMD64 VM with 2 GB of RAM and 1 vCPU using a Docker Compose file. The page load time was normal, and no slowness was observed.

  4. I deployed a new BookStack stack (LinuxServer images) on a Raspberry Pi 4 with 8 GB of RAM, using a Docker Compose file. The page load time was between 500-800ms.

  5. SolidNerd only provides a prebuilt image for AMD64, so I had to build a Docker image for ARM64 using the Dockerfile provided by them. I deployed a new BookStack stack (image built with SolidNerd's Dockerfile) on a Raspberry Pi 4 with 8 GB of RAM, using a Docker Compose file. The page load time was between 500–800 ms.

Based on the listed test cases, I believe the issue is somehow limited to running BookStack on ARM64 devices. Therefore, I tried the following steps to determine whether the issue lies with the Docker image or the BookStack application:

  1. I created a new PHP script named test.php (ChatGPT) and placed the file in /app/www/public.`
<?php

// Start measuring execution time
$start = microtime(true);

// Simulate a database query
try {
   $mysqli = new mysqli("testdb", "root", "securepasswd", "bookstack02");
   if ($mysqli->connect_error) {
       die("Database connection failed: " . $mysqli->connect_error);
   }

   $result = $mysqli->query("SELECT COUNT(*) AS total FROM role_permissions");
   $row = $result->fetch_assoc();
   echo "Total rows in table: " . $row['total'] . "\n";

   $mysqli->close();
} catch (Exception $e) {
   echo "Database error: " . $e->getMessage() . "\n";
}

// Simulate file I/O operations
$file = 'test_file.txt';
file_put_contents($file, str_repeat("Hello, World!\n", 1000));
$content = file_get_contents($file);
unlink($file);
echo "File operations completed.\n";

// Simulate heavy computations
function calculateFactorial($num) {
   return ($num == 1 || $num == 0) ? 1 : $num * calculateFactorial($num - 1);
}
$factorial = calculateFactorial(15);
echo "Factorial of 15: " . $factorial . "\n";

// End measuring execution time
$end = microtime(true);
$executionTime = $end - $start;
echo "Execution time: {$executionTime} seconds\n";

?>
  1. I appended /config/nginx/site-confs/default.conf to allow loading the test.php file.

  2. Test accessing the test page from browser. The page load time was normal, and no slowness was observed.

Screenshot from 2025-01-03 15-06-48

Screenshot from 2025-01-03 16-59-51

I am reporting the issue here because I feel it is very specific to the BookStack app and not the Docker environment in which it is contained. Most of the time, the Docker image maintainer can't be helpful with these types of issues. I have tried my best to gather the details and summarize them for your reference. I have also performed many other troubleshooting procedures, including database optimization and PHP optimization, but I felt they were no longer relevant to the issue.

Steps to Reproduce

I have explained the steps in the description.

Expected Behaviour

The page loads much quicker since the server is on the LAN.

Screenshots or Additional Context

The Docker Compose file used to deploy the containers:

services:
  us-bookstack02-app:
    image: linuxserver/bookstack:v24.12-ls184
    container_name: us-bookstack02-app
    hostname: us-bookstack02-app
    networks:
      - us-bookstack02
    ports:
      - "8116:80"
    volumes:
      - /us-bookstack02/us-bookstack02/app:/config
    environment:
      PUID: 1000
      PGID: 1000
      APP_URL: "http://192.168.10.1:8116"
      APP_KEY: "secret"
      DB_HOST: "us-bookstack02-db"
      DB_DATABASE: bookstack02
      DB_USERNAME: dbadmin
      DB_PASSWORD: "secret"
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    depends_on:
      - us-bookstack02-db
  
  us-bookstack02-db:
    image: linuxserver/mariadb:10.11.10
    container_name: us-bookstack02-db
    hostname: us-bookstack02-db
    networks:
      - us-bookstack02      
    expose:
      - 3306
    volumes:
      - /us-bookstack02/us-bookstack02/db:/config
    environment:  
      PUID: 1000
      PGID: 1000
      MYSQL_ROOT_PASSWORD: "secret"
      MYSQL_DATABASE: bookstack02
      MYSQL_USER: dbadmin
      MYSQL_PASSWORD: "secret"
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
     
networks:
  us-bookstack02:
    name: us-bookstack02
    driver: bridge

Browser Details

No response

Exact BookStack Version

v24.12

@ssddanbrown
Copy link
Member

Handily, I just had reflashed my Pi4 in the last few days so have one to hand for debugging.
There's a /status endpoint which can be used for testing the timings of the base app load requirements (avoid any variance of instance-specific content).

Raspi Hardware/OS info

Model: Raspberry Pi 4 Model B Rev 1.1
RAM: 2GB
Linux raspberrypi 6.6.62+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64 GNU/Linux
Revision: b03111

Test A - Vanilla install

Setup a manual install using system packages, with nginx, default-mysql-server and PHP8.2.
PHP apt packages installed as follows: apt install php8.2-{common,mbstring,gd,zip,xml,mysql,curl,fpm,cli}
Found that opcache was active and installed as part of these.

Benchmark

ab -n 50 -c 5 http://192.168.1.20/status 
This is ApacheBench, Version 2.3 <$Revision: 1913912 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.20 (be patient).....done


Server Software:        nginx/1.22.1
Server Hostname:        192.168.1.20
Server Port:            80

Document Path:          /status
Document Length:        45 bytes

Concurrency Level:      5
Time taken for tests:   1.209 seconds
Complete requests:      50
Failed requests:        0
Total transferred:      73650 bytes
HTML transferred:       2250 bytes
Requests per second:    41.34 [#/sec] (mean)
Time per request:       120.936 [ms] (mean)
Time per request:       24.187 [ms] (mean, across all concurrent requests)
Transfer rate:          59.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    4  15.1      2     109
Processing:    51   97  24.9     90     145
Waiting:       51   97  25.0     89     145
Total:         54  102  29.7     92     213

Percentage of the requests served within a certain time (ms)
  50%     92
  66%    124
  75%    130
  80%    130
  90%    133
  95%    140
  98%    213
  99%    213
 100%    213 (longest request)

Re-running, the above is on the high-end for me, I'm seeing a range of max from 100-200 generally.
Adding opcache-JIT didn't look to improve things much.

Running some (undocumented/unadvised) app cache commands (php artisan config:cache && php artisan route:cache && php artisan view:cache) takes this
down a little to the 60-160ms range.

Test B - Docker compose

Installed docker using system packages (apt install docker-compose docker.io).
Using a composer file based on that from the OP:

compose file
services:
  us-bookstack02-app:
    image: linuxserver/bookstack:v24.12-ls184
    container_name: us-bookstack02-app
    hostname: us-bookstack02-app
    networks:
      - us-bookstack02
    ports:
      - "8080:80"
    volumes:
      - ./app:/config
    environment:
      PUID: 1000
      PGID: 1000
      APP_URL: "http://192.168.1.20:8080"
      APP_KEY: "base64:JBlR67RL9gx/z7QjMLHFgcwMLn2+q4udjHhyiBxpJv4="
      DB_HOST: "us-bookstack02-db"
      DB_DATABASE: bookstack02
      DB_USERNAME: dbadmin
      DB_PASSWORD: "secret"
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    depends_on:
      - us-bookstack02-db
  
  us-bookstack02-db:
    image: linuxserver/mariadb:10.11.10
    container_name: us-bookstack02-db
    hostname: us-bookstack02-db
    networks:
      - us-bookstack02      
    expose:
      - 3306
    volumes:
      - ./db:/config
    environment:  
      PUID: 1000
      PGID: 1000
      MYSQL_ROOT_PASSWORD: "secret"
      MYSQL_DATABASE: bookstack02
      MYSQL_USER: dbadmin
      MYSQL_PASSWORD: "secret"
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
     
networks:
  us-bookstack02:
    name: us-bookstack02
    driver: bridge
Benchmark

ab -n 50 -c 5 http://192.168.1.20:8080/status
This is ApacheBench, Version 2.3 <$Revision: 1913912 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.20 (be patient).....done


Server Software:        nginx
Server Hostname:        192.168.1.20
Server Port:            8080

Document Path:          /status
Document Length:        45 bytes

Concurrency Level:      5
Time taken for tests:   7.526 seconds
Complete requests:      50
Failed requests:        0
Total transferred:      74600 bytes
HTML transferred:       2250 bytes
Requests per second:    6.64 [#/sec] (mean)
Time per request:       752.589 [ms] (mean)
Time per request:       150.518 [ms] (mean, across all concurrent requests)
Transfer rate:          9.68 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    3   0.9      3       6
Processing:   394  692 156.6    656    1065
Waiting:      393  691 156.8    656    1065
Total:        399  695 156.5    660    1068

Percentage of the requests served within a certain time (ms)
  50%    660
  66%    740
  75%    792
  80%    826
  90%    937
  95%   1032
  98%   1068
  99%   1068
 100%   1068 (longest request)

So this is about 5x as slow, and feels it in app too.

Test C - Docker compose with opcache added

In test B, I noticed opcache was disabled for the running PHP, so I enabled it by accessing the container
and running a apk add php83-opcache before restarting that same container instance.

This seemed to help:

Benchmark

ab -n 50 -c 5 http://192.168.1.20:8080/status
This is ApacheBench, Version 2.3 <$Revision: 1913912 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.20 (be patient).....done


Server Software:        nginx
Server Hostname:        192.168.1.20
Server Port:            8080

Document Path:          /status
Document Length:        45 bytes

Concurrency Level:      5
Time taken for tests:   1.321 seconds
Complete requests:      50
Failed requests:        0
Total transferred:      85680 bytes
HTML transferred:       2250 bytes
Requests per second:    37.85 [#/sec] (mean)
Time per request:       132.095 [ms] (mean)
Time per request:       26.419 [ms] (mean, across all concurrent requests)
Transfer rate:          63.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    3   1.5      3       7
Processing:    72  115  27.9    115     188
Waiting:       72  114  28.2    113     187
Total:         76  118  28.2    117     195

Percentage of the requests served within a certain time (ms)
  50%    117
  66%    140
  75%    145
  80%    146
  90%    148
  95%    153
  98%    195
  99%    195
 100%    195 (longest request)

Brings it down to numbers similar to Test A results.

Rough Evaluation/Thoughts

Without spending time debugging into this (getting a bit late here) my guess is that this isn't ARM specific, but the app is slow without the opcache optimizations due to using quite slow storage (assuming you're using an SD card too) while an app like BookStack relies on loading many different app/framework/library files in a short period.
Enabling the opcache then massively helps to avoid the mass reloading of files on each request.
While your test file did perform some IO, it was a simple single file write and read (which may have benefit from system buffers) compared to reading much more from many different files.

Again, didn't specifically test this theory but when debugging the compose setup I checked app-level timings and saw the actual app & database logic was minimal, but there was large proceeding app times, pointing roughly to app loading/boot.

Try hacking in the opcache extension on your instance like I did above and see if that dramatically improves things for you too.

@codenamek83
Copy link
Author

codenamek83 commented Jan 4, 2025

Hi @ssddanbrown,

Thanks for your quick insights and recommendation. For the Raspberry Pi 4, I am not using an SD card but an SSD drive connected via the USB 3.0 interface with a USB-to-SATA adapter. The performance is definitely better than an SD card, although it is limited by what the USB port can offer. The VM used for testing is deployed on an NVMe drive.

Based on your recommendation, I built a new image using the Dockerfile provided by LinuxServer.io, with a modification to include php83-opcache. Below are the test results comparing the original LinuxServer image and the custom image I built with opcache.

Without Opcache
This is ApacheBench, Version 2.3 <$Revision: 1913912 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.10.1 (be patient).....done


Server Software:        nginx
Server Hostname:        192.168.10.1
Server Port:            8116

Document Path:          /status
Document Length:        45 bytes

Concurrency Level:      5
Time taken for tests:   8.212 seconds
Complete requests:      50
Failed requests:        0
Total transferred:      74600 bytes
HTML transferred:       2250 bytes
Requests per second:    6.09 [#/sec] (mean)
Time per request:       821.232 [ms] (mean)
Time per request:       164.246 [ms] (mean, across all concurrent requests)
Transfer rate:          8.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:   381  776 201.1    766    1292
Waiting:      380  775 200.9    766    1292
Total:        382  776 201.0    767    1293

Percentage of the requests served within a certain time (ms)
  50%    767
  66%    813
  75%    924
  80%    947
  90%   1037
  95%   1137
  98%   1293
  99%   1293
 100%   1293 (longest request)
With Opcache
This is ApacheBench, Version 2.3 <$Revision: 1913912 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.10.1 (be patient).....done


Server Software:        nginx
Server Hostname:        192.168.10.1
Server Port:            8116

Document Path:          /status
Document Length:        45 bytes

Concurrency Level:      5
Time taken for tests:   1.003 seconds
Complete requests:      50
Failed requests:        0
Total transferred:      74600 bytes
HTML transferred:       2250 bytes
Requests per second:    49.86 [#/sec] (mean)
Time per request:       100.277 [ms] (mean)
Time per request:       20.055 [ms] (mean, across all concurrent requests)
Transfer rate:          72.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:    53   91  21.7     87     172
Waiting:       52   90  21.7     86     171
Total:         55   91  21.7     87     173

Percentage of the requests served within a certain time (ms)
  50%     87
  66%     97
  75%    101
  80%    104
  90%    120
  95%    123
  98%    173
  99%    173
 100%    173 (longest request)
Without Opcache - VM
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 0.0.0.0 (be patient).....done


Server Software:        nginx
Server Hostname:        0.0.0.0
Server Port:            8116

Document Path:          /status
Document Length:        45 bytes

Concurrency Level:      5
Time taken for tests:   2.636 seconds
Complete requests:      50
Failed requests:        0
Total transferred:      74600 bytes
HTML transferred:       2250 bytes
Requests per second:    18.97 [#/sec] (mean)
Time per request:       263.611 [ms] (mean)
Time per request:       52.722 [ms] (mean, across all concurrent requests)
Transfer rate:          27.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   105  250  48.5    250     374
Waiting:      105  250  48.4    250     373
Total:        106  250  48.5    250     374

Percentage of the requests served within a certain time (ms)
  50%    250
  66%    260
  75%    271
  80%    275
  90%    301
  95%    317
  98%    374
  99%    374
 100%    374 (longest request)
With Opcache - VM
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 0.0.0.0 (be patient).....done


Server Software:        nginx
Server Hostname:        0.0.0.0
Server Port:            8116

Document Path:          /status
Document Length:        45 bytes

Concurrency Level:      5
Time taken for tests:   0.362 seconds
Complete requests:      50
Failed requests:        0
Total transferred:      74600 bytes
HTML transferred:       2250 bytes
Requests per second:    138.13 [#/sec] (mean)
Time per request:       36.198 [ms] (mean)
Time per request:       7.240 [ms] (mean, across all concurrent requests)
Transfer rate:          201.26 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:    17   33   8.8     32      52
Waiting:       16   33   8.8     32      52
Total:         17   33   8.8     32      52

Percentage of the requests served within a certain time (ms)
  50%     32
  66%     36
  75%     38
  80%     40
  90%     48
  95%     50
  98%     52
  99%     52
 100%     52 (longest request)

I have also updated my primary Bookstack instance and observed a drastic performance gain while navigating between pages. Do you intend to perform any other tests to confirm this theory on your end? Let me know if you would like me to test anything else.

@ssddanbrown
Copy link
Member

I am not using an SD card but an SSD drive connected via the USB 3.0 interface with a USB-to-SATA adapter.

Ah, okay, might still just be the underlying impact of loading many files in a non-opcache environment on this kinda hardware. Also wouldn't be surpised if non-opcache php is less optimized on ARM, since ARM is less common and has gained its popularity while opcache is an option.

Do you intend to perform any other tests to confirm this theory on your end?

Probably not, we know what helps here, and opcache is a pretty common default in most production setups these days.
Might be good to request that opcache is to the linuxserver image by default.

@codenamek83
Copy link
Author

Thank you for your help, @ssddanbrown. I have reported your feedback to the LinuxServer team. Hopefully, they will consider including this extension by default, so users won’t need to build the images locally with every update or install it using a custom script.

@ssddanbrown
Copy link
Member

@codenamek83 Yeah, I saw they already have a PR on the go (linuxserver/docker-bookstack#254).

Thanks for the detail you provided and the time you spent on this with a view to help.
I'll therefore close this off since there's no changes due to the core codebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

2 participants