Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build script stops with building cryptography (and some more errors...) #99

Closed
nihilistaX opened this issue Dec 26, 2021 · 9 comments
Closed

Build script stops with building cryptography (and some more errors...) #99

nihilistaX opened this issue Dec 26, 2021 · 9 comments
Labels
enhancement

Comments

@nihilistaX
Copy link

I found that building 9.0.2 for buster stops with error while building cryptography.
Here >>> i see, that latest version (36.0.1) didnt work with buster, so i manually set cryptography==36.0.0 within requirements.txt.
Before i had to comment out downloading of requirements.txt from server to prevent overwriting with using build script.
Now after this build script runs with success....but only till build-server.py script, this has been moved to /seahub/scripts.
I changed one line in buildscript and got it running, but ended with error...so gave up for now
@nihilistaX nihilistaX changed the title Build script stops with building cryptography Build script stops with building cryptography (and some more errors...) Dec 26, 2021
@Gustl22
Copy link
Contributor

Gustl22 commented Dec 27, 2021

Thanks for the hint. Currently also having problems building for buster based distros:
https://github.com/haiwen/seafile-rpi/pull/100/files
I try to solve it, but time is as always the limiting factor. If you come with a solution first, I'd glad to hear ;)

@nihilistaX
Copy link
Author

Great to see you changed something. It was just luck to find out whats the problem with cryptography, but i'm a noob in coding, so i'm very limited while fixing those bugs ;-). But i can test it and hopefully there will be a chance to get scrpit working finde as before.

@nihilistaX
Copy link
Author

I managed to build server package now. What i did: Cloned this repo with latest changes, manually set cryptography version (like above), intstalled gvm, installed go1.14 with that and set that to default (with go version 1.14.x from apt it didn't work).
After that build script is running completely... ;-) (Didnt test that package till now, but if not i will report)

@jobenvil
Copy link
Member

I've check the Go version on the lxc container.

seafile@rpi-buster:~/go$ go version
go version go1.15.9 linux/arm

This is comming from buster-backports, as indicated in his repository

seafile@rpi-buster:~/go$ cat /etc/apt/sources.list
# Generated by distrobuilder
deb http://deb.debian.org/debian buster main
deb http://security.debian.org/debian-security buster/updates main
deb http://deb.debian.org/debian buster-backports main

To sum up: It is unsual that the build works with older Go libraries but not with newer (you installed go1.14 with gvm). @nihilistaX Could you please try to append the buster-backports repository and try again the build and having the cryptography==36.0.0? Or maybe not, since you will trigger an update of other packages and you maybe don't wanna do that. I prefer always to enable "backports" since debian is per se not having the high edge packages. But this is up to you.

jobenvil added a commit to jobenvil/rpi-build-seafile that referenced this issue Dec 28, 2021
@jobenvil
Test to check the build v9.0.2
6bfb84e 
See rpi issue haiwen/seafile-rpi#99
@jobenvil
Copy link
Member

I tested and the thirdparty built worked with your advise cryptography==36.0.0:


   Install Seahub and SeafDAV thirdparty requirements

+ python3 -m pip install -r /home/seafile/haiwen-build/seahub_thirdparty/requirements.txt --target /home/seafile/haiwen-build/seahub_thirdparty --no-cache --upgrade
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting Django==3.2.6
  Downloading https://www.piwheels.org/simple/django/Django-3.2.6-py3-none-any.whl (7.9 MB)
     |████████████████████████████████| 7.9 MB 2.2 MB/s
Collecting future
  Downloading https://www.piwheels.org/simple/future/future-0.18.2-py3-none-any.whl (491 kB)
     |████████████████████████████████| 491 kB 5.9 MB/s
Collecting captcha
  Downloading https://www.piwheels.org/simple/captcha/captcha-0.3-py3-none-any.whl (101 kB)
     |████████████████████████████████| 101 kB 4.4 MB/s
Collecting django-statici18n
  Downloading https://www.piwheels.org/simple/django-statici18n/django_statici18n-2.2.0-py2.py3-none-any.whl (10 kB)
Collecting django-webpack_loader==0.7.0
  Downloading https://www.piwheels.org/simple/django-webpack-loader/django_webpack_loader-0.7.0-py2.py3-none-any.whl (14 kB)
Collecting gunicorn
  Downloading https://www.piwheels.org/simple/gunicorn/gunicorn-20.1.0-py3-none-any.whl (81 kB)
     |████████████████████████████████| 81 kB 11.7 MB/s
Collecting mysqlclient
  Downloading https://www.piwheels.org/simple/mysqlclient/mysqlclient-2.1.0-cp37-cp37m-linux_armv7l.whl (107 kB)
     |████████████████████████████████| 107 kB 4.9 MB/s
Collecting django-picklefield==2.1.1
  Downloading https://www.piwheels.org/simple/django-picklefield/django_picklefield-2.1.1-py2.py3-none-any.whl (9.4 kB)
Collecting openpyxl
  Downloading https://www.piwheels.org/simple/openpyxl/openpyxl-3.0.9-py2.py3-none-any.whl (242 kB)
     |████████████████████████████████| 242 kB 4.5 MB/s
Collecting qrcode
  Downloading https://www.piwheels.org/simple/qrcode/qrcode-7.3.1-py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 10.7 MB/s
Collecting django-formtools==2.3
  Downloading https://www.piwheels.org/simple/django-formtools/django_formtools-2.3-py3-none-any.whl (148 kB)
     |████████████████████████████████| 148 kB 4.0 MB/s
Collecting django-simple-captcha
  Downloading https://www.piwheels.org/simple/django-simple-captcha/django_simple_captcha-0.5.14-py2.py3-none-any.whl (93 kB)
     |████████████████████████████████| 93 kB 9.5 MB/s
Collecting djangorestframework==3.11.1
  Downloading https://www.piwheels.org/simple/djangorestframework/djangorestframework-3.11.1-py3-none-any.whl (915 kB)
     |████████████████████████████████| 915 kB 4.7 MB/s
Collecting python-dateutil
  Downloading https://www.piwheels.org/simple/python-dateutil/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB 9.8 MB/s
Collecting requests
  Downloading https://www.piwheels.org/simple/requests/requests-2.26.0-py2.py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 7.4 MB/s
Collecting pillow
  Downloading https://www.piwheels.org/simple/pillow/Pillow-8.4.0-cp37-cp37m-linux_armv7l.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 1.8 MB/s
Collecting pyjwt==2.1.*
  Downloading https://www.piwheels.org/simple/pyjwt/PyJWT-2.1.0-py3-none-any.whl (16 kB)
Collecting pycryptodome==3.12.0
  Downloading https://www.piwheels.org/simple/pycryptodome/pycryptodome-3.12.0-cp35-abi3-linux_armv7l.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB 2.5 MB/s
Collecting requests_oauthlib
  Downloading https://www.piwheels.org/simple/requests-oauthlib/requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting chardet
  Downloading https://www.piwheels.org/simple/chardet/chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 9.9 MB/s
Collecting python-cas
  Downloading https://www.piwheels.org/simple/python-cas/python_cas-1.6.0-py2.py3-none-any.whl (10 kB)
Collecting djangosaml2==0.20.0
  Downloading https://www.piwheels.org/simple/djangosaml2/djangosaml2-0.20.0-py2.py3-none-any.whl (73 kB)
     |████████████████████████████████| 73 kB 7.8 MB/s
Collecting pysaml2==6.5.1
  Downloading https://www.piwheels.org/simple/pysaml2/pysaml2-6.5.1-py2.py3-none-any.whl (401 kB)
     |████████████████████████████████| 401 kB 4.1 MB/s
Collecting cffi==1.14.0
  Downloading https://www.piwheels.org/simple/cffi/cffi-1.14.0-cp37-cp37m-linux_armv7l.whl (344 kB)
     |████████████████████████████████| 344 kB 2.3 MB/s
Collecting cryptography==36.0.0
  Downloading https://www.piwheels.org/simple/cryptography/cryptography-36.0.0-cp37-cp37m-linux_armv7l.whl (2.2 MB)
     |████████████████████████████████| 2.2 MB 2.0 MB/s
Collecting defusedxml~=0.5
  Downloading https://www.piwheels.org/simple/defusedxml/defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting Jinja2~=2.10
  Downloading https://www.piwheels.org/simple/jinja2/Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 3.8 MB/s
Collecting json5~=0.8.5
  Downloading https://www.piwheels.org/simple/json5/json5-0.8.5-py2.py3-none-any.whl (26 kB)
Collecting python-pam~=1.8
  Downloading https://www.piwheels.org/simple/python-pam/python_pam-1.8.4-py2.py3-none-any.whl (5.8 kB)
Collecting PyYAML~=5.1
  Downloading https://www.piwheels.org/simple/pyyaml/PyYAML-5.4.1-cp37-cp37m-linux_armv7l.whl (45 kB)
     |████████████████████████████████| 45 kB 5.8 MB/s
Collecting six~=1.13
  Downloading https://www.piwheels.org/simple/six/six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting lxml
  Downloading https://www.piwheels.org/simple/lxml/lxml-4.7.1-cp37-cp37m-linux_armv7l.whl (6.0 MB)
     |████████████████████████████████| 6.0 MB 2.1 MB/s
Collecting sqlalchemy
  Downloading https://www.piwheels.org/simple/sqlalchemy/SQLAlchemy-1.4.29-cp37-cp37m-linux_armv7l.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 1.9 MB/s
Collecting pytz
  Downloading https://www.piwheels.org/simple/pytz/pytz-2021.3-py3-none-any.whl (511 kB)
     |████████████████████████████████| 511 kB 4.1 MB/s
Collecting asgiref<4,>=3.3.2
  Downloading https://www.piwheels.org/simple/asgiref/asgiref-3.4.1-py3-none-any.whl (25 kB)
Collecting sqlparse>=0.2.2
  Downloading https://www.piwheels.org/simple/sqlparse/sqlparse-0.4.2-py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 304 kB/s
Collecting xmlschema
  Downloading https://www.piwheels.org/simple/xmlschema/xmlschema-1.9.2-py3-none-any.whl (273 kB)
     |████████████████████████████████| 273 kB 2.2 MB/s
Collecting importlib-resources
  Downloading https://www.piwheels.org/simple/importlib-resources/importlib_resources-5.4.0-py3-none-any.whl (28 kB)
Collecting pyOpenSSL
  Downloading https://www.piwheels.org/simple/pyopenssl/pyOpenSSL-21.0.0-py2.py3-none-any.whl (55 kB)
     |████████████████████████████████| 55 kB 3.1 MB/s
Collecting pycparser
  Downloading https://www.piwheels.org/simple/pycparser/pycparser-2.21-py2.py3-none-any.whl (119 kB)
     |████████████████████████████████| 119 kB 5.3 MB/s
Collecting django-appconf>=1.0
  Downloading https://www.piwheels.org/simple/django-appconf/django_appconf-1.0.5-py3-none-any.whl (6.4 kB)
Collecting setuptools>=3.0
  Downloading https://www.piwheels.org/simple/setuptools/setuptools-60.1.0-py3-none-any.whl (952 kB)
     |████████████████████████████████| 952 kB 1.9 MB/s
Collecting et-xmlfile
  Downloading https://www.piwheels.org/simple/et-xmlfile/et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)
Collecting django-ranged-response==0.2.0
  Downloading https://www.piwheels.org/simple/django-ranged-response/django_ranged_response-0.2.0-py3-none-any.whl (3.9 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading https://www.piwheels.org/simple/urllib3/urllib3-1.26.7-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 3.4 MB/s
Collecting certifi>=2017.4.17
  Downloading https://www.piwheels.org/simple/certifi/certifi-2021.10.8-py2.py3-none-any.whl (151 kB)
     |████████████████████████████████| 151 kB 6.7 MB/s
Collecting charset-normalizer~=2.0.0
  Downloading https://www.piwheels.org/simple/charset-normalizer/charset_normalizer-2.0.9-py3-none-any.whl (43 kB)
     |████████████████████████████████| 43 kB 10.2 MB/s
Collecting idna<4,>=2.5
  Downloading https://www.piwheels.org/simple/idna/idna-3.3-py3-none-any.whl (64 kB)
     |████████████████████████████████| 64 kB 8.8 MB/s
Collecting oauthlib>=3.0.0
  Downloading https://www.piwheels.org/simple/oauthlib/oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
     |████████████████████████████████| 146 kB 3.9 MB/s
Collecting MarkupSafe>=0.23
  Downloading https://www.piwheels.org/simple/markupsafe/MarkupSafe-2.0.1-cp37-cp37m-linux_armv7l.whl (31 kB)
Collecting importlib-metadata
  Downloading https://www.piwheels.org/simple/importlib-metadata/importlib_metadata-4.10.0-py3-none-any.whl (17 kB)
Collecting typing-extensions
  Downloading https://www.piwheels.org/simple/typing-extensions/typing_extensions-4.0.1-py3-none-any.whl (22 kB)
Collecting zipp>=0.5
  Downloading https://www.piwheels.org/simple/zipp/zipp-3.6.0-py3-none-any.whl (5.3 kB)
Collecting elementpath<3.0.0,>=2.4.0
  Downloading https://www.piwheels.org/simple/elementpath/elementpath-2.4.0-py3-none-any.whl (163 kB)
     |████████████████████████████████| 163 kB 1.8 MB/s
Installing collected packages: pycparser, typing-extensions, cffi, zipp, urllib3, sqlparse, six, pytz, idna, elementpath, cryptography, charset-normalizer, certifi, asgiref, xmlschema, requests, python-dateutil, pyOpenSSL, importlib-resources, Django, defusedxml, setuptools, pysaml2, pillow, oauthlib, MarkupSafe, lxml, importlib-metadata, et-xmlfile, django-ranged-response, django-appconf, sqlalchemy, requests-oauthlib, qrcode, PyYAML, python-pam, python-cas, pyjwt, pycryptodome, openpyxl, mysqlclient, json5, Jinja2, gunicorn, future, djangosaml2, djangorestframework, django-webpack-loader, django-statici18n, django-simple-captcha, django-picklefield, django-formtools, chardet, captcha
Successfully installed Django-3.2.6 Jinja2-2.11.3 MarkupSafe-2.0.1 PyYAML-5.4.1 asgiref-3.4.1 captcha-0.3 certifi-2021.10.8 cffi-1.14.0 chardet-4.0.0 charset-normalizer-2.0.9 cryptography-36.0.0 defusedxml-0.7.1 django-appconf-1.0.5 django-formtools-2.3 django-picklefield-2.1.1 django-ranged-response-0.2.0 django-simple-captcha-0.5.14 django-statici18n-2.2.0 django-webpack-loader-0.7.0 djangorestframework-3.11.1 djangosaml2-0.20.0 elementpath-2.4.0 et-xmlfile-1.1.0 future-0.18.2 gunicorn-20.1.0 idna-3.3 importlib-metadata-4.10.0 importlib-resources-5.4.0 json5-0.8.5 lxml-4.7.1 mysqlclient-2.1.0 oauthlib-3.1.1 openpyxl-3.0.9 pillow-8.4.0 pyOpenSSL-21.0.0 pycparser-2.21 pycryptodome-3.12.0 pyjwt-2.1.0 pysaml2-6.5.1 python-cas-1.6.0 python-dateutil-2.8.2 python-pam-1.8.4 pytz-2021.3 qrcode-7.3.1 requests-2.26.0 requests-oauthlib-1.3.0 setuptools-60.1.0 six-1.16.0 sqlalchemy-1.4.29 sqlparse-0.4.2 typing-extensions-4.0.1 urllib3-1.26.7 xmlschema-1.9.2 zipp-3.6.0

The build command:

seafile@rpi-buster:~$ ./build.sh -D -A -v 9.0.2 -h https://raw.githubusercontent.com/jobenvil/rpi-build-seafile/main/seahub_requirements_v9.0.2.txt

See seahub_requirements_v9.0.2.txt

@nihilistaX
Copy link
Author 


To sum up: It is unsual that the build works with older `Go` libraries but not with newer (you installed go1.14 with gvm). @nihilistaX Could you please try to append the `buster-backports` repository and try again the build and having the `cryptography==36.0.0`? Or maybe not, since you will trigger an update of other packages and you maybe don't wanna do that. I prefer always to enable "backports" since debian is per se not having the high edge packages. But this is up to you.

I think i have installed from backports, there are some different versions, but that didn't wortk for me. Maybe i did someting wrong, but with gvm it was really easy and that works.

@jobenvil
Copy link
Member

FWIW golang in buster-backports is equal to 1.15 as indicated here.

@nihilistaX
Copy link
Author

nihilistaX commented Dec 28, 2021 •
edited
Loading

FWIW golang in buster-backports is equal to 1.15 as indicated here.

You are right, but i have installed 1.14 (and tried with 1.15 too) from backports, and that didn't work, so i installled 1.14 with gvm, what works. Don't know why it didn't with golang from backports repo.

Edit: OK, i tried on my second device, with go1.15 from backports and this is what i get:
./merge.go:241:17: invalid operation: 1 << i (shift count type int, must be unsigned integer) note: module requires Go 1.14error: Build seafile-server (go_fileserver) failed

Edit1: Now i have installed 3 versions of go with apt, 1.11, 1.14 + 1.15. Found this about managing multiple versions and switched to 1.14 with update-alternatives --config go.........and voila, no error.
But then later: [ERROR] failed to copy /root/build/seafile-rpi/built-seafile-sources/R9.0.2/fileserver to /tmp/seafile-server-build/seafile-server/seafile/bin: [Errno 2] No such file or directory: '/root/build/seafile-rpi/built-seafile-sources/R9.0.2/fileserver'

Edit2: Switched to go 1.15, missing 'fileserver' now is in place and build was OK. Don't know why build on my other device with gvm/go 1.14 was completed with no error....

@jobenvil
Copy link
Member

jobenvil commented Dec 29, 2021 •
edited
Loading

@nihilistaX Thanks for your comments and your honesty.

@Gustl22 I made some research building the seahub_thirdparts python modules (requirements.txt) and this is the working outcome:

#Django==3.2.6
Django<4
future
captcha
django-statici18n
django-webpack_loader==0.7.0
#django-webpack_loader>=0.7.0 tried but failed
gunicorn
mysqlclient
#django-picklefield==2.1.1
django-picklefield>=2.1.1
openpyxl
qrcode
#django-formtools==2.3
django-formtools>=2.3
django-simple-captcha
#djangorestframework==3.11.1
djangorestframework>=3.11.1
python-dateutil
requests
pillow
#pyjwt==2.1.*
pyjwt>=2.1.*
#pycryptodome==3.12.0
pycryptodome>=3.12.0
requests_oauthlib
chardet
python-cas
#djangosaml2==0.20.0
djangosaml2>=0.20.0
#pysaml2==6.5.1
pysaml2>=6.5.1
#cffi==1.14.0
cffi>=1.14.0
# input from @nihilistaX see https://github.com/haiwen/seafile-rpi/issues/99
cryptography==36.0.0

The commented lines are the original from the official seahub repository. The uncommented one are the adapted by me, which I tested for build success. Here few comments why I tried to do so:

  1. Original requirement Django==3.2.6 has the vulnerability CVE-2021-44420. Django>4 breaks seahub, since the django-admin.py issue. We stick less than version 4. The Django matched version is 3.2.10 and is no vulnerable. In January 2022 will be the new 3.2.11.
  2. django-webpack_loader==0.7.0 This still must be true. If we update to higher versions (>1). Seahub breaks.
  3. django-picklefield>=2.1.1, django-formtools>=2.3, djangorestframework>=3.11.1, pyjwt>=2.1.*, pycryptodome>=3.12.0, djangosaml2>=0.20.0, pysaml2>=6.5.1, cffi>=1.14.0 could use newer version than the official one (theoretically) and I used them succesfully for my tests. The properly approach should be to use newer module versions if it doesn't break seahub. Unfortunatedly I don't have a CI/CD pipeline to check every seahub thirdpart module against his function inside seahub. Therefore "theoretically".

Edit: Here the updated versions of the modules, which were sucesfully built and which I'm using atm for v9.0.2:

Successfully installed Django-3.2.10 Jinja2-2.11.3 MarkupSafe-2.0.1 PyYAML-5.4.1 asgiref-3.4.1 captcha-0.3 certifi-2021.10.8 cffi-1.15.0 chardet-4.0.0 charset-normalizer-2.0.9 cryptography-36.0.0 defusedxml-0.7.1 django-appconf-1.0.5 django-formtools-2.3 django-picklefield-3.0.1 django-ranged-response-0.2.0 django-simple-captcha-0.5.14 django-statici18n-2.2.0 django-webpack-loader-0.7.0 djangorestframework-3.13.1 djangosaml2-1.3.5 elementpath-2.4.0 et-xmlfile-1.1.0 future-0.18.2 gunicorn-20.1.0 idna-3.3 importlib-resources-5.4.0 json5-0.8.5 lxml-4.7.1 mysqlclient-2.1.0 oauthlib-3.1.1 openpyxl-3.0.9 pillow-8.4.0 pyOpenSSL-21.0.0 pycparser-2.21 pycryptodome-3.12.0 pyjwt-2.3.0 pysaml2-7.1.0 python-cas-1.6.0 python-dateutil-2.8.2 python-pam-1.8.4 pytz-2021.3 qrcode-7.3.1 requests-2.26.0 requests-oauthlib-1.3.0 setuptools-60.1.1 six-1.16.0 sqlalchemy-1.4.29 sqlparse-0.4.2 urllib3-1.26.7 xmlschema-1.9.2 zipp-3.6.0

@jobenvil jobenvil pinned this issue Jan 7, 2022
Gustl22 added a commit to Gustl22/seafile-rpi that referenced this issue Jun 29, 2022
@Gustl22
feat: Add custom seahub and seafdav requirements (closes haiwen#99)
ba65075
@ChatDeBlofeld ChatDeBlofeld mentioned this issue Mar 18, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jobenvil @nihilistaX @Gustl22