-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate to Tebako packager #126
Conversation
@maxirmx here's the real error from https://github.com/metanorma/packed-mn/runs/4419965948?check_suite_focus=true:
|
Thank you, @ronaldtse !
I can see that the file is in memfs but ruby cannot find it so it shall be an issue with libdwarfs-wr. I think I would rather test it bottom up: |
Also this is the usual cost payed by the people who try to cut the corners. |
Usually it's better to pay the price early on rather than run into a crash later. Agree 😉 |
ce4c512
to
35183e5
Compare
Strange to see usage of so much swap... how are we loading the DwarFS image in the binary? |
It happens prior to loading image to the binary. Memory is required for mkdwarfs to create compressed image. Libzstd We are loading by INCBIN but it is not related |
Isn't it strange to require 25GB of swap for an image of few hundred MBs...? |
Actually it worked with 17 gb |
https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md Note memory limit option. |
Is the large memory size limit just used for read ahead purposes? If we specify a lower limit for memory we should still be able to package the image right? ruby-packer has the 15GB swap issue because it is trying to serialize the binary package as a C array text file... |
Thye large memory size can be explained by the history of dwarfs and explanations of the author
I believe he keeps the whole copy of filesystem to be packages in memory in order to look for duplicates, similar box, optimize cross-page boundaries and do other similar things. It is reasonable if you consider 300 sequential versions of something liek perl (or ruby ) |
Further he adds:
Sure, advanced features of C++ are very cool. However, it is insidious. There is high risk that you the program be creating several copies of data unintentionally. I think it may be the case. Not only the full filesystem to be packaged, but several copies of it are loaded to memory. |
@ronaldtse @CAMOBAP @opoudjis Thank you |
Not a problem from my point of view @maxirmx I have two questions sofar:
|
Actually it is sh script already. Everything else is about prerequisites. We can probably use some package mamger to make one-liner. There is a brief document here: |
I have preliminary estimates only. I think I will be able to have MacOS version of the same functionality in approx 2 weeks. Windows ~ 6 weeks, but it is very prilimiary and shaky estimate. I did not start either of them yet. |
here is a brief document here:
No, the status today is as follows:
https://github.com/wheybags/glibc_version_header But it wil be one more hack that is hard to estimate. |
Not sure why macOS build is failing here:
But hopefully we are able to migrate macOS to Tebako soon. |
@ronaldtse |
Good find. I wonder why that is the case... seems that the name got an additional prefix. |
@maxirmx could you help resolve the merge conflicts and merge this PR? Thanks! |
|
Thanks @maxirmx ! Now, technically, the three remaining failures are due to metanorma/isodoc#367. This is a bug in the metanorma-iso gem, that requires the “sassc” gem. However, it would be useful if we can package the “sassc” gem as well so that users can use custom SASS/SCSS files. Let me make that a separate issue. |
Metanorma PR checklist