Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random build failures in CI #119

Open
skalee opened this issue Jan 10, 2019 · 5 comments
Open

Random build failures in CI #119

skalee opened this issue Jan 10, 2019 · 5 comments

Comments

@skalee
Copy link
Contributor

skalee commented Jan 10, 2019

Tests do fail from time to time in CI due to some "bad file descriptor" error (Errno::EBADF). For example:

It is unclear whether this issue is related to current project, or to rspec-pgp_matchers.

@skalee
Copy link
Contributor Author

skalee commented Jan 16, 2019

Handling temporary files in riboseinc/rspec-pgp_matchers has been improved, and it helped a lot. However, the problem is not fully resolved — this failures still happen, though very rarely. See this comment: riboseinc/rspec-pgp_matchers#31 (comment).

@skalee
Copy link
Contributor Author

skalee commented Jun 8, 2019

I become more and more convinced that these issues are caused by some race condition in Ruby's Open3#popen3. This method uses UNIX pipes for capturing standard input and output, and I have a strong suspicion that file descriptors returned from IO.pipe method are not guaranteed to be ready for immediate use.

@skalee
Copy link
Contributor Author

skalee commented Jun 12, 2019

After a lot of thinking I came to conclusion that the current CI setup does not suit us well. I am going to replace it with a Docker image.

  1. Firstly, these race condition issues happen only in Travis CI. I cannot reproduce them elsewhere, and they are almost certainly related to this particular environment. Therefore, running builds in Docker container should help. Also, backtrace inspection clearly indicates that the problem is not a bug in EnMail, but is rather specific to our custom RSpec matchers. So it's not like kicking a can into a long grass.
  2. Secondly, Travis build environment is very difficult to replicate on local computer, which makes build debugging quite difficult. A big advantage of Docker is that images are painless to host on any computer.
  3. Thirdly, although proof-of-concept builds took up to twenty minutes each, that is twice longer than on current master, I expect that this change will eventually result with a substantial speed-up, perhaps by an order of magnitude. This can be achieved by caching prebuilt images (e.g. pushing them to Docker Hub), or by using more recent Linux distributions, which provide more recent versions of required software, so we don't need to build that ourselv
  4. Travis CI environment is capable of running Docker containers. However, we are not bound to that, and can easily move to another provider if we want to.
  5. Finally, I am hoping for easier re-using this Docker images across various Ribose projects.

skalee added a commit that referenced this issue Jun 12, 2019
We hope that switching from Ubuntu to MacOS in Travis CI will solve
strange random failures we are experiencing.

See: #119
@skalee
Copy link
Contributor Author

skalee commented Jun 13, 2019

Meanwhile I have tried with OS X in Travis. I was hoping for an easy fix for this issue as well as for a substantial speed-up (GnuPG and all RNP's dependencies can be installed with Homebrew in no time). Unfortunately, the same file descriptor failure has occurred: https://travis-ci.org/riboseinc/enmail/jobs/545273825.

@ronaldtse
Copy link
Contributor

@skalee any updates in this area? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants