Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide stand-alone pbench server container for external integration #3081

Merged
merged 12 commits into from
Jan 5, 2023

Conversation

portante
Copy link
Member

@portante portante commented Nov 19, 2022

Provide a stand-alone pbench server container for external integration.

A number of the previous commits in this PR were pulled out into separate PRs, most notably PR2 #3108, #3112, #3130, #3131, #3132, and #3133.

Below are the individual itemized commits we are working with to finalize stand-alone Pbench Server container image:

  1. For now OIDC server is optional
  2. Remove use of unused default-host
    Remove use of unused `default-host` and anything that depended on it.
    This allows us to avoid fixing up the pbench server configuration file
    for the in-a-can server container.
    
    We also make a change to always copy the HTTPD pbench server conf file
    into place, but only fix it up with the right host name for the in-a-can
    environment.
    
  3. Add rsyslog for observability
  4. Do not be so aggressive and run cleanup on every ERRor of a shell command
  5. Remove redundant PB_POD_NAME setting
  6. Bring jenkins/runlocal inline with CI pipeline
  7. Use mod_logio formatting for TTFB, in, and out
    If we have the module, mod_logio, enhance the HTTPD access logs to emit
    time-to-first-byte sent in microseconds, # of bytes received (including
    request and headers), and # of bytes sent.
    
  8. Replace Apache2 with Nginx
    We also move the document root for serving files to be:
    
        /srv/pbench/public_html
    
    This has a number of advantages.  It gives us one external mount point
    for the production deployment, so that the dashboard is served from the
    same volume.  It also means the `404.html` and `50x.html` pages that
    are referenced in the Nginx configuration can eventually provided by
    the dashboard.
    
    We can also take advantage of a `maintenance.json` "page", where the
    presence of that file will cause Nginx to server a 597 code, and its
    JSON payload will be pulled from that file.
    
    Finally, this allows us to provide a method for serving the old legacy
    pages for the Agent generated HTML.
    

@portante portante added bug Server packaging Issues related to software packaging Containerization Of and relating to the process of setting up and maintaining container images labels Nov 19, 2022
@portante portante added this to the v0.72 milestone Nov 19, 2022
@portante portante self-assigned this Nov 19, 2022
Copy link
Member

@webbnh webbnh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a few small things to attend to.

server/containers/images/.gitignore Outdated Show resolved Hide resolved
server/pbenchinacan/container-build.sh Outdated Show resolved Hide resolved
server/pbenchinacan/pod-nodata.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pod-nodata.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pbench-server-only.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pbench-server-only.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pbench-server-only.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pbench-server-only.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pbench-server-only.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pbench-server-only.yml Outdated Show resolved Hide resolved
@portante portante force-pushed the PBENCH-944 branch 3 times, most recently from 8ef5b98 to e744ecc Compare November 22, 2022 16:08
Copy link
Member

@webbnh webbnh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few concerns:

  • I think it is premature to be removing support from the CI for using both internal and external container image registries.
  • I'm not sure how the container will function without the Pip installs.
  • I don't think that combining the various pod spec YAML files is actually producing a benefit -- I think the proliferation of parameters makes it unusable for humans without a wrapper script (which isn't included in this PR). (I think it would probably be better to just delete the previous pod.yml file, as I don't think we're going to require a pod with pre-loaded data.)

And, there are some other smaller items and questions.

jenkins/Pipeline.gy Outdated Show resolved Hide resolved
server/pbenchinacan/container-build-sa.sh Outdated Show resolved Hide resolved
server/pbenchinacan/container-build-ci.sh Outdated Show resolved Hide resolved
server/pbenchinacan/container-build-ci.sh Outdated Show resolved Hide resolved
server/pbenchinacan/container-build-ci.sh Outdated Show resolved Hide resolved
server/pbenchinacan/container-build-ci.sh Outdated Show resolved Hide resolved
server/pbenchinacan/pod.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pod.yml Outdated Show resolved Hide resolved
server/pbenchinacan/pod.yml Outdated Show resolved Hide resolved
server/pbenchinacan/container-build-sa.sh Outdated Show resolved Hide resolved
server/pbenchinacan/pod.yml Outdated Show resolved Hide resolved
@portante
Copy link
Member Author

FWIW, I've pulled a few of these commits off into PR #3087 since they are independent of the direction suggested by this PR.

jenkins/run-server-func-tests Outdated Show resolved Hide resolved
jenkins/run-server-func-tests Outdated Show resolved Hide resolved
jenkins/run-server-func-tests Outdated Show resolved Hide resolved
jenkins/run-server-func-tests Outdated Show resolved Hide resolved
jenkins/run-server-func-tests Outdated Show resolved Hide resolved
jenkins/run-server-func-tests Outdated Show resolved Hide resolved
jenkins/Pipeline.gy Outdated Show resolved Hide resolved
@portante portante force-pushed the PBENCH-944 branch 4 times, most recently from 2e62d8f to 498d267 Compare November 29, 2022 05:50
@lgtm-com

This comment was marked as resolved.

webbnh
webbnh previously approved these changes Jan 4, 2023
Copy link
Member

@webbnh webbnh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Member Author

@portante portante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe I have addressed all out-standing review comments. Thank you, everyone!

webbnh
webbnh previously approved these changes Jan 4, 2023
Copy link
Member

@webbnh webbnh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

portante and others added 12 commits January 4, 2023 20:20
Remove use of unused `default-host` and anything that depended on it.
This allows us to avoid fixing up the pbench server configuration file
for the in-a-can server container.

We also make a change to always copy the HTTPD pbench server conf file
into place, but only fix it up with the right host name for the in-a-can
environment.
If we have the module, mod_logio, enhance the HTTPD access logs to emit
time-to-first-byte sent in microseconds, # of bytes received (including
request and headers), and # of bytes sent.
We also move the document root for serving files to be:

    /srv/pbench/public_html

This has a number of advantages.  It gives us one external mount point
for the production deployment, so that the dashboard is served from the
same volume.  It also means the `404.html` and `50x.html` pages that
are referenced in the Nginx configuration can eventually provided by
the dashboard.

We can also take advantage of a `maintenance.json` "page", where the
presence of that file will cause Nginx to server a 597 code, and its
JSON payload will be pulled from that file.

Finally, this allows us to provide a method for serving the old legacy
pages for the Agent generated HTML.
Replace the commented out `buildah login` command line with a more
informative description of how to login using an application token.
Copy link
Member

@webbnh webbnh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@portante portante merged commit ab85a91 into distributed-system-analysis:main Jan 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Containerization Of and relating to the process of setting up and maintaining container images packaging Issues related to software packaging Server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants