Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

draft: major refactor #90

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

draft: major refactor #90

wants to merge 7 commits into from

Conversation

jptosso
Copy link
Member

@jptosso jptosso commented Sep 2, 2023

New features

  • Amazing performance improvements:
    • New variable parsing with better performance
    • Logging improvements
    • New cache eviction system
    • New sync.Pools for often used resources
  • Reload system: a HUP signal will safely reload all applications and rules, shutdown will safely finish ongoing transactions
  • Advanced concurrency manager to avoid DOS
  • Switched to zerolog
  • Implements CRS by default
  • More code coverage, the new SPOE engine is easier to test
  • Now you can have multiple configuration files, for example, you can have one application per file inside a directory

TODO:

  • Implement Response processing
  • Implement contextual logging
  • Test configurations merging
  • Implement HUP and KILL signal handlers
  • Implement eviction policy

@jptosso jptosso changed the title major refactor draft: major refactor Sep 2, 2023
@jptosso jptosso marked this pull request as draft September 2, 2023 21:37
return fmt.Sprintf("interrupted with status %d and action %s", e.Interruption.Status, e.Interruption.Action)
}

func (e *ErrInterrupted) Is(target error) bool {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is wrong. Is is not meant to see if two errors are the same, instead check error wrapping.

Copy link
Member Author

@jptosso jptosso Sep 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm playing around with errors.As, apparently that's a shortcut to compare and cast at the same time.
It requires the implementation of Is, and it is working and passing tests

@jptosso jptosso marked this pull request as ready for review September 3, 2023 14:16
@jptosso
Copy link
Member Author

jptosso commented Sep 3, 2023

@sts could you help us fix the e2e for this branch? I'm not sure how it works

const defaultExpire = time.Second * 10
const defaultEvictionInterval = time.Second * 1

func Add(tx types.Transaction, ttl time.Duration) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func Add(tx types.Transaction, ttl time.Duration) {
func Set(tx types.Transaction, ttl time.Duration) {

It will override existing entries with the same ID.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although you are right, this is transparent, as the ID MUST always be unique, by design

// It should only be called when the server is shutting down
// And the SPOE server is already shutdown
// Otherwise the server might crash
func Shutdown() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func Shutdown() {
func RemoveAll() {

A function called Shutdown would imply that after calling it, prevent the usage of the cache. As everything is manipulating a global variable inside this package, I would expose it with the same name as the called Function

"github.com/stretchr/testify/assert"
)

func TestCache(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: please add some newlines between the different stages

"github.com/rs/zerolog"
)

func New(level string, file string) (*zerolog.Logger, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: A package just for setting up zerolog is kinda overkill. I would prefer it to only be configured inside the main

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, originally that was a full logger implementation, but it ended up as a wrapper for zerolog. I'm moving it

url.WriteString(req.Query)
}
tx.ProcessURI(url.String(), req.Method, "HTTP/"+req.Version)
if err := readHeaders(req.Headers, func(key, value string) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also love inlining error checks but with newlines its fairly hard to read. Maybe cant you give it the function directly?

tx.ProcessLogging()
tx.Close()
// TODO does remove forces eviction?
cache.Remove(id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

even if it does force eviction, we maybe want that to prevent the memory filling up

"github.com/negasus/haproxy-spoe-go/message"
)

func unmarshalMessage(msg *message.Message, out interface{}) error {
Copy link
Contributor

@fionera fionera Sep 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please no reflection inside the hotpath, especially when you can hardcode all fields

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fionera I think the second sentence summarizes your needs. Please refrain from using the first sentence in a public place like this 🙏

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed it ❤️


var requestPool = sync.Pool{
New: func() interface{} {
return &applicationRequest{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return &applicationRequest{
return &applicationRequest{}


var responsePool = sync.Pool{
New: func() interface{} {
return &applicationResponse{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return &applicationResponse{
return &applicationResponse{}

"strings"
)

func readHeaders(headers string, callback func(key string, value string)) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just throw this into application.go, as its not complicated/big enough to deserve its own file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants