Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis pipe for request open and close #101

Open
jace opened this issue Feb 20, 2017 · 1 comment
Open

Redis pipe for request open and close #101

jace opened this issue Feb 20, 2017 · 1 comment

Comments

@jace
Copy link
Member

jace commented Feb 20, 2017

HasGeek apps now make extensive use of Redis, calling it multiple times in each request. Redis is much faster if we call it just once, pipelining all requests. We use this in Hasjob when retrieving viewcounts, which are stored as hundreds of individual keys (so that each of them expire automatically).

Coaster should provide a pipeline framework that allows multiple sources to pipeline a request and fire them at once, during request construction (usually retrieval) and teardown (usually storage).

The construction phase isn't well defined as our code doesn't operate on the basis of callbacks, but we can assume that for teardown:

  1. Coaster offers a pipe proxy that can be used like this:

    def foo(results):
        pass
    
    with redis_pipe(callback=foo) as pipe:
        pipe.operation()
        …
  2. The pipe proxy (a) passes the calls through to the actual pipe object, and (b) tracks the calls so it knows which result belongs to which caller.

  3. When the pipe is finally executed, results are split into batches and sent to each callback function.

  4. Coaster offers the framework but not the app-level frame, something that creates the stacks and handles the execution. That's up to the app itself or a helper lib like Baseframe.

@jace
Copy link
Member Author

jace commented Jul 29, 2019

Since point 4 is unclear, it'll work like this in an app:

def before_request():
	g.redis_pipe = RedisPipe()  # This creates the pipe

def view():
	with g.redis_pipe(callback=foo) as pipe:
		pipe.operation()

def after_request():
	g.redis_pipe.commit()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant