Various improvements to Serve Request Batching tutorial (ray-project#50400)

## Why are these changes needed?

    * separated inference discussion to make article flow better
    * added discusion on named deployments
    * added recreating new serve handle from different processes
    * added example using curl directly to HTTP endpoint
    * fixed imports on serve API example
    * cleaned up massive block of outputs
    * Added prerequisites section to tutorial
* Clarify that we are teaching two separate options for serving models
    *  Fix orphaned `)`
  
## Related issue number

n/a

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [x] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Ricardo Decal <[email protected]>
Signed-off-by: Ricardo Decal <[email protected]>
Co-authored-by: Edward Oakes <[email protected]>
Co-authored-by: angelinalg <[email protected]>

Author: 3 people

.gitignore

@@ -126,6 +126,7 @@ scripts/nodes.txt
 .idea/**/tasks.xml
 .idea/dictionaries
 .llvm-local.bazelrc
+.aider*
 
 # Sensitive or high-churn files:
 .idea/**/dataSources/

.vale/styles/config/vocabularies/General/accept.txt

@@ -23,5 +23,6 @@ Alibaba
 LSH
 BTS
 [Mm]ultimodal
+Pythonic
 [Gg]rafana
 CLI

doc/source/serve/tutorials/batch.md

@@ -6,17 +6,25 @@ orphan: true
 
 # Serve a Text Generator with Request Batching
 
-This example deploys a simple text generator that takes in
-a batch of queries and processes them at once. In particular, it shows:
+This tutorial shows how to deploy a text generator that processes multiple queries simultaneously using batching. 
+Learn how to:
 
-- How to implement and deploy a Ray Serve deployment that accepts batches.
-- How to configure the batch size.
-- How to query the model in Python.
+- Implement a Ray Serve deployment that handles batched requests
+- Configure and optimize batch processing
+- Query the model from HTTP and Python
 
-This tutorial is a guide for serving online queries when your model can take advantage of batching. For example, linear regressions and neural networks use CPU and GPU's vectorized instructions to perform computation in parallel. Performing inference with batching can increase the *throughput* of the model as well as *utilization* of the hardware.
+Batching can significantly improve performance when your model supports parallel processing like GPU acceleration or vectorized operations.
+It increases both throughput and hardware utilization by processing multiple requests together.
 
-For _offline_ batch inference with large datasets, see [batch inference with Ray Data](batch_inference_home).
+:::{note}
+This tutorial focuses on online serving with batching. For offline batch processing of large datasets, see [batch inference with Ray Data](batch_inference_home).
+:::
+
+## Prerequisites
 
+```python
+pip install "ray[serve] transformers"
+```
 
 ## Define the Deployment
 Open a new Python file called `tutorial_batch.py`. First, import Ray Serve and some other helpers.
@@ -26,41 +34,47 @@ Open a new Python file called `tutorial_batch.py`. First, import Ray Serve and s
 :start-after: __doc_import_begin__
 ```
 
-You can use the `@serve.batch` decorator to annotate a function or a method.
-This annotation automatically causes calls to the function to be batched together.
-The function must handle a list of objects and is called with a single object.
-This function must also be `async def` so that you can handle multiple queries concurrently:
+Ray Serve provides the `@serve.batch` decorator to automatically batch individual requests to
+a function or class method.
+
+The decorated method:
+- Must be `async def` to handle concurrent requests
+- Receives a list of requests to process together
+- Returns a list of results of equal length, one for each request
 
 ```python
 @serve.batch
 async def my_batch_handler(self, requests: List):
-    pass
+    # Process multiple requests together
+    results = []
+    for request in requests:
+        results.append(request)  # processing logic here
+    return results
 ```
 
-The batch handler can then be called from another `async def` method in your deployment.
-These calls together are batched and executed together, but return an individual result as if
-they were a normal function call:
+You can call the batch handler from another `async def` method in your deployment.
+Ray Serve batches and executes these calls together, but returns individual results just like
+normal function calls:
 
 ```python
 class BatchingDeployment:
     @serve.batch
     async def my_batch_handler(self, requests: List):
         results = []
         for request in requests:
-            results.append(request.json())
+            results.append(request.json())  # processing logic here
         return results
 
     async def __call__(self, request):
         return await self.my_batch_handler(request)
 ```
 
 :::{note}
-By default, Ray Serve performs *opportunistic batching*. This means that as
-soon as the batch handler is called, the method is executed without
-waiting for a full batch. If there are more queries available after this call
-finishes, the larger batch may be executed. You can tune this behavior using the
-`batch_wait_timeout_s` option to `@serve.batch` (defaults to 0). Increasing this
-timeout may improve throughput at the cost of latency under low load.
+Ray Serve uses *opportunistic batching* by default - executing requests as 
+soon as they arrive without waiting for a full batch. You can adjust this behavior using 
+`batch_wait_timeout_s` in the `@serve.batch` decorator to trade increased latency
+for increased throughput (defaults to 0). Increasing this value may improve throughput
+at the cost of latency under low load.
 :::
 
 Next, define a deployment that takes in a list of input strings and runs 
@@ -80,113 +94,85 @@ the maximum possible batch size that Ray Serve executes at once.
 :start-after: __doc_deploy_begin__
 ```
 
-## Deploy the Deployment
-Deploy the deployment by running the following through the terminal.
+## Deployment Options
+
+You can deploy your app in two ways:
+
+### Option 1: Deploying with the Serve Command-Line Interface
+```console
+$ serve run tutorial_batch:generator --name "Text-Completion-App"
+```
+
+### Option 2: Deploying with the Python API
+
+Alternatively, you can deploy the app using the Python API using the `serve.run` function. 
+This command returns a handle that you can use to query the deployment.
+
+```python
+from ray.serve.handle import DeploymentHandle
+
+handle: DeploymentHandle = serve.run(generator, name="Text-Completion-App")
+```
+
+You can now use this handle to query the model. See the [Querying the Model](#querying-the-model) section below.
+
+
+## Querying the Model
+
+There are multiple ways to interact with your deployed model:
+
+### 1. Simple HTTP Queries
+For basic testing, use curl:
+
 ```console
-$ serve run tutorial_batch:generator
+$ curl "http://localhost:8000/?text=Once+upon+a+time"
 ```
 
-Define a [Ray remote task](ray-remote-functions) to send queries in
-parallel. While Serve is running, open a separate terminal window, and run the 
-following in an interactive Python shell or a separate Python script:
+### 2. Send HTTP requests in parallel with Ray
+For higher throughput, use [Ray remote tasks](ray-remote-functions) to send parallel requests:
 
 ```python
 import ray
 import requests
-import numpy as np
 
 @ray.remote
 def send_query(text):
-    resp = requests.get("http://localhost:8000/?text={}".format(text))
+    resp = requests.post("http://localhost:8000/", params={"text": text})
     return resp.text
 
-# Use Ray to send all queries in parallel
+# Example batch of queries
 texts = [
     'Once upon a time,',
     'Hi my name is Lewis and I like to',
-    'My name is Mary, and my favorite',
-    'My name is Clara and I am',
-    'My name is Julien and I like to',
-    'Today I accidentally',
-    'My greatest wish is to',
     'In a galaxy far far away',
-    'My best talent is',
 ]
-results = ray.get([send_query.remote(text) for text in texts])
-print("Result returned:", results)
-```
-
-You should get an output like the following. The first batch has a 
-batch size of 1, and the subsequent queries have a batch size of 4. Even though the client script issues each 
-query independently, Ray Serve evaluates them in batches.
-```python
-(pid=...) Our input array has length: 1
-(pid=...) Our input array has length: 4
-(pid=...) Our input array has length: 4
-Result returned: [
-    'Once upon a time, when I got to look at and see the work of my parents (I still can\'t stand them,) they said, "Boys, you\'re going to like it if you\'ll stay away from him or make him look',
-
-    "Hi my name is Lewis and I like to look great. When I'm not playing against, it's when I play my best and always feel most comfortable. I get paid by the same people who make my games, who work hardest for me.", 
-
-    "My name is Mary, and my favorite person in these two universes, the Green Lantern and the Red Lantern, are the same, except they're two of the Green Lanterns, but they also have their own different traits. Now their relationship is known", 
-
-    'My name is Clara and I am married and live in Philadelphia. I am an English language teacher and translator. I am passionate about the issues that have so inspired me and my journey. My story begins with the discovery of my own child having been born', 
 
-    'My name is Julien and I like to travel with my son on vacations... In fact I really prefer to spend more time with my son."\n\nIn 2011, the following year he was diagnosed with terminal Alzheimer\'s disease, and since then,', 
-
-    "Today I accidentally got lost and went on another tour in August. My story was different, but it had so many emotions that it made me happy. I'm proud to still be able to go back to Oregon for work.\n\nFor the longest", 
-
-    'My greatest wish is to return your loved ones to this earth where they can begin their own free and prosperous lives. This is true only on occasion as it is not intended or even encouraged to be so.\n\nThe Gospel of Luke 8:29', 
-
-    'In a galaxy far far away, the most brilliant and powerful beings known would soon enter upon New York, setting out to restore order to the state. When the world turned against them, Darth Vader himself and Obi-Wan Kenobi, along with the Jedi', 
-
-    'My best talent is that I can make a movie with somebody who really has a big and strong voice. I do believe that they would be great writers. I can tell you that to make sure."\n\n\nWith this in mind, "Ghostbusters'
-]
+# Send all queries in parallel
+results = ray.get([send_query.remote(text) for text in texts])
 ```
 
-## Deploy the Deployment using Python API
-If you want to evaluate a whole batch in Python, Ray Serve allows you to send
-queries with the Python API. A batch of queries can either come from the web server
-or the Python API.
-
-To query the deployment with the Python API, use `serve.run()`, which is part
-of the Python API, instead of running `serve run` from the console. Add the following
-to the Python script `tutorial_batch.py`:
+### 3. Sending requests using DeploymentHandle
+For a more Pythonic way to query the model, you can use the deployment handle directly:
 
 ```python
-from ray.serve.handle import DeploymentHandle
-
-handle: DeploymentHandle = serve.run(generator)
-)
-```
-
-Generally, to enqueue a query, you can call `handle.method.remote(data)`. This call 
-immediately returns a `DeploymentResponse`. You can call `.result()` to 
-retrieve the result. Add the following to the same Python script.
+import ray
+from ray import serve
 
-```python
 input_batch = [
     'Once upon a time,',
     'Hi my name is Lewis and I like to',
-    'My name is Mary, and my favorite',
-    'My name is Clara and I am',
-    'My name is Julien and I like to',
-    'Today I accidentally',
-    'My greatest wish is to',
     'In a galaxy far far away',
-    'My best talent is',
 ]
-print("Input batch is", input_batch)
 
-import ray
-responses = [handle.handle_batch.remote(batch) for batch in input_batch]
+# initialize using the 'auto' option to connect to the already-running Ray cluster
+ray.init(address="auto")
+
+handle = serve.get_deployment_handle("BatchTextGenerator", app_name="Text-Completion-App")
+responses = [handle.handle_batch.remote(text) for text in input_batch]
 results = [r.result() for r in responses]
-print("Result batch is", results)
 ```
 
-Finally, run the script.
-```console
-$ python tutorial_batch.py
-```
+## Performance Considerations
 
-You should get an output similar to the previous example.
+- Increase `max_batch_size` if you have sufficient memory and want higher throughput - this may increase latency
+- Increase `batch_wait_timeout_s` if throughput is more important than latency