feat: Dynamic Service creation

[holzweber]

What does this PR address?

Fixes #4460

Before submitting:

• Does the Pull Request follow Conventional Commits specification naming? Here are GitHub's
  guide on how to create a pull request.
• Does the code follow BentoML's code style, pre-commit run -a script has passed (instructions)?
• Did you read through contribution guidelines and follow development guidelines?
• [] Did your changes require updates to the documentation? Have you updated
  those accordingly? Here are documentation guidelines and tips on writting docs.
• Did you write tests to cover your changes?

[bojiang]

Hi holzweber. We are deprecating Runner API from 1.2. But I see the values of the dynamic servive example. Would you like update this once we finished the 1.2 DOC?

[bojiang]

Thus may I mark this as merge-hold?

[holzweber]

@bojiang i just checked the new documentation. but i do not know if my idea will work any longer, as i can not access the service object with the new service-annotation idea. i am scared that we really need to generate a pyhton file for dynamic services.. which is kind of an ugly solution...

any other idea how to get this working with 1.2?

[aarnphm]

Dynamic service in 1.2 will still work, as OpenLLM has to do something similar like this.

You can probably hijack directly into the service object, since bentoml.Service will return the new service object, which treats all runnable as a normal python class.

I think one huge difference here is that the lifecycle is just a class, so probably a lot simpler.

[aarnphm]

Will send more example once I finish openllm revamp 😄

[holzweber]

@aarnphm ... Any more updates on this?
I was trying to do something like this, but it does not work as only the last services is in the service list afterwards (based on what i see in openapi)....

def wrap_service_methods(model: bentoml.Model,
                         targets: Any,
                         predict_route: str,
                         predict_name: str,
                         predict_proba_route: str,
                         predict_proba_name: str,
                         ):
    """Wrap models in service methods and annotate as api."""
    @bentoml.api(route=predict_route, name=predict_name)
    async def predict(input_doc: str):
        predictions = await model.predict.async_run([input_doc])
        return {"result": targets[predictions[0]]}

    @bentoml.api(route=predict_proba_route, name=predict_proba_name)
    async def predict_proba(input_doc: str):
        predictions = await model.predict_proba.async_run([input_doc])
        return predictions[0]


    return predict, predict_proba


@bentoml.service(
workers=1, resources={"cpu": "1"}
)
class DynamicService:
    def __init__(self):
        """Nothing to do here."""
        pass
    for idx, available_model in enumerate(bentoml.models.list()):
        if "twenty_news_group" in available_model.tag.name:
            print(f"Creating Endpoint {idx}")
            bento_model = bentoml.sklearn.get(f"{available_model.tag.name}:latest")
            print(bento_model)
            target_names = bento_model.custom_objects["target_names"]
            path_predict = f"predict_model_{idx}"
            path_predict_proba = f"predict_proba_model_{idx}"
            predict, predict_proba = wrap_service_methods(bento_model,
                                                          target_names,
                                                          predict_route=path_predict,
                                                          predict_name=path_predict,
                                                          predict_proba_route=path_predict_proba,
                                                          predict_proba_name=path_predict_proba,
                                                          )

[holzweber]

Edit: i added the methods via the locals() function. can you check and resolve conversation, if this is okey?

[aarnphm]

you can probably use types.new_class here, or even type() to construct subclass.

[holzweber]

true, i tried it with type() and it seems to work. updated my latest push.. so it is visible now :)