Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/stream request python engine #1829

Merged
merged 56 commits into from
Dec 31, 2024
Merged
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
21cde00
chore: add document
namchuai Dec 1, 2024
bd1bf91
feat: update engine interface
namchuai Dec 2, 2024
08fbb8a
chore: add document
namchuai Dec 1, 2024
7d9cf3b
feat: update engine interface
namchuai Dec 2, 2024
3548342
Feat: init python engine
nguyenhoangthuan99 Dec 10, 2024
8486533
Merge branch 'j/engine-improvement' into feat/python-engine
nguyenhoangthuan99 Dec 10, 2024
6958db8
Fix: conflict
nguyenhoangthuan99 Dec 10, 2024
ff2c02d
feat: add python engine implementation
nguyenhoangthuan99 Dec 11, 2024
52d8105
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 11, 2024
accec0a
Fix: CI build window
nguyenhoangthuan99 Dec 11, 2024
e0f75ad
Merge branch 'feat/python-engine' of github.com:janhq/cortex.cpp into…
nguyenhoangthuan99 Dec 11, 2024
6a8bebf
Fix: CI build window
nguyenhoangthuan99 Dec 11, 2024
36f29bf
feat: support download python model from cortexso
nguyenhoangthuan99 Dec 12, 2024
6e3965c
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 12, 2024
10d53a1
feat: add inference interface
nguyenhoangthuan99 Dec 12, 2024
389dd88
feat: integrate to cortex cpp
nguyenhoangthuan99 Dec 14, 2024
d15351f
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 14, 2024
860c2c3
merge dev
nguyenhoangthuan99 Dec 14, 2024
e6324c2
fix: remove pythone engine load engine option
nguyenhoangthuan99 Dec 16, 2024
8bdca01
Merge branch 'feat/python-engine' of github.com:janhq/cortex.cpp into…
nguyenhoangthuan99 Dec 16, 2024
3838a36
Feat: init environment interface
nguyenhoangthuan99 Dec 16, 2024
34237d6
feat: move virtual environment inside model
nguyenhoangthuan99 Dec 17, 2024
56a4f74
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 18, 2024
7ce7eb7
Update CMakeLists.txt
nguyenhoangthuan99 Dec 18, 2024
c2b1118
Update CMakeLists.txt
nguyenhoangthuan99 Dec 18, 2024
7f9ded0
fix: CI build
nguyenhoangthuan99 Dec 18, 2024
27d5097
fix: move log of python to cortex logs folder
nguyenhoangthuan99 Dec 19, 2024
f95cfef
fix: unitest for remote engine because change location of template re…
nguyenhoangthuan99 Dec 19, 2024
2ea032b
fix: CI build windows
nguyenhoangthuan99 Dec 19, 2024
5959980
fix: CI build windows
nguyenhoangthuan99 Dec 19, 2024
0139bbd
Merge branch 'dev' of github.com:janhq/cortex.cpp into feat/python-en…
nguyenhoangthuan99 Dec 19, 2024
09b56ad
feat: add depends model.yml for python engine
nguyenhoangthuan99 Dec 23, 2024
b4c9bb4
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 23, 2024
30b6ae1
Merge branch 'feat/python-engine' of github.com:janhq/cortex.cpp into…
nguyenhoangthuan99 Dec 23, 2024
4fb3688
fix: CI build
nguyenhoangthuan99 Dec 23, 2024
75ad024
merge dev
nguyenhoangthuan99 Dec 26, 2024
2c7756f
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 26, 2024
d525719
stream response
nguyenhoangthuan99 Dec 26, 2024
2bcedf6
update set permission api
nguyenhoangthuan99 Dec 26, 2024
7694e40
Merge branch 'feat/python-engine' of github.com:janhq/cortex.cpp into…
nguyenhoangthuan99 Dec 26, 2024
5ce13f7
Merge branch 'feat/python-engine' into feat/stream-request-python-engine
nguyenhoangthuan99 Dec 27, 2024
4b9e6dc
Fix: comment
nguyenhoangthuan99 Dec 27, 2024
8d1080c
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 27, 2024
b84f04c
Feat: stream response
nguyenhoangthuan99 Dec 27, 2024
839cce4
fix: run concurrent request with stream mode
nguyenhoangthuan99 Dec 28, 2024
2797f1e
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 30, 2024
084c27c
Merge branch 'dev' into feat/python-engine
nguyenhoangthuan99 Dec 30, 2024
035f2d5
Fix: remove unnecessary interface
nguyenhoangthuan99 Dec 30, 2024
dc10a21
Merge branch 'feat/python-engine' of github.com:janhq/cortex.cpp into…
nguyenhoangthuan99 Dec 30, 2024
75625bb
Fix comment
nguyenhoangthuan99 Dec 30, 2024
dabc154
Fix: comment review
nguyenhoangthuan99 Dec 30, 2024
a591be5
Merge branch 'feat/python-engine' into feat/stream-request-python-engine
nguyenhoangthuan99 Dec 31, 2024
4b2f1fc
fix comment
nguyenhoangthuan99 Dec 31, 2024
b14d6bb
Merge branch 'dev' into feat/stream-request-python-engine
nguyenhoangthuan99 Dec 31, 2024
082cfe9
fix comment
nguyenhoangthuan99 Dec 31, 2024
aa42e9d
Merge branch 'feat/stream-request-python-engine' of github.com:janhq/…
nguyenhoangthuan99 Dec 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Feat: init python engine
nguyenhoangthuan99 committed Dec 10, 2024
commit 3548342ebab9cdcc7d4becbb499a95d20298f976
1 change: 1 addition & 0 deletions engine/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -142,6 +142,7 @@ file(APPEND "${CMAKE_CURRENT_BINARY_DIR}/cortex_openapi.h"
add_executable(${TARGET_NAME} main.cc
${CMAKE_CURRENT_SOURCE_DIR}/utils/cpuid/cpu_info.cc
${CMAKE_CURRENT_SOURCE_DIR}/utils/file_logger.cc
${CMAKE_CURRENT_SOURCE_DIR}/extensions/template_renderer.cc
)

target_include_directories(${TARGET_NAME} PRIVATE ${CMAKE_CURRENT_BINARY_DIR})
316 changes: 316 additions & 0 deletions engine/config/model_config.h
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
#pragma once

#include <json/json.h>
#include <yaml-cpp/yaml.h>
#include <fstream>
#include <iostream>
#include <limits>
#include <sstream>
#include <string>
@@ -343,4 +346,317 @@ struct ModelConfig {
}
};

struct Endpoint {
std::string method;
std::string path;
std::string transform_request;
std::string transform_response;
};

struct PythonModelConfig {
// General Metadata
std::string id;
std::string model;
std::string name;
int version;

// Inference Parameters
Endpoint load_model;
Endpoint destroy;
Endpoint inference;
Endpoint heath_check;
std::vector<Endpoint> extra_endpoints;

// Model Load Parameters
int port;
std::string log_path;
std::string log_level;
std::string environments;
std::vector<std::string> command; // New command field
std::string engine;
Json::Value extra_params; // Accept dynamic extra parameters

// Method to convert C++ struct to YAML
std::string ToYaml() const {
YAML::Emitter out;
out << YAML::BeginMap;

out << YAML::Key << "id" << YAML::Value << id;
out << YAML::Key << "model" << YAML::Value << model;
out << YAML::Key << "name" << YAML::Value << name;
out << YAML::Key << "version" << YAML::Value << version;

// Inference Parameters
out << YAML::Key << "load_model" << YAML::Value << YAML::BeginMap;
out << YAML::Key << "method" << YAML::Value << load_model.method;
out << YAML::Key << "path" << YAML::Value << load_model.path;
out << YAML::Key << "transform_request" << YAML::Value
<< load_model.transform_request;
out << YAML::Key << "transform_response" << YAML::Value
<< load_model.transform_response;
out << YAML::EndMap;

out << YAML::Key << "destroy" << YAML::Value << YAML::BeginMap;
out << YAML::Key << "method" << YAML::Value << destroy.method;
out << YAML::Key << "path" << YAML::Value << destroy.path;
out << YAML::EndMap;

out << YAML::Key << "inference" << YAML::Value << YAML::BeginMap;
out << YAML::Key << "method" << YAML::Value << inference.method;
out << YAML::Key << "path" << YAML::Value << inference.path;
out << YAML::EndMap;

out << YAML::Key << "extra_endpoints" << YAML::Value << YAML::BeginSeq;
for (const auto& endpoint : extra_endpoints) {
out << YAML::BeginMap;
out << YAML::Key << "method" << YAML::Value << endpoint.method;
out << YAML::Key << "path" << YAML::Value << endpoint.path;
out << YAML::EndMap;
}
out << YAML::EndSeq;

// Model Load Parameters
out << YAML::Key << "port" << YAML::Value << port;
out << YAML::Key << "log_path" << YAML::Value << log_path;
out << YAML::Key << "log_level" << YAML::Value
<< log_level;
out << YAML::Key << "environments" << YAML::Value
<< environments;

// Serialize command as YAML list
out << YAML::Key << "command" << YAML::Value << YAML::BeginSeq;
for (const auto& cmd : command) {
out << cmd;
}
out << YAML::EndSeq;

out << YAML::Key << "engine" << YAML::Value << engine;

// Serialize extra_params as YAML
out << YAML::Key << "extra_params" << YAML::Value << YAML::BeginMap;
for (Json::ValueConstIterator iter = extra_params.begin();
iter != extra_params.end(); ++iter) {
out << YAML::Key << iter.key().asString() << YAML::Value
<< iter->asString();
}
out << YAML::EndMap;
return out.c_str();
}

// Method to populate struct from YAML file
void ReadFromYaml(const std::string& filePath) {
YAML::Node config = YAML::LoadFile(filePath);

if (config["id"])
id = config["id"].as<std::string>();
if (config["model"])
model = config["model"].as<std::string>();
if (config["name"])
name = config["name"].as<std::string>();
if (config["version"])
version = config["version"].as<int>();

// Inference Parameters

auto ip = config;
if (ip["load_model"]) {
load_model.method =
ip["load_model"]["method"].as<std::string>();
load_model.path =
ip["load_model"]["path"].as<std::string>();
load_model.transform_request =
ip["load_model"]["transform_request"].as<std::string>();
load_model.transform_response =
ip["load_model"]["transform_response"].as<std::string>();
}
if (ip["destroy"]) {
destroy.method =
ip["destroy"]["method"].as<std::string>();
destroy.path =
ip["destroy"]["path"].as<std::string>();
}
if (ip["inference"]) {
inference.method =
ip["inference"]["method"].as<std::string>();
inference.path =
ip["inference"]["path"].as<std::string>();
}
if (ip["extra_endpoints"] && ip["extra_endpoints"].IsSequence()) {
for (const auto& endpoint : ip["extra_endpoints"]) {
Endpoint e;
e.method = endpoint["method"].as<std::string>();
e.path = endpoint["path"].as<std::string>();
extra_endpoints.push_back(e);
}
}


// Model Load Parameters

auto mlp = config;
if (mlp["port"])
port = mlp["port"].as<int>();
if (mlp["log_path"])
log_path = mlp["log_path"].as<std::string>();
if (mlp["log_level"])
log_level = mlp["log_level"].as<std::string>();
if (mlp["environments"])
environments = mlp["environments"].as<std::string>();
if (mlp["engine"])
engine = mlp["engine"].as<std::string>();

if (mlp["command"] && mlp["command"].IsSequence()) {
for (const auto& cmd : mlp["command"]) {
command.push_back(cmd.as<std::string>());
}
}

if (mlp["extra_params"]) {
for (YAML::const_iterator it = mlp["extra_params"].begin();
it != mlp["extra_params"].end(); ++it) {
extra_params[it->first.as<std::string>()] =
it->second.as<std::string>();
}
}

}

// Method to convert the struct to JSON
std::string ToJson() const {
Json::Value root;

root["id"] = id;
root["model"] = model;
root["name"] = name;
root["version"] = version;

// Inference Parameters
root["inference_parameters"]["load_model"]["method"] =
load_model.method;
root["inference_parameters"]["load_model"]["path"] =
load_model.path;
root["inference_parameters"]["load_model"]["transform_request"] =
load_model.transform_request;
root["inference_parameters"]["load_model"]["transform_response"] =
load_model.transform_response;

root["inference_parameters"]["destroy"]["method"] =
destroy.method;
root["inference_parameters"]["destroy"]["path"] =
destroy.path;

root["inference_parameters"]["inference"]["method"] =
inference.method;
root["inference_parameters"]["inference"]["path"] =
inference.path;

for (const auto& endpoint : extra_endpoints) {
Json::Value e;
e["method"] = endpoint.method;
e["path"] = endpoint.path;
root["inference_parameters"]["extra_endpoints"].append(e);
}

// Model Load Parameters
root["model_load_params"]["port"] = port;
root["model_load_params"]["log_path"] = log_path;
root["model_load_params"]["log_level"] = log_level;
root["model_load_params"]["environments"] = environments;

// Serialize command as JSON array
for (const auto& cmd : command) {
root["model_load_params"]["command"].append(cmd);
}

root["model_load_params"]["engine"] = engine;
root["model_load_params"]["extra_params"] =
extra_params; // Serialize the JSON value directly

Json::StreamWriterBuilder writer;
return Json::writeString(writer, root);
}

// Method to populate struct from JSON
void FromJson(const std::string& jsonString) {
Json::CharReaderBuilder reader;
Json::Value root;
std::string errs;
std::istringstream s(jsonString);

if (!Json::parseFromStream(reader, s, &root, &errs)) {
std::cerr << "Error parsing JSON: " << errs << std::endl;
return;
}

if (root.isMember("id"))
id = root["id"].asString();
if (root.isMember("model"))
model = root["model"].asString();
if (root.isMember("name"))
name = root["name"].asString();
if (root.isMember("version"))
version = root["version"].asInt();

// Inference Parameters
if (root.isMember("inference_parameters")) {
const Json::Value& ip = root["inference_parameters"];
if (ip.isMember("load_model")) {
load_model.method =
ip["load_model"]["method"].asString();
load_model.path =
ip["load_model"]["path"].asString();
load_model.transform_request =
ip["load_model"]["transform_request"].asString();
load_model.transform_response =
ip["load_model"]["transform_response"].asString();
}
if (ip.isMember("destroy")) {
destroy.method =
ip["destroy"]["method"].asString();
destroy.path = ip["destroy"]["path"].asString();
}
if (ip.isMember("inference")) {
inference.method =
ip["inference"]["method"].asString();
inference.path =
ip["inference"]["path"].asString();
}
if (ip.isMember("extra_endpoints")) {
for (const auto& endpoint : ip["extra_endpoints"]) {
Endpoint e;
e.method = endpoint["method"].asString();
e.path = endpoint["path"].asString();
extra_endpoints.push_back(e);
}
}
}

// Model Load Parameters
if (root.isMember("model_load_params")) {
const Json::Value& mlp = root["model_load_params"];
if (mlp.isMember("port"))
port = mlp["port"].asInt();
if (mlp.isMember("log_path"))
log_path = mlp["log_path"].asString();
if (mlp.isMember("log_level"))
log_level = mlp["log_level"].asString();
if (mlp.isMember("environments"))
environments = mlp["environments"].asString();
if (mlp.isMember("engine"))
engine = mlp["engine"].asString();

if (mlp.isMember("command")) {
for (const auto& cmd : mlp["command"]) {
command.push_back(cmd.asString());
}
}

if (mlp.isMember("extra_params")) {
extra_params =
mlp["extra_params"]; // Directly assign the JSON value
}
}
}
};

} // namespace config
Empty file.
Empty file.
136 changes: 136 additions & 0 deletions engine/extensions/template_renderer.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
#if defined(_WIN32) || defined(_WIN64)
#define NOMINMAX
#undef min
#undef max
#endif
#include "template_renderer.h"
#include <regex>
#include <stdexcept>
#include "utils/logging_utils.h"
namespace remote_engine {
TemplateRenderer::TemplateRenderer() {
// Configure Inja environment
env_.set_trim_blocks(true);
env_.set_lstrip_blocks(true);

// Add tojson function for all value types
env_.add_callback("tojson", 1, [](inja::Arguments& args) {
if (args.empty()) {
return nlohmann::json(nullptr);
}
const auto& value = *args[0];

if (value.is_string()) {
return nlohmann::json(std::string("\"") + value.get<std::string>() +
"\"");
}
return value;
});
}

std::string TemplateRenderer::Render(const std::string& tmpl,
const Json::Value& data) {
try {
// Convert Json::Value to nlohmann::json
auto json_data = ConvertJsonValue(data);

// Create the input data structure expected by the template
nlohmann::json template_data;
template_data["input_request"] = json_data;

// Debug output
LOG_DEBUG << "Template: " << tmpl;
LOG_DEBUG << "Data: " << template_data.dump(2);

// Render template
std::string result = env_.render(tmpl, template_data);

// Clean up any potential double quotes in JSON strings
result = std::regex_replace(result, std::regex("\\\"\\\""), "\"");

LOG_DEBUG << "Result: " << result;

// Validate JSON
auto parsed = nlohmann::json::parse(result);

return result;
} catch (const std::exception& e) {
LOG_ERROR << "Template rendering failed: " << e.what();
LOG_ERROR << "Template: " << tmpl;
throw std::runtime_error(std::string("Template rendering failed: ") +
e.what());
}
}

nlohmann::json TemplateRenderer::ConvertJsonValue(const Json::Value& input) {
if (input.isNull()) {
return nullptr;
} else if (input.isBool()) {
return input.asBool();
} else if (input.isInt()) {
return input.asInt();
} else if (input.isUInt()) {
return input.asUInt();
} else if (input.isDouble()) {
return input.asDouble();
} else if (input.isString()) {
return input.asString();
} else if (input.isArray()) {
nlohmann::json arr = nlohmann::json::array();
for (const auto& element : input) {
arr.push_back(ConvertJsonValue(element));
}
return arr;
} else if (input.isObject()) {
nlohmann::json obj = nlohmann::json::object();
for (const auto& key : input.getMemberNames()) {
obj[key] = ConvertJsonValue(input[key]);
}
return obj;
}
return nullptr;
}

Json::Value TemplateRenderer::ConvertNlohmannJson(const nlohmann::json& input) {
if (input.is_null()) {
return Json::Value();
} else if (input.is_boolean()) {
return Json::Value(input.get<bool>());
} else if (input.is_number_integer()) {
return Json::Value(input.get<int>());
} else if (input.is_number_unsigned()) {
return Json::Value(input.get<unsigned int>());
} else if (input.is_number_float()) {
return Json::Value(input.get<double>());
} else if (input.is_string()) {
return Json::Value(input.get<std::string>());
} else if (input.is_array()) {
Json::Value arr(Json::arrayValue);
for (const auto& element : input) {
arr.append(ConvertNlohmannJson(element));
}
return arr;
} else if (input.is_object()) {
Json::Value obj(Json::objectValue);
for (auto it = input.begin(); it != input.end(); ++it) {
obj[it.key()] = ConvertNlohmannJson(it.value());
}
return obj;
}
return Json::Value();
}

std::string TemplateRenderer::RenderFile(const std::string& template_path,
const Json::Value& data) {
try {
// Convert Json::Value to nlohmann::json
auto json_data = ConvertJsonValue(data);

// Load and render template
return env_.render_file(template_path, json_data);
} catch (const std::exception& e) {
throw std::runtime_error(std::string("Template file rendering failed: ") +
e.what());
}
}
} // namespace remote_engine
40 changes: 40 additions & 0 deletions engine/extensions/template_renderer.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#pragma once

#include <memory>

#include <string>
#include "json/json.h"
#include "trantor/utils/Logger.h"
// clang-format off
#if defined(_WIN32) || defined(_WIN64)
#define NOMINMAX
#undef min
#undef max
#endif
#include <nlohmann/json.hpp>
#include <inja/inja.hpp>
// clang-format on
namespace remote_engine {
class TemplateRenderer {
public:
TemplateRenderer();
~TemplateRenderer() = default;

// Convert Json::Value to nlohmann::json
static nlohmann::json ConvertJsonValue(const Json::Value& input);

// Convert nlohmann::json to Json::Value
static Json::Value ConvertNlohmannJson(const nlohmann::json& input);

// Render template with data
std::string Render(const std::string& tmpl, const Json::Value& data);

// Load template from file and render
std::string RenderFile(const std::string& template_path,
const Json::Value& data);

private:
inja::Environment env_;
};

} // namespace remote_engine
2 changes: 1 addition & 1 deletion engine/utils/config_yaml_utils.h
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ const std::vector<std::string> kDefaultEnabledOrigins{
"http://localhost:39281", "http://127.0.0.1:39281", "http://0.0.0.0:39281"};
constexpr const auto kDefaultNoProxy = "example.com,::1,localhost,127.0.0.1";
const std::vector<std::string> kDefaultSupportedEngines{
kLlamaEngine, kOnnxEngine, kTrtLlmEngine};
kLlamaEngine, kOnnxEngine, kTrtLlmEngine, kPythonEngine};

struct CortexConfig {
std::string logFolderPath;
1 change: 1 addition & 0 deletions engine/utils/engine_constants.h
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
constexpr const auto kOnnxEngine = "onnxruntime";
constexpr const auto kLlamaEngine = "llama-cpp";
constexpr const auto kTrtLlmEngine = "tensorrt-llm";
constexpr const auto kPythonEngine = "python-engine";

constexpr const auto kOnnxRepo = "cortex.onnx";
constexpr const auto kLlamaRepo = "cortex.llamacpp";
3 changes: 3 additions & 0 deletions engine/utils/environment_constants.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#pragma once

constexpr const auto kWhisperVQEnvironment = "whispervq";
3 changes: 2 additions & 1 deletion engine/vcpkg.json
Original file line number Diff line number Diff line change
@@ -13,6 +13,7 @@
"sqlitecpp",
"trantor",
"indicators",
"lfreist-hwinfo"
"lfreist-hwinfo",
"inja"
]
}