使用secretnote查询scql报notebook解析错误

[EtanWatson]

secretnote和scql都是用的p2p模式部署在一个单机上面

[EtanWatson]

[EtanWatson]

看着像是解析notebook的格式有问题

[wangzul]

看着像是解析notebook的格式有问题

通过brokerctl命令可以执行查询语句吗？
brokerctl 使用方法https://www.secretflow.org.cn/zh-CN/docs/scql/0.8.1b1/intro/p2p-tutorial

[EtanWatson]

用客户端提交，报这个错

[wangzul]

用客户端提交，报这个错

方便补全一下输入的命令吗。

[wangzul]

用客户端提交，报这个错

同时我想了解一下，你部署的scql是否是参考的https://www.secretflow.org.cn/zh-CN/docs/scql/0.8.1b1/reference/p2p-deploy-config 文档，如果是的话可以反馈一下使用的版本号。

[EtanWatson]

./brokerctl run "SELECT * from ta" --project-id "demo" --host http://localhost:50050 用这个命令执行的
我直接用快速开始改了一下端口启动的，用docker-compose，进入的是scql-0.8.0b2里面的scql/scql-0.8.0b2/examples/p2p-tutorial里面的yml文件

[EtanWatson]

./brokerctl run "SELECT * from ta" --project-id "demo" --host http://localhost:50050 用这个命令执行的 我直接用快速开始改了一下端口启动的，用docker-compose，进入的是scql-0.8.0b2里面的scql/scql-0.8.0b2/examples/p2p-tutorial里面的yml文件

./brokerctl run "SELECT * from ta" --project-id "demo" --host http://localhost:50050 ----timeout 3 用的是这个

[EtanWatson]

我发现我用客户端提交和secretnote提交broker里面的日志不一样，用客户端提交，broker里面是报错的，用secretnote提交不报错

用brokerctl提交broker日志
2024-07-31 07:31:02.7317 WARNING server.go:131 |GIN|status=400|method=POST|path=/intra/query|ip=10.172.230.193|latency=321.578µs|
2024-07-31 07:33:21.7317 INFO server.go:121 GIN | Start handling request method=POST path=/intra/query client=10.172.230.193
2024-07-31 07:33:21.7317 ERROR common.go:139 |RequestID:|RequestParty:|SessionID:|ActionName:|CostTime:187.955µs|Reason:|ErrorMsg:got unexpected error="proto: (line 1:431): unknown field "enable_session_logger_separation"" while unmarshal proto from content={"header":null, "project_id":"demo", "query":"select * from ta", "debug_opts":{"enable_psi_detail_log":false}, "dry_run":false, "job_config":{"session_expire_seconds":"0", "time_zone":"", "unbalance_psi_ratio_threshold":"0", "unbalance_psi_larger_party_rows_count_threshold":"0", "psi_curve_type":0, "http_max_payload_size":"0", "link_recv_timeout_sec":"0", "link_throttle_window_size":"0", "link_chunked_send_parallel_size":"0", "enable_session_logger_separation":false}}|Request:
2024-07-31 07:33:21.7317 WARNING server.go:131 |GIN|status=400|method=POST|path=/intra/query|ip=10.172.230.193|latency=454.526µs|

用secretnote提交broker日志
2024-07-31 07:42:34.7317 INFO query_handler.go:108 create session 738f643d-4f10-11ef-bdb4-0242c0a89003 with query 'SELECT * from ta' in project demo
2024-07-31 07:42:34.7317 INFO query_handler.go:496 create query runner for job 738f643d-4f10-11ef-bdb4-0242c0a89003
2024-07-31 07:42:34.7317 INFO query_handler.go:501 get source tables [{dbName: tableName:ta dbType:0}] in project demo from storage
2024-07-31 07:42:34.7317 INFO query_handler.go:509 work parties: [alice]; data parties: [alice] for job 738f643d-4f10-11ef-bdb4-0242c0a89003
2024-07-31 07:42:34.7317 INFO checksum.go:128 save local checksum table schema: [139 101 10 115 74 218 159 43 229 206];ccl: [87 33 60 99 161 81 72 170 253 195] for party alice
2024-07-31 07:42:34.7317 INFO query_handler.go:538 distribute query completed for job 738f643d-4f10-11ef-bdb4-0242c0a89003
2024-07-31 07:42:34.7317 INFO query_runner.go:413 Execution Plan:
digraph G {
0 [label="runsql:{in:[],out:[Out:{t_0,t_1,t_2,t_3,},],attr:[sql:select user_credit.ID,user_credit.age,user_credit.credit_rank,user_credit.income from alice.user_credit,table_refs:[alice.user_credit],],party:[alice,]}"]
1 [label="publish:{in:[In:{t_0,t_1,t_2,t_3,},],out:[Out:{t_4,t_5,t_6,t_7,},],attr:[],party:[alice,]}"]
0 -> 1 [label = "t_0:{ID:PRIVATE:STRING}"]
0 -> 1 [label = "t_1:{age:PRIVATE:INT64}"]
0 -> 1 [label = "t_2:{credit_rank:PRIVATE:INT64}"]
0 -> 1 [label = "t_3:{income:PRIVATE:INT64}"]
}

2024-07-31 07:42:34.7317 INFO checksum.go:103 get local checksum table schema: [139 101 10 115 74 218 159 43 229 206];ccl: [87 33 60 99 161 81 72 170 253 195] for party alice
2024-07-31 07:42:34.7317 INFO executor.go:159 |RequestID:|SessionID:738f643d-4f10-11ef-bdb4-0242c0a89003|ActionName:Executor@RunExecutionPlan|CostTime:49.590087ms|Reason:|ErrorMsg:|Request:{"job_params":{"party_code":"alice","parties":[{"code":"alice","name":"alice","host":"engine_alice:8003","public_key":"MCowBQYDK2VwAyEARXHvfPvm4VSl/gvPCkbo/qzJ2XR6BJl1FXL8PCdL6GA="}],"job_id":"738f643d-4f10-11ef-bdb4-0242c0a89003","spu_runtime_cfg":{"protocol":"SEMI2K","field":"FM64"},"link_cfg":{},"psi_cfg":{}},"graph":{"nodes":{"0":{"node_name":"runsql.0","op_type":"RunSQL","outputs":{"Out":{"tensors":[{"name":"demo.ta.ID.0","elem_type":"STRING","option":"REFERENCE","annotation":{"status":"TENSORSTATUS_PRIVATE"},"ref_num":1},{"name":"demo.ta.age.1","elem_type":"INT64","option":"REFERENCE","annotation":{"status":"TENSORSTATUS_PRIVATE"},"ref_num":1},{"name":"demo.ta.credit_rank.2","elem_type":"INT64","option":"REFERENCE","annotation":{"status":"TENSORSTATUS_PRIVATE"},"ref_num":1},{"name":"demo.ta.income.3","elem_type":"INT64","option":"REFERENCE","annotation":{"status":"TENSORSTATUS_PRIVATE"},"ref_num":1}]}},"attributes":{"sql":{"t":{"name":".0","elem_type":"STRING","string_data":["select user_credit.ID,user_credit.age,user_credit.credit_rank,user_credit.income from alice.user_credit"]}},"table_refs":{"t":{"name":".0","shape":{"dim":[{"dim_value":"1"}]},"elem_type":"STRING","string_data":["alice.user_credit"]}}}},"1":{"node_name":"publish.1","op_type":"Publish","inputs":{"In":{"tensors":[{"name":"demo.ta.ID.0","elem_type":"STRING","option":"REFERENCE","annotation":{"status":"TENSORSTATUS_PRIVATE"},"ref_num":1},{"name":"demo.ta.age.1","elem_type":"INT64","option":"REFERENCE","annotation":{"status":"TENSORSTATUS_PRIVATE"},"ref_num":1},{"name":"demo.ta.credit_rank.2","elem_type":"INT64","option":"REFERENCE","annotation":{"status":"TENSORSTATUS_PRIVATE"},"ref_num":1},{"name":"demo.ta.income.3","elem_type":"INT64","option":"REFERENCE","annotation":{"status":"TENSORSTATUS_PRIVATE"},"ref_num":1}]}},"outputs":{"Out":{"tensors":[{"name":"ID.4","elem_type":"STRING","annotation":{"status":"TENSORSTATUS_PRIVATE"},"string_data":["ID"]},{"name":"age.5","elem_type":"STRING","annotation":{"status":"TENSORSTATUS_PRIVATE"},"string_data":["age"]},{"name":"credit_rank.6","elem_type":"STRING","annotation":{"status":"TENSORSTATUS_PRIVATE"},"string_data":["credit_rank"]},{"name":"income.7","elem_type":"STRING","annotation":{"status":"TENSORSTATUS_PRIVATE"},"string_data":["income"]}]}}}},"policy":{"worker_num":1,"subdags":[{"jobs":[{"node_ids":["0"]}]},{"jobs":[{"node_ids":["1"]}]}]},"sub_graph_checksum":"c857950d2f7c9b23c23115d50e8e05b267c238b7fce6b76a7db796debcf0a1ac"},"graph_checksum":{"check_graph_checksum":true,"sub_graph_checksums":{"0":"c857950d2f7c9b23c23115d50e8e05b267c238b7fce6b76a7db796debcf0a1ac"},"whole_graph_checksum":"f2d39ce2f36130a2ff5c2286ab57b9c2b0968ecf6e53e2c286341f35a4ffc71d"},"callback_url":"http://broker_alice:8080/intra/cb/engine"}|PartyCode:alice|Url:https://engine_alice:8003/SCQLEngineService/RunExecutionPlan
2024-07-31 07:42:34.7317 INFO executor.go:100 |RequestID:|SessionID:738f643d-4f10-11ef-bdb4-0242c0a89003|ActionName:EngineStub@RunExecutionPlan|CostTime:50.710091ms|Reason:|ErrorMsg:|Request:
2024-07-31 07:42:34.7317 INFO common.go:141 |RequestID:|RequestParty:|SessionID:|ActionName:Intra@DoQuery|CostTime:63.322069ms|Reason:|ErrorMsg:|Request:project_id:"demo" query:"SELECT * from ta"
2024-07-31 07:42:34.7317 INFO server.go:135 |GIN|status=200|method=POST|path=/intra/query|ip=192.168.144.1|latency=63.57077ms|

用secretnote提交engine日志
2024-07-31 07:51:22.224 [info] [engine_service_impl.cc:RunExecutionPlan:223] [sciengine] Start handling RunExecutionPlan request client=192.168.144.3:58100, request={"job_params":{"party_code":"alice","parties":[{"code":"alice","name":"alice","host":"engine_alice:8003","rank":0,"public_key":"MCowBQYDK2VwAyEARXHvfPvm4VSl/gvPCkbo/qzJ2XR6BJl1FXL8PCdL6GA="}],"job_id":"ade1a586-4f11-11ef-bdb4-0242c0a89003","spu_runtime_cfg":{"protocol":2,"field":2,"fxp_fraction_bits":"0","max_concurrency":0,"enable_action_trace":false,"enable_type_checker":false,"enable_pphlo_trace":false,"enable_runtime_snapshot":false,"snapshot_dump_dir":"","enable_pphlo_profile":false,"enable_hal_profile":false,"public_random_seed":"0","share_max_chunk_size":"0","fxp_div_goldschmidt_iters":"0","fxp_exp_mode":0,"fxp_exp_iters":"0","fxp_log_mode":0,"fxp_log_iters":"0","fxp_log_orders":"0","sigmoid_mode":0,"enable_lower_accuracy_rsqrt":false,"sine_cosine_iters":"0","beaver_type":0,"trunc_allow_msb_error":false,"experimental_disable_mmul_split":false,"experimental_enable_inter_op_par":false,"experimental_enable_intra_op_par":false,"experimental_disable_vectorization":false,"experimental_inter_op_concurrency":"0","experimental_enable_colocated_optimization":false},"time_zone":"","link_cfg":{"link_recv_timeout_sec":"0","link_throttle_window_size":"0","link_chunked_send_parallel_size":"0","http_max_payload_size":"0"},"psi_cfg":{"unbalance_psi_ratio_threshold":"0","unbalance_psi_larger_party_rows_count_threshold":"0","psi_curve_type":0}},"graph":{"nodes":{"0":{"node_name":"runsql.0","op_type":"RunSQL","inputs":{},"outputs":{"Out":{"tensors":[{"name":"demo.ta.ID.0","elem_type":8,"option":1,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":[],"data_validity":[],"ref_num":1},{"name":"demo.ta.age.1","elem_type":4,"option":1,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":[],"data_validity":[],"ref_num":1},{"name":"demo.ta.credit_rank.2","elem_type":4,"option":1,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":[],"data_validity":[],"ref_num":1},{"name":"demo.ta.income.3","elem_type":4,"option":1,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":[],"data_validity":[],"ref_num":1}]}},"attributes":{"sql":{"t":{"name":".0","elem_type":8,"option":0,"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":["select user_credit.ID,user_credit.age,user_credit.credit_rank,user_credit.income from alice.user_credit"],"data_validity":[],"ref_num":0}},"table_refs":{"t":{"name":".0","shape":{"dim":[{"dim_value":"1"}]},"elem_type":8,"option":0,"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":["alice.user_credit"],"data_validity":[],"ref_num":0}}}},"1":{"node_name":"publish.1","op_type":"Publish","inputs":{"In":{"tensors":[{"name":"demo.ta.ID.0","elem_type":8,"option":1,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":[],"data_validity":[],"ref_num":1},{"name":"demo.ta.age.1","elem_type":4,"option":1,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":[],"data_validity":[],"ref_num":1},{"name":"demo.ta.credit_rank.2","elem_type":4,"option":1,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":[],"data_validity":[],"ref_num":1},{"name":"demo.ta.income.3","elem_type":4,"option":1,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":[],"data_validity":[],"ref_num":1}]}},"outputs":{"Out":{"tensors":[{"name":"ID.4","elem_type":8,"option":0,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":["ID"],"data_validity":[],"ref_num":0},{"name":"age.5","elem_type":8,"option":0,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":["age"],"data_validity":[],"ref_num":0},{"name":"credit_rank.6","elem_type":8,"option":0,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":["credit_rank"],"data_validity":[],"ref_num":0},{"name":"income.7","elem_type":8,"option":0,"annotation":{"status":1},"int32_data":[],"int64_data":[],"float_data":[],"double_data":[],"bool_data":[],"string_data":["income"],"data_validity":[],"ref_num":0}]}},"attributes":{}}},"policy":{"worker_num":1,"subdags":[{"jobs":[{"worker_id":0,"node_ids":["0"]}],"need_call_barrier_after_jobs":false},{"jobs":[{"worker_id":0,"node_ids":["1"]}],"need_call_barrier_after_jobs":false}]},"sub_graph_checksum":"c857950d2f7c9b23c23115d50e8e05b267c238b7fce6b76a7db796debcf0a1ac"},"graph_checksum":{"check_graph_checksum":true,"sub_graph_checksums":{"0":"c857950d2f7c9b23c23115d50e8e05b267c238b7fce6b76a7db796debcf0a1ac"},"whole_graph_checksum":"f2d39ce2f36130a2ff5c2286ab57b9c2b0968ecf6e53e2c286341f35a4ffc71d"},"async":false,"callback_url":"http://broker_alice:8080/intra/cb/engine"}
2024-07-31 07:51:22.224 [info] [session_manager.cc:CreateSession:144] [sciengine] create session(ade1a586-4f11-11ef-bdb4-0242c0a89003) succ, cost(0ms), current running session=1
2024-07-31 07:51:22.225 [info] [engine_service_impl.cc:CheckGraphChecksum:171] [sciengine] skip checking graph checksum
2024-07-31 07:51:22.225 [info] [engine_service_impl.cc:RunPlanCore:448] [sciengine] session(ade1a586-4f11-11ef-bdb4-0242c0a89003) start to execute node(runsql.0), op(RunSQL)
2024-07-31 07:51:22.254 [info] [run_sql.cc:Execute:74] [sciengine] get result row=19, column=4
2024-07-31 07:51:22.254 [info] [engine_service_impl.cc:RunPlanCore:460] [sciengine] session(ade1a586-4f11-11ef-bdb4-0242c0a89003) finished executing node(runsql.0), op(RunSQL), cost(29)ms
2024-07-31 07:51:22.254 [info] [engine_service_impl.cc:RunPlanCore:448] [sciengine] session(ade1a586-4f11-11ef-bdb4-0242c0a89003) start to execute node(publish.1), op(Publish)
RecordPublishNodeDetail Start
2024-07-31 07:51:22.255 [info] [session.cc:DelTensors:291] [sciengine] remove tensor demo.ta.ID.0
2024-07-31 07:51:22.255 [info] [session.cc:DelTensors:291] [sciengine] remove tensor demo.ta.age.1
2024-07-31 07:51:22.255 [info] [session.cc:DelTensors:291] [sciengine] remove tensor demo.ta.credit_rank.2
2024-07-31 07:51:22.255 [info] [session.cc:DelTensors:291] [sciengine] remove tensor demo.ta.income.3
2024-07-31 07:51:22.255 [info] [engine_service_impl.cc:RunPlanCore:460] [sciengine] session(ade1a586-4f11-11ef-bdb4-0242c0a89003) finished executing node(publish.1), op(Publish), cost(0)ms
2024-07-31 07:51:22.255 [info] [engine_service_impl.cc:RunPlanCore:489] [sciengine] session(ade1a586-4f11-11ef-bdb4-0242c0a89003) run plan policy succ
2024-07-31 07:51:22.255 [info] [engine_service_impl.cc:RunPlanSync:522] [sciengine] RunExecutionPlan success, sessionID=ade1a586-4f11-11ef-bdb4-0242c0a89003
2024-07-31 07:51:22.255 [info] [session_manager.cc:RemoveSession:228] [sciengine] session(ade1a586-4f11-11ef-bdb4-0242c0a89003) removed, running_cost(30ms), current running session=0
2024-07-31 07:51:22.257 [info] [engine_service_impl.cc:StopJob:149] [sciengine] EngineServiceImpl::StopJob(ade1a586-4f11-11ef-bdb4-0242c0a89003), reason()
2024-07-31 07:51:22.257 [warning] [session_manager.cc:StopSession:176] [sciengine] session(ade1a586-4f11-11ef-bdb4-0242c0a89003) not exists.

[wangzul]

./brokerctl run "SELECT * from ta" --project-id "demo" --host http://localhost:50050 用这个命令执行的 我直接用快速开始改了一下端口启动的，用docker-compose，进入的是scql-0.8.0b2里面的scql/scql-0.8.0b2/examples/p2p-tutorial里面的yml文件

./brokerctl run "SELECT * from ta" --project-id "demo" --host http://localhost:50050 ----timeout 3 用的是这个

你确定一下是否漏配置了，我根据文档跑了一下是正常的。

[EtanWatson]

我把scql换了一个最新的镜像，用客户端brokerctl可以正常访问了，但是secretnote还是不行，我用的是secretnote:1.6.1-amd64版本的，我看最新有一个1.6.1的镜像，我更新下镜像试一下

[EtanWatson]

果然换了镜像就好了，谢谢啦