部署完第一次执行PSI失败 #189
Comments
|
从日志中获取的job信息为wgjf kubectl get pod -a 查看一下wgjf相关pod信息，然后提供一下kubectl get pod -n xx {name} -oyaml日志 |
kubectl get pod -a 看下有没有pod信息，如果有可以执行kubectl get pod -n xx {name} -oyaml查询一下信息。 |
|
|
可以提供一下下方内容：
|
|
|
看起来是没问题的，执行一下kubectl get kj -n cross-domain wgjf -oyaml |
|
wgjf是很早的job，新的job是vhau： |
|
|
|
|
|
|
尝试重新运行一个新任务，然后获取一下最新的kuscia日志看下 |
|
或者尝试stop 几个暂时不使用的docker 容器【清理一下内存】后在此运行PSI任务。 |
|
停了2个容器后，再执行PSI任务成功了，为什么？是内存资源不足吗？可以通过设置request limit资源限额解决 不通容器之间的资源隔离问题吗 |
|
任务发起节点是，提示错误的节点吗？还是合作方提示的错误。我需要确认一下。 |
|
任务发起节点是提示错误的节点。合作方节点的pod是正常running的，不过双方都是在同一个宿主机，不同的容器。 |
|
重新部署了一次，再次出现第一次PSI卡住。 |
|
重新部署后使用的部署模式有改变吗？ |
|
部署模式没变，还是P2P |
Mem: 31Gi 8.8Gi 5.2Gi 39Mi 17Gi 21Gi
|
|
由于您长时间没有回复，这边先关闭这个问题。后续有其他问题，欢迎再沟通！ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue Type
Running
Have you searched for existing documents and issues?
Yes
OS Platform and Distribution
centos 8
All_in_one Version
v1.11.0
Module type
secretpad
Module version
secretpad:0.12.0b0
What happend and What you expected to happen.
Log output.
bash-5.2# kubectl get kj -A NAMESPACE NAME STARTTIME COMPLETIONTIME LASTRECONCILETIME PHASE bob wgjf cross-domain wgjf 4m54s 4m53s Running bash-5.2# bash-5.2# bash-5.2# kubectl -n cross-domain get kt NAME STARTTIME COMPLETIONTIME LASTRECONCILETIME PHASE wgjf-zispqgwa-node-3 5m16s 2m18s Pending bash-5.2# bash-5.2# kubectl -n cross-domain describe kt wgjf-zispqgwa-node-3 Name: wgjf-zispqgwa-node-3 Namespace: cross-domain Labels: kuscia.secretflow/controller=kuscia-job kuscia.secretflow/job-uid=383d2f6a-cbed-4679-85f7-d5bc5e580026 Annotations: kuscia.secretflow/initiator: alice kuscia.secretflow/interconn-bfia-parties: kuscia.secretflow/interconn-kuscia-parties: bob kuscia.secretflow/interconn-self-parties: alice kuscia.secretflow/job-id: wgjf kuscia.secretflow/self-cluster-as-initiator: true kuscia.secretflow/self-cluster-as-participant: true kuscia.secretflow/task-alias: wgjf-zispqgwa-node-3 API Version: kuscia.secretflow/v1alpha1 Kind: KusciaTask Metadata: Creation Timestamp: 2025-01-03T02:30:13Z Generation: 1 Managed Fields: API Version: kuscia.secretflow/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kuscia.secretflow/initiator: f:kuscia.secretflow/interconn-bfia-parties: f:kuscia.secretflow/interconn-kuscia-parties: f:kuscia.secretflow/interconn-self-parties: f:kuscia.secretflow/job-id: f:kuscia.secretflow/self-cluster-as-initiator: f:kuscia.secretflow/self-cluster-as-participant: f:kuscia.secretflow/task-alias: f:labels: .: f:kuscia.secretflow/controller: f:kuscia.secretflow/job-uid: f:ownerReferences: .: k:{"uid":"383d2f6a-cbed-4679-85f7-d5bc5e580026"}: f:spec: .: f:initiator: f:parties: f:scheduleConfig: f:taskInputConfig: Manager: kuscia Operation: Update Time: 2025-01-03T02:30:13Z API Version: kuscia.secretflow/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:allocatedPorts: f:conditions: f:lastReconcileTime: f:partyTaskStatus: f:phase: f:podStatuses: .: f:alice/wgjf-zispqgwa-node-3-0: .: f:createTime: f:message: f:namespace: f:nodeName: f:podName: f:podPhase: f:reason: f:startTime: f:serviceStatuses: .: f:alice/wgjf-zispqgwa-node-3-0-fed: .: f:createTime: f:namespace: f:portName: f:portNumber: f:readyTime: f:scope: f:serviceName: f:alice/wgjf-zispqgwa-node-3-0-global: .: f:createTime: f:namespace: f:portName: f:portNumber: f:readyTime: f:scope: f:serviceName: f:alice/wgjf-zispqgwa-node-3-0-inference: .: f:createTime: f:namespace: f:portName: f:portNumber: f:readyTime: f:scope: f:serviceName: f:alice/wgjf-zispqgwa-node-3-0-spu: .: f:createTime: f:namespace: f:portName: f:portNumber: f:readyTime: f:scope: f:serviceName: f:startTime: Manager: kuscia Operation: Update Subresource: status Time: 2025-01-03T02:33:11Z Owner References: API Version: kuscia.secretflow/v1alpha1 Block Owner Deletion: true Controller: true Kind: KusciaJob Name: wgjf UID: 383d2f6a-cbed-4679-85f7-d5bc5e580026 Resource Version: 4375 UID: 7bd5c437-eeaf-48c0-b99b-4687af6efe28 Spec: Initiator: alice Parties: App Image Ref: secretflow-image Domain ID: bob Template: Spec: App Image Ref: secretflow-image Domain ID: alice Template: Spec: Schedule Config: Task Input Config: { "sf_datasource_config": { "bob": { "id": "default-data-source" }, "alice": { "id": "default-data-source" } }, "sf_cluster_desc": { "parties": ["bob", "alice"], "devices": [{ "name": "spu", "type": "spu", "parties": ["bob", "alice"], "config": "{\"runtime_config\":{\"protocol\":\"SEMI2K\",\"field\":\"FM128\"},\"link_desc\":{\"connect_retry_times\":60,\"connect_retry_interval_ms\":1000,\"brpc_channel_protocol\":\"http\",\"brpc_channel_connection_type\":\"pooled\",\"recv_timeout_ms\":1200000,\"http_timeout_ms\":1200000}}" }, { "name": "heu", "type": "heu", "parties": ["bob", "alice"], "config": "{\"mode\": \"PHEU\", \"schema\": \"paillier\", \"key_size\": 2048}" }], "ray_fed_config": { "cross_silo_comm_backend": "brpc_link" } }, "sf_node_eval_param": { "domain": "data_prep", "name": "psi", "version": "1.0.0", "attr_paths": ["input/input_ds1/keys", "input/input_ds2/keys", "protocol", "sort_result", "receiver_parties", "allow_empty_result", "join_type", "input_ds1_keys_duplicated", "input_ds2_keys_duplicated"], "attrs": [{ "is_na": false, "ss": ["id1"] }, { "is_na": false, "ss": ["id2"] }, { "is_na": false, "s": "PROTOCOL_RR22" }, { "b": true, "is_na": false }, { "is_na": false, "ss": ["alice", "bob"] }, { "is_na": true }, { "is_na": false, "s": "inner_join" }, { "b": true, "is_na": false }, { "b": true, "is_na": false }], "inputs": [{ "type": "sf.table.individual", "meta": { "@type": "type.googleapis.com/secretflow.spec.v1.IndividualTable", "line_count": "-1" }, "data_refs": [{ "uri": "alice_1462036507.csv", "party": "alice", "format": "csv" }] }, { "type": "sf.table.individual", "meta": { "@type": "type.googleapis.com/secretflow.spec.v1.IndividualTable", "line_count": "-1" }, "data_refs": [{ "uri": "bob_1005531837.csv", "party": "bob", "format": "csv" }] }], "checkpoint_uri": "ckwgjf-zispqgwa-node-3-output-0" }, "sf_output_uris": ["wgjf_zispqgwa_node_3_output_0", "wgjf_zispqgwa_node_3_output_1"], "sf_input_ids": ["waeprdtm", "tcjoqqyl"], "sf_input_partitions_spec": ["", ""], "sf_output_ids": ["wgjf-zispqgwa-node-3-output-0", "wgjf-zispqgwa-node-3-output-1"], "table_attrs": [{ "table_id": "waeprdtm", "column_attrs": [{ "col_name": "id1", "col_type": "feature" }, { "col_name": "age", "col_type": "feature" }, { "col_name": "education", "col_type": "feature" }, { "col_name": "default", "col_type": "feature" }, { "col_name": "balance", "col_type": "feature" }, { "col_name": "housing", "col_type": "feature" }, { "col_name": "loan", "col_type": "feature" }, { "col_name": "day", "col_type": "feature" }, { "col_name": "duration", "col_type": "feature" }, { "col_name": "campaign", "col_type": "feature" }, { "col_name": "pdays", "col_type": "feature" }, { "col_name": "previous", "col_type": "feature" }, { "col_name": "job_blue-collar", "col_type": "feature" }, { "col_name": "job_entrepreneur", "col_type": "feature" }, { "col_name": "job_housemaid", "col_type": "feature" }, { "col_name": "job_management", "col_type": "feature" }, { "col_name": "job_retired", "col_type": "feature" }, { "col_name": "job_self-employed", "col_type": "feature" }, { "col_name": "job_services", "col_type": "feature" }, { "col_name": "job_student", "col_type": "feature" }, { "col_name": "job_technician", "col_type": "feature" }, { "col_name": "job_unemployed", "col_type": "feature" }, { "col_name": "marital_divorced", "col_type": "feature" }, { "col_name": "marital_married", "col_type": "feature" }, { "col_name": "marital_single", "col_type": "feature" }] }, { "table_id": "tcjoqqyl", "column_attrs": [{ "col_name": "id2", "col_type": "feature" }, { "col_name": "contact_cellular", "col_type": "feature" }, { "col_name": "contact_telephone", "col_type": "feature" }, { "col_name": "contact_unknown", "col_type": "feature" }, { "col_name": "month_apr", "col_type": "feature" }, { "col_name": "month_aug", "col_type": "feature" }, { "col_name": "month_dec", "col_type": "feature" }, { "col_name": "month_feb", "col_type": "feature" }, { "col_name": "month_jan", "col_type": "feature" }, { "col_name": "month_jul", "col_type": "feature" }, { "col_name": "month_jun", "col_type": "feature" }, { "col_name": "month_mar", "col_type": "feature" }, { "col_name": "month_may", "col_type": "feature" }, { "col_name": "month_nov", "col_type": "feature" }, { "col_name": "month_oct", "col_type": "feature" }, { "col_name": "month_sep", "col_type": "feature" }, { "col_name": "poutcome_failure", "col_type": "feature" }, { "col_name": "poutcome_other", "col_type": "feature" }, { "col_name": "poutcome_success", "col_type": "feature" }, { "col_name": "poutcome_unknown", "col_type": "feature" }, { "col_name": "y", "col_type": "feature" }] }] } Status: Allocated Ports: Domain ID: alice Named Port: wgjf-zispqgwa-node-3-0/client-server: 23831 wgjf-zispqgwa-node-3-0/fed: 23834 wgjf-zispqgwa-node-3-0/global: 23835 wgjf-zispqgwa-node-3-0/inference: 23832 wgjf-zispqgwa-node-3-0/node-manager: 23836 wgjf-zispqgwa-node-3-0/object-manager: 23830 wgjf-zispqgwa-node-3-0/spu: 23833 Conditions: Last Transition Time: 2025-01-03T02:30:13Z Status: True Type: PortsAllocated Last Transition Time: 2025-01-03T02:30:14Z Status: True Type: ResourceCreated Last Reconcile Time: 2025-01-03T02:33:11Z Party Task Status: Domain ID: bob Phase: Pending Domain ID: alice Phase: Pending Phase: Pending Pod Statuses: alice/wgjf-zispqgwa-node-3-0: Create Time: 2025-01-03T02:30:14Z Message: container[secretflow] waiting state reason: "CreateContainerError", message: "context deadline exceeded" Namespace: alice Node Name: root-kuscia-autonomy-alice-localhost-localdomain Pod Name: wgjf-zispqgwa-node-3-0 Pod Phase: Pending Reason: CreateContainerError Start Time: 2025-01-03T02:30:15Z Service Statuses: alice/wgjf-zispqgwa-node-3-0-fed: Create Time: 2025-01-03T02:30:14Z Namespace: alice Port Name: fed Port Number: 23834 Ready Time: 2025-01-03T02:32:21Z Scope: Cluster Service Name: wgjf-zispqgwa-node-3-0-fed alice/wgjf-zispqgwa-node-3-0-global: Create Time: 2025-01-03T02:30:14Z Namespace: alice Port Name: global Port Number: 23835 Ready Time: 2025-01-03T02:32:21Z Scope: Domain Service Name: wgjf-zispqgwa-node-3-0-global alice/wgjf-zispqgwa-node-3-0-inference: Create Time: 2025-01-03T02:30:14Z Namespace: alice Port Name: inference Port Number: 23832 Ready Time: 2025-01-03T02:32:21Z Scope: Cluster Service Name: wgjf-zispqgwa-node-3-0-inference alice/wgjf-zispqgwa-node-3-0-spu: Create Time: 2025-01-03T02:30:14Z Namespace: alice Port Name: spu Port Number: 23833 Ready Time: 2025-01-03T02:32:21Z Scope: Cluster Service Name: wgjf-zispqgwa-node-3-0-spu Start Time: 2025-01-03T02:30:13Z Events: <none> bash-5.2# bash-5.2# bash-5.2# kubectl -n alice get po NAME READY STATUS RESTARTS AGE dataproxy-alice-6d8899569d-g9t5k 1/1 Running 0 37m wgjf-zispqgwa-node-3-0 0/1 CreateContainerError 0 6m14s bash-5.2# bash-5.2# bash-5.2# kubectl -n alice describe po wgjf-zispqgwa-node-3-0 Name: wgjf-zispqgwa-node-3-0 Namespace: alice Priority: 0 Service Account: default Node: root-kuscia-autonomy-alice-localhost-localdomain/172.20.0.2 Start Time: Fri, 03 Jan 2025 10:30:15 +0800 Labels: kuscia.secretflow/communication-role-client=true kuscia.secretflow/communication-role-server=true kuscia.secretflow/controller=kusciatask kuscia.secretflow/pod-identity=7bd5c437-eeaf-48c0-b99b-4687af6efe28-0 kuscia.secretflow/pod-role= kuscia.secretflow/task-resource-group-uid=39675f44-5793-4494-bf64-277ea6ab7fea kuscia.secretflow/task-resource-uid=cd1e6f2a-05d3-486e-806f-c498fdab023c kuscia.secretflow/task-uid=7bd5c437-eeaf-48c0-b99b-4687af6efe28 Annotations: kuscia.secretflow/config-template-value-cm-name: wgjf-zispqgwa-node-3-kuscia-gen-conf kuscia.secretflow/config-template-volumes: config-template kuscia.secretflow/initiator: alice kuscia.secretflow/task-id: wgjf-zispqgwa-node-3 kuscia.secretflow/task-resource: wgjf-zispqgwa-node-3-d0d2eb4ad63a kuscia.secretflow/task-resource-group: wgjf-zispqgwa-node-3 kuscia.secretflow/taskresource-reserving-timestamp: 2025-01-03T10:30:15+08:00 Status: Pending IP: 10.88.0.3 IPs: IP: 10.88.0.3 Containers: secretflow: Container ID: Image: secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/secretflow-lite-anolis8:1.11.0b1 Image ID: Ports: 23833/TCP, 23834/TCP, 23835/TCP, 23836/TCP, 23830/TCP, 23831/TCP, 23832/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP Command: sh Args: -c python -m secretflow.kuscia.entry ./kuscia/task-config.conf State: Waiting Reason: CreateContainerError Ready: False Restart Count: 0 Environment: KUSCIA_PORT_GLOBAL_NUMBER: 23835 KUSCIA_PORT_NODE_MANAGER_NUMBER: 23836 KUSCIA_PORT_OBJECT_MANAGER_NUMBER: 23830 KUSCIA_PORT_CLIENT_SERVER_NUMBER: 23831 KUSCIA_PORT_INFERENCE_NUMBER: 23832 KUSCIA_PORT_SPU_NUMBER: 23833 KUSCIA_PORT_FED_NUMBER: 23834 Mounts: ./kuscia/task-config.conf from config-template (rw,path="task-config.conf") Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: config-template: Type: ConfigMap (a volume populated by a ConfigMap) Name: wgjf-zispqgwa-node-3-configtemplate Optional: false QoS Class: BestEffort Node-Selectors: kuscia.secretflow/namespace=alice Tolerations: kuscia.secretflow/agent:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 6m26s kuscia-scheduler 0/1 nodes are available: waiting for task resource. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling., can not find related task resource. Normal Scheduled 6m24s kuscia-scheduler Successfully assigned alice/wgjf-zispqgwa-node-3-0 to root-kuscia-autonomy-alice-localhost-localdomain Warning Failed 4m20s Agent Error: context deadline exceeded Warning MissingClusterDNS 4m19s (x3 over 6m25s) Agent pod: "wgjf-zispqgwa-node-3-0_alice(8eec2f78-1cb8-449c-bc01-4a48fa44be67)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. Normal Pulled 4m19s (x2 over 6m20s) Agent Container image "secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/secretflow-lite-anolis8:1.11.0b1" already present on machine Warning Failed 4m17s Agent Error: failed to reserve container name "secretflow_wgjf-zispqgwa-node-3-0_alice_8eec2f78-1cb8-449c-bc01-4a48fa44be67_0": name "secretflow_wgjf-zispqgwa-node-3-0_alice_8eec2f78-1cb8-449c-bc01-4a48fa44be67_0" is reserved for "4dc0b5734c4effa344e54c539a7f3b4a6be24013da99f6c8d9e144be44fdeef1" bash-5.2#The text was updated successfully, but these errors were encountered: