Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement ftp to send logs from supervisor to client #342

Open
jenny011 opened this issue Dec 19, 2023 · 0 comments
Open

Implement ftp to send logs from supervisor to client #342

jenny011 opened this issue Dec 19, 2023 · 0 comments

Comments

@jenny011
Copy link
Collaborator

In #317 , pipeline logs are not changed and are stored at the supervisor.
We would like to implement proper per-pipeline logging, as well as ftp to send logs from supervisor to client
@jenny011 jenny011 mentioned this issue Dec 19, 2023
Merged
MaxiBoether pushed a commit that referenced this issue Jan 5, 2024
@jenny011
Implement a Modyn Client CLI (#333)
b0dfc47 
This PR implements the modyn client part of #317. 
Summary: One client instance starts one pipeline and polls for pipeline
status until exit.
1. Add a modynclient/ on the same level as modyn/. modynclient/ imports
grpc code from modyn/.
2. Add GetPipelineStatus message to supervisor grpc proto.
3. Add IPC queues to store status messages from PipelineExecutors.
Supervisor keeps three queues per pipeline: pipeline_status,
training_status, evaluation_status. Currently, status messages are added
sequentially. Supervisor keeps those queues separate as a preparation
for overlapping training and evaluation in future optimizations. Upon
one GetPipelineStatusRequest, Supervisor sends back all the status
messages in the queues of the requesting pipeline.
4. Add TrainingStatusReporter and EvaluationStatusReporter to
supervisor. Remove pipeline monitor from supervisor.
5. Move TrainingStatusTracker and EvaluationStatusTracker to client.
Implement enlighten progress counter and status bar at client.
6. Update the Dummy model: run one nn.Linear layer instead of
nn.Identity so that the optimizer (CrossEntropyLoss) accepts the model
output. Input non-batched tensor shape: (1,2).
7. Update supervisor and client integration tests. Both supervisor and
client tests run the same Dummy model pipeline, using a tiny dataset
instead of the images dataset. In client test, run the pipeline through
client entrypoint.
8. Update python to >= 3.11

TODOs
1. Implement ftp in a separate PR in the future. #342 
2. Implement tensorboard in a separate PR in the future. #339 
3. Rethink GRPCHandler wait_for_training_completion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jenny011