Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smaller job defaults requirements #4785

Merged
merged 3 commits into from
Dec 18, 2024
Merged

smaller job defaults requirements #4785

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e4c3408
smaller job defaults requirements
wdbaruni Dec 18, 2024
11d9d2c
Merge branch 'main' into eng-192-compute-node-fails-to-start-on-small…
wdbaruni Dec 18, 2024
91c40cb
fix swagger
wdbaruni Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions cmd/cli/node/columns.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"fmt"
"strings"

"github.com/c2h5oh/datasize"
"github.com/dustin/go-humanize"
"github.com/jedib0t/go-pretty/v6/table"
"github.com/jedib0t/go-pretty/v6/text"
"github.com/samber/lo"
Expand Down Expand Up @@ -100,13 +100,17 @@ var toggleColumns = map[string][]output.TableColumn[*models.NodeState]{
{
ColumnConfig: table.ColumnConfig{Name: "memory", WidthMax: len("10.0 GB / "), WidthMaxEnforcer: text.WrapSoft},
Value: ifComputeNode(func(cni models.ComputeNodeInfo) string {
return fmt.Sprintf("%s / %s", datasize.ByteSize(cni.AvailableCapacity.Memory).HR(), datasize.ByteSize(cni.MaxCapacity.Memory).HR())
return fmt.Sprintf("%s / %s",
humanize.Bytes(cni.AvailableCapacity.Memory),
humanize.Bytes(cni.MaxCapacity.Memory))
}),
},
{
ColumnConfig: table.ColumnConfig{Name: "disk", WidthMax: len("100.0 GB / "), WidthMaxEnforcer: text.WrapSoft},
Value: ifComputeNode(func(cni models.ComputeNodeInfo) string {
return fmt.Sprintf("%s / %s", datasize.ByteSize(cni.AvailableCapacity.Disk).HR(), datasize.ByteSize(cni.MaxCapacity.Disk).HR())
return fmt.Sprintf("%s / %s",
humanize.Bytes(cni.AvailableCapacity.Disk),
humanize.Bytes(cni.MaxCapacity.Disk))
}),
},
{
Expand Down
14 changes: 7 additions & 7 deletions pkg/config/defaults.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,9 @@ var Default = types.Bacalhau{
Interval: 15 * types.Second,
},
AllocatedCapacity: types.ResourceScaler{
CPU: "70%",
Memory: "70%",
Disk: "70%",
CPU: "80%",
Memory: "80%",
Disk: "80%",
GPU: "100%",
},
},
Expand All @@ -63,7 +63,7 @@ var Default = types.Bacalhau{
Task: types.BatchTaskDefaultConfig{
Resources: types.ResourcesConfig{
CPU: "500m",
Memory: "1Gb",
Memory: "512Mb",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

Memory configuration appears to be reduced from previous defaults

Based on the search results, I can make the following observations:

  • The production configuration (pkg/config_legacy/configenv/production.go) previously used 1Gb memory
  • Test configurations (pkg/test/compute/resourcelimits_test.go and pkg/config/types/bacalhau_test.go) use 1Gb memory
  • The new default of 512Mb is a significant reduction from the previous 1Gb setting

Since this is a significant reduction in memory allocation and there's evidence of higher memory usage in production and test configurations, this warrants attention.

  • pkg/config_legacy/configenv/production.go: Previously used 1Gb
  • pkg/config/defaults.go: New default of 512Mb
  • pkg/test/compute/resourcelimits_test.go: Tests expecting 1Gb
🔗 Analysis chain

Verify if 512Mb is sufficient for typical workloads

While reducing memory requirements to 512Mb aligns with the goal of running on smaller EC2 instances, we should verify this doesn't impact typical workloads.

Consider implementing memory auto-scaling or providing documentation for users to override these defaults when needed for memory-intensive workloads.

Also applies to: 75-75, 84-84, 93-93

Let me try a different approach to gather information about memory usage patterns in the codebase.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for memory-related issues or discussions in recent PRs and issues
gh search issues "memory OOM out of memory" --repo bacalhau-project/bacalhau --state all --limit 10

# Look for memory configuration mentions in tests
rg -l "Memory.*[0-9]+[MGT]b" "test" || true

Length of output: 2860

Script:

#!/bin/bash
# Search for memory configurations in the entire codebase
rg -A 2 -B 2 'Memory.*[0-9]+[MGT]b' || true

# Look for memory-related configurations and defaults
fd -e yaml -e yml | xargs rg 'memory|Memory' || true

# Look for memory-related test configurations
ast-grep --pattern 'Memory: "$_"'

Length of output: 4163

},
},
},
Expand All @@ -72,7 +72,7 @@ var Default = types.Bacalhau{
Task: types.BatchTaskDefaultConfig{
Resources: types.ResourcesConfig{
CPU: "500m",
Memory: "1Gb",
Memory: "512Mb",
},
},
},
Expand All @@ -81,7 +81,7 @@ var Default = types.Bacalhau{
Task: types.LongRunningTaskDefaultConfig{
Resources: types.ResourcesConfig{
CPU: "500m",
Memory: "1Gb",
Memory: "512Mb",
},
},
},
Expand All @@ -90,7 +90,7 @@ var Default = types.Bacalhau{
Task: types.LongRunningTaskDefaultConfig{
Resources: types.ResourcesConfig{
CPU: "500m",
Memory: "1Gb",
Memory: "512Mb",
},
},
},
Expand Down
1 change: 0 additions & 1 deletion pkg/swagger/docs.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion pkg/swagger/swagger.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1817,7 +1817,6 @@
}
},
"NodeID": {
"description": "TODO replace all access on this field with the `ID()` method",
"type": "string"
},
"NodeType": {
Expand Down
1 change: 0 additions & 1 deletion webui/lib/api/schema/swagger.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1817,7 +1817,6 @@
}
},
"NodeID": {
"description": "TODO replace all access on this field with the `ID()` method",
"type": "string"
},
"NodeType": {
Expand Down
Loading