feat: indexable `Blob`s by ggreif · Pull Request #424 · caffeinelabs/motoko-core

ggreif · 2025-11-18T10:52:59Z

See caffeinelabs/motoko#5018.

TODO:

also do this for base!
at method
get method
doctests

https://github.com/dfinity/motoko/pull/5018/files TODO: - [ ] also do this for `base`! - [ ] `index` method - [ ] test

github-actions · 2025-11-18T10:54:04Z

✨ Documentation preview for bda9424:

https://dfinity.github.io/motoko-core/pull/424 (source code)

github-actions · 2025-11-18T10:54:35Z

Benchmark Results

bench/ArrayBuilding.bench.mo $({\color{green}-0.19\%})$

Large known-size array building

Compares performance of different data structures for building arrays of known size.

Instructions: ${\color{green}-0.13\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{green}-0.05\%}$

Instructions

	1000	100000	1000000
List	542_157 $({\color{green}-1.11\%})$	47_994_641 $({\color{green}-0.68\%})$	475_069_697 $({\color{green}-0.65\%})$
Buffer	342_007 $({\color{red}+0.00\%})$	33_903_437 $({\color{red}+0.00\%})$	339_003_652 $({\color{red}+0.00\%})$
pure/List	302_137 $({\color{red}+0.00\%})$	30_003_592 $({\color{red}+0.00\%})$	300_055_974 $({\color{red}+0.00\%})$
VarArray ?T	180_528 $({\color{red}+0.00\%})$	17_802_958 $({\color{red}+0.00\%})$	178_003_173 $({\color{red}+0.00\%})$
VarArray T	160_815 $({\color{red}+0.00\%})$	15_803_245 $({\color{red}+0.00\%})$	158_003_460 $({\color{red}+0.00\%})$
Array (baseline)	42_697 $({\color{red}+0.00\%})$	4_003_127 $({\color{red}+0.00\%})$	40_003_342 $({\color{red}+0.00\%})$

Heap

	1000	100000	1000000
List	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
Buffer	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
pure/List	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
VarArray ?T	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
VarArray T	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
Array (baseline)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	1000	100000	1000000
List	9.96 KiB $({\color{green}-0.97\%})$	797.46 KiB $({\color{green}-0.01\%})$	7.67 MiB $({\color{green}-0.00\%})$
Buffer	8.71 KiB $({\color{gray}0\%})$	782.15 KiB $({\color{gray}0\%})$	7.63 MiB $({\color{gray}0\%})$
pure/List	19.95 KiB $({\color{gray}0\%})$	1.91 MiB $({\color{gray}0\%})$	19.07 MiB $({\color{gray}0\%})$
VarArray ?T	8.24 KiB $({\color{gray}0\%})$	781.68 KiB $({\color{gray}0\%})$	7.63 MiB $({\color{gray}0\%})$
VarArray T	8.23 KiB $({\color{gray}0\%})$	781.67 KiB $({\color{gray}0\%})$	7.63 MiB $({\color{gray}0\%})$
Array (baseline)	4.3 KiB $({\color{gray}0\%})$	391.02 KiB $({\color{gray}0\%})$	3.82 MiB $({\color{gray}0\%})$

bench/FromIters.bench.mo $({\color{gray}0\%})$

Benchmarking the fromIter functions

Columns describe the number of elements in the input iter.

Instructions: ${\color{gray}0\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{gray}0\%}$

Instructions

	100	10_000	100_000
Array.fromIter	48_764 $({\color{gray}0\%})$	4_712_025 $({\color{gray}0\%})$	47_103_135 $({\color{gray}0\%})$
List.fromIter	31_698 $({\color{gray}0\%})$	3_061_541 $({\color{gray}0\%})$	30_603_553 $({\color{gray}0\%})$
List.fromIter . Iter.reverse	50_297 $({\color{gray}0\%})$	4_832_563 $({\color{gray}0\%})$	48_305_477 $({\color{gray}0\%})$

Heap

	100	10_000	100_000
Array.fromIter	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
List.fromIter	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
List.fromIter . Iter.reverse	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	100	10_000	100_000
Array.fromIter	2.76 KiB $({\color{gray}0\%})$	234.79 KiB $({\color{gray}0\%})$	2.29 MiB $({\color{gray}0\%})$
List.fromIter	3.51 KiB $({\color{gray}0\%})$	312.88 KiB $({\color{gray}0\%})$	3.05 MiB $({\color{gray}0\%})$
List.fromIter . Iter.reverse	5.11 KiB $({\color{gray}0\%})$	469.17 KiB $({\color{gray}0\%})$	4.58 MiB $({\color{gray}0\%})$

bench/ListBufferNewArray.bench.mo $({\color{green}-6.13\%})$

List vs. Buffer for creating known-size arrays

Performance comparison between List and Buffer for creating a new array.

Instructions: ${\color{green}-1.94\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{green}-4.19\%}$

Instructions

	0 (baseline)	1	5	10	100 (for loop)
List	1_367 $({\color{green}-11.64\%})$	2_687 $({\color{green}-7.85\%})$	8_667 $({\color{green}-4.19\%})$	13_462 $({\color{green}-3.48\%})$	73_164 $({\color{green}-1.88\%})$
pure/List	1_247 $({\color{gray}0\%})$	1_355 $({\color{gray}0\%})$	2_439 $({\color{gray}0\%})$	3_801 $({\color{gray}0\%})$	31_868 $({\color{gray}0\%})$
Buffer	2_119 $({\color{gray}0\%})$	2_271 $({\color{gray}0\%})$	3_518 $({\color{gray}0\%})$	5_085 $({\color{gray}0\%})$	36_640 $({\color{gray}0\%})$

Heap

	0 (baseline)	1	5	10	100 (for loop)
List	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
pure/List	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
Buffer	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	0 (baseline)	1	5	10	100 (for loop)
List	476 B $({\color{green}-17.36\%})$	516 B $({\color{green}-16.23\%})$	676 B $({\color{green}-12.89\%})$	784 B $({\color{green}-11.31\%})$	1.84 KiB $({\color{green}-5.05\%})$
pure/List	360 B $({\color{gray}0\%})$	380 B $({\color{gray}0\%})$	460 B $({\color{gray}0\%})$	560 B $({\color{gray}0\%})$	2.3 KiB $({\color{gray}0\%})$
Buffer	856 B $({\color{gray}0\%})$	864 B $({\color{gray}0\%})$	896 B $({\color{gray}0\%})$	936 B $({\color{gray}0\%})$	1.62 KiB $({\color{gray}0\%})$

bench/PriorityQueues.bench.mo $({\color{green}-2.68\%})$

Different priority queue implementations

_Compare the performance of the following priority queue implementations:

PriorityQueue: Binary heap implementation over List.
PriorityQueueSet: Wrapper over Set<(T, Nat)>._

Instructions: ${\color{green}-2.70\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{red}+0.02\%}$

Instructions

	A) PriorityQueue	B) PriorityQueueSet
1.) 100000 operations (push:pop = 1:1)	568_913_057 $({\color{green}-4.79\%})$	522_729_861 $({\color{gray}0\%})$
2.) 100000 operations (push:pop = 2:1)	707_495_424 $({\color{green}-4.77\%})$	809_693_415 $({\color{gray}0\%})$
3.) 100000 operations (push:pop = 10:1)	336_409_578 $({\color{green}-6.01\%})$	873_181_028 $({\color{gray}0\%})$
4.) 100000 operations (only push)	176_982_954 $({\color{green}-8.02\%})$	886_824_792 $({\color{gray}0\%})$
5.) 50000 pushes, then 50000 pops	745_226_615 $({\color{green}-4.04\%})$	961_776_534 $({\color{gray}0\%})$
6.) 50000 pushes, then 25000 "pop;push"es	504_254_228 $({\color{green}-4.76\%})$	922_137_111 $({\color{gray}0\%})$

Heap

	A) PriorityQueue	B) PriorityQueueSet
1.) 100000 operations (push:pop = 1:1)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
2.) 100000 operations (push:pop = 2:1)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
3.) 100000 operations (push:pop = 10:1)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
4.) 100000 operations (only push)	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
5.) 50000 pushes, then 50000 pops	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$
6.) 50000 pushes, then 25000 "pop;push"es	272 B $({\color{gray}0\%})$	272 B $({\color{gray}0\%})$

Garbage Collection

	A) PriorityQueue	B) PriorityQueueSet
1.) 100000 operations (push:pop = 1:1)	15.07 MiB $({\color{red}+0.24\%})$	17.43 MiB $({\color{gray}0\%})$
2.) 100000 operations (push:pop = 2:1)	19.73 MiB $({\color{gray}0\%})$	19.32 MiB $({\color{gray}0\%})$
3.) 100000 operations (push:pop = 10:1)	8.67 MiB $({\color{gray}0\%})$	12.64 MiB $({\color{gray}0\%})$
4.) 100000 operations (only push)	3.87 MiB $({\color{gray}0\%})$	9.96 MiB $({\color{gray}0\%})$
5.) 50000 pushes, then 50000 pops	22.03 MiB $({\color{red}+0.02\%})$	26.2 MiB $({\color{gray}0\%})$
6.) 50000 pushes, then 25000 "pop;push"es	14.22 MiB $({\color{gray}0\%})$	18.44 MiB $({\color{gray}0\%})$

bench/PureListStackSafety.bench.mo $({\color{gray}0\%})$

List Stack safety

Check stack-safety of the following pure/List-related functions.

Instructions: ${\color{gray}0\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{gray}0\%}$

Instructions


pure/List.split	24_602_524 $({\color{gray}0\%})$
pure/List.all	7_901_014 $({\color{gray}0\%})$
pure/List.any	8_001_390 $({\color{gray}0\%})$
pure/List.map	23_103_767 $({\color{gray}0\%})$
pure/List.filter	21_104_188 $({\color{gray}0\%})$
pure/List.filterMap	27_404_742 $({\color{gray}0\%})$
pure/List.partition	21_304_994 $({\color{gray}0\%})$
pure/List.join	33_105_326 $({\color{gray}0\%})$
pure/List.flatten	24_805_667 $({\color{gray}0\%})$
pure/List.take	24_605_664 $({\color{gray}0\%})$
pure/List.drop	9_904_119 $({\color{gray}0\%})$
pure/List.foldRight	19_105_768 $({\color{gray}0\%})$
pure/List.merge	31_808_584 $({\color{gray}0\%})$
pure/List.chunks	51_510_344 $({\color{gray}0\%})$
pure/Queue	142_662_505 $({\color{gray}0\%})$

Heap


pure/List.split	272 B $({\color{gray}0\%})$
pure/List.all	272 B $({\color{gray}0\%})$
pure/List.any	272 B $({\color{gray}0\%})$
pure/List.map	272 B $({\color{gray}0\%})$
pure/List.filter	272 B $({\color{gray}0\%})$
pure/List.filterMap	272 B $({\color{gray}0\%})$
pure/List.partition	272 B $({\color{gray}0\%})$
pure/List.join	272 B $({\color{gray}0\%})$
pure/List.flatten	272 B $({\color{gray}0\%})$
pure/List.take	272 B $({\color{gray}0\%})$
pure/List.drop	272 B $({\color{gray}0\%})$
pure/List.foldRight	272 B $({\color{gray}0\%})$
pure/List.merge	272 B $({\color{gray}0\%})$
pure/List.chunks	272 B $({\color{gray}0\%})$
pure/Queue	272 B $({\color{gray}0\%})$

Garbage Collection


pure/List.split	3.05 MiB $({\color{gray}0\%})$
pure/List.all	328 B $({\color{gray}0\%})$
pure/List.any	328 B $({\color{gray}0\%})$
pure/List.map	3.05 MiB $({\color{gray}0\%})$
pure/List.filter	3.05 MiB $({\color{gray}0\%})$
pure/List.filterMap	3.05 MiB $({\color{gray}0\%})$
pure/List.partition	3.05 MiB $({\color{gray}0\%})$
pure/List.join	3.05 MiB $({\color{gray}0\%})$
pure/List.flatten	3.05 MiB $({\color{gray}0\%})$
pure/List.take	3.05 MiB $({\color{gray}0\%})$
pure/List.drop	328 B $({\color{gray}0\%})$
pure/List.foldRight	1.53 MiB $({\color{gray}0\%})$
pure/List.merge	4.58 MiB $({\color{gray}0\%})$
pure/List.chunks	7.63 MiB $({\color{gray}0\%})$
pure/Queue	18.31 MiB $({\color{gray}0\%})$

bench/Queues.bench.mo $({\color{gray}0\%})$

Different queue implementations

Compare the performance of the following queue implementations:

pure/Queue: The default immutable double-ended queue implementation.
- Pros: Good amortized performance, meaning that the average cost of operations is low O(1).
- Cons: In worst case, an operation can take O(size) time rebuilding the queue as demonstrated in the Pop front 2 elements scenario.
pure/RealTimeQueue
- Pros: Every operation is guaranteed to take at most O(1) time and space.
- Cons: Poor amortized performance: Instruction cost is on average 3x for pop and 8x for push compared to pure/Queue.
mutable Queue
- Pros: Also O(1) guarantees with a lower constant factor than pure/RealTimeQueue. Amortized performance is comparable to pure/Queue.
- Cons: It is mutable and cannot be used in shared types (not shareable).

Instructions: ${\color{gray}0\%}$
Heap: ${\color{gray}0\%}$
Stable Memory: ${\color{gray}0\%}$
Garbage Collection: ${\color{gray}0\%}$

Instructions

	pure/Queue	pure/RealTimeQueue	mutable Queue
Initialize with 2 elements	3_092 $({\color{gray}0\%})$	2_304 $({\color{gray}0\%})$	3_040 $({\color{gray}0\%})$
Push 500 elements	90_713 $({\color{gray}0\%})$	744_219 $({\color{gray}0\%})$	219_284 $({\color{gray}0\%})$
Pop front 2 elements	86_966 $({\color{gray}0\%})$	4_446 $({\color{gray}0\%})$	3_847 $({\color{gray}0\%})$
Pop 150 front&back	92_095 $({\color{gray}0\%})$	304_908 $({\color{gray}0\%})$	124_581 $({\color{gray}0\%})$

Heap

	pure/Queue	pure/RealTimeQueue	mutable Queue
Initialize with 2 elements	324 B $({\color{gray}0\%})$	300 B $({\color{gray}0\%})$	352 B $({\color{gray}0\%})$
Push 500 elements	8.08 KiB $({\color{gray}0\%})$	8.17 KiB $({\color{gray}0\%})$	19.8 KiB $({\color{gray}0\%})$
Pop front 2 elements	240 B $({\color{gray}0\%})$	240 B $({\color{gray}0\%})$	192 B $({\color{gray}0\%})$
Pop 150 front&back	-4.42 KiB $({\color{gray}0\%})$	-492 B $({\color{gray}0\%})$	-11.45 KiB $({\color{gray}0\%})$

Garbage Collection

	pure/Queue	pure/RealTimeQueue	mutable Queue
Initialize with 2 elements	508 B $({\color{gray}0\%})$	444 B $({\color{gray}0\%})$	456 B $({\color{gray}0\%})$
Push 500 elements	10.1 KiB $({\color{gray}0\%})$	137.84 KiB $({\color{gray}0\%})$	344 B $({\color{gray}0\%})$
Pop front 2 elements	12.19 KiB $({\color{gray}0\%})$	528 B $({\color{gray}0\%})$	424 B $({\color{gray}0\%})$
Pop 150 front&back	15.61 KiB $({\color{gray}0\%})$	49.66 KiB $({\color{gray}0\%})$	12.1 KiB $({\color{gray}0\%})$

Note: Renamed benchmarks cannot be compared. Refer to the current baseline for manual comparison.

rvanasa · 2025-11-18T15:26:27Z

I think we should use the same pattern as established in List, i.e. a get function which returns an optional and an at function which traps outside of the index range.

rvanasa · 2025-11-19T15:26:12Z

src/Blob.mo

+  /// ```
+  ///
+  /// Runtime: `O(1)`
+  public func at(blob : Blob, index : Nat) : Nat8 = blob.get index;


Realizing this might be confusing with contextual dot notation...

Oh, you mean the missing parens?

Thinking of the discrepancy where blob.get(i) returns a Nat8 whereas Blob.get(blob, i) would return an optional ?Nat8.

Not sure if there's much we can do about this aside from a breaking change to the compiler or using a different convention for the Blob module compared to List (which might confuse LLMs and humans).

Oh, I get it now. But the same situation probably for Text if that provides indexing at all. That wouldn't be O(1), though.

Double-checked that Text currently doesn't provide indexing. Maybe we could change blob.get(i) to return an optional as part of Motoko 1.0? This seems like a more useful design, since the current behavior is also available through blob[i].

Makes sense. But then we should change all .get() accessors (e.g. array). Also we have to be careful to not suddenly generate garbage...

This should be ok due to the change causing type checking errors that give devs a chance to update their code before compiling again.

feat: indexable Blobs

6072003

https://github.com/dfinity/motoko/pull/5018/files TODO: - [ ] also do this for `base`! - [ ] `index` method - [ ] test

func get

7ff6663

ggreif force-pushed the gabor/blob-index branch from c713824 to 7ff6663 Compare November 18, 2025 21:56

ggreif added 3 commits November 18, 2025 22:58

API

6e3d1de

fix

19df92a

func at

fb3adce

ggreif force-pushed the gabor/blob-index branch from 55ce12f to fb3adce Compare November 18, 2025 22:09

update Changelog.md

bda9424

ggreif self-assigned this Nov 18, 2025

ggreif added the enhancement New feature or request label Nov 18, 2025

ggreif marked this pull request as ready for review November 18, 2025 22:19

ggreif requested a review from a team as a code owner November 18, 2025 22:19

ggreif requested review from christoph-dfinity, crusso and rvanasa November 19, 2025 14:25

rvanasa reviewed Nov 19, 2025

View reviewed changes

Conversation

ggreif commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 18, 2025

Benchmark Results

Large known-size array building

Benchmarking the fromIter functions

List vs. Buffer for creating known-size arrays

Different priority queue implementations

List Stack safety

Different queue implementations

Uh oh!

rvanasa commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rvanasa Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

ggreif Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

rvanasa Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggreif Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

rvanasa Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggreif Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rvanasa Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

ggreif commented Nov 18, 2025 •

edited

Loading

github-actions bot commented Nov 18, 2025 •

edited

Loading

rvanasa commented Nov 18, 2025 •

edited

Loading

rvanasa Nov 19, 2025 •

edited

Loading

rvanasa Nov 26, 2025 •

edited

Loading

ggreif Nov 26, 2025 •

edited

Loading