Skip to content

test: Extend Spark Array functions: array_repeat , shuffle and slice test coverage #20420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
comphead merged 1 commit into from
Feb 20, 2026
Merged

test: Extend Spark Array functions: array_repeat , shuffle and slice test coverage #20420

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions datafusion/sqllogictest/test_files/spark/array/array_repeat.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,26 @@ SELECT array_repeat('123', -1);
----
[]

query ?
SELECT array_repeat('123', CAST('2' AS INT));
----
[123, 123]

query ?
SELECT array_repeat(123, 3);
----
[123, 123, 123]

query ?
SELECT array_repeat('2001-09-28T01:00:00'::timestamp, 2);
----
[2001-09-28T01:00:00, 2001-09-28T01:00:00]

query ?
SELECT array_repeat(array_repeat('123', CAST('2' AS INT)), CAST('3' AS INT));
----
[[123, 123], [123, 123], [123, 123]]

query ?
SELECT array_repeat(['123'], 2);
----
Expand Down
20 changes: 20 additions & 0 deletions datafusion/sqllogictest/test_files/spark/array/shuffle.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,26 @@ SELECT shuffle(column1, 1) FROM test_shuffle_fixed_size;
[9, NULL, 8]
NULL

query ?
SELECT shuffle(['2001-09-28T01:00:00'::timestamp, '2001-08-28T01:00:00'::timestamp, '2001-07-28T01:00:00'::timestamp, '2001-06-28T01:00:00'::timestamp, '2001-05-28T01:00:00'::timestamp], 1);
Copy link
Contributor

@comphead comphead Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not related to the PR, but how it comes shuffle which is non deterministic https://spark.apache.org/docs/latest/api/sql/#shuffle has stable output 🤔

Copy link
Contributor

@Jefffrey Jefffrey Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're providing a seed for deterministic output

Copy link
Member Author

@erenavsarogullari erenavsarogullari Feb 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, shuffle function is non-deterministic. However, it can also return deterministic result by passing seed for specially this kind of cases (e.g: test verification).

For reference:

shuffle(array)
shuffle(array, seed)

----
[2001-09-28T01:00:00, 2001-06-28T01:00:00, 2001-07-28T01:00:00, 2001-08-28T01:00:00, 2001-05-28T01:00:00]

query ?
SELECT shuffle(shuffle([1, 20, NULL, 3, 100, NULL, 98, 99], 1), 1);
----
[1, 99, NULL, 98, 100, NULL, 3, 20]

query ?
SELECT shuffle([' ', NULL, 'abc'], 1);
----
[ , NULL, abc]

query ?
SELECT shuffle([1, 2, 3, 4], CAST('2' AS INT));
----
[1, 4, 2, 3]

# Clean up
statement ok
DROP TABLE test_shuffle_list_types;
Expand Down
15 changes: 15 additions & 0 deletions datafusion/sqllogictest/test_files/spark/array/slice.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,18 @@ FROM VALUES
NULL
NULL
NULL

query ?
SELECT slice(['2001-09-28T01:00:00'::timestamp, '2001-08-28T01:00:00'::timestamp, '2001-07-28T01:00:00'::timestamp, '2001-06-28T01:00:00'::timestamp, '2001-05-28T01:00:00'::timestamp], 1, 3);
----
[2001-09-28T01:00:00, 2001-08-28T01:00:00, 2001-07-28T01:00:00]

query ?
SELECT slice(slice([1, 2, 3, 4], 1, 3), 1, 2);
----
[1, 2]

query ?
SELECT slice([1, 2, 3, 4], CAST('2' AS INT), 4);
----
[2, 3, 4]
4 changes: 3 additions & 1 deletion docs/source/contributor-guide/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@ DataFusion's SQL implementation is tested using [sqllogictest](https://github.co
cargo test --profile=ci --test sqllogictests
# Run a specific test file
cargo test --profile=ci --test sqllogictests -- aggregate.slt
# Run and update expected outputs
# Run a specific test file and update expected outputs
cargo test --profile=ci --test sqllogictests -- aggregate.slt --complete
Copy link
Contributor

@comphead comphead Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks unrelated to the PR

Copy link
Contributor

@Jefffrey Jefffrey Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems a nice improvement regardless

Copy link
Member Author

@erenavsarogullari erenavsarogullari Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have covered this doc update here because of a minor change

# Run and update expected outputs for all test files
cargo test --profile=ci --test sqllogictests -- --complete
```

Expand Down