Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Treat data links even more like symlinks, allow crossing using / #11926

Merged
merged 56 commits into from
Jan 8, 2025
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
0e1c169
un-pend more tests
radeusgd Dec 9, 2024
64d2e7a
make the test more comprehensive
radeusgd Dec 9, 2024
ac153c1
make size work across datalink
radeusgd Dec 9, 2024
2cade74
docs for size
radeusgd Dec 9, 2024
14e1a39
WIP docs
radeusgd Dec 10, 2024
7123a26
doc
radeusgd Dec 16, 2024
481d9b4
fix - missing arg in error
radeusgd Dec 16, 2024
b53d907
remove check - it was problematic
radeusgd Dec 16, 2024
e61f3f8
tests
radeusgd Dec 16, 2024
ffc807d
is regular file & is directory
radeusgd Dec 16, 2024
dc98452
S3 is_directory
radeusgd Dec 16, 2024
ab58e29
split resolution into parts and check for data link
radeusgd Dec 18, 2024
b372853
missing () in tests
radeusgd Dec 18, 2024
7d8d956
optimization: do a cheap check before an expensive oen
radeusgd Dec 18, 2024
73739d5
fix wrong paths in test
radeusgd Dec 18, 2024
e8df977
refactor data link handling to ignore directories that match the name
radeusgd Dec 19, 2024
e7486e7
simplify some calls
radeusgd Dec 19, 2024
5f1ef38
more simpl
radeusgd Dec 19, 2024
6fe45dc
fixes
radeusgd Dec 19, 2024
e4ad514
allow windows paths sometimes
radeusgd Dec 19, 2024
443299f
clearer error message, add missing conversion
radeusgd Dec 19, 2024
82e5560
follow links in `new` for Enso Cloud and S3
radeusgd Dec 19, 2024
b76c78a
follow links in `new` for local files
radeusgd Dec 19, 2024
8c76283
fix for S3
radeusgd Dec 19, 2024
04c89ef
changelog
radeusgd Dec 19, 2024
8883807
heuristics for distinguishing datalinks from directories on S3
radeusgd Dec 19, 2024
8704ea7
fix test
radeusgd Dec 19, 2024
7881933
some edge cases
radeusgd Dec 19, 2024
41d4f01
fix is_directory
radeusgd Dec 20, 2024
d09880a
remove 'go to root' - it was breaking other cases and I don't think i…
radeusgd Dec 20, 2024
ca5559c
refactor - make helper private, allow seting bucket through / on `s3://`
radeusgd Dec 20, 2024
7668476
report time in nicer form than ms
radeusgd Dec 20, 2024
b6e4c96
cache is data link because it is heavy and used a lot (makes tests ~2…
radeusgd Dec 20, 2024
8dc432d
Revert "report time in nicer form than ms"
radeusgd Dec 20, 2024
4fc1cfb
Merge branch 'develop' into wip/radeusgd/datalinks-as-symlinks
radeusgd Dec 20, 2024
abe14c2
fix warning propagation in cache
radeusgd Dec 20, 2024
e98c6a4
fix datalink heuristic
radeusgd Dec 20, 2024
ad91b02
CR move file like conversion to the only other place it can be
radeusgd Dec 20, 2024
c741c54
whitespace
radeusgd Dec 20, 2024
7af1dfb
javafmt
radeusgd Dec 20, 2024
6cc2a51
remove debug print
radeusgd Dec 20, 2024
62b9c4e
Merge branch 'develop' into wip/radeusgd/datalinks-as-symlinks
radeusgd Dec 20, 2024
0788dc1
try dir builtin
radeusgd Dec 20, 2024
425ea26
fixing error handling, add one more test case
radeusgd Dec 20, 2024
4368160
move builtins that should be hidden into a private module
radeusgd Dec 20, 2024
62bc402
javafmt
radeusgd Dec 20, 2024
e497e7e
fix wrong method usage
radeusgd Dec 20, 2024
5b4ff91
remove type signature that was too strict - if not supported can be Any
radeusgd Dec 20, 2024
ef1b1d6
Merge branch 'develop' into wip/radeusgd/datalinks-as-symlinks
radeusgd Dec 27, 2024
adfb136
add test for File.new relative path
radeusgd Dec 27, 2024
15b66e1
change local file logic to fix relative paths
radeusgd Dec 27, 2024
96e7dda
fix anomaly in S3 test
radeusgd Dec 30, 2024
953ff1d
Merge branch 'develop' into wip/radeusgd/datalinks-as-symlinks
radeusgd Dec 30, 2024
9976cf2
Merge branch 'develop' into wip/radeusgd/datalinks-as-symlinks
radeusgd Dec 30, 2024
ccd289d
Merge branch 'develop' into wip/radeusgd/datalinks-as-symlinks
radeusgd Jan 8, 2025
77f5471
CR missing PRIVATE
radeusgd Jan 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@
[11889]: https://github.com/enso-org/enso/pull/11889
[11836]: https://github.com/enso-org/enso/pull/11836

#### Enso Standard Library

- [Allow using `/` to access files inside a directory reached through a data
link.][11926]

[11926]: https://github.com/enso-org/enso/pull/11926

#### Enso Language & Runtime

- [Promote broken values instead of ignoring them][11777].
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
private

from Standard.Base import all
import Standard.Base.Errors.Illegal_Argument.Illegal_Argument
import Standard.Base.Internal.Path_Helpers

import project.Internal.S3_Path.S3_Path

type Path_Entry
Directory (name : Text)

File (name : Text)

is_directory self -> Boolean = case self of
Path_Entry.Directory _ -> True
Path_Entry.File _ -> False

type Decomposed_S3_Path
Value (parts : Vector Path_Entry)

## Reconstructs the original path.
key self -> Text =
add_directory_suffix = self.parts.not_empty && self.parts.last.is_directory
suffix = if add_directory_suffix then S3_Path.delimiter else ""
self.parts.map .name . join separator=S3_Path.delimiter suffix=suffix

parse (key : Text) -> Decomposed_S3_Path =
has_directory_suffix = key.ends_with S3_Path.delimiter
parts = key.split S3_Path.delimiter . filter (p-> p.is_empty.not)
entries = case has_directory_suffix of
True -> parts.map Path_Entry.Directory
False ->
if parts.is_empty then [] else
(parts.drop (..Last 1) . map Path_Entry.Directory) + [Path_Entry.File parts.last]
Decomposed_S3_Path.Value entries

join (paths : Vector Decomposed_S3_Path) -> Decomposed_S3_Path =
if paths.is_empty then Error.throw (Illegal_Argument.Error "Cannot join an empty list of paths.") else
flattened = paths.flat_map .parts
# Any `File` parts from the middle are now transformed to `Directory`:
aligned = flattened.map_with_index ix-> part-> case part of
Path_Entry.Directory _ -> part
Path_Entry.File name ->
is_last = ix == flattened.length-1
if is_last then part else Path_Entry.Directory name
Decomposed_S3_Path.Value aligned

normalize self -> Decomposed_S3_Path =
new_parts = Path_Helpers.normalize_segments self.parts .name
Decomposed_S3_Path.Value new_parts

parent self -> Decomposed_S3_Path | Nothing =
if self.parts.is_empty then Nothing else
new_parts = self.parts.drop (..Last 1)
Decomposed_S3_Path.Value new_parts

is_empty self -> Boolean = self.parts.is_empty

first_part self -> Path_Entry | Nothing =
if self.parts.is_empty then Nothing else
self.parts.first

drop_first_part self -> Decomposed_S3_Path =
if self.parts.is_empty then self else
new_parts = self.parts.drop 1
Decomposed_S3_Path.Value new_parts
88 changes: 21 additions & 67 deletions distribution/lib/Standard/AWS/0.0.0-dev/src/Internal/S3_Path.enso
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
from Standard.Base import all
import Standard.Base.Errors.Illegal_Argument.Illegal_Argument
import Standard.Base.Internal.Path_Helpers

import project.Errors.S3_Error
import project.Internal.Decomposed_S3_Path.Decomposed_S3_Path
import project.S3.S3

## PRIVATE
Expand All @@ -25,8 +25,7 @@ type S3_Path
bucket = (without_prefix.take first_slash_index)
if bucket == "" then Error.throw (Illegal_Argument.Error "Invalid S3 path: empty bucket name with key name.") else
key = (without_prefix.drop first_slash_index+1)
normalized = Decomposed_S3_Path.parse key . normalize . key
S3_Path.Value bucket normalized
S3_Path.Value bucket key

## PRIVATE
to_text self -> Text =
Expand All @@ -43,6 +42,20 @@ type S3_Path
Checks if this path represents a directory.
is_directory self -> Boolean = self.is_root || (self.key.ends_with S3_Path.delimiter)

## PRIVATE
private set_new_path self new_path:Decomposed_S3_Path -> S3_Path =
# Handle the edge case of resolving `s3://` path without bucket - first part of the key becomes the bucket name
has_no_bucket = self.bucket == ""
set_new_bucket = has_no_bucket && new_path.is_empty.not
case set_new_bucket of
True ->
new_bucket = new_path.first_part.name
new_key = new_path.drop_first_part.normalize.key
S3_Path.Value new_bucket new_key
False ->
new_key = new_path.normalize.key
S3_Path.Value self.bucket new_key

## PRIVATE
Resolves a subdirectory entry.
This only makes logical sense for paths for which `path.is_directory == True`,
Expand All @@ -52,15 +65,12 @@ type S3_Path
if `subpath` ends with the delimiter.
resolve self (subpath : Text) -> S3_Path =
joined = Decomposed_S3_Path.join [Decomposed_S3_Path.parse self.key, Decomposed_S3_Path.parse subpath]
new_key = joined.normalize.key
S3_Path.Value self.bucket new_key
self.set_new_path joined

## PRIVATE
join self (subpaths : Vector) -> S3_Path =
joined = Decomposed_S3_Path.join (([self.key]+subpaths).map Decomposed_S3_Path.parse)
new_key = joined.normalize.key
S3_Path.Value self.bucket new_key

self.set_new_path joined

## PRIVATE
Returns the parent directory.
Expand Down Expand Up @@ -94,65 +104,9 @@ type S3_Path
path delimiter. In the future we could allow customizing it.
delimiter -> Text = "/"

## PRIVATE
type Path_Entry
## PRIVATE
Directory (name : Text)

## PRIVATE
File (name : Text)

## PRIVATE
is_directory self -> Boolean = case self of
Path_Entry.Directory _ -> True
Path_Entry.File _ -> False

## PRIVATE
type Decomposed_S3_Path
## PRIVATE
Value (parts : Vector Path_Entry) (go_to_root : Boolean)

## PRIVATE
Reconstructs the original path.
key self -> Text =
add_directory_suffix = self.parts.not_empty && self.parts.last.is_directory
suffix = if add_directory_suffix then S3_Path.delimiter else ""
self.parts.map .name . join separator=S3_Path.delimiter suffix=suffix

## PRIVATE
parse (key : Text) -> Decomposed_S3_Path =
has_directory_suffix = key.ends_with S3_Path.delimiter
has_root_prefix = key.starts_with S3_Path.delimiter
parts = key.split S3_Path.delimiter . filter (p-> p.is_empty.not)
entries = case has_directory_suffix of
True -> parts.map Path_Entry.Directory
False ->
if parts.is_empty then [] else
(parts.drop (..Last 1) . map Path_Entry.Directory) + [Path_Entry.File parts.last]
Decomposed_S3_Path.Value entries has_root_prefix

## PRIVATE
join (paths : Vector Decomposed_S3_Path) -> Decomposed_S3_Path =
if paths.is_empty then Error.throw (Illegal_Argument.Error "Cannot join an empty list of paths.") else
last_root_ix = paths.last_index_of (.go_to_root)
without_ignored_paths = if last_root_ix.is_nothing then paths else
paths.drop last_root_ix
flattened = without_ignored_paths.flat_map .parts
# Any `File` parts from the middle are now transformed to `Directory`:
aligned = flattened.map_with_index ix-> part-> case part of
Path_Entry.Directory _ -> part
Path_Entry.File name ->
is_last = ix == flattened.length-1
if is_last then part else Path_Entry.Directory name
Decomposed_S3_Path.Value aligned (last_root_ix.is_nothing.not)

## PRIVATE
normalize self -> Decomposed_S3_Path ! S3_Error =
new_parts = Path_Helpers.normalize_segments self.parts .name
Decomposed_S3_Path.Value new_parts self.go_to_root
bucket_root self -> S3_Path = S3_Path.Value self.bucket ""

## PRIVATE
parent self -> Decomposed_S3_Path | Nothing =
if self.parts.is_empty then Nothing else
new_parts = self.parts.drop (..Last 1)
Decomposed_S3_Path.Value new_parts self.go_to_root
without_trailing_slash self -> S3_Path =
if self.key.ends_with S3_Path.delimiter then S3_Path.Value self.bucket (self.key.drop (..Last 1)) else self
Loading
Loading