Proposal: Use Rclone's FS Interface for Object Store Layer

[ankitrgadiya]

The project contains implementations for object store providers. I want to propose an alternative to that by using Rclone's FS interface. Rclone has implementations for several providers and so by using that UtahFS can immediately support multiple providers. A simple wrapper around Rclone's FS can be implemented which satisfies the ObjectStore interface internally.

As far as I understood, Object Layer is simple Getters and Setters and since all of the additional security is implemented by above layers, a third party implementation should suffice. But I might be wrong here.

Disclaimer: I'm not in anyway associated with Rclone project.

[ncw]

There are several ways of joining utahfs and rclone.

One is as above. Implementing Object Storage interface is very easy. I'd be happy to give it a go if you are interested.

That does tie this project to rclone which may not be desirable though.

It would be possible to use rclone the same way restic uses rclone. So rclone could implement an "rclone serve utahfs" to serve a remote cloud storage and utahfs could start an rclone process and communicate to it over stdin/stdout.

However the the requirements are quite low on the cloud storage so probably one of rclone's existing servers would work fine.

For this way, if you wanted to setup rclone with utahfs you'd install rclone, set up your backend and then use the rclone backend in utahfs.

Does the key have slashes in? General purpose cloud storage isn't good at thousands of files in a directory.

PS I'm the rclone project lead 😃

[Bren2010]

The approach I had in mind was more like the first one you describe:

• Vendor rclone as a dependency so that we're not immediately affected by breaking changes
• Implement the ObjectStorage interface over rclone's own abstraction
• Setup UtahFS's config logic to be able to construct any of rclone's providers

The keys don't have slashes in them, they're hex-encoded numbers.

[ankitrgadiya]

I agree with @Bren2010. I'd say dependency on Rclone will be there either way, so why not use the Interface. Also, pardon my ignorance as I'm not familiar with Rclone's serve sub-command, but if another process is required, that might not be good user experience.

Another point for discussion may be the Configuration for various Cloud Providers. As I understand Rclone implements it using the Mapper interface. The user is required to put all the configuration fields in this map, and the individual FS implementation takes the relevant values from it. So, it is the library's user's responsibility to populate this map with required values. Depending on how you are looking at it, this may or may not be trivial.

@Bren2010 I would suggest switching to Go modules over vendoring the dependencies. It supports explicitly pinning the dependencies to the versions. The newer versions of Go are also configured to use Go module proxy.

[ncw]

@Bren2010 wrote:

The approach I had in mind was more like the first one you describe:

• Vendor rclone as a dependency so that we're not immediately affected by breaking changes
• Implement the ObjectStorage interface over rclone's own abstraction
• Setup UtahFS's config logic to be able to construct any of rclone's providers

The config is the hard part I think and it is the main reason I haven't split the backends out into their own package.

Config for some backends, eg s3 is reasonably straight forward. Config for some backends involves getting an oauth token and a bit of interaction with the user.

It is possible to generate the necessary tokens can be done with the rclone authorize command and potentially I could make the code for that easily accessible so that could be called from utafs with a sub command (handwaving).

Note that rclone needs to be able to set things in the config too - the tokens it gets back over oauth only last 1 hour normally and need to be refreshed. The rclone authorize stub could set tokens (and other things) in the config directly.

The keys don't have slashes in them, they're hex-encoded numbers.

So a naive implementation would put those hex encoded numbers all in one directory.

How many keys will there be? I'm assuming that there might be millions?

S3 will deal with millions of files in one directory just fine, but the non bucket based systems (eg Google drive, dropbox, box etc) won't deal with that at all well so it would be necessary to split the hex-encoded numbers up with some slashes to make directories.

@ankitrgadiya wrote:

Another point for discussion may be the Configuration for various Cloud Providers. As I understand Rclone implements it using the Mapper interface. The user is required to put all the configuration fields in this map, and the individual FS implementation takes the relevant values from it. So, it is the library's user's responsibility to populate this map with required values. Depending on how you are looking at it, this may or may not be trivial.

Yes that is a problem which needs solving!

For reference the mapper interface is quite simple

// Getter provides an interface to get config items
type Getter interface {
	// Get should get an item with the key passed in and return
	// the value. If the item is found then it should return true,
	// otherwise false.
	Get(key string) (value string, ok bool)
}

// Setter provides an interface to set config items
type Setter interface {
	// Set should set an item into persistent config store.
	Set(key, value string)
}

// Mapper provides an interface to read and write config
type Mapper interface {
	Getter
	Setter
}

So in general rclone would like a map[string]string to store config details.

It is known in advance what the config keys are (and rclone can dump them in JSON) so it would be relatively easy to auto generate something which could be included in StorageStruct.

utahfs/cmd/internal/config/config.go

Line 25 in 278bc1d

type StorageProvider struct {

Rclone's config keys are "snake_case" but it would be easy to convert them into "rclone-backend-kebab-case", eg "rclone-drive-hard-delete". The Mapper could then use a bit of reflect to find the right things to get and set in the config struct which doesn't sound too hard.

This assumes you never want more than one backend configured at once of course.

Alternatively you could make a backend type called "Rclone" and use rclone's config file system. So this backend would just name the backend config that it wanted to use (maybe with a path too).