Skip to content

Commit

Permalink
update datasafe-storage-api Readme
Browse files Browse the repository at this point in the history
update: datasafe-storage-api and improve test-coverage
  • Loading branch information
AssahBismarkabah committed Jul 24, 2024
1 parent 89da57d commit 296cb62
Show file tree
Hide file tree
Showing 6 changed files with 383 additions and 7 deletions.
125 changes: 122 additions & 3 deletions datasafe-storage/datasafe-storage-api/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,124 @@
# Storage API

This module exposes storage API used by other modules. Use
[StorageService](src/main/java/de/adorsys/datasafe/storage/api/StorageService.java) interface, provided by this module,
if you want to write your own adapter.
This module provides an abstraction layer for various storage operations, enabling interaction with different types
of storage backends through a unified interface. It allows for the reading, writing, listing, and removal of data stored in different locations such as S3 buckets, local file systems, based on URI schemes or patterns.
It exposes storage API used by other modules.

- Use [StorageService](src/main/java/de/adorsys/datasafe/storage/api/StorageService.java) interface, provided by this module,
if you want to write your own adapter.

## Key Types and Interfaces

#### StorageCheckService
- **Purpose:** To check if a specified resource exists at a given location.
- **Key Method:**
```java
boolean objectExists(AbsoluteLocation location);
```
#### StorageListService
- **Purpose:** To list resources at a given location.
- **Key Method:**
```java
Stream<AbsoluteLocation<ResolvedResource>> list(AbsoluteLocation location);
```
#### StorageReadService
- **Purpose:** To read data from a specified resource location.
- **Key Method:**
```java
InputStream read(AbsoluteLocation location);
```

#### StorageRemoveService
- **Purpose:** To remove a specified resource location.
- **Key Method:**
```java
void remove(AbsoluteLocation location);
```
#### StorageWriteService
- **Purpose:** To write data to a specified resource location
- **Key Method:**
```java
OutputStream write(WithCallback<AbsoluteLocation, ? extends ResourceWriteCallback> locationWithCallback);
```
- **Additional Method:**
```java
default Optional<Integer> flushChunkSize(AbsoluteLocation location) { ... }
```
#### StorageService
- **Purpose:** Combines all storage operations into a single interface.
- **Implements:**
* StorageCheckService
* StorageListService
* StorageReadService
* StorageRemoveService
* StorageWriteService

## Key Classes

#### BaseDelegatingStorage
- **Purpose:** Abstract base class that delegates storage operations to actual storage implementations.
- **Method:** Implements methods from StorageService and delegates them to an abstract service method.
```java
protected abstract StorageService service(AbsoluteLocation location);
```
#### RegexDelegatingStorage
- **Purpose:** Delegates storage operations based on regex matching of URIs.

- **Key Fields:**
```java
private final Map<Pattern, StorageService> storageByPattern;
```
- **Implementation of service method**
```java
protected StorageService service(AbsoluteLocation location) { ... }
```
#### SchemeDelegatingStorage
- **Purpose:** Delegates storage operations based on URI schemes.

- **Key Fields:**
```java
private final Map<String, StorageService> storageByScheme;
```
- **Implementation of service method**
```java
protected StorageService service(AbsoluteLocation location) { ... }
```
#### UriBasedAuthStorageService
- **Purpose:** Manages storage connections based on URIs containing credentials, such as S3 URIs.

- **Key Fields:**
```java
private final Map<AccessId, StorageService> clientByItsAccessKey = new ConcurrentHashMap<>();
private final Function<URI, String> bucketExtractor;
private final Function<URI, String> regionExtractor;
private final Function<URI, String> endpointExtractor;
private final Function<AccessId, StorageService> storageServiceBuilder;
```
- **Key Inner Class: AccessId**
```java
private final String accessKey;
private final String secretKey;
private final String region;
private final String bucketName;
private final String endpoint;
private final URI withoutCreds;
private final URI onlyHostPart;
```

#### UserBasedDelegatingStorage
- **Purpose:** Delegates storage operations based on user-specific bucket mappings.

- **Key Fields:**
```java
private final Map<String, StorageService> clientByBucket = new ConcurrentHashMap<>();
private final List<String> amazonBuckets;
private final Function<String, StorageService> storageServiceBuilder;
```
### URI Routing Flowchart:
This flowchart depicts how storage operations are routed based on URIs and patterns:
![URI Routing flowchart](http://www.plantuml.com/plantuml/dpng/dL1DImCn4BtlhnZtj0N1GtiKQTdMAXHR3D9Z6PFPDfWcaang_VTck-lgKWNn56RcVUIzSM3q7FSckz1McgW8hilHLJdQbCuo7VacorRaWxD53EGl8NzAJzwy0NX7C4L6WHL1OETnIp1PtUU3JBm7fdsXqZMaQtjCn0ullk7JVkNTGQiaYX2jhZGfq9R9LoW9AkUR2ILhkuKtpJiueDSkXixu6UKBMHKwzytio4KO9l79Me0OrZQbSL5rb1Jce2Nr6PKs54vZmY-SH0EtQGKD9E-MhKYVx58d_YljiXwxgAArIuSvMV9Q_l2Jx95Cs_PvUtNjDJsL1YKQKuUjyMV8K-mf6TeYDvJFJonVoIDhPt_bzWhufqQ_Xp-ePE9kkTuiPlFPmxGOP6EoAkxD1m00)
## Conclusion
This module provides a robust and flexible framework for managing data storage in a system with multiple storage backends. It offers several key benefits:
- **Pluggability:** Easily add new storage implementations without modifying existing code.
- **Testability:** Simplify testing with a clear API and mocking capabilities.
- **Maintainability:** Centralize storage logic and reduce code duplication.
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@
public class UriBasedAuthStorageService extends BaseDelegatingStorage {

private final Map<AccessId, StorageService> clientByItsAccessKey = new ConcurrentHashMap<>();
private final Function<URI, String> bucketExtractor;
private final Function<URI, String> regionExtractor;
private final Function<URI, String> endpointExtractor;
final Function<URI, String> bucketExtractor;
final Function<URI, String> regionExtractor;
final Function<URI, String> endpointExtractor;

// Builder to create S3 or other kind of Storage service
private final Function<AccessId, StorageService> storageServiceBuilder;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
package de.adorsys.datasafe.storage.api;
import de.adorsys.datasafe.types.api.resource.AbsoluteLocation;
import de.adorsys.datasafe.types.api.resource.BasePrivateResource;
import de.adorsys.datasafe.types.api.resource.WithCallback;
import de.adorsys.datasafe.types.api.shared.BaseMockitoTest;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.Mock;

import java.util.Collections;
import java.util.Map;
import java.util.regex.Pattern;

import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.verify;

public class RegexDelegatingStorageTest extends BaseMockitoTest{

@Mock
private StorageService service;
private RegexDelegatingStorage tested;
private AbsoluteLocation location;

@BeforeEach
void setUp() {
Map<Pattern, StorageService> storageByPattern = Collections.singletonMap(Pattern.compile("s3://.*"), service);
tested = new RegexDelegatingStorage(storageByPattern);
location = new AbsoluteLocation<>(BasePrivateResource.forPrivate("s3://bucket"));
}
@Test
void objectExists() {
tested.objectExists(location);
verify(service).objectExists(location);
}
@Test
void list() {
tested.list(location);
verify(service).list(location);
}
@Test
void read() {
tested.read(location);
verify(service).read(location);
}
@Test
void remove() {
tested.remove(location);
verify(service).remove(location);
}
@Test
void write() {
tested.write(WithCallback.noCallback(location));
verify(service).write(any(WithCallback.class));
}
@Test
void objectExistsWithNoMatch() {
AbsoluteLocation badlocation = new AbsoluteLocation<>(BasePrivateResource.forPrivate("file://bucket"));
assertThrows(IllegalArgumentException.class, () -> tested.objectExists(badlocation));
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@

import de.adorsys.datasafe.types.api.resource.AbsoluteLocation;
import de.adorsys.datasafe.types.api.resource.BasePrivateResource;
import de.adorsys.datasafe.types.api.resource.PrivateResource;
import de.adorsys.datasafe.types.api.resource.WithCallback;
import de.adorsys.datasafe.types.api.shared.BaseMockitoTest;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.ToString;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
import org.junit.jupiter.params.provider.ValueSource;
Expand Down Expand Up @@ -42,8 +44,49 @@ void init() {
when(getService.apply(argumentCaptor.capture())).thenReturn(storage);
tested = new UriBasedAuthStorageService(getService);
}
@Test
void testDefaultConstructor() {
UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService);
assertThat(service).isNotNull();
}

@MethodSource("fixture")
@Test
void testCustomConstructor() {
Function<URI, String[]> segmentator = uri -> new String[] {"region", "bucket"};
UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService, segmentator);
assertThat(service).isNotNull();
}
@Test
void testRegionExtractor() {
Function<URI, String[]> segmentator = uri -> new String[] {"region", "bucket"};
UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService, segmentator);

URI uri = URI.create("http://host.com/region/bucket");
String region = service.regionExtractor.apply(uri);
assertThat(region).isEqualTo("region");
}

@Test
void testBucketExtractor() {
Function<URI, String[]> segmentator = uri -> new String[] {"region", "bucket"};
UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService, segmentator);

URI uri = URI.create("http://host.com/region/bucket");
String bucket = service.bucketExtractor.apply(uri);
assertThat(bucket).isEqualTo("bucket");
}

@Test
void testEndpointExtractor() {
Function<URI, String[]> segmentator = uri -> new String[] {"region", "bucket"};
UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService, segmentator);

URI uri = URI.create("http://host.com:8080/region/bucket");
String endpoint = service.endpointExtractor.apply(uri);
assertThat(endpoint).isEqualTo("http://host.com:8080/");
}

@MethodSource("fixture")
@ParameterizedTest
void objectExists(MappedItem item) {
tested.objectExists(item.getUri());
Expand Down
Loading

0 comments on commit 296cb62

Please sign in to comment.