Merge pull request #338 from adorsys/analyze-document-and-present-dat…

…asafe-storage-api

update: datasafe-storage-api and improve test-coverage

datasafe-storage/datasafe-storage-api/README.md

@@ -1,5 +1,124 @@
 # Storage API
 
-This module exposes storage API used by other modules. Use 
-[StorageService](src/main/java/de/adorsys/datasafe/storage/api/StorageService.java) interface, provided by this module,
-if you want to write your own adapter.
+This module provides an abstraction layer for various storage operations, enabling interaction with different types
+of storage backends through a unified interface. It allows for the reading, writing, listing, and removal of data stored in different locations such as S3 buckets, local file systems, based on URI schemes or patterns.
+It exposes storage API used by other modules. 
+
+- Use [StorageService](src/main/java/de/adorsys/datasafe/storage/api/StorageService.java) interface, provided by this module,
+if you want to write your own adapter.
+
+## Key Types and Interfaces
+
+#### StorageCheckService
+- **Purpose:** To check if a specified resource exists at a given location.
+- **Key Method:**
+```java
+boolean objectExists(AbsoluteLocation location);
+```
+#### StorageListService
+- **Purpose:** To list resources at a given location.
+- **Key Method:**
+```java
+Stream<AbsoluteLocation<ResolvedResource>> list(AbsoluteLocation location);
+```
+#### StorageReadService
+- **Purpose:** To read data from a specified resource location.
+- **Key Method:**
+```java
+InputStream read(AbsoluteLocation location);
+```
+
+#### StorageRemoveService
+- **Purpose:** To remove a specified resource location.
+- **Key Method:**
+```java
+void remove(AbsoluteLocation location);
+```
+#### StorageWriteService
+- **Purpose:** To write data to a specified resource location
+- **Key Method:**
+```java
+OutputStream write(WithCallback<AbsoluteLocation, ? extends ResourceWriteCallback> locationWithCallback);
+```
+- **Additional Method:**
+```java
+default Optional<Integer> flushChunkSize(AbsoluteLocation location) { ... }
+```
+#### StorageService
+- **Purpose:** Combines all storage operations into a single interface.
+- **Implements:**
+* StorageCheckService
+* StorageListService
+* StorageReadService
+* StorageRemoveService
+* StorageWriteService
+
+## Key Classes
+
+#### BaseDelegatingStorage
+- **Purpose:** Abstract base class that delegates storage operations to actual storage implementations.
+- **Method:** Implements methods from StorageService and delegates them to an abstract service method.
+```java
+protected abstract StorageService service(AbsoluteLocation location);
+```
+#### RegexDelegatingStorage
+- **Purpose:** Delegates storage operations based on regex matching of URIs.
+
+- **Key Fields:** 
+```java
+private final Map<Pattern, StorageService> storageByPattern;
+```
+- **Implementation of service method**
+```java
+protected StorageService service(AbsoluteLocation location) { ... }
+```
+#### SchemeDelegatingStorage
+- **Purpose:** Delegates storage operations based on URI schemes.
+
+- **Key Fields:**
+```java
+private final Map<String, StorageService> storageByScheme;
+```
+- **Implementation of service method**
+```java
+protected StorageService service(AbsoluteLocation location) { ... }
+```
+#### UriBasedAuthStorageService
+- **Purpose:** Manages storage connections based on URIs containing credentials, such as S3 URIs.
+
+- **Key Fields:**
+```java
+private final Map<AccessId, StorageService> clientByItsAccessKey = new ConcurrentHashMap<>();
+private final Function<URI, String> bucketExtractor;
+private final Function<URI, String> regionExtractor;
+private final Function<URI, String> endpointExtractor;
+private final Function<AccessId, StorageService> storageServiceBuilder;
+```
+- **Key Inner Class: AccessId**
+```java
+private final String accessKey;
+private final String secretKey;
+private final String region;
+private final String bucketName;
+private final String endpoint;
+private final URI withoutCreds;
+private final URI onlyHostPart;
+```
+
+#### UserBasedDelegatingStorage
+- **Purpose:** Delegates storage operations based on user-specific bucket mappings.
+
+- **Key Fields:**
+```java
+private final Map<String, StorageService> clientByBucket = new ConcurrentHashMap<>();
+private final List<String> amazonBuckets;
+private final Function<String, StorageService> storageServiceBuilder;
+```
+### URI Routing Flowchart:
+This flowchart depicts how storage operations are routed based on URIs and patterns:
+![URI Routing flowchart](http://www.plantuml.com/plantuml/dpng/dL1DImCn4BtlhnZtj0N1GtiKQTdMAXHR3D9Z6PFPDfWcaang_VTck-lgKWNn56RcVUIzSM3q7FSckz1McgW8hilHLJdQbCuo7VacorRaWxD53EGl8NzAJzwy0NX7C4L6WHL1OETnIp1PtUU3JBm7fdsXqZMaQtjCn0ullk7JVkNTGQiaYX2jhZGfq9R9LoW9AkUR2ILhkuKtpJiueDSkXixu6UKBMHKwzytio4KO9l79Me0OrZQbSL5rb1Jce2Nr6PKs54vZmY-SH0EtQGKD9E-MhKYVx58d_YljiXwxgAArIuSvMV9Q_l2Jx95Cs_PvUtNjDJsL1YKQKuUjyMV8K-mf6TeYDvJFJonVoIDhPt_bzWhufqQ_Xp-ePE9kkTuiPlFPmxGOP6EoAkxD1m00)
+## Conclusion
+This module provides a robust and flexible framework for managing data storage in a system with multiple storage backends. It offers several key benefits:
+- **Pluggability:** Easily add new storage implementations without modifying existing code.
+- **Testability:** Simplify testing with a clear API and mocking capabilities.
+- **Maintainability:** Centralize storage logic and reduce code duplication.

datasafe-storage/datasafe-storage-api/src/main/java/de/adorsys/datasafe/storage/api/UriBasedAuthStorageService.java

@@ -19,9 +19,9 @@
 public class UriBasedAuthStorageService extends BaseDelegatingStorage {
 
  private final Map<AccessId, StorageService> clientByItsAccessKey = new ConcurrentHashMap<>();
- private final Function<URI, String> bucketExtractor;
- private final Function<URI, String> regionExtractor;
- private final Function<URI, String> endpointExtractor;
+ final Function<URI, String> bucketExtractor;
+ final Function<URI, String> regionExtractor;
+ final Function<URI, String> endpointExtractor;
 
  // Builder to create S3 or other kind of Storage service
  private final Function<AccessId, StorageService> storageServiceBuilder;

datasafe-storage/datasafe-storage-api/src/test/java/de/adorsys/datasafe/storage/api/RegexDelegatingStorageTest.java

@@ -0,0 +1,62 @@
+package de.adorsys.datasafe.storage.api;
+import de.adorsys.datasafe.types.api.resource.AbsoluteLocation;
+import de.adorsys.datasafe.types.api.resource.BasePrivateResource;
+import de.adorsys.datasafe.types.api.resource.WithCallback;
+import de.adorsys.datasafe.types.api.shared.BaseMockitoTest;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mock;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.verify;
+
+public class RegexDelegatingStorageTest extends BaseMockitoTest{
+
+ @Mock
+ private StorageService service;
+ private RegexDelegatingStorage tested;
+ private AbsoluteLocation location;
+
+ @BeforeEach
+ void setUp() {
+ Map<Pattern, StorageService> storageByPattern = Collections.singletonMap(Pattern.compile("s3://.*"), service);
+ tested = new RegexDelegatingStorage(storageByPattern);
+ location = new AbsoluteLocation<>(BasePrivateResource.forPrivate("s3://bucket"));
+ }
+ @Test
+ void objectExists() {
+ tested.objectExists(location);
+ verify(service).objectExists(location);
+ }
+ @Test
+ void list() {
+ tested.list(location);
+ verify(service).list(location);
+ }
+ @Test
+ void read() {
+ tested.read(location);
+ verify(service).read(location);
+ }
+ @Test
+ void remove() {
+ tested.remove(location);
+ verify(service).remove(location);
+ }
+ @Test
+ void write() {
+ tested.write(WithCallback.noCallback(location));
+ verify(service).write(any(WithCallback.class));
+ }
+ @Test
+ void objectExistsWithNoMatch() {
+ AbsoluteLocation badlocation = new AbsoluteLocation<>(BasePrivateResource.forPrivate("file://bucket"));
+ assertThrows(IllegalArgumentException.class, () -> tested.objectExists(badlocation));
+ }
+
+}

datasafe-storage/datasafe-storage-api/src/test/java/de/adorsys/datasafe/storage/api/UriBasedAuthStorageServiceTest.java

@@ -2,12 +2,14 @@
 
 import de.adorsys.datasafe.types.api.resource.AbsoluteLocation;
 import de.adorsys.datasafe.types.api.resource.BasePrivateResource;
+import de.adorsys.datasafe.types.api.resource.PrivateResource;
 import de.adorsys.datasafe.types.api.resource.WithCallback;
 import de.adorsys.datasafe.types.api.shared.BaseMockitoTest;
 import lombok.Getter;
 import lombok.RequiredArgsConstructor;
 import lombok.ToString;
 import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
 import org.junit.jupiter.params.ParameterizedTest;
 import org.junit.jupiter.params.provider.MethodSource;
 import org.junit.jupiter.params.provider.ValueSource;
@@ -42,8 +44,49 @@ void init() {
  when(getService.apply(argumentCaptor.capture())).thenReturn(storage);
  tested = new UriBasedAuthStorageService(getService);
  }
+ @Test
+ void testDefaultConstructor() {
+ UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService);
+ assertThat(service).isNotNull();
+ }
 
- @MethodSource("fixture")
+ @Test
+ void testCustomConstructor() {
+ Function<URI, String[]> segmentator = uri -> new String[] {"region", "bucket"};
+ UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService, segmentator);
+ assertThat(service).isNotNull();
+ }
+ @Test
+ void testRegionExtractor() {
+ Function<URI, String[]> segmentator = uri -> new String[] {"region", "bucket"};
+ UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService, segmentator);
+
+ URI uri = URI.create("http://host.com/region/bucket");
+ String region = service.regionExtractor.apply(uri);
+ assertThat(region).isEqualTo("region");
+ }
+
+ @Test
+ void testBucketExtractor() {
+ Function<URI, String[]> segmentator = uri -> new String[] {"region", "bucket"};
+ UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService, segmentator);
+
+ URI uri = URI.create("http://host.com/region/bucket");
+ String bucket = service.bucketExtractor.apply(uri);
+ assertThat(bucket).isEqualTo("bucket");
+ }
+
+ @Test
+ void testEndpointExtractor() {
+ Function<URI, String[]> segmentator = uri -> new String[] {"region", "bucket"};
+ UriBasedAuthStorageService service = new UriBasedAuthStorageService(getService, segmentator);
+
+ URI uri = URI.create("http://host.com:8080/region/bucket");
+ String endpoint = service.endpointExtractor.apply(uri);
+ assertThat(endpoint).isEqualTo("http://host.com:8080/");
+ }
+
+ @MethodSource("fixture")
  @ParameterizedTest
  void objectExists(MappedItem item) {
  tested.objectExists(item.getUri());