API: v7.11.0

v7.11.0 - 2024-09-26

Fixes

• Removed dependency on batect
• Removed the need to copy the access token when running the UI locally
• Refactored the e2e tests

v7.0.9 - 2024-02-06

Features

• Ability to pass a custom regex for username validation. See the documentation on the custom_user_name_regex variable.
• Decoupled API & SDK into separate releases.
• New optional infrastructure variables to increase cpu and memory limits for the API container. See the infrastructure variables for more information.
• Upgraded browserify-sign from 4.2.1 to 4.2.2.
• Upgraded @adobe/css-tools from 4.3.1 to 4.3.2.

Fixes

• Issue with the last updated date on datasets being 'Never Updated'.

Breaking Changes

Migration

v7.0.7 - 2023-11-07

Fixes

• Hitting maximum security group rules for the load balancer.
• Documentation improvements and removes any references to the old deprecated repositories.

Closes relevant GitHub issues

• #50
• #59
• #54
• #51

v7.0.4 - 2023-09-20

Features

• Improved release process
• Added Athena workgroup and database as outputs of the rAPId module.

Fixes

• Updated terraform default application_version and ui_version variables.
• Migration script and documentation.

v7.0.2 - 2023-09-14

Fixes

• Update UI repo references.

v7.0.0 - 2023-09-12

Features

• Layers have been introduced to rAPId. These are now the highest level of grouping for your data. They allow you to separate your data into areas that relate to the layers in your data architecture e.g raw, curated, presentation. You will need to specify your layers when you create or migrate a rAPId instance.
• All the code is now in this monorepo. The previous Infrastructure, UI and API repos are now deprecated. This will ease the use and development of rAPId.
• Schemas are now stored in DynamoDB, rather than S3. This offers speed and usability improvements, as well as making rAPId easier to extend.
• Code efficiency improvements. There were several areas in rAPId where we were executing costly operations that caused performance to degrade at scale. We've fixed these inefficiencies, taking us from O(n²) -> O(n) in these areas.
• Glue Crawlers have been removed, with Athena tables are created directly by the API instead. Data is now available to query immediately after it is uploaded, rather than the previous wait (approximately 3 mins) while crawlers ran. It also offers scalability benefits because without crawlers we are not dependant on the number of free IPs within the subnet.
• Improved UI testing with Playwright.

Breaking Changes

• All dataset endpoints will be prefixed with layer. Typically going from domain/dataset to layer/domain/dataset.

Migration

• See the migration doc for details on how to migrate to v7 from v6.

SDK: v0.1.9

v0.1.9 - 2024-09-12

Features

• Expanded rAPId sdk metadata to include: update_behaviour, is_latest_version and description.

v0.1.7 - 2023-02-06

Features

• Ability to now create a protected domain via the sdk.

v0.1.5 - 2023-11-07

Fixes

• Issue within the sdk upload_and_create_dataset function where schema metadata wasn't being correctly overridden.
• Documentation improvements and removes any references to the old deprecated repositories.

v0.1.3 - 2023-09-20

Fixes

• Fix the behaviour of the dataset pattern functions in the SDK.

v0.1.1- 2023-09-12

Features

• Layers have been introduced to rAPId. These are now the highest level of grouping for your data. They allow you to separate your data into areas that relate to the layers in your data architecture e.g raw, curated, presentation. You will need to specify your layers when you create or migrate a rAPId instance.
• All the code is now in this monorepo. The previous Infrastructure, UI and API repos are now deprecated. This will ease the use and development of rAPId.
• Schemas are now stored in DynamoDB, rather than S3. This offers speed and usability improvements, as well as making rAPId easier to extend.
• Code efficiency improvements. There were several areas in rAPId where we were executing costly operations that caused performance to degrade at scale. We've fixed these inefficiencies, taking us from O(n²) -> O(n) in these areas.
• Glue Crawlers have been removed, with Athena tables are created directly by the API instead. Data is now available to query immediately after it is uploaded, rather than the previous wait (approximately 3 mins) while crawlers ran. It also offers scalability benefits because without crawlers we are not dependant on the number of free IPs within the subnet.
• Improved UI testing with Playwright.

Breaking Changes

• All dataset endpoints will be prefixed with layer. Typically going from domain/dataset to layer/domain/dataset.
• All sdk functions that interact with datasets will now require an argument for layer.

Migration

• See the migration doc for details on how to migrate to v7 from v6.

API: v7.10.0

v7.10.0 - 2024-03-21

Fixes

• Issues with the documentation where links were not re-directing successfully.

Features

• Extra infrastructure output blocks

v7.0.8 - 2023-11-15

Fixes

• Issue with date types when editing a schema on the UI because of no option to apply format column and therefore getting an all fields are required error.
• Tweaked UI design when adding permissions to subject.
• Updated NextJS and Zod package version.

Features

• Data bucket now has EventBridge notifications enabled by default.

Closes relevant GitHub issues

• #57

v7.0.6 - 2023-10-18

Features

• New UI page that allows for the ability to delete users and clients easily.
• Clients can now be created and deleted via the sdk.

Fixes

• Where dataset info was being called on columns with a date type, this was causing an issue with the Pydantic validation.
• Tweaked the documentation to implement searching for column heading style guide to match what the API returns in the error message.

v7.0.3 - 2023-09-15

Fixes

• Fixes issue where permissions were not being correctly read and causing api functionality to fail

v7.0.1 - 2023-09-13

Fixes

• Date types were being stored as strings which caused issues when querying with Athena. They are now stored as date types.

SDK: v0.1.8

v0.1.8 - 2024-03-21

Features

• Ability to now perform the following rAPId functions via the sdk; create user, delete user, list subjects, list layers, list protected domains and delete dataset.

v0.1.6 - 2023-11-15

Fixes

• SDK not uploading a Pandas Dataframe with a date field set correctly.

Closes relevant GitHub issues

• #57

v0.1.4 - 2023-10-18

Features

• Clients can now be created and deleted via the sdk.

Fixes

• Fixed an issue with the sdk not showing schemas were created successfully due to a wrong response code.
• Where dataset info was being called on columns with a date type, this was causing an issue with the Pydantic validation.
• Tweaked the documentation to implement searching for column heading style guide to match what the API returns in the error message.

v0.1.2 - 2023-09-13

Fixes

• Date types were being stored as strings which caused issues when querying with Athena. They are now stored as date types.
• Rename the rAPId sdk method generate_info to fetch_dataset_info and remove an unnecessary argument.

API: v7.0.9

v7.0.9 - 2024-02-06

See v7.0.9 changes

Features

• Ability to pass a custom regex for username validation. See the documentation on the custom_user_name_regex variable.
• Decoupled API & SDK into separate releases.
• New optional infrastructure variables to increase cpu and memory limits for the API container. See the infrastructure variables for more information.
• Upgraded browserify-sign from 4.2.1 to 4.2.2.
• Upgraded @adobe/css-tools from 4.3.1 to 4.3.2.

Fixes

• Issue with the last updated date on datasets being 'Never Updated'.

Breaking Changes

Migration

v7.0.7 - 2023-11-07

Fixes

• Hitting maximum security group rules for the load balancer.
• Documentation improvements and removes any references to the old deprecated repositories.

Closes relevant GitHub issues

• #50
• #59
• #54
• #51

v7.0.4 - 2023-09-20

Features

• Improved release process
• Added Athena workgroup and database as outputs of the rAPId module.

Fixes

• Updated terraform default application_version and ui_version variables.
• Migration script and documentation.

v7.0.2 - 2023-09-14

Fixes

• Update UI repo references.

v7.0.0 - 2023-09-12

Features

• Layers have been introduced to rAPId. These are now the highest level of grouping for your data. They allow you to separate your data into areas that relate to the layers in your data architecture e.g raw, curated, presentation. You will need to specify your layers when you create or migrate a rAPId instance.
• All the code is now in this monorepo. The previous Infrastructure, UI and API repos are now deprecated. This will ease the use and development of rAPId.
• Schemas are now stored in DynamoDB, rather than S3. This offers speed and usability improvements, as well as making rAPId easier to extend.
• Code efficiency improvements. There were several areas in rAPId where we were executing costly operations that caused performance to degrade at scale. We've fixed these inefficiencies, taking us from O(n²) -> O(n) in these areas.
• Glue Crawlers have been removed, with Athena tables are created directly by the API instead. Data is now available to query immediately after it is uploaded, rather than the previous wait (approximately 3 mins) while crawlers ran. It also offers scalability benefits because without crawlers we are not dependant on the number of free IPs within the subnet.
• Improved UI testing with Playwright.

Breaking Changes

• All dataset endpoints will be prefixed with layer. Typically going from domain/dataset to layer/domain/dataset.

Migration

• See the migration doc for details on how to migrate to v7 from v6.

SDK: v0.1.7

v0.1.7 - 2023-02-06

Features

• Ability to now create a protected domain via the sdk.

v0.1.5 - 2023-11-07

Fixes

• Issue within the sdk upload_and_create_dataset function where schema metadata wasn't being correctly overridden.
• Documentation improvements and removes any references to the old deprecated repositories.

v0.1.3 - 2023-09-20

Fixes

• Fix the behaviour of the dataset pattern functions in the SDK.

v0.1.1- 2023-09-12

Features

• Layers have been introduced to rAPId. These are now the highest level of grouping for your data. They allow you to separate your data into areas that relate to the layers in your data architecture e.g raw, curated, presentation. You will need to specify your layers when you create or migrate a rAPId instance.
• All the code is now in this monorepo. The previous Infrastructure, UI and API repos are now deprecated. This will ease the use and development of rAPId.
• Schemas are now stored in DynamoDB, rather than S3. This offers speed and usability improvements, as well as making rAPId easier to extend.
• Code efficiency improvements. There were several areas in rAPId where we were executing costly operations that caused performance to degrade at scale. We've fixed these inefficiencies, taking us from O(n²) -> O(n) in these areas.
• Glue Crawlers have been removed, with Athena tables are created directly by the API instead. Data is now available to query immediately after it is uploaded, rather than the previous wait (approximately 3 mins) while crawlers ran. It also offers scalability benefits because without crawlers we are not dependant on the number of free IPs within the subnet.
• Improved UI testing with Playwright.

Breaking Changes

• All dataset endpoints will be prefixed with layer. Typically going from domain/dataset to layer/domain/dataset.
• All sdk functions that interact with datasets will now require an argument for layer.

Migration

• See the migration doc for details on how to migrate to v7 from v6.

v7.0.8

v7.0.8 / v0.1.6 (sdk) - 2023-11-15

Fixes

• Issue with date types when editing a schema on the UI because of no option to apply format column and therefore getting an all fields are required error.
• Tweaked UI design when adding permissions to subject.
• SDK not uploading a Pandas Dataframe with a date field set correctly.
• Updated NextJS and Zod package version.

Features

• Data bucket now has EventBridge notifications enabled by default.

Closes relevant GitHub issues

• #57

v7.0.6 / v0.1.4 (sdk) - 2023-10-18

Features

• New UI page that allows for the ability to delete users and clients easily.
• Clients can now be created and deleted via the sdk.

Fixes

• Fixed an issue with the sdk not showing schemas were created successfully due to a wrong response code.
• Where dataset info was being called on columns with a date type, this was causing an issue with the Pydantic validation.
• Tweaked the documentation to implement searching for column heading style guide to match what the API returns in the error message.

v7.0.4 / v0.1.2 (sdk) - 2023-09-20

Features

• Improved release process
• Added Athena workgroup and database as outputs of the rAPId module.

Fixes

• Updated terraform default application_version and ui_version variables.
• Migration script and documentation.

v7.0.2 / v0.1.2 (sdk) - 2023-09-14

Fixes

• Update UI repo references.

v7.0.0 / v0.1.1 (sdk) - 2023-09-12

Features

• Layers have been introduced to rAPId. These are now the highest level of grouping for your data. They allow you to separate your data into areas that relate to the layers in your data architecture e.g raw, curated, presentation. You will need to specify your layers when you create or migrate a rAPId instance.
• All the code is now in this monorepo. The previous Infrastructure, UI and API repos are now deprecated. This will ease the use and development of rAPId.
• Schemas are now stored in DynamoDB, rather than S3. This offers speed and usability improvements, as well as making rAPId easier to extend.
• Code efficiency improvements. There were several areas in rAPId where we were executing costly operations that caused performance to degrade at scale. We've fixed these inefficiencies, taking us from O(n²) -> O(n) in these areas.
• Glue Crawlers have been removed, with Athena tables are created directly by the API instead. Data is now available to query immediately after it is uploaded, rather than the previous wait (approximately 3 mins) while crawlers ran. It also offers scalability benefits because without crawlers we are not dependant on the number of free IPs within the subnet.
• Improved UI testing with Playwright.

Breaking Changes

• All dataset endpoints will be prefixed with layer. Typically going from domain/dataset to layer/domain/dataset.
• All sdk functions that interact with datasets will now require an argument for layer.

Migration

• See the migration doc for details on how to migrate to v7 from v6.

v7.0.7

v7.0.7 / v0.1.5 (sdk) - 2023-11-07

Fixes

• Issue within the sdk upload_and_create_dataset function where schema metadata wasn't being correctly overridden.
• Hitting maximum security group rules for the load balancer.
• Documentation improvements and removes any references to the old deprecated repositories.

Closes relevant GitHub issues

• #50
• #59
• #54
• #51

v7.0.5 / v0.1.3 (sdk) - 2023-09-20

Fixes

• Fix the behaviour of the dataset pattern functions in the SDK.

v7.0.3 / v0.1.2 (sdk) - 2023-09-15

Fixes

• Fixes issue where permissions were not being correctly read and causing api functionality to fail

v7.0.1 / v0.1.2 (sdk) - 2023-09-13

Fixes

• Date types were being stored as strings which caused issues when querying with Athena. They are now stored as date types.
• Rename the rAPId sdk method generate_info to fetch_dataset_info and remove an unnecessary argument.

v7.0.6

v7.0.6 / v0.1.4 (sdk) - 2023-10-18

Features

• New UI page that allows for the ability to delete users and clients easily.
• Clients can now be created and deleted via the sdk.

Fixes

• Fixed an issue with the sdk not showing schemas were created successfully due to a wrong response code.
• Where dataset info was being called on columns with a date type, this was causing an issue with the Pydantic validation.
• Tweaked the documentation to implement searching for column heading style guide to match what the API returns in the error message.

v7.0.4 / v0.1.2 (sdk) - 2023-09-20

Features

• Improved release process
• Added Athena workgroup and database as outputs of the rAPId module.

Fixes

• Updated terraform default application_version and ui_version variables.
• Migration script and documentation.

v7.0.2 / v0.1.2 (sdk) - 2023-09-14

Fixes

• Update UI repo references.

v7.0.0 / v0.1.1 (sdk) - 2023-09-12

Features

• Layers have been introduced to rAPId. These are now the highest level of grouping for your data. They allow you to separate your data into areas that relate to the layers in your data architecture e.g raw, curated, presentation. You will need to specify your layers when you create or migrate a rAPId instance.
• All the code is now in this monorepo. The previous Infrastructure, UI and API repos are now deprecated. This will ease the use and development of rAPId.
• Schemas are now stored in DynamoDB, rather than S3. This offers speed and usability improvements, as well as making rAPId easier to extend.
• Code efficiency improvements. There were several areas in rAPId where we were executing costly operations that caused performance to degrade at scale. We've fixed these inefficiencies, taking us from O(n²) -> O(n) in these areas.
• Glue Crawlers have been removed, with Athena tables are created directly by the API instead. Data is now available to query immediately after it is uploaded, rather than the previous wait (approximately 3 mins) while crawlers ran. It also offers scalability benefits because without crawlers we are not dependant on the number of free IPs within the subnet.
• Improved UI testing with Playwright.

Breaking Changes

• All dataset endpoints will be prefixed with layer. Typically going from domain/dataset to layer/domain/dataset.
• All sdk functions that interact with datasets will now require an argument for layer.

Migration

• See the migration doc for details on how to migrate to v7 from v6.

v7.0.4

v7.0.4 / v0.1.2 (sdk) - 2023-09-20

Features

• Improved release process
• Added Athena workgroup and database as outputs of the rAPId module.

Fixes

• Updated terraform default application_version and ui_version variables.
• Migration script and documentation.

v7.0.2 / v0.1.2 (sdk) - 2023-09-14

Fixes

• Update UI repo references.

v7.0.0 / v0.1.1 (sdk) - 2023-09-12

Features

• Layers have been introduced to rAPId. These are now the highest level of grouping for your data. They allow you to separate your data into areas that relate to the layers in your data architecture e.g raw, curated, presentation. You will need to specify your layers when you create or migrate a rAPId instance.
• All the code is now in this monorepo. The previous Infrastructure, UI and API repos are now deprecated. This will ease the use and development of rAPId.
• Schemas are now stored in DynamoDB, rather than S3. This offers speed and usability improvements, as well as making rAPId easier to extend.
• Code efficiency improvements. There were several areas in rAPId where we were executing costly operations that caused performance to degrade at scale. We've fixed these inefficiencies, taking us from O(n²) -> O(n) in these areas.
• Glue Crawlers have been removed, with Athena tables are created directly by the API instead. Data is now available to query immediately after it is uploaded, rather than the previous wait (approximately 3 mins) while crawlers ran. It also offers scalability benefits because without crawlers we are not dependant on the number of free IPs within the subnet.
• Improved UI testing with Playwright.

Breaking Changes

• All dataset endpoints will be prefixed with layer. Typically going from domain/dataset to layer/domain/dataset.
• All sdk functions that interact with datasets will now require an argument for layer.

Migration

• See the migration doc for details on how to migrate to v7 from v6.