…5013)

# Description

This PR is the first one related with distribution task feature, adding
the following changes:
* Added `distribution` JSON column to `datasets` table:
* This column is non-nullable so a value is always required when a
dataset is created.
* By default old datasets will have the value `{"strategy": "overlap",
"min_submitted": 1}`.
* Added `distribution` attribute to `DatasetCreate` schema:
  * None is not a valid value.
* If no value is specified for this attribute
`DatasetOverlapDistributionCreate` with `min_submitted` to `1` is used.
* `DatasetOverlapDistributionCreate` only allows values greater or equal
than `1` for `min_submitted` attributed.
* Now the context `create_dataset` function is receiving a dictionary
instead of `DatasetCreate` schema.
* Moved dataset creation validations to a new `DatasetCreateValidator`
class.

Update of `distribution` attribute for datasets will be done in a
different issue.

Closes #5005 

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (change restructuring the codebase without changing
functionality)
- [ ] Improvement (change adding some improvement to an existing
functionality)
- [ ] Documentation update

**How Has This Been Tested**

(Please describe the tests that you ran to verify your changes. And
ideally, reference `tests`)

- [x] Adding new tests and passing old ones.
- [x] Check that migration works as expected with old datasets and
SQLite.
- [x] Check that migration works as expected with old datasets and
PostgreSQL.

**Checklist**

- [ ] I added relevant documentation
- [ ] follows the style guidelines of this project
- [ ] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [ ] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Paco Aranda <[email protected]>

…nly (#5148)

# Description

Add changes to `responses_submitted` relationship to avoid problems with
existent `responses` relationship and avoid a warning message that
SQLAlchemy was reporting.

Refs #5000 

**Type of change**

- Improvement (change adding some improvement to an existing
functionality)

**How Has This Been Tested**

- [x] Warning is not showing anymore.
- [x] Test are passing.

**Checklist**

- I added relevant documentation
- follows the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

# Description

This PR adds changes to the endpoints to get the dataset progress and
current user metrics in the following way:

## `GET /datasets/:dataset_id/progress`

I have changed the endpoint to support the new business logic behind the
distribution task. Responding with only `completed` and `pending` type
of records and using `total` as the sum of the two types of records.

Old response without distribution task:

```json
{
  "total": 8,
  "submitted": 2,
  "discarded": 2,
  "conflicting": 1,
  "pending": 3
}
```

New response with the changes from this PR supporting distribution task:

* The `completed` attribute will have the count of all the records with
status as `completed` for the dataset.
* The `pending` attribute will have the count of all the records with
status as `pending` for the dataset.
* The `total` attribute will have the sum of the `completed` and
`pending` attributes.

```json
{
  "total": 5
  "completed": 2,
  "pending": 3,
}
```

@damianpumar some changes are required on the frontend to support this
new endpoint structure.

## `GET /me/datasets/:dataset_id/metrics`

Old response without distribution task:

```json
{
  "records": {
    "count": 7
  },
  "responses": {
    "count": 4,
    "submitted": 1,
    "discarded": 2,
    "draft": 1
  }
}
```

New response with the changes from this PR supporting distribution task:

* `records` section has been eliminated because is not necessary
anymore.
* `responses` `count` section has been renamed to `total`.
* `pending` section has been added to the `responses` section.

```json
{
  "responses": {
    "total": 7,
    "submitted": 1,
    "discarded": 2,
    "draft": 1,
    "pending": 3
  }
}
```

The logic behind these attributes is the following:
* `total` is the sum of `submitted`, `discarded`, `draft` and `pending`
attribute values.
* `submitted` is the count of all responses belonging to the current
user in the specified dataset with `submitted` status.
* `discarded` is the count of all responses belonging to the current
user in the specified dataset with `discarded` status.
* `draft` is the count of all responses belonging to the current user in
the specified dataset with `draft` status.
* `pending` is the count of all records with `pending` status for the
dataset that has not responses belonging to the current user.

@damianpumar some changes are required on the frontend to support this
new endpoint structure as well.

Closes #5139 

**Type of change**

- Breaking change (fix or feature that would cause existing
functionality to not work as expected)

**How Has This Been Tested**

- [x] Modifying existent tests.
- [x] Running test suite with SQLite and PostgreSQL.

**Checklist**

- I added relevant documentation
- follows the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Paco Aranda <[email protected]>
Co-authored-by: Damián Pumar <[email protected]>

#5153)

<!-- Please include a summary of the changes and the related issue.
Please also include relevant motivation and context. List any
dependencies that are required for this change. -->

This PR is the first of a series of PRs for cleaning the
listing-records-related endpoints.

This PR removes the `GET /api/v1/me/datasets/:dataset_id/records`
endpoint since the only client was the frontend application and now is
using the equivalent search endpoint.

**Type of change**
<!-- Please delete options that are not relevant. Remember to title the
PR according to the type of change -->

- Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- Improvement (change adding some improvement to an existing
functionality)

**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->

**Checklist**
<!-- Please go over the list and make sure you've taken everything into
account -->

- I added relevant documentation
- follows the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

…aram (#5156)

# Description
<!-- Please include a summary of the changes and the related issue.
Please also include relevant motivation and context. List any
dependencies that are required for this change. -->

> [!NOTE]
> This PR must be merged after
#5153

This PR removes support for filtering using metadata as a query param:

-  This filter is not available anymore for list endpoints
- The metadata filter can be defined as part of the request body for
search filters.

**Type of change**
<!-- Please delete options that are not relevant. Remember to title the
PR according to the type of change -->

- Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- Refactor (change restructuring the codebase without changing
functionality)

**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->

**Checklist**
<!-- Please go over the list and make sure you've taken everything into
account -->

- I added relevant documentation
- follows the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

…upport (#5163)

# Description
<!-- Please include a summary of the changes and the related issue.
Please also include relevant motivation and context. List any
dependencies that are required for this change. -->

> [!NOTE]
> This PR must be merged after
#5156

This PR removes support for filtering records with response_status query
param:

- This filter is removed for listing records endpoints
- The response status filter is available for search endpoints using the
filter request body.

**Type of change**
<!-- Please delete options that are not relevant. Remember to title the
PR according to the type of change -->

- Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- Refactor (change restructuring the codebase without changing
functionality)

**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->

**Checklist**
<!-- Please go over the list and make sure you've taken everything into
account -->

- I added relevant documentation
- I followed the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: José Francisco Calvo <[email protected]>

…notated datasets (#5171)

# Description
<!-- Please include a summary of the changes and the related issue.
Please also include relevant motivation and context. List any
dependencies that are required for this change. -->

This PR changes the current validator when updating the distribution
task to allow updating the distribution task settings for datasets with
records without ANY response.

cc @nataliaElv 

**Type of change**
<!-- Please delete options that are not relevant. Remember to title the
PR according to the type of change -->

- Improvement (change adding some improvement to an existing
functionality)

**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->

**Checklist**
<!-- Please go over the list and make sure you've taken everything into
account -->

- I added relevant documentation
- I followed the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

…5166)

# Description
<!-- Please include a summary of the changes and the related issue.
Please also include relevant motivation and context. List any
dependencies that are required for this change. -->

This PR removes support of `sort_by` query param for list/search records
endpoints.

**Type of change**
<!-- Please delete options that are not relevant. Remember to title the
PR according to the type of change -->

- Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- Refactor (change restructuring the codebase without changing
functionality)

**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->

**Checklist**
<!-- Please go over the list and make sure you've taken everything into
account -->

- I added relevant documentation
- I followed the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

# Description

This PR add a new `get_serializable_async_db` function helper that
returns a session using isolation leve as `SERIALIZABLE`.

This session can be used on some handlers where we require that specific
isolation level.

As example I have added that session helper for handler deleting
responses and PostgreSQL is showing the following received queries:

```sql
2024-07-04 17:09:40.417 CEST [83566] LOG:  statement: BEGIN ISOLATION LEVEL READ COMMITTED;
2024-07-04 17:09:40.418 CEST [83566] LOG:  execute __asyncpg_stmt_e__: SELECT users.first_name, users.last_name, users.username, users.role, users.api_key, users.password_hash, users.id, users.inserted_at, users.updated_at
	FROM users
	WHERE users.api_key = $1::VARCHAR
2024-07-04 17:09:40.418 CEST [83566] DETAIL:  parameters: $1 = 'argilla.apikey'
2024-07-04 17:09:40.422 CEST [83566] LOG:  execute __asyncpg_stmt_12__: SELECT users_1.id AS users_1_id, workspaces.name AS workspaces_name, workspaces.id AS workspaces_id, workspaces.inserted_at AS workspaces_inserted_at, workspaces.updated_at AS workspaces_updated_at
	FROM users AS users_1 JOIN workspaces_users AS workspaces_users_1 ON users_1.id = workspaces_users_1.user_id JOIN workspaces ON workspaces.id = workspaces_users_1.workspace_id
	WHERE users_1.id IN ($1::UUID) ORDER BY workspaces_users_1.inserted_at ASC
2024-07-04 17:09:40.422 CEST [83566] DETAIL:  parameters: $1 = 'ed2d570f-cc9f-4d53-a433-74aa7a286a52'
2024-07-04 17:09:40.426 CEST [83566] LOG:  execute __asyncpg_stmt_13__: SELECT users.first_name, users.last_name, users.username, users.role, users.api_key, users.password_hash, users.id, users.inserted_at, users.updated_at
	FROM users
	WHERE users.username = $1::VARCHAR
2024-07-04 17:09:40.426 CEST [83566] DETAIL:  parameters: $1 = 'argilla'
2024-07-04 17:09:40.428 CEST [83566] LOG:  execute __asyncpg_stmt_12__: SELECT users_1.id AS users_1_id, workspaces.name AS workspaces_name, workspaces.id AS workspaces_id, workspaces.inserted_at AS workspaces_inserted_at, workspaces.updated_at AS workspaces_updated_at
	FROM users AS users_1 JOIN workspaces_users AS workspaces_users_1 ON users_1.id = workspaces_users_1.user_id JOIN workspaces ON workspaces.id = workspaces_users_1.workspace_id
	WHERE users_1.id IN ($1::UUID) ORDER BY workspaces_users_1.inserted_at ASC
2024-07-04 17:09:40.428 CEST [83566] DETAIL:  parameters: $1 = 'ed2d570f-cc9f-4d53-a433-74aa7a286a52'
2024-07-04 17:09:40.430 CEST [83563] LOG:  statement: BEGIN ISOLATION LEVEL SERIALIZABLE;
2024-07-04 17:09:40.430 CEST [83563] LOG:  execute __asyncpg_stmt_14__: SELECT responses.values, responses.status, responses.record_id, responses.user_id, responses.id, responses.inserted_at, responses.updated_at
	FROM responses
	WHERE responses.id = $1::UUID
2024-07-04 17:09:40.430 CEST [83563] DETAIL:  parameters: $1 = 'fdea95a0-ee9a-43ea-b093-2e13f2473c19'
2024-07-04 17:09:40.431 CEST [83566] LOG:  statement: ROLLBACK;
2024-07-04 17:09:40.432 CEST [83563] LOG:  statement: ROLLBACK;
```

We can clearly see that there are two nested transaction:
1. The main one to get current user using default `get_async_db` helper.
2. A nested one using `get_serializable_async_db` (and setting
`SERIALIZABLE` isolation level) trying to find the response by id.

The response id used is fake so the transaction ends there and the
deletion is not done.

## Missing things on this PR

- [x] Fix some failing tests.
- [ ] Tests are passing but still not changing the isolation level to
`SERIALIZABLE`.
- [ ] Check that this works as expected and does not affect SQLite.
- [ ] Check that this works as expected with PostgreSQL (no concurrency
errors).


Closes #5155

**Type of change**

- New feature (non-breaking change which adds functionality)
- Improvement (change adding some improvement to an existing
functionality)


**How Has This Been Tested**

- [x] Manually seeing PostgreSQL logs.

**Checklist**

- I added relevant documentation
- I followed the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

# Description
<!-- Please include a summary of the changes and the related issue.
Please also include relevant motivation and context. List any
dependencies that are required for this change. -->

This PR removes deprecated endpoints working with records to avoid
creating records with a proper record status computation. The affected
endpoints are:

`POST /api/v1/datasets/:dataset_id/records`
`PATCH /api/v1/datasets/:dataset_id/records`


**Type of change**
<!-- Please delete options that are not relevant. Remember to title the
PR according to the type of change -->

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- Refactor (change restructuring the codebase without changing
functionality)
- Improvement (change adding some improvement to an existing
functionality)
- Documentation update

**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->

**Checklist**
<!-- Please go over the list and make sure you've taken everything into
account -->

- I added relevant documentation
- I followed the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

# Description
<!-- Please include a summary of the changes and the related issue.
Please also include relevant motivation and context. List any
dependencies that are required for this change. -->

This PR adds the record status as a read-only property in the `Record`
resource class.

Closes #5141

**Type of change**
<!-- Please delete options that are not relevant. Remember to title the
PR according to the type of change -->

- Improvement (change adding some improvement to an existing
functionality)

**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->

**Checklist**
<!-- Please go over the list and make sure you've taken everything into
account -->

- I added relevant documentation
- I followed the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

…ctor/cleaning-list-records-endpoints

…/argilla-server/list-records-endpoint-using-db

…ctor/cleaning-list-records-endpoints

…/argilla-server/list-records-endpoint-using-db

…oint-using-db

for more information, see https://pre-commit.ci

…/useDatasetProgressViewModel.ts

…oint-using-db

…oint-using-db

…oint-using-db

…oint-using-db

…oint-using-db

…oint-using-db

for more information, see https://pre-commit.ci

…oint-using-db

…oint-using-db

…oint-using-db

…oint-using-db

… of github.com:argilla-io/argilla into refactor/argilla-server/list-records-endpoint-using-db

…oint-using-db

…oint-using-db

for more information, see https://pre-commit.ci

…oint-using-db