Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch asset invalidations #11937

Open
wants to merge 104 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
139613e
Formatting
somebody1234 Dec 16, 2024
3bef491
Batch `deleteAsset` calls
somebody1234 Dec 16, 2024
f48ccde
Cancel refetching directories naively by skipping refetch if any muta…
somebody1234 Dec 16, 2024
a1e2eb5
Add invalidations to batched delete mutation
somebody1234 Dec 16, 2024
3c3bfc5
Fail batched delete mutation if any requests fail
somebody1234 Dec 16, 2024
79af793
Get rid of `AssetEventType.delete`
somebody1234 Dec 16, 2024
defc5c7
Batched restore mutation and remove `AssetEventType.restore`
somebody1234 Dec 16, 2024
8a9f8b5
Replace `AssetEventType.deleteForever` with calls to `deleteAssetsMut…
somebody1234 Dec 16, 2024
6992664
Replace `AssetEventType.removeSelf` with hook
somebody1234 Dec 16, 2024
01867ff
Remove obsolete `AssetListEventType.emptyTrash`
somebody1234 Dec 16, 2024
b14a1b3
Replace `AssetListEventType.copy` with hook
somebody1234 Dec 17, 2024
4a38163
Replace `AssetListEventType.move` with hook
somebody1234 Dec 17, 2024
0f8514f
Fix `transferBetweenCategories`
somebody1234 Dec 17, 2024
b33fd6f
Remove `AssetListEventType.delete`
somebody1234 Dec 17, 2024
16f36af
Replace `AssetListEventType.clearTrash` with hook
somebody1234 Dec 17, 2024
1ce159c
Replace `AssetListEventType.duplicateProject` with hook
somebody1234 Dec 17, 2024
587f976
Remove `AssetListEvent`s completely; move `useCutAndPaste` to `cutAnd…
somebody1234 Dec 17, 2024
8aaa6b0
Formatting
somebody1234 Dec 17, 2024
a6487f8
Remove some section headers
somebody1234 Dec 17, 2024
1210d7c
Remove `move` and `delete` events and callbacks
somebody1234 Dec 17, 2024
383289d
Formatting
somebody1234 Dec 17, 2024
298686c
Replace `AssetEventType.download` with hook
somebody1234 Dec 17, 2024
5207e8a
Simplify `setCanDownload` subscriber code
somebody1234 Dec 17, 2024
1dea1d6
Formatting
somebody1234 Dec 17, 2024
354e025
WIP: Include more selection info to bring back `downloadSelected`
somebody1234 Dec 17, 2024
7588040
Fix type errors
somebody1234 Dec 18, 2024
dc1daf2
Replace `AssetEventType.downladSelected` with hook
somebody1234 Dec 18, 2024
1288aa2
Add hooks to add and remove labels
somebody1234 Dec 18, 2024
0a2791e
Add invalidations to label mutation hooks
somebody1234 Dec 18, 2024
156479a
Remove unused `addLabels` and `removeLabels` events
somebody1234 Dec 18, 2024
ef2f309
Remove `AssetEvent`s
somebody1234 Dec 18, 2024
98ffd84
(hopefully) Bring back display of temporary labels
somebody1234 Dec 18, 2024
b2f9801
Fix errors
somebody1234 Dec 18, 2024
6ef379d
Fix type errors; remove `key`, `directoryKey` and `directoryId` from …
somebody1234 Dec 18, 2024
2f9b0e1
Merge branch 'develop' into wip/sb/batch-invalidations
somebody1234 Dec 18, 2024
6ec4eab
Remove obsolete comment
somebody1234 Dec 18, 2024
5f31a8e
Remove usages of `directoryKey`
somebody1234 Dec 18, 2024
9112bbc
WIP: Fix labels drag
somebody1234 Dec 18, 2024
c3d8035
Fixes for directories opening on drag hover
somebody1234 Dec 18, 2024
600cf8c
Fix labels drag and drop not working
somebody1234 Dec 18, 2024
7d6c3db
Fix labels column display when dragging labels
somebody1234 Dec 18, 2024
153f511
WIP
somebody1234 Dec 19, 2024
c23b975
Create node from port button (#11836)
vitvakatu Dec 18, 2024
d4343f0
Improve code editor performance with snapshot-less synchronization (#…
kazcw Dec 18, 2024
6e68237
Allow a block to contain mixed indentation levels (#11870)
kazcw Dec 18, 2024
6ecd687
Fix issue with inline stings. (#11907)
jdunkerley Dec 18, 2024
083a507
Sort output of `File.list` (#11752)
GregoryTravis Dec 18, 2024
377b8db
Read MapBox API token at runtime (#11889)
vitvakatu Dec 18, 2024
5751277
Implement `Table.add_group_number`, with operations `Unique` and `Equ…
GregoryTravis Dec 18, 2024
d4e2861
Clear excel cache on reload (#11872)
GregoryTravis Dec 18, 2024
7764e28
Run workflows depending on changed files (#11723)
4e6 Dec 19, 2024
edad736
Improve Sentry support (#11776)
somebody1234 Dec 19, 2024
c7de620
Add missing entries. (#11914)
jdunkerley Dec 19, 2024
5b32aac
Add `useBackendMutation`
somebody1234 Dec 19, 2024
dd36e35
Refactor all `useMutation(backendMutationOptions())` to `useBackendMu…
somebody1234 Dec 19, 2024
a85c7e6
Add `removeAssetsLabelsMutationOptions`
somebody1234 Dec 20, 2024
b21db3c
Minor optimizations to `useBackendQuery` calls
somebody1234 Dec 20, 2024
24dfcad
Remove unnecessary `useMemo` calls in `backendHooks`
somebody1234 Dec 20, 2024
46a3243
Replace empty parameter arrays in `useBackendQuery` with `EMPTY_ARRAY`
somebody1234 Dec 20, 2024
75e94f1
Remove some usages of `useMemo`
somebody1234 Dec 20, 2024
7920957
Switch some backend mutation hooks to mutation options functions
somebody1234 Dec 20, 2024
5733250
Fix some type errors
somebody1234 Dec 20, 2024
3d7ad94
Project `listDirectory` query to `getAsset` query
somebody1234 Dec 23, 2024
726e0ea
Remove redundant functions
somebody1234 Dec 23, 2024
822e470
Remove unnecessary code
somebody1234 Dec 23, 2024
23917a3
Proper optimistic state for batch delete and restore
somebody1234 Dec 23, 2024
d9e8ced
Fix issue with labels being incorrectly added/removed due to outdated…
somebody1234 Dec 24, 2024
66ded33
Merge branch 'develop' into wip/sb/batch-invalidations
somebody1234 Dec 24, 2024
75ef031
Fix lint errors
somebody1234 Dec 24, 2024
779403f
Remove `useBackendMutation`
somebody1234 Dec 30, 2024
4ad4269
Reduce changed files
somebody1234 Dec 30, 2024
1e772c0
Merge branch 'develop' into wip/sb/batch-invalidations
somebody1234 Dec 30, 2024
88d5b1c
Switch to `useIsMutating`
somebody1234 Dec 30, 2024
05d6968
Fade assets being deleted instead of hiding them
somebody1234 Dec 30, 2024
f95a5fc
Fix bad merge
somebody1234 Dec 30, 2024
0aa1f9e
Merge branch 'develop' into wip/sb/batch-invalidations
somebody1234 Jan 2, 2025
25ce57d
Fix errors
somebody1234 Jan 2, 2025
a1d8517
Remove overzealous use of `EMPTY_ARRAY`
somebody1234 Jan 3, 2025
78886e3
Address some CRs
somebody1234 Jan 3, 2025
53c9ade
Un-lift directories query subscriber
somebody1234 Jan 3, 2025
59ca5bc
Address more CRs
somebody1234 Jan 3, 2025
c830ea6
Split `backendHooks`
somebody1234 Jan 3, 2025
bd824ca
Rename `backendHooks`
somebody1234 Jan 3, 2025
cd3417b
Fix errors
somebody1234 Jan 3, 2025
0a6fdf5
Fix errors
somebody1234 Jan 3, 2025
8b199f6
Merge branch 'develop' into wip/sb/batch-invalidations
somebody1234 Jan 3, 2025
2208ae4
Move download logic to `RemoteBackend` and `LocalBackend`
somebody1234 Jan 3, 2025
938337d
Avoid wrapping mutation in mutation for `useRemoveSelfPermissionMutat…
somebody1234 Jan 3, 2025
03297f9
Refactor `useDownloadAssetsMutation` to `downloadAssetsMutationOptions`
somebody1234 Jan 3, 2025
c828375
Refactor `useUploadFileMutation` to return a proper mutation result
somebody1234 Jan 3, 2025
ae968e5
Return `UseQueryResult` from `useListUserGroupsWithUsers`
somebody1234 Jan 3, 2025
e9de7af
Fix errors
somebody1234 Jan 6, 2025
6f28ad4
Fix logic error
somebody1234 Jan 6, 2025
4b9f8fd
Remove `clearTrashMutation` instead of `ensureQuery` + `deleteAssetsM…
somebody1234 Jan 6, 2025
46c57e5
Refactor `useDuplicateProjectMutation` to `duplicateProjectMutationOp…
somebody1234 Jan 6, 2025
3760ca4
Merge branch 'develop' into wip/sb/batch-invalidations
somebody1234 Jan 7, 2025
5c7e43e
Fix "Clear Trash"
somebody1234 Jan 7, 2025
4424e4d
Fix regression
somebody1234 Jan 7, 2025
18b0749
Switch from sync to async callback in `ConfirmDeleteModalProps`
somebody1234 Jan 7, 2025
52c8297
Add `meta.refetchType` for mutations
somebody1234 Jan 7, 2025
c84a6b3
Add `refetchType: 'all'` to all mutations that invalidate `listDirect…
somebody1234 Jan 7, 2025
74be4b0
Fix regression causing asset rows to re-render every time
somebody1234 Jan 7, 2025
deeae0a
Fix infinite recursion error
somebody1234 Jan 7, 2025
6c3a972
Fix re-render issues?
somebody1234 Jan 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -191,3 +191,9 @@ test-results/
test-traces/
playwright-report/
playwright/.cache/

#########
## Git ##
#########

/.mailmap
10 changes: 6 additions & 4 deletions app/common/src/backendQuery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ export type BackendMethods = object.ExtractKeys<Backend, object.MethodOf<Backend

/** For each backend method, an optional function defining how to create a query key from its arguments. */
type BackendQueryNormalizers = {
[Method in BackendMethods]?: (...args: Parameters<Backend[Method]>) => queryCore.QueryKey
[Method in BackendMethods]?: (
...args: Readonly<Parameters<Backend[Method]>>
) => queryCore.QueryKey
}

const NORMALIZE_METHOD_QUERY: BackendQueryNormalizers = {
Expand All @@ -22,7 +24,7 @@ const NORMALIZE_METHOD_QUERY: BackendQueryNormalizers = {
/** Creates a partial query key representing the given method and arguments. */
function normalizeMethodQuery<Method extends BackendMethods>(
method: Method,
args: Parameters<Backend[Method]>,
args: Readonly<Parameters<Backend[Method]>>,
) {
return NORMALIZE_METHOD_QUERY[method]?.(...args) ?? args
}
Expand All @@ -31,7 +33,7 @@ function normalizeMethodQuery<Method extends BackendMethods>(
export function backendQueryOptions<Method extends BackendMethods>(
backend: Backend | null,
method: Method,
args: Parameters<Backend[Method]>,
args: Readonly<Parameters<Backend[Method]>>,
keyExtra?: queryCore.QueryKey | undefined,
): {
queryKey: queryCore.QueryKey
Expand All @@ -47,7 +49,7 @@ export function backendQueryOptions<Method extends BackendMethods>(
export function backendQueryKey<Method extends BackendMethods>(
backend: Backend | null,
method: Method,
args: Parameters<Backend[Method]>,
args: Readonly<Parameters<Backend[Method]>>,
keyExtra?: queryCore.QueryKey | undefined,
): queryCore.QueryKey {
return [backend?.type, method, ...normalizeMethodQuery(method, args), ...(keyExtra ?? [])]
Expand Down
4 changes: 4 additions & 0 deletions app/common/src/queryClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ declare module '@tanstack/query-core' {
* @default false
*/
readonly awaitInvalidates?: queryCore.QueryKey[] | boolean
readonly refetchType?: queryCore.InvalidateQueryFilters['refetchType']
}

readonly queryMeta: {
Expand Down Expand Up @@ -98,6 +99,7 @@ export function createQueryClient<TStorageValue = string>(
mutationCache: new queryCore.MutationCache({
onSuccess: (_data, _variables, _context, mutation) => {
const shouldAwaitInvalidates = mutation.meta?.awaitInvalidates ?? false
const refetchType = mutation.meta?.refetchType ?? 'active'
const invalidates = mutation.meta?.invalidates ?? []
const invalidatesToAwait = (() => {
if (Array.isArray(shouldAwaitInvalidates)) {
Expand All @@ -113,6 +115,7 @@ export function createQueryClient<TStorageValue = string>(
for (const queryKey of invalidatesToIgnore) {
void queryClient.invalidateQueries({
predicate: query => queryCore.matchQuery({ queryKey }, query),
refetchType,
})
}

Expand All @@ -121,6 +124,7 @@ export function createQueryClient<TStorageValue = string>(
invalidatesToAwait.map(queryKey =>
queryClient.invalidateQueries({
predicate: query => queryCore.matchQuery({ queryKey }, query),
refetchType,
}),
),
)
Expand Down
22 changes: 15 additions & 7 deletions app/common/src/services/Backend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1070,6 +1070,18 @@ export function assetIsType<Type extends AssetType>(type: Type) {
return (asset: AnyAsset): asset is Extract<AnyAsset, Asset<Type>> => asset.type === type
}

/** Extract the type of an id and return a discriminated union containing both id and type. */
export function extractTypeFromId(id: AssetId): AnyAsset extends infer T ?
T extends T ?
Pick<T, ('id' | 'type') & keyof T>
: never
: never {
return {
type: id.match(/^(.+?)-/)?.[1],
id,
} as never
}

/** Creates a new placeholder asset id for the given asset type. */
export function createPlaceholderAssetId<Type extends AssetType>(
type: Type,
Expand Down Expand Up @@ -1674,11 +1686,7 @@ export default abstract class Backend {
title: string,
): Promise<CreatedProject>
/** Return project details. */
abstract getProjectDetails(
projectId: ProjectId,
directoryId: DirectoryId | null,
getPresignedUrl?: boolean,
): Promise<Project>
abstract getProjectDetails(projectId: ProjectId, getPresignedUrl?: boolean): Promise<Project>
/** Return Language Server logs for a project session. */
abstract getProjectSessionLogs(
projectSessionId: ProjectSessionId,
Expand Down Expand Up @@ -1767,8 +1775,8 @@ export default abstract class Backend {
projectId?: string | null,
metadata?: object | null,
): Promise<void>
/** Download from an arbitrary URL that is assumed to originate from this backend. */
abstract download(url: string, name?: string): Promise<void>
/** Download an asset. */
abstract download(assetId: AssetId, title: string): Promise<void>

/**
* Get the URL for the customer portal.
Expand Down
4 changes: 0 additions & 4 deletions app/common/src/text/english.json
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@
"editDescriptionError": "Could not edit description",
"canOnlyDownloadFilesError": "You currently can only download files.",
"noProjectSelectedError": "First select a project to download.",
"downloadInvalidTypeError": "You can only download files, projects, and Datalinks",
"downloadProjectError": "Could not download project '$0'",
"downloadFileError": "Could not download file '$0'",
"downloadDatalinkError": "Could not download Datalink '$0'",
Expand All @@ -64,9 +63,6 @@
"nameShouldNotContainInvalidCharacters": "Name should not contain invalid characters",
"invalidEmailValidationError": "Please enter a valid email address",

"projectHasNoSourceFilesPhrase": "project has no source files",
"fileNotFoundPhrase": "file not found",

"noNewProfilePictureError": "Could not upload a new profile picture because no image was found",

"registrationError": "Something went wrong! Please try again or contact the administrators.",
Expand Down
2 changes: 1 addition & 1 deletion app/common/src/utilities/data/array.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/** @file Utilities for manipulating arrays. */

export const EMPTY_ARRAY: readonly never[] = []
export const EMPTY_ARRAY: readonly [] = []

// ====================
// === shallowEqual ===
Expand Down
9 changes: 9 additions & 0 deletions app/common/src/utilities/data/object.ts
Original file line number Diff line number Diff line change
Expand Up @@ -219,3 +219,12 @@ export function useObjectId() {
* can be used to splice together objects without the risk of collisions.
*/
export type DisjointKeysUnion<A, B> = keyof A & keyof B extends never ? A & B : never

/**
* Merge types of values of an object union. Useful to return an object that UNSAFELY
* (at runtime) conforms to the shape of a discriminated union.
* Especially useful for things like Tanstack Query results.
*/
export type MergeValuesOfObjectUnion<T> = {
[K in `${keyof T & string}`]: T[K & keyof T]
}
5 changes: 1 addition & 4 deletions app/gui/src/dashboard/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -571,10 +571,6 @@ function AppRouter(props: AppRouterProps) {
)
}

// ====================================
// === LocalBackendPathSynchronizer ===
// ====================================

/** Keep `localBackend.rootPath` in sync with the saved root path state. */
function LocalBackendPathSynchronizer() {
const [localRootDirectory] = localStorageProvider.useLocalStorageState('localRootDirectory')
Expand All @@ -586,5 +582,6 @@ function LocalBackendPathSynchronizer() {
localBackend.resetRootPath()
}
}

return null
}
Loading
Loading