feat: tag cleanup job

[jrasm91]

• Add a job to remove empty tags
• Add a button in the UI to manually run a few different clean up jobs

Light	Dark

[jrasm91]

@mertalev if you know a better way to do this, feel free to replace it.

[danieldietzler]

Code looks slick 🚀

[mertalev]

Suggested change

	await this.dataSource.transaction(async (manager) => {
	const ids = new Set<string>();
	const tags = await manager.find(TagEntity);
	for (const tag of tags) {
	const count = await manager
	.createQueryBuilder('assets', 'asset')
	.innerJoin(
	'asset.tags',
	'asset_tags',
	'asset_tags.id IN (SELECT id_descendant FROM tags_closure WHERE id_ancestor = :tagId)',
	{ tagId: tag.id },
	)
	.getCount();
	if (count === 0) {
	this.logger.debug(`Found empty tag: ${tag.id} - ${tag.value}`);
	ids.add(tag.id);
	}
	}
	if (ids.size > 0) {
	await manager.delete(TagEntity, { id: In([...ids]) });
	this.logger.log(`Deleted ${ids.size} empty tags`);
	}
	});
	await this.dataSource.query(`
	delete from tags
	where not exists(
	select 1
	from tag_asset
	where tag_asset."tagsId" = tags.id
	)`);

[jrasm91]

If I have a tag Parent/Child I don't want to delete Parent or Child if Child is connected to an asset. Does this handle that case?

[mertalev]

Yes, assuming that if the child has a row in tag_asset, the parent also has a row.

[jrasm91]

The parent doesn't necessarily have a row.

[mertalev]

Hm, why not? It isn't a parent/child relationship if the parent doesn't always include the child.

[jrasm91]

We link a single tag "Event/Year/Title" to a single asset. We don't want to have to link every tag in the hierarchy, eg Event and Event/Year and Event/Year/Title - that is implied by the fact that Event/Year/Title is the tag name. In terms of the database we track that relationship in a separate table. We have one link per tag hierarchy, instead of one record per asset/nested/tag combination, which would be a lot more records to read, write, and sync.

[jrasm91]

I don't know how to exactly write the not exists query with the current query builder, but I can try to take a look at this in the future.

[alextran1502]

I find the button Create job confusing. Maybe we can have a Clean up job card with three buttons corresponding to the three jobs we have here?

[jrasm91]

Maybe. I think there are plans to add more manual jobs and potentially trigger all jobs to be run through a separate interface like this. I'm open to different UX though.

The current problem is that we only have two options - force=true or force=false - and not all of our jobs work with just that option.

Some examples:

• all the clean up jobs
• metadata extraction just needs to be run on videos
• any job just needs to run on assets processed after a certain date
• changing between thumbnail formats for one thumbnail type doesn't need to regenerate everything

Maybe we can work towards separating the queue stats from the way we queue jobs and the types of jobs we can queue.

[alextran1502]

@jrasm91 sounds good, we can think of a better way of representing these new jobs in a different PR