Feature/s3 support

[Mike-Solar]

Add S3 support for the project. This version does not contains Chinese support.

[Mike-Solar]

@elonen

[elonen]

Thanks for this, S3 storage is a great addition!
I'm working through the PR and will be submitting patches to address a few things, as this is a rather major change.

Diff Cleanup

I've pushed a commit (d7e025b) that separates formatting changes from the actual S3 feature code. The original PR had ~12,000 lines changed in server/src/, but most were rustfmt reformatting mixed with semantic changes. The cleanup:

• Reverts formatting-only changes to preserve the original code style
• Restores Cargo.lock to version control (for reproducible builds)
• Pins AWS SDK versions compatible with Rust 1.87

S3 functionality is unchanged; this just makes the diff reviewable (~1800 lines vs ~12000).

My Concerns & Probable Changes

Browser Compatibility

Transcoding exists primarily because of limited video playback support in browsers, not for bandwidth/storage. I'm thinking:

• Add a config option to disable allowing users to skip transcoding
• Add validation when "Skip transcode" is selected to check that user know what they are doing: warn about formats that won't play in browsers (ProRes, AVI, etc.)
• Audio and image files should still be processed since they require transcoding to become playable
• Update the client UI label to clarify the risks of skipping transcoding

Configuration & Security

• ✅ DONE - Add environment variable support for S3 credentials via clap's env attribute by letting AWS SDK handle auth in the many ways it supports (avoiding secrets in CLI args/process lists)

Code Quality

• Probably refactor S3 operations to use async proper instead of creating a blocking runtime
• ✅ DONE - Add integration tests for the S3 storage path (using MinIO or similar)

Media management

• What does / should this do when a video is trashed?

Dependencies

The AWS SDK versions (aws-sdk-s3 1.115.0, aws-sdk-sts 1.94.0) require Rust 1.88+. The pinned versions in Cargo.lock (aws-sdk-s3 1.103.0) work with Rust 1.87. I'll keep 1.87 support for now, and pin lower version. Maybe update later.

What Looks Good

• Clean StorageBackend abstraction
• No breaking changes to existing installations (folder structure, DB schema, gRPC API intact - can maybe reveal S3 to Organizers later)
• basic_folders plugin remains compatible without modification
• Progress reporting and multipart upload handling are well-implemented

I'll submit patches as they're ready, feel free to comment and discuss.

[elonen]

Thoughts on user-specified transcoding mode

Bug: Default behavior changed from Auto to Force

Currently the checkbox defaults to checked (true), which sends "true" → TranscodePreference::Force. This would mean all HTTP uploads are transcoded by default, even browser-compatible H.264/MP4 files that previously weren't transcoded (unless too high bitrate).

First, let's make this an option in server config, disabled by default.
When enabled, this would fix the abovementioned problem:

1. Client: Replace checkbox with dropdown (Auto/Force/Skip)

A dropdown restores Auto as the default while offering all three options:

<script>
    let transcodeOption: "auto" | "force" | "skip" = $state("auto");
</script>

<div class="flex justify-end items-center text-sm mb-2 gap-2">
    <select
        bind:value={transcodeOption}
        class="bg-slate-800 border border-slate-600 text-gray-300 text-xs rounded px-2 py-1 focus:outline-none focus:border-slate-500"
    >
        <option value="auto">Auto transcode</option>
        <option value="force">Always transcode</option>
        <option value="skip">Skip transcoding</option>
    </select>
    {#if transcodeOption === "skip"}
        <span class="text-amber-400 text-xs">⚠ Without transcoding, file will only play if it's already browser-compatible</span>
    {/if}
</div>

Update the header to send the string value:

ajax.setRequestHeader("X-CLAPSHOT-TRANSCODE", transcodeOption);  // "auto", "force", or "skip"

(The cookie can be removed since the header is authoritative.)

2. Server: Update header parsing to match

In file_upload.rs:

let transcode_preference = hdrs
    .get("x-clapshot-transcode")
    .and_then(|v| v.to_str().ok())
    .map(|s| match s.to_ascii_lowercase().as_str() {
        "force" => TranscodePreference::Force,
        "skip" => TranscodePreference::Skip,
        _ => TranscodePreference::Auto,
    })
    .unwrap_or(TranscodePreference::Auto);

3. Server: Soft validation + notification when Skip is used on (potentially) incompatible formats

In video_pipeline/mod.rs, around line 430:

let requested_transcode = match md.transcode_preference {
    TranscodePreference::Force => {
        Some(("user requested transcoding".to_string(), target_bitrate))
    }
    TranscodePreference::Skip => {
        // Soft validate: warn if format may not be browser-compatible
        if let Some((reason, _)) = auto_transcoding_need(md, target_bitrate) {
            user_msg_tx
                .send(UserMessage {
                    topic: UserMessageTopic::Ok,
                    msg: "Transcoding skipped as requested".to_string(),
                    details: Some(format!(
                        "Note: This file may not play in all browsers ({}). \
                         If playback fails, re-upload with transcoding enabled.",
                        reason
                    )),
                    user_id: Some(md.user_id.clone()),
                    media_file_id: Some(media_id.to_string()),
                    ..Default::default()
                })
                .ok();
        }
        None
    }
    TranscodePreference::Auto => auto_transcoding_need(md, target_bitrate),
};

=> give users full control while providing clear feedback when they upload something that might not play.

[Mike-Solar]

#106

[elonen]

⚠️ This also really needs playback authentication, S3 pre-signed endpoints and/or proxy through Clapshot server (secure and foolproof but would make Server a bandwidth bottleneck).

[elonen]

Stretch goal: Make this able to co-exist with local files.
Idea: store S3 info in <videoid>/S3.json, which would signify that the video is not locally stored, but in object storage.

(Maybe do #109 while at it)

[Mike-Solar]

⚠️ This also really needs playback authentication, S3 pre-signed endpoints and/or proxy through Clapshot server (secure and foolproof but would make Server a bandwidth bottleneck).

In my design, S3 bucket is a public bucket and there's no authentication. But this actually leads security issues. I just realized this.

[Mike-Solar]

Hey, I think we can return a temporary link for S3 backend. I may do this these days. Authentication is still required.