Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FG-2933: Add CompressedAudio schema #159

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:
branches: [main]
tags: ["releases/**"]
pull_request:
branches: ["*"]
branches: ["**"]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this so that CI runs on my branches with a slash in the branch name. I'm happy to remove this if you're not interested in this change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 I've been using slashes in my branch names and didn't have an issue with CI running: #158

Copy link
Contributor Author

@fgwt202412 fgwt202412 Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is targeting a branch other than main (my other branch, fgwt202412/ubuntu-22_04-python-3_7), which does have a slash in it.

Docs

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, makes sense, this is for the target branch rather than the head branch. In some other repos we use pull_request: {} which may do the same thing by default? Anyhow, this seems fine


jobs:
build:
Expand Down
35 changes: 35 additions & 0 deletions internal/__snapshots__/exportTypeScriptSchemas.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,40 @@ export type CompressedVideo = {
*/
format: string;
};
",
"CompressedAudio" => "// Generated by https://github.com/foxglove/schemas
// Options: {}

import { Time } from "./Time";

/** A single frame of a compressed audio bitstream */
export type CompressedAudio = {
/** Timestamp of audio frame */
timestamp: Time;

/** Compressed audio frame data. */
data: Uint8Array;

/**
* Audio format.
*
* Supported values: \`opus\`.
*/
format: string;

/**
* Frame type.
*
* Supported values: \`key\`, \`delta\`.
*/
type: string;

/** Number of audio samples per second. */
sample_rate: number;

/** Number of audio channels in the input. */
number_of_channels: number;
};
",
"CylinderPrimitive" => "// Generated by https://github.com/foxglove/schemas
// Options: {}
Expand Down Expand Up @@ -1104,6 +1138,7 @@ export enum PositionCovarianceType {
export * from "./CameraCalibration";
export * from "./CircleAnnotation";
export * from "./Color";
export * from "./CompressedAudio";
export * from "./CompressedImage";
export * from "./CompressedVideo";
export * from "./CubePrimitive";
Expand Down
39 changes: 39 additions & 0 deletions internal/schemas.ts
Original file line number Diff line number Diff line change
Expand Up @@ -843,6 +843,44 @@ Specifically, the requirements for different \`format\` values are:
],
};

const CompressedAudio: FoxgloveMessageSchema = {
type: "message",
name: "CompressedAudio",
description: "A single frame of a compressed audio bitstream",
fields: [
{
name: "timestamp",
type: { type: "primitive", name: "time" },
description: "Timestamp of audio frame",
},
{
name: "data",
type: { type: "primitive", name: "bytes" },
description: `Compressed audio frame data.`,
},
{
name: "format",
type: { type: "primitive", name: "string" },
description: "Audio format.\n\nSupported values: `opus`.",
},
{
name: "type",
type: { type: "primitive", name: "string" },
description: "Frame type.\n\n Supported values: `key`, `delta`.",
},
Comment on lines +866 to +870
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We historically chose not to include this for CompressedVideo, although I don't think I know the whole reasoning behind the decision. It's certainly caused us a lot of pain having to detect key frames ourselves, but on the other hand it's nicer for users. cc @defunctzombie @snosenzo for any thoughts / additional historical context

Copy link
Contributor Author

@fgwt202412 fgwt202412 Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In practice with the MCAP recording demo as it is in foxglove/mcap#1293, the frames are all keyframes, which would make this field somewhat unnecessary. That might not be the case for all users though.

It definitely pains me to include this extra field in every message (more data => more storage costs), but it does mean there is no logic of detecting keyframes like you said. A cursory search did not come up with how to detect audio keyframes, so I'd be worried about the complexity of the logic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We historically chose not to include this for CompressedVideo, although I don't think I know the whole reasoning behind the decision. It's certainly caused us a lot of pain having to detect key frames ourselves, but on the other hand it's nicer for users. cc @defunctzombie @snosenzo for any thoughts / additional historical context

I pushed for fewer fields (like dropping isKeyframe and other metadata config). My arguments were similar to those discussed here: a) easier for the user and b) fewer fields that did not matter 99.9% of the time c) information which was already present in the data.

I would have the same arguments for type here. It seems like we should be able to identify this from the stream (and the proposed panel code does not use this field so why add it now - we can add it later).

sample_rate and num_channels seem to behave differently and are not present in the data itself which means they need to be present somewhere. One could imagine this information could have been something in the mcap channel metadata but we have not leaned on that and it makes it harder to reason about how to craft the messages.

Do we need to support this "description" concept? https://developer.mozilla.org/en-US/docs/Web/API/AudioDecoder/configure#description

{
name: "sample_rate",
type: { type: "primitive", name: "uint32" },
description: "Number of audio samples per second.",
},
{
name: "number_of_channels",
type: { type: "primitive", name: "uint32" },
description: "Number of audio channels in the input.",
},
],
};

const RawImage: FoxgloveMessageSchema = {
type: "message",
name: "RawImage",
Expand Down Expand Up @@ -1482,6 +1520,7 @@ export const foxgloveMessageSchemas = {
Color,
CompressedImage,
CompressedVideo,
CompressedAudio,
CylinderPrimitive,
CubePrimitive,
FrameTransform,
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@foxglove/schemas",
"version": "1.6.6",
"version": "1.6.7",
"description": "Foxglove-defined message schemas for ROS, Protobuf, FlatBuffers, OMG IDL, and JSON",
"license": "MIT",
"repository": {
Expand Down
26 changes: 26 additions & 0 deletions ros_foxglove_msgs/ros1/CompressedAudio.msg

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 26 additions & 0 deletions ros_foxglove_msgs/ros2/CompressedAudio.msg

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

95 changes: 95 additions & 0 deletions schemas/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

32 changes: 32 additions & 0 deletions schemas/flatbuffer/CompressedAudio.fbs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

47 changes: 47 additions & 0 deletions schemas/jsonschema/CompressedAudio.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading