Skip to content

Commit

Permalink
Include two CSV dumps of test vector metadata (related #16).
Browse files Browse the repository at this point in the history
  - Add coverage.csv which shows which sections are covered by which test vectors.
  - Add test_summary.csv which is a dump of several human readable fields for test vectors.
  - Using GitHub workflows to automatically generate this is deferred for a future PR.
  • Loading branch information
jwcullen committed Jun 16, 2023
1 parent ef23228 commit d10b655
Show file tree
Hide file tree
Showing 2 changed files with 116 additions and 0 deletions.
24 changes: 24 additions & 0 deletions tests/coverage.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
"primary_tested_spec_section","tests"
"2.1","test_000018"
"2.2","test_000023, test_000012, test_000005, test_000003, test_000017"
"2.5","test_000007"
"2.6","test_000000_3, test_000022"
"2.7.2","test_000217, test_000218, test_000202, test_000219, test_000200, test_000208, test_000215, test_000207, test_000204, test_000203, test_000216, test_000206, test_000201, test_000056, test_000205, test_000212, test_000220, test_000050, test_000051, test_000214, test_000036, test_000049, test_000059, test_000053, test_000054, test_000052, test_000055, test_000213"
"2.7.3","test_000107, test_000106, test_000102, test_000043, test_000039, test_000115, test_000045, test_000113, test_000108, test_000114, test_000112, test_000109, test_000105, test_000111, test_000110, test_000104, test_000044, test_000042, test_000103, test_000040, test_000048, test_000100, test_000038, test_000046, test_000101"
"2.8","test_000060, test_000058"
"2.9","test_000015, test_000019, test_000002"
"2.9.3","test_000059"
"2.10","test_000035, test_000045, test_000034, test_000023, test_000020, test_000024, test_000031, test_000021, test_000027, test_000033, test_000056, test_000025, test_000018, test_000050, test_000051, test_000028, test_000014, test_000026, test_000049, test_000046, test_000053, test_000054, test_000032, test_000052, test_000055"
"2.11","test_000006"
"2.12.1","test_000035, test_000045, test_000034, test_000023, test_000020, test_000024, test_000021, test_000027, test_000033, test_000056, test_000025, test_000050, test_000051, test_000028, test_000014, test_000026, test_000049, test_000046, test_000053, test_000054, test_000032, test_000052, test_000055"
"2.12.4","test_000029, test_000031, test_000030"
"3.1","test_000013"
"3.2","test_000231, test_000058, test_000006"
"3.10","test_000022"
"3.13.1","test_000022"
"4.1.2","test_000016"
"5","test_000037, test_000050"
"5.2.2","test_000000_3"
"6.2.3","test_000059"
"6.3.2.1","test_000223, test_000209, test_000221, test_000211"
"6.3.3","test_000058"
92 changes: 92 additions & 0 deletions tests/test_summary.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
"test","base","primary_tested_spec_sections","is_valid","description"
"test_000000_3","None","2.6/num_samples_per_frame, 5.2.2/ia_sample_duration",False,"A simple profile stereo IAMF stream with 1 substream. The final audio frame incorrectly has fewer samples than all other frames because it does not have sufficient `num_samples_to_trim_at_end`"
"test_000002","test_000000_3","2.9",True,"A stereo IAMF stream with a no-op Parameter Block OBU added."
"test_000003","None","2.2/num_samples_to_trim_at_end",True,"A simple example of a stereo IAMF stream with 1 substream and no parameter blocks."
"test_000005","test_000002","2.2/num_samples_to_trim_at_end",True,"A frame-aligned stereo IAMF stream with no trimming."
"test_000006","test_000005","2.11, 3.2/temporal_delimiter_obu",True,"A stereo base profile IAMF stream with temporal delimiters."
"test_000007","test_000005","2.5/ia_code",False,"A stereo IAMF stream with which is invalid because `ia_code` is case-sensitive and should be lowercase."
"test_000012","test_000005","2.2/num_samples_to_trim_at_end",True,"A stereo IAMF stream. Additional samples are trimmed from the end of input wav file to make it shorter."
"test_000013","test_000002","3.1",False,"A stereo IAMF stream with inconsistent values for `param_definition_mode` which should fail."
"test_000014","test_000005","2.10, 2.12.1/pre_skip",False,"A stereo IAMF stream with no trimming encoded using Opus. It is invalid because `pre_skip` is incorrectly set to 0."
"test_000015","test_000005","2.9",False,"A stereo IAMF stream that includes an extra parameter block which is not used by any audio element."
"test_000016","test_000005","4.1.2",False,"A stereo IAMF stream. It has a parameter block not aligned with the audio frames, ending 100 samples earlier. This is invalid because parameter blocks should cover the whole duration of the audio frames that use the parameter."
"test_000017","test_000005","2.2/num_samples_to_trim_at_end",False,"A stereo IAMF stream. It is invalid because it has more than one frame of samples trimmed off the end."
"test_000018","test_000005","2.1/obu_type, 2.10/explicit_audio_substream_id",True,"A stereo IAMF stream. The substream has an ID larger than 21 which requires the ID to be explicitly listed in the OBU."
"test_000019","test_000005","2.9",True,"A stereo IAMF stream that has two parameter blocks with the same ID occuring one after another."
"test_000020","test_000014","2.10, 2.12.1",True,"A stereo IAMF stream encoded using 20 ms Opus frames which encodes a 500 ms sine wav with a frequency of 1000 Hz."
"test_000021","test_000020","2.10, 2.12.1",True,"A stereo IAMF stream encoded using 40 ms Opus frames."
"test_000022","test_000020","2.6/audio_roll_distance, 3.10, 3.13.1",False,"A stereo IAMF stream encoded using 20 ms Opus frames, but the audio roll distance is incorrect."
"test_000023","test_000020","2.2/num_samples_to_trim_at_start, 2.10, 2.12.1",True,"A stereo IAMF stream encoded using 5 ms Opus frames which requires fully trimmed frames at the start."
"test_000024","test_000020","2.10, 2.12.1",True,"A stereo IAMF stream encoded using 60 ms Opus frames."
"test_000025","test_000020","2.10, 2.12.1/version",False,"A stereo IAMF stream encoded using Opus, with an incorrect Opus version."
"test_000026","test_000020","2.10, 2.12.1/output_channel_count",False,"A stereo IAMF stream encoded using Opus, with an incorrect `output_channel_count`."
"test_000027","test_000020","2.10, 2.12.1/output_gain",False,"A stereo IAMF stream encoded using Opus, with an incorrect `output_gain`."
"test_000028","test_000020","2.10, 2.12.1/channel_mapping_family",False,"A stereo IAMF stream encoded using Opus, with an incorrect `channel_mapping_family`."
"test_000029","test_000005","2.12.4/sample_rate",True,"A stereo IAMF stream encoded using LPCM with a sample rate of 48 kHz."
"test_000030","test_000029","2.12.4/sample_rate",True,"A stereo IAMF stream with a sample rate of 44.1 kHz."
"test_000031","test_000029","2.10, 2.12.4/sample_size",True,"A stereo IAMF stream with a bit-depth of 24."
"test_000032","test_000020","2.10, 2.12.1",True,"A stereo IAMF stream encoded using a target bit rate of 24000 for libopus."
"test_000033","test_000020","2.10, 2.12.1",True,"A stereo IAMF stream encoded using a target bit rate of 96000 for libopus."
"test_000034","test_000020","2.10, 2.12.1",True,"A stereo IAMF stream encoded using a VOIP mode for libopus."
"test_000035","test_000020","2.10, 2.12.1",True,"A stereo IAMF stream encoded using low-delay mode for libopus."
"test_000036","test_000005","2.7.2/channel_audio_layer_config",True,"A 2-layer IAMF stream with the last layer being 5.1."
"test_000037","test_000020","5",True,"A stereo IAMF stream encoding ~22 seconds of audio using Opus."
"test_000038","test_000005","2.7.3/ambisonics_mono_config",True,"A first-order ambisonics IAMF stream encoded using `ambisonics_mode` = `MONO`."
"test_000039","test_000038","2.7.3/ambisonics_mono_config",True,"A third-order ambisonics IAMF stream encoded using `ambisonics_mode` = `MONO`."
"test_000040","test_000038","2.7.3/ambisonics_mono_config",False,"A first-order ambisonics IAMF stream encoded using `ambisonics_mode` = `MONO`. That has a wrong number of channels for an ambisonic audio stream."
"test_000042","test_000038","2.7.3/ambisonics_projection_config",True,"A first-order ambisonics IAMF stream encoded using `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000043","test_000038","2.7.3/ambisonics_projection_config",True,"A third-order ambisonics IAMF stream encoded using `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000044","test_000043","2.7.3/ambisonics_projection_config",True,"An IAMF stream encoded using `ambisonics_mode` = `PROJECTION`. This test vector tests mixed-order ambisonics representation by using third-order ambisonics to represent a lower order (first-order) ambisonics by using a demixing matrix."
"test_000045","test_000038","2.7.3/ambisonics_mono_config, 2.10, 2.12.1",True,"A first-order ambisonics IAMF stream encoded using Opus and`ambisonics_mode` = `MONO`."
"test_000046","test_000043","2.7.3/ambisonics_mono, 2.10, 2.12.1",True,"A third-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000048","test_000042","2.7.3/ambisonics_projection_config",True,"A variant of `test_000042.textproto` that encodes in a projection mode with Opus as coupled stereo channels."
"test_000049","test_000036","2.7.2/channel_audio_layer_config, 2.10, 2.12.1",True,"A 2-layer IAMF stream encoded using Opus with the last layer being 5.1."
"test_000050","test_000037","2.7.2/channel_audio_layer_config, 2.10, 2.12.1, 5",True,"A 4-layer IAMF stream encoded using Opus with the last layer being 7.1.4."
"test_000051","test_000050","2.7.2/channel_audio_layer_config, 2.10, 2.12.1",True,"A 3-layer IAMF stream encoded using Opus with the last layer being 7.1.2."
"test_000052","test_000051","2.7.2/channel_audio_layer_config, 2.10, 2.12.1",True,"A 2-layer IAMF stream encoded using Opus with the last layer being 3.1.2."
"test_000053","test_000051","2.7.2/channel_audio_layer_config, 2.10, 2.12.1",True,"A 2-layer IAMF stream encoded using Opus with the last layer being 7.1."
"test_000054","test_000050","2.7.2/channel_audio_layer_config, 2.10, 2.12.1",True,"A 4-layer IAMF stream encoded using Opus with the last layer being 5.1.4."
"test_000055","test_000054","2.7.2/channel_audio_layer_config, 2.10, 2.12.1",True,"A 3-layer IAMF stream encoded using Opus with the last layer being 5.1.2."
"test_000056","test_000055","2.7.2/channel_audio_layer_config, 2.10, 2.12.1",True,"A 2-layer IAMF stream encoded using Opus with the last layer being 5.1.0."
"test_000058","test_000005","2.8/num_audio_elements, 3.2, 6.3.3",True,"An IAMF stream with a mix presentation which mixes 2 stereo audio elements."
"test_000059","test_000056","2.7.2, 2.9.3, 6.2.3",True,"A 2-layer IAMF stream encoded using Opus, with Parameter Block OBUs defining non-default recon gains."
"test_000060","test_000005","2.8/count_label, 2.8/language_label, 2.8/mix_presentation_annotations",True,"A stereo IAMF stream with two language labels."
"test_000100","test_000038","2.7.3/ambisonics_mono_config",True,"A zero-order ambisonics IAMF stream encoded using `ambisonics_mode` = `MONO`."
"test_000101","test_000038","2.7.3/ambisonics_mono_config",True,"A first-order ambisonics IAMF stream encoded using `ambisonics_mode` = `MONO`."
"test_000102","test_000038","2.7.3/ambisonics_mono_config",True,"A second-order ambisonics IAMF stream encoded using `ambisonics_mode` = `MONO`."
"test_000103","test_000039","2.7.3/ambisonics_mono_config",True,"A third-order ambisonics IAMF stream encoded using `ambisonics_mode` = `MONO`."
"test_000104","test_000042","2.7.3/ambisonics_projection_config",True,"A zero-order ambisonics IAMF stream encoded using `ambisonics_mode` = `PROJECTION`."
"test_000105","test_000042","2.7.3/ambisonics_projection_config",True,"A first-order ambisonics IAMF stream encoded using `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000106","test_000042","2.7.3/ambisonics_projection_config",True,"A second-order ambisonics IAMF stream encoded using `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000107","test_000043","2.7.3/ambisonics_projection_config",True,"A third-order ambisonics IAMF stream encoded using `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000108","test_000100","2.7.3/ambisonics_mono_config",True,"A zero-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `MONO`."
"test_000109","test_000101","2.7.3/ambisonics_mono_config",True,"A first-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `MONO`."
"test_000110","test_000102","2.7.3/ambisonics_mono_config",True,"A second-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `MONO`."
"test_000111","test_000103","2.7.3/ambisonics_mono_config",True,"A third-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `MONO`."
"test_000112","test_000104","2.7.3/ambisonics_projection_config",True,"A zero-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `PROJECTION`."
"test_000113","test_000105","2.7.3/ambisonics_projection_config",True,"A first-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000114","test_000106","2.7.3/ambisonics_projection_config",True,"A second-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000115","test_000107","2.7.3/ambisonics_projection_config",True,"A third-order ambisonics IAMF stream encoded using Opus and `ambisonics_mode` = `PROJECTION` with an (almost) identity demixing matrix."
"test_000200","test_000003","2.7.2/loudspeaker_layout",True,"A simple example of a mono IAMF stream and no parameter blocks."
"test_000201","test_000003","2.7.2/loudspeaker_layout",True,"A simple example of a stereo IAMF stream and no parameter blocks."
"test_000202","test_000201","2.7.2/loudspeaker_layout",True,"A simple example of a 3.1.2 IAMF stream and no parameter blocks."
"test_000203","test_000202","2.7.2/loudspeaker_layout",True,"A simple example of a 5.1 IAMF stream and no parameter blocks."
"test_000204","test_000203","2.7.2/loudspeaker_layout",True,"A simple example of a 5.1.2 IAMF stream and no parameter blocks."
"test_000205","test_000204","2.7.2/loudspeaker_layout",True,"A simple example of a 5.1.4 IAMF stream and no parameter blocks."
"test_000206","test_000205","2.7.2/loudspeaker_layout",True,"A simple example of a 7.1 IAMF stream and no parameter blocks."
"test_000207","test_000206","2.7.2/loudspeaker_layout",True,"A simple example of a 7.1.2 IAMF stream and no parameter blocks."
"test_000208","test_000207","2.7.2/loudspeaker_layout",True,"A simple example of a 7.1.4 IAMF stream and no parameter blocks."
"test_000209","test_000208","6.3.2.1/Rendering_a_channel-based_audio_element_to_loudspeakers",True,"A 7.1.4 IAMF stream to use default demixing parameter"
"test_000211","test_000208","6.3.2.1/Rendering_a_channel-based_audio_element_to_loudspeakers",True,"A 7.1.4 IAMF stream to use libear or static down-mix matrix for demixing"
"test_000212","test_000200","2.7.2/loudspeaker_layout",True,"A simple example of a mono IAMF stream with 1 substream and no parameter blocks."
"test_000213","test_000201","2.7.2/loudspeaker_layout",True,"A simple example of a stereo IAMF stream with 1 substream and no parameter blocks."
"test_000214","test_000202","2.7.2/loudspeaker_layout",True,"A simple example of a 3.1.2 IAMF stream and no parameter blocks."
"test_000215","test_000203","2.7.2/loudspeaker_layout",True,"A simple example of a 5.1 IAMF stream and no parameter blocks."
"test_000216","test_000204","2.7.2/loudspeaker_layout",True,"A simple example of a 5.1.2 IAMF stream and no parameter blocks."
"test_000217","test_000205","2.7.2/loudspeaker_layout",True,"A simple example of a 5.1.4 IAMF stream and no parameter blocks."
"test_000218","test_000206","2.7.2/loudspeaker_layout",True,"A simple example of a 7.1 IAMF stream and no parameter blocks."
"test_000219","test_000207","2.7.2/loudspeaker_layout",True,"A simple example of a 7.1.2 IAMF stream and no parameter blocks."
"test_000220","test_000208","2.7.2/loudspeaker_layout",True,"A simple example of a 7.1.4 IAMF stream and no parameter blocks."
"test_000221","test_000209","6.3.2.1/Rendering_a_channel-based_audio_element_to_loudspeakers",True,"A 7.1.4 IAMF stream to use default demixing parameter."
"test_000223","test_000223","6.3.2.1/Rendering_a_channel-based_audio_element_to_loudspeakers",True,"A 7.1.4 IAMF stream to use libear or static down-mix matrix for demixing."
"test_000231","test_000201","3.2/num_samples_to_trim_at_end",True,"A simple example of a 32bit stereo IAMF stream and no parameter blocks."

0 comments on commit d10b655

Please sign in to comment.