forked from vgmstream/vgmstream
-
Notifications
You must be signed in to change notification settings - Fork 0
v research
bnnm edited this page Apr 9, 2023
·
1 revision
- general API
- plugin-like decoders
- mixing chain
- resamplers
-
filter chain with resamples/etc need to alter output samples
-
decoder generates N samples but mixing may consume/make other samples
-
on next calls only decode if all has been consumed ("drain")
- effect may have samples to generate
- may need its own decoder delay?
-
on a mixing chain, different parts may need to hold the samples
-
this could be done with some struct like sbuf, to pass around
- each part/decoder could have its own buf and uses sbuf to swap it around (drain()?)
-
decoder generates 1024 samples
-
resamples takes that and makes 2048 samples
- but also depends on previous samples, so may "eat" all samples and return 0
-
output may consume 1000 samples
example (simple case, no mixing)
- decode ogg to sbuf_ogg_f32[1024], sets sbuf = sbuf_ogg_f32
- vgmstream returns sbuf (now pointing to sbuf_ogg_f32)
- external plays all 1024 samples, or just 600 samples and marks those consumed
- call render_vgmstream (returns same sbuf if not all was consumed)
- vgmstream sets up a base sbuf
- decode ogg to sbuf_ogg_f32[1024], sets sbuf = sbuf_ogg_f32
- ...
after (complex case)
- layout decodes layer 1
- decoder ogg to sbuf_ogg_f32[1024], sets own sbuf = sbuf_ogg_f32
- layout decodes layer 2
- decoder ogg to sbuf_ogg_f32[1024], sets own sbuf = sbuf_ogg_f32
- layout mixes both sbuf into sbuf_f32
- mixing applies volume over sbuf (now sbuf_f32)
- mixing applies resampling, changes sbuf = sbuf_f32_resample
- returns 512 samples and keeps part of the 1024
- on next call instead of decoding return other samples
typedef enum {
SFMT_S16,
SFMT_S24,
SFMT_F32,
SFMT_S16P,
SFMT_S24P,
SFMT_F32P,
} sfmt_t;
typedef struct {
void* buf; /* point to current sample buffer */
int filled; /* max samples in buffer */
int consumed; /* used samples (optional) */
int size; /* buffer max (may be bigger than current samples) */
int channels; /* step */
sfmt_t fmt; /* buffer type */
/* interleaved: buffer for all channels = [ch*s] = (ch1 ch2 ch1 ch2 ch1 ch2 ch1 ch2 ...) */
/* planar: buffer per channel = [ch][s] = (c1 c1 c1 c1 ...) (c2 c2 c2 c2 ...) */
int planar;
void* drain; //?
} sbuf_t;
helpers
- sbuf_utils.c: utils to transform between types
- sbuf_copy: copy/mmove if possible (same buf), takes format into account
- get_format? (for ffmpeg)
possible steps
- sample types, helpers and stuff
- setup: create internal buffer
- render start: check if there are samples in sbuf, call sbuf_copy
- decoder: change vorbis/mpeg/ffmpeg to return non s16
- render: convert + copy samples to out-sbuf
- if needed only (in-sbuf = last sbuf? + same type no need to do anything)
- render: change to output any sample amount
- other stuff:
- mixing order?
- fix dual stereo bufs
- loop: adjust so current loop sbuf is invalid
- need to provide external buffer (for winamp's double-for-DSP buffer) + copy on last step
check generated lists on compile to clean meta inits
- rockbox: metadata.c
- ffmpeg: allcodecs.c, allformats.c demuxer_list.ccodec_desc.c
c defines:
//c:
const struct_t blah = {
.callback = internal_function,
.callback2 = internal_function2,
...
};
//h:
extern const struct_t blah;
- WaveTrack.h: lanes
- SampleBlock.h: audio chunks
- MixAndRender.h: converts lanes
- MIDIPlay.h: audio engine
- no real-time effects
- ImportFFmpeg.cpp: handles conversion
- envelope editor
- sox_i.h > internal header
effect chain
-
create effect (sox_create_effect)
- effects: channels, rate, flangeretc
-
pass args (sox_effect_options)
-
add effect to chain (sox_add_effect)
-
first effect must be a input / sample handler
- out-> encoding = ...; out->signal.rate = ...;
-
render is called "flow" (sox_flow_effects)
- does loop if needed
-
effects must provide
- "drain" function: read "osamp" info "obuf" from "in"
- "flow" function: write "isam" total "ibuf" into "out"
-
sox_effect_t has base def + priv, that can be alloc'd on create/stop
-
sox_effect_handler_t in each file defines: create/start/stop/drain/flow
- drain: optional, called to finish getting output after input is complete
- flow: called to process samples (all of them)
- start: called before flow
-
effects have priv_t with their own init stuff
-
effects.c sox_flow_effects main proc
example
int feed_loop(...) {
do {
if (need_input) {
ilen = fread(ibuf, ..)
}
// produce as much as possible
error = soxr_process(ctx, ibuf, ilen, obuf, olen);
...
// when less than requested
need_input = odone < olen && ibuf
} while (!error && need_input)
}
// optional feeding functions
typedef size_t(* soxr_input_fn_t)( /* Supply data to be resampled. (return data_len) */
void* input_fn_state, /* As given to soxr_set_input_fn (below). */
soxr_in_t* data, /* Returned data; see below. N.B. ptr to ptr(s)*/
size_t requested_len /* Samples per channel, >= returned data_len.
);
soxr_error_t soxr_set_input_fn( /* Set (or reset) an input function.*/
soxr_t resampler, /* As returned by soxr_create. */
soxr_input_fn_t, /* Function to supply data to be resampled.*/
void * input_fn_state, /* If needed by the input function. */
size_t max_ilen /* Maximum value for input fn. requested_len.*/
);
audio loop
- read packet (av_parser_parse2)
- send packet (avcodec_send_packet)
- get samples (avcodec_receive_frame)
- access to AVFrame frame->data or extended_data if plannar
- write samples
filter loop
- create filtergraph (chain)
- create abuffer filter (feed samples)
- create other filters (changes format, channels, freq, etc)
- set filter options
- create abuffersink filter (endpoint to read samples after chain)
- link filters manually
- alloc AVFrame to store data
model: uses "libvlc"
- audio_output: mixer (pulse audio, wasapi, etc)
- definitions to handle output
- plugins define open/close/play/flush
- input: input modules
- misc: misc utils
- modules: main plugins
- audio_filter: equalizer
- audio_mixer: downmixers, etc
- audio_output: out
- decoder: adpcm.c, theora
- OpenDecoder: inits "decoder_t", calcs samples per block, etc
- Flush: .
- DecodeBlock: decodes into block->buffer
- demux: mkv
block internals
- vlc_frame.h/vlc_block.h: frame_t (blocks of binary data, to avoid passing around)
- for data, reallocs/allocs bufs as needed
- linked list for next blocks
- audio also uses p_buffer
- "BLOCK_FLAG_PREROLL : mark this block to be decoded (no matter what)."
mixing loop
- drain(): trailing samples
- owner->filters (delay check?)
- interleave/deinterleave helpers
- buf[maxch][smpl] vs buf[maxch*smpl]
- audio filters:
- callbacks: open/close/process
- converters: change for f32 to s16, etc
- resamplers: change srate
- mixers: change 5.1 to 2.0
- a filter can do multiple of the above
- filters specify if buffer is reused or needs a new one
- https://github.com/videolan/vlc/tree/master/modules/audio_filter/resampler
- https://github.com/videolan/vlc/blob/master/modules/audio_mixer/integer.c
sf_open / sf_open_file / sf_open_virtual: returns SNDFILE*
sf_close
sf_format_check
sf_seek
sf_read* / sf_write*
sf_version
sf_error
SF_INFO *sfinfo:
frames ;//samples.
samplerate
channels
format //OR of meta WAV/RAW/FLAC/etc + codec PCM8/16/FLOAT/etc + endian
sections
seekable
SF_VIRTUAL_IO *sfvirtual
sf_vio_get_filelen get_filelen ;
sf_vio_seek seek ;
sf_vio_read read ;
sf_vio_write write ;
sf_vio_tell tell ;
#define SF_BUFFER_LEN (8192)
typedef union
{
double dbuf [SF_BUFFER_LEN / sizeof (double)] ;
int64_t lbuf [SF_BUFFER_LEN / sizeof (int64_t)] ;
float fbuf [SF_BUFFER_LEN / sizeof (float)] ;
int ibuf [SF_BUFFER_LEN / sizeof (int)] ;
short sbuf [SF_BUFFER_LEN / sizeof (short)] ;
char cbuf [SF_BUFFER_LEN / sizeof (char)] ;
signed char scbuf [SF_BUFFER_LEN / sizeof (signed char)] ;
unsigned char ucbuf [SF_BUFFER_LEN / sizeof (signed char)] ;
} BUF_UNION ;