DP modules deadline calculations #10177

marcinszkudlinski · 2025-08-20T07:27:00Z

This PR introduces proper Deadlines calculations, based on current pipeline state.
Verbose description provided in first commit

it allows to calculate deadlines even in case of DP to DP connections

Thinks todo

Finally get rid of comp_buffer in pipeline, replace it with abstract "class" audio_buffer. This will allow DP to DP conection within pipeline
cross core access to module's internals. Required for deadline calculations based on next module state, that may be on diffrent core
modules LongestProcessingTime (see 1st commit for definitions) based on data from manifest

also a tiny change in Zephyr is required - thread deadline set to absolute timestamp, not always to "now",
zephyrproject-rtos/zephyr#94731

EDIT: documentation from .c file moved here:

thesofproject/sof-docs#515

Copilot

Pull Request Overview

This PR introduces proper deadline calculations for DP (Data Processing) modules using Earliest Deadline First (EDF) scheduling, allowing DP to DP connections within pipelines. The implementation calculates deadlines based on current pipeline state and buffer conditions rather than simple timing assumptions.

Implements comprehensive deadline calculation algorithm based on buffer Latest Feeding Time (LFT) and module Longest Processing Time (LPT)
Adds pre-run notifier events to capture timestamps for deadline calculations
Introduces new sink API for calculating Latest Feeding Time (LFT) for buffers

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
west.yml	Updates Zephyr dependency to use custom fork with thread deadline improvements
src/schedule/zephyr_ll.c	Adds LL_PRE_RUN notifier event before task execution
src/schedule/ll_schedule.c	Adds LL_PRE_RUN notifier event before task execution
src/schedule/zephyr_dp_schedule.c	Major refactoring to implement EDF scheduling with proper deadline calculations
src/include/sof/lib/notifier.h	Adds NOTIFIER_ID_LL_PRE_RUN event type
src/include/sof/audio/module_adapter/module/generic.h	Adds module deadline and LPT calculation APIs
src/include/sof/audio/audio_buffer.h	Adds LFT calculation function declaration
src/include/module/audio/sink_api.h	Adds get_lft operation to sink API
src/audio/module_adapter/module/generic.c	Implements module deadline calculation logic
src/audio/buffers/ring_buffer.c	Adds LFT operation to ring buffer sink ops
src/audio/buffers/comp_buffer.c	Adds LFT operation to comp buffer sink ops
src/audio/buffers/audio_buffer.c	Implements comprehensive LFT calculation with DP-to-DP support

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/schedule/zephyr_dp_schedule.c

src/audio/buffers/audio_buffer.c

src/schedule/zephyr_dp_schedule.c

src/include/module/audio/sink_api.h

lyakh · 2025-08-21T06:54:10Z

src/schedule/zephyr_dp_schedule.c

+ *	a DEADLINE is the latest moment >>a module<< must finish processing to feed all target
+ *	buffers before their LFTs
+ *	Calculation is simple
+ *	  - module's deadline is the nearest LFT of all target buffers


"output buffers" / "sink buffers"?

lyakh · 2025-08-21T07:01:13Z

src/schedule/zephyr_dp_schedule.c

+ * extra care for 32bit overflows or use slow 64bit operations). Also all modules have the same
+ * timestamp as "NOW", regardless of moment in the cycle the deadlines are calculated
+ *
+ * EXAMPLE1 (data source period is longer or equal to data source)


is this correct? "data source period is longer or equal to data source?"

src/schedule/zephyr_dp_schedule.c

lyakh · 2025-08-21T07:21:45Z

src/schedule/zephyr_dp_schedule.c

+ *
+ *
+ *
+ * EXAMPLE2 (data source period is shorter than data receiver)


would be good to use consistent names "source," "sink," "input," "output"

I'll change it to convention

sink/source as API exposed by a buffer or by a module

data producer/data consumer as module that is processing data using sink/source API

sink/source of data may be misleading as it has opposite meaning, depending if we're talking about a module (receiving data from source API / sending to sink API) or a buffer (receiving data from sink API / sending to source API)

src/schedule/zephyr_dp_schedule.c

lyakh · 2025-08-21T07:46:44Z

src/include/sof/audio/module_adapter/module/generic.h

+ *  - a worst case - module period
+ *    note that module period may be calculated reasonably late, i.e. in prepare() method
+ */
+static inline uint32_t module_get_LPT(struct processing_module *mod)


lower case please. You spell-out "last_feeding_time" in other function names, but use an abbreviation for "lpt." Let's do this consistently

module_get_longest_processing_time is really long, but ok

lyakh · 2025-08-21T07:47:52Z

src/audio/buffers/audio_buffer.c

+	uint32_t us_in_buffer =
+			1000 * source_get_data_available(&buffer->_source_api) / bytes_per_ms;
+
+	return us_in_buffer;


please remove the code below, we can add it when needed.

I think it would be better to leave it, it has been tested, just cannot be enabled because of cross core and comp_buffer
I'm afraid when I remove it it will be lost - at least very hard to find

@marcinszkudlinski I'd then put it under some #ifdef GOOD_CODE_JUST_NOT_ENABLED_YET or similar macro with a comment. As is it's confusing both people and compilers IMHO

lyakh · 2025-08-21T07:54:22Z

src/schedule/ll_schedule.c

@@ -313,6 +313,9 @@ static void schedule_ll_tasks_run(void *data)

 	perf_cnt_init(&sch->pcd);

+	notifier_event(sch, NOTIFIER_ID_LL_PRE_RUN,
+		       NOTIFIER_TARGET_CORE_LOCAL, NULL, 0);


Is this for cmocka / tb simulations?

lyakh · 2025-08-21T08:01:28Z

src/schedule/zephyr_ll.c

@@ -176,6 +176,9 @@ static void zephyr_ll_run(void *data)
 	struct list_item *list, *tmp, task_head = LIST_INIT(task_head);
 	uint32_t flags;

+	notifier_event(sch, NOTIFIER_ID_LL_PRE_RUN,
+		       NOTIFIER_TARGET_CORE_LOCAL, NULL, 0);


this notifier is only directed to the same core as the one this LL scheduler is running on. So the notification handler will be called inline. Is this really what we want?

as for now - yes.
Later we need to unify all DP schedulers for all cores - as one DP may affect other regardless of a core they are located on.

@marcinszkudlinski but that also means then that those notification handlers on this core will run in the LL scheduler context, thus delaying LL tasks

yes, - and what's wrong about it? LL has one principle - it must FINISH within 1ms. It does not matter how

lyakh · 2025-08-21T08:07:19Z

src/schedule/zephyr_dp_schedule.c

@@ -808,6 +851,7 @@ int scheduler_dp_init(void)
 	if (ret)
 		return ret;

+	notifier_register(NULL, NULL, NOTIFIER_ID_LL_PRE_RUN, scheduler_dp_ll_tick, 0);
 	notifier_register(NULL, NULL, NOTIFIER_ID_LL_POST_RUN, scheduler_dp_ll_tick, 0);


in fact I'm just noticing this - the DP scheduler registers an LL task, but its callback is a NOP, while the actual processing is done in a notifier, which also is running inline in the LL scheduler thread. Can we not just use the LL task callback and remove notifiers? Just use the lowest priority to run it last?

I believe we can do it in several ways, but notifications are really for such things like DP recalculations. It is not an "LL task", but a schedule decision operation deep in SOF internals
Notifier is a right place in my opinion

@marcinszkudlinski I actually think that running notifiers inline isn't a good idea. Notifiers should be asynchronous - something like a work queue or a semaphore. Triggering a notifier should be atomic and have a well bounded duration, while processing takes place later and can take a long time. In particular this isn't good because it's asymmetric whether the handler is on the same or a different core, while in fact it should be symmetric. We've discussed it before, and I still find this wrong now...

Introducing extra LL task(s) for DP calculations won't change a bit. DP calculations will be executed in exactly same way - in LL context, before first LL processing task and after last LL processing task. Timings will be the same. Just more complicated and less clear how it goes - because it will be hidden under priorities
I would even think about hard-coded calls to DP, not by notifications. @softwarecki - you're the owner now ;)

@marcinszkudlinski I'd prefer a hard-coded call to DP over notifiers, I don't think this is a good use-case for them. But an ordinary LL task still looks like a good solution for this too to me, potentially a better one.

lyakh · 2025-08-21T08:11:49Z

the last commit shouldn't be merged obviously, adding a label.

marcinszkudlinski · 2025-08-25T13:14:49Z

changes - all "resolved" issues. Still todo: documentation

marcinszkudlinski · 2025-08-25T13:18:39Z

rebase only

lgirdwood

Its not clear how the IBS/OBS are configured with the source/sink buffers around headroom. Also questions on DP task priorities with EDF.

lgirdwood · 2025-08-26T14:53:30Z

src/schedule/zephyr_dp_schedule.c

+ *	   would never provide data in time (if LPT = period that means a module required 100% of
+ *	   CPU for processing, so it is really the worst possible case)
+ *
+ *	   Example: if a data rate is 48samples/msec and OBS = 480samples, the "worst case" period


Should we say frames instead of samples ?

src/schedule/zephyr_dp_schedule.c

lgirdwood · 2025-08-26T15:07:39Z

src/schedule/zephyr_dp_schedule.c

+ * Delayed start means that:
+ *  - when a DP module becomes ready for a first time, its deadline set to NOW
+ *  - even if DP module provides data early, the data will be hold in the buffer
+ *    till first LPT passes since DP become ready for the first time
+ *


How does this EDF work when the DP tasks have different priorities ?

There's no such thing as DP priorities. The rule is simple - DP (or DP chain) must provide data before the LL at the end of chain drain its buffer.
Note that priority in pipeline does not mean "one module or pipeline is more important than another", just "this must go first or the next one won't be able to proceed". Anyway - all LLs must finish in 1ms

In case of DP "a deadline" is in fact "priority" - all DP threads are on the same zephyr priority and the one with earliest deadline gets CPU (for Zephyr a deadline is just a number - the lowest one goes first)
In case of LL - see here
https://marcinszkudlinski.github.io/sof-docs/PAGES/architectures/firmware/sof-common/pipeline_2_0/pipeline2_0_discussion.html#module-creation-and-pipeline-iteration

Ok, I think we need to spell that out in the comment and probably also the IPC header i.e. IPC priority is ignored for DP EDF modules.

src/schedule/zephyr_dp_schedule.c

lgirdwood · 2025-08-26T15:38:36Z

src/include/sof/audio/module_adapter/module/generic.h

+/**
+ * Get a Longest Processing Time estimation for the module, in us
+ * It is either
+ *  - a value taken from the manifest or estimated from module period (TODO)


we should also be able to take from topology.

lgirdwood · 2025-08-26T15:48:31Z

src/audio/buffers/audio_buffer.c

+			deadline_correction =
+					module_get_LPT(data_producer_mod) * deadline_correction;
+	}
+


@marcinszkudlinski I think it might be worth adding some LOG_DBG() here for debugging/integration.

I reccomend some tracing - when enabled (kconfig?) report every deadline calculation, every DP thread state change. Logs may be not fast enough for this...

src/audio/buffers/audio_buffer.c

this commit adds a stubbed "get_LFT" operation to sink_api Stub of the function is needed because, of circular dependency. As described, calculation of LFT requires calculation of (next) module deadline, which requires calculation of (next) buffer LFT Signed-off-by: Marcin Szkudlinski <[email protected]>

as described in zephyr_dp_scheduler.c TODO: get estimation of LPT from module manifest currently only a "worst case" is calculated Signed-off-by: Marcin Szkudlinski <[email protected]>

This commit introduces calculating of LFT for a module following description from zephyr_dp_scheduler.c The implementation is ready for DP to DP deadline calculations, however, the rest of the code is not. Therefore the DP to DP part has been deactivated with proper comment Signed-off-by: Marcin Szkudlinski <[email protected]>

Recalculation of DP tasks states - readiness and deadlines - need to be called every time situation may change, that includes: - start of LL processing - end of LL processing - end of processing of any DP module this commit adds NOTIFIER_ID_LL_PRE_RUN to allow pre-ll call to DP scheduler Signed-off-by: Marcin Szkudlinski <[email protected]>

this commit enables (re) calculation of all DP modules deadlines at every DP scheduler run. It calculates all deadlines based on current buffers status and modules needs, as described Signed-off-by: Marcin Szkudlinski <[email protected]>

for testing and compilation Signed-off-by: Marcin Szkudlinski <[email protected]>

lgirdwood · 2025-09-08T14:17:51Z

@marcinszkudlinski zephyr update just merged, pls check its good for you.

marcinszkudlinski · 2025-09-09T08:58:03Z

@marcinszkudlinski zephyr update just merged, pls check its good for you.

I see zephyrproject-rtos/zephyr#94731 is still open

marcinszkudlinski force-pushed the DP_EXTENSIONS branch from bf56dec to 4416c7c Compare August 20, 2025 07:43

marcinszkudlinski changed the title ~~Dp extensions~~ DP modules deadline calculations Aug 20, 2025

marcinszkudlinski added pipeline2.0 dp_scheduler labels Aug 20, 2025

marcinszkudlinski marked this pull request as ready for review August 20, 2025 12:58

Copilot AI review requested due to automatic review settings August 20, 2025 12:58

marcinszkudlinski requested review from pblaszko, dbaluta, LaurentiuM1234, lgirdwood, plbossart, mmaka1, lbetlej and kv2019i as code owners August 20, 2025 12:58

Copilot AI reviewed Aug 20, 2025

View reviewed changes

lyakh reviewed Aug 21, 2025

View reviewed changes

lyakh added the DNM Do Not Merge tag label Aug 21, 2025

marcinszkudlinski force-pushed the DP_EXTENSIONS branch from bf5074a to 3e00247 Compare August 25, 2025 13:13

marcinszkudlinski force-pushed the DP_EXTENSIONS branch from 3e00247 to f597a87 Compare August 25, 2025 13:18

lgirdwood reviewed Aug 26, 2025

View reviewed changes

marcinszkudlinski added 4 commits September 4, 2025 15:40

DP: add get_deadline and get_LPT to module API

347eddd

as described in zephyr_dp_scheduler.c TODO: get estimation of LPT from module manifest currently only a "worst case" is calculated Signed-off-by: Marcin Szkudlinski <[email protected]>

marcinszkudlinski force-pushed the DP_EXTENSIONS branch from f597a87 to 82d9dfe Compare September 4, 2025 15:12

marcinszkudlinski added 2 commits September 5, 2025 08:18

DP: use real module deadline calculation in DP

64b52ff

this commit enables (re) calculation of all DP modules deadlines at every DP scheduler run. It calculates all deadlines based on current buffers status and modules needs, as described Signed-off-by: Marcin Szkudlinski <[email protected]>

[DNM] use Zephyr from PR branch

f78f2bc

for testing and compilation Signed-off-by: Marcin Szkudlinski <[email protected]>

marcinszkudlinski force-pushed the DP_EXTENSIONS branch from 82d9dfe to f78f2bc Compare September 5, 2025 06:33

lyakh mentioned this pull request Sep 9, 2025

[BUG] Notifiers shouldn't run in the caller's context when triggered on the same core #10225

Open

DP modules deadline calculations #10177

Are you sure you want to change the base?

DP modules deadline calculations #10177

Uh oh!

Conversation

marcinszkudlinski commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcinszkudlinski Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcinszkudlinski Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lyakh commented Aug 21, 2025

Uh oh!

marcinszkudlinski commented Aug 25, 2025

Uh oh!

marcinszkudlinski commented Aug 25, 2025

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcinszkudlinski Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

marcinszkudlinski commented Aug 20, 2025 •

edited

Loading

marcinszkudlinski Aug 25, 2025 •

edited

Loading

marcinszkudlinski Sep 9, 2025 •

edited

Loading

marcinszkudlinski Aug 27, 2025 •

edited

Loading