Skip to content

Conversation

@minhmo1620
Copy link
Contributor

@minhmo1620 minhmo1620 commented Jan 9, 2026

Problem Statement

When a store is marked as isFlinkVeniceViewsEnabled=true, controller should only create the view topic and broadcast START_OF_PUSH.

Currently, when there is a push job for a store, the server will ingest data into views topics, and controller will send out END_OF_PUSH and VERSION_SWAP messages.

In Flink-based views, data records and control messages (EOP/VS) will be populated through Flink job to ensure the correctness for the CM.

Solution

To block the unnecessary write to view topic, we will use store properties isFlinkVeniceViewsEnabled config, which is introduced in #2357 and #2356.

There are two main places we will block:

  1. clients/venice-push-job/src/main/java/com/linkedin/venice/hadoop/task/datawriter/AbstractPartitionWriter.java.

In this writer, we use the flag through a push job config that is only active when there is materialized view configured for the store. This push config will stop CompositeVeniceWriter creation, which will write to view topic before writing to version topic.

  1. LeaderFollowerStoreIngestionTask

In this SIT, we will not create viewWriter, hence, will not queueUpVersionTopicWritesWithViewWriters. This prevents data write in native replication process.

Code changes

  • Added new code behind a config. If so list the config names and their default values in the PR description.
  • Introduced new log lines.
    • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

  • Code has no race conditions or thread safety issues.
  • Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
  • No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
  • Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
  • Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

  • New unit tests added.
  • New integration tests added.
  • Modified or extended existing tests.
  • Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

  • No. You can skip the rest of this section.
  • Yes. Clearly explain the behavior change and its impact.

@minhmo1620 minhmo1620 changed the title [controller][vpj] Use isFlinkVeniceViewsEnabled flag to block unncessary write to view topic [controller][vpj][dvc] Use isFlinkVeniceViewsEnabled flag to block unncessary write to view topic Jan 9, 2026
@minhmo1620 minhmo1620 marked this pull request as ready for review January 9, 2026 03:13
@minhmo1620 minhmo1620 changed the title [controller][vpj][dvc] Use isFlinkVeniceViewsEnabled flag to block unncessary write to view topic [controller][vpj][dvc] Use isFlinkVeniceViewsEnabled flag to block unnecessary write to view topic Jan 9, 2026
@minhmo1620 minhmo1620 requested a review from xunyin8 January 9, 2026 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant