KAFKA-16339: [1/4 KStream#transform] Remove Deprecated "transformer" methods and classes #17198

fonsdant · 2024-09-15T00:44:23Z

Hi, @mjsax! I have removed the transform methods from the main and test classes. However, I have asked myself whether the approach I have used for StandbyTaskEOSIntegrationTest and StreamThreadTest is correct. Should I replace transform with process as suggested in transform Javadoc or remove these tests? Thanks!

fonsdant · 2024-09-16T00:41:22Z

I have missed removing the references to transform on Javadocs too. I will push another commit for it.

fonsdant · 2024-09-17T22:52:56Z

I have noticed that KStreamTransformIntegrationTest.TestTransformer could be removed as it has become unused.

fonsdant · 2024-09-17T23:23:23Z

For now, I have removed the build with deduplicaiton topology test until we decide how to replace the transform method inside it.

fonsdant · 2024-09-20T00:37:42Z

I have replaced tranform with process as suggested by docs. I have implemented Processor based on StandbyTaskEOSMultiRebalanceIntegrationTest.buildWithUniqueIdAssignmentTopology, but shouldWipeOutStandbyStateDirectoryIfCheckpointIsMissing has failed.

fonsdant · 2024-09-20T01:22:19Z

I have managed to fix the test! :D

fonsdant · 2024-09-23T16:15:19Z

I have removed the references to transform from docs. I am wondering if it would be better to replace transform with process, instead of removing it. I have started to think about it while removing the flatTransformValues (the third method to remove). @mjsax, could you help me with this decision?

mjsax

Thanks for the PR. Made a pass.

Btw: is there anything in docs/streams/... that we need to update/remove, too (we can also do this as a single docs-only PR for the whole ticket, ie, all transformXxx() method)

mjsax · 2024-09-28T15:55:34Z

streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java

@@ -179,8 +178,7 @@ <KR> KStream<KR, V> selectKey(final KeyValueMapper<? super K, ? super V, ? exten
     * altered arbitrarily).
     * The provided {@link KeyValueMapper} is applied to each input record and computes a new output record.
     * Thus, an input record {@code <K,V>} can be transformed into an output record {@code <K':V'>}.
-     * This is a stateless record-by-record operation (cf. {@link #transform(TransformerSupplier, String...)} for
-     * stateful record transformation).
+     * This is a stateless record-by-record operation.


Yes, it think we should link to #process(...) here, instead of removing (same elsewhere where it make sense).

Btw: feel free to skip updating JavaDocs of other deprecated methods which will be remove in follow up PRs of this ticket.

streams/src/main/java/org/apache/kafka/streams/kstream/Transformer.java

streams/src/main/java/org/apache/kafka/streams/kstream/TransformerSupplier.java

streams/src/main/java/org/apache/kafka/streams/kstream/ValueTransformerSupplier.java

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamImpl.java

mjsax · 2024-09-28T16:11:35Z

streams/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java

@@ -360,11 +368,9 @@ public KeyValue<Integer, Integer> transform(final Integer key, final Integer val
                            throw new RuntimeException("Injected test error");
                        }

-                        return KeyValue.pair(key, value);
+                        store.put(key, value);


We do we add a put here?

Similar to the return null replacement with delete. Since we would return a pair with key and value, now I put it in store.

mjsax · 2024-09-28T16:15:08Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java

-    @ParameterizedTest
-    @MethodSource("data")        
-    @SuppressWarnings("deprecation")
-    public void shouldPunctuateWithTimestampPreservedInProcessorContext(final boolean stateUpdaterEnabled, final boolean processingThreadsEnabled) {


I think we need to keep this test, but rewrite using #process().

mjsax · 2024-09-28T16:16:49Z

streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/TopologyTest.scala

@@ -276,61 +274,6 @@ class TopologyTest {
    assertEquals(getTopologyScala, getTopologyJava)
  }

-  @nowarn
-  @Test
-  def shouldBuildIdenticalTopologyInJavaNScalaTransform(): Unit = {


Wondering if we have a test gap, as we don't have a similar test for process() -- should we add a new test for #process() like this one (or rewrite this test for #process())?

I have rewritten this test with process, but have missed to rewrite the test name. I will fix it.

mjsax · 2024-09-28T16:18:02Z

streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/kstream/KStreamTest.scala

@@ -222,42 +221,6 @@ class KStreamTest extends TestDriver {
    testDriver.close()
  }

-  @nowarn
-  @Test
-  def testTransformCorrectlyRecords(): Unit = {


Same question as above. Seems we are lacking a similar test for #process() ?

Hm... I have replaced Transformer with ProcessorSupplier and transform with process. Have I missed something else?

mjsax · 2024-09-28T16:18:20Z

streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/kstream/KStreamTest.scala

-  @nowarn
-  @Test
-  def testSettingNameOnTransform(): Unit = {
-    class TestTransformer extends Transformer[String, String, KeyValue[String, String]] {


fonsdant · 2024-10-08T11:38:59Z

I will push some commits, one for each file from which I have removed the transform link, replacing it with process.

fonsdant · 2024-10-14T21:14:21Z

I have rebased my branch on trunk and this caused GitHub to close the PR. I am implementing the last review points again.

fonsdant · 2024-10-15T12:21:32Z

This time, I have focused on removing the methods first and doing refactorings. After this, I will proceed with Javadoc updates.

I have made the removing part. Now, I will do the refactoring part.

mjsax

Thanks for updating the PR. LGTM.

Just triggered CI. Can merge after it finished and passed.

mjsax · 2024-10-18T22:48:03Z

@fonsdant -- seems there is some compilation error:

StandbyTaskEOSIntegrationTest.java:332: error: cannot find symbol

            .transform(
> Task :streams:compileTestJava
            ^
  symbol:   method transform(()->new or[...]{ } },String)
  location: interface KStream<Integer,Integer>

Seem we need to rewrite this test to use the new process() instead of transform()

fonsdant · 2024-10-19T19:29:25Z

Thanks, @mjsax, still need to refactor it. Meanwhile, do you prefer that I convert this PR to draft? Or should I keep it as it is?

mjsax · 2024-10-22T00:29:38Z

We can keep this PR as-is -- you can also to the refactoring as part of this PR if you want.

mjsax · 2024-10-30T06:12:27Z

@fonsdant Any update on this PR? -- We are approaching AK 4.0 release deadlines, and there is 3 more PRs for this ticket...

fonsdant · 2024-10-30T10:19:56Z

@mjsax, sorry for the delay, I am still refactoring the tests, replacing transform with process.

Given the coming deadline, I agree it would be adequate someone more experienced to resume it.

fonsdant · 2024-10-30T23:27:17Z

I will push the work I done this passed week.

mjsax · 2024-10-31T00:45:16Z

Given the coming deadline, I agree it would be adequate someone more experienced to resume it.

As long as you have enough time to keep working, I think we are a good. Of course, if you are blocked, let us know.

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

...-tests/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java

streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java

mjsax · 2024-11-04T21:00:24Z

streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/kstream/KStreamTest.scala

+//
+//    val transformNode = builder.build().describe().subtopologies().asScala.head.nodes().asScala.toList(1)
+//    assertEquals("my-name", transformNode.name())
+//  }


Should we rewrite this to testSettingNameOnProcess?

Oh, sorry! I have missed it up. Of course, I will rewrite it!

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

fonsdant · 2024-11-05T01:31:04Z

@mjsax, I have removed the adding lines in buildTopologyWithDeduplication in StandbyTaskEOSIntegrationTest to match the previous implementation with transform (but still with process). Also, removed the comments of the KStreamTest scala test and rewritten it. Finally, removed var usage.

mjsax

Thanks for the quick turn around. I think the Scala test can be simplified significantly. In the end, we only verify the name of he processor node in the Topology.

mjsax · 2024-11-05T02:06:59Z

streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/kstream/KStreamTest.scala

+  @Test
+  def testSettingNameOnTransform(): Unit = {
+    val processorSupplier: ProcessorSupplier[String, String, String, String] =
+      new api.ProcessorSupplier[String, String, String, String] {


Why do we need an explicit ProcessorSupplier for this test? And we use process(() => new TestProcessor) similar to the old code?

mjsax · 2024-11-05T02:07:19Z

streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/kstream/KStreamTest.scala

+  def testSettingNameOnTransform(): Unit = {
+    val processorSupplier: ProcessorSupplier[String, String, String, String] =
+      new api.ProcessorSupplier[String, String, String, String] {
+        private val storeName = "store-name"


Why do we add a store?

mjsax · 2024-11-05T02:08:40Z

streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/kstream/KStreamTest.scala

+              val processedKey = s"$key-processed"
+              val processedValue = s"$value-processed"
+              store.put(processedKey, processedValue)
+              context.forward(new api.Record(processedKey, processedValue, record.timestamp()))


What the Processor does is actually pretty meaningless for the test, so I think the while implementation could actually be empty? For this case, we don't even need init() method, as we don't need access to the context either.

fonsdant changed the title ~~KAFKA-16339: [1/4 KStream#tranform] Remove Deprecated "transformer" methods and classes~~ KAFKA-16339: [1/4 KStream#transform] Remove Deprecated "transformer" methods and classes Sep 15, 2024

fonsdant force-pushed the kstream-transform-remove-deprecated-transformer-methods-and-classes branch 2 times, most recently from bafaa10 to f28ac72 Compare September 17, 2024 23:17

github-actions bot added the streams label Sep 20, 2024

fonsdant mentioned this pull request Sep 20, 2024

KAFKA-16339: [2/4 KStream#flatTransform] Remove Deprecated "transformer" methods and classes #17245

Draft

mjsax reviewed Sep 28, 2024

View reviewed changes

kamalcph mentioned this pull request Oct 12, 2024

KAFKA-15859: Make RemoteListOffsets call an async operation #16602

Merged

3 tasks

fonsdant closed this Oct 14, 2024

fonsdant force-pushed the kstream-transform-remove-deprecated-transformer-methods-and-classes branch from 35fd141 to 203f323 Compare October 14, 2024 20:48

github-actions bot added the small Small PRs label Oct 14, 2024

fonsdant reopened this Oct 14, 2024

github-actions bot removed the small Small PRs label Oct 14, 2024

mjsax added the ci-approved label Oct 16, 2024

mjsax approved these changes Oct 16, 2024

View reviewed changes

fonsdant added 18 commits November 4, 2024 08:58

remove kstream transform integration test

b1d4970

remove kstream transform test

13c86d9

remove transform from streams builder test

287e25d

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

remove transform from kstream impl test

1bb09f1

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

remove transformer supplier adapter

b93f88f

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

remove transformer supplier adapter test

6bb1b84

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

refator stand by task eos integration test

cf9d7fc

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

refator stream thread test

c61ccc0

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

refactor scala topology test

2e2ec57

refactor scala kstream test

b090991

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

adjust topology test for spotless scala

46cf8d1

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

adjust kstream test for spotless scala

05d3d7c

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

update javadoc of transformer

5db97e2

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

update javadoc of transformer supplier

42f8635

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

update javadoc of value transformer supplier

38cc9c3

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

update javadoc of value transformer with key supplier

7b51fd1

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

update javadoc of connected store provider

532f9f2

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

update javadoc of kstream

7a65aed

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

fonsdant force-pushed the kstream-transform-remove-deprecated-transformer-methods-and-classes branch from 46a01a6 to 7a65aed Compare November 4, 2024 11:59

mjsax reviewed Nov 4, 2024

View reviewed changes

...-tests/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java Outdated Show resolved Hide resolved

mjsax reviewed Nov 4, 2024

View reviewed changes

...-tests/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java Outdated Show resolved Hide resolved

mjsax reviewed Nov 4, 2024

View reviewed changes

...-tests/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java Outdated Show resolved Hide resolved

mjsax reviewed Nov 4, 2024

View reviewed changes

...-tests/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java Outdated Show resolved Hide resolved

mjsax reviewed Nov 4, 2024

View reviewed changes

streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java Outdated Show resolved Hide resolved

mjsax reviewed Nov 4, 2024

View reviewed changes

fonsdant added 3 commits November 4, 2024 22:15

refactor standby task eos integration test

a77fdda

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

refactor stream thread test

e5bafbd

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

refactor scala kstream test

f915758

Signed-off-by: Joao Pedro Fonseca Dantas <[email protected]>

mjsax reviewed Nov 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-16339: [1/4 KStream#transform] Remove Deprecated "transformer" methods and classes #17198

KAFKA-16339: [1/4 KStream#transform] Remove Deprecated "transformer" methods and classes #17198

fonsdant commented Sep 15, 2024

fonsdant commented Sep 16, 2024

fonsdant commented Sep 17, 2024

fonsdant commented Sep 17, 2024

fonsdant commented Sep 20, 2024

fonsdant commented Sep 20, 2024

fonsdant commented Sep 23, 2024

mjsax left a comment •

edited

Loading

mjsax Sep 28, 2024 •

edited

Loading

mjsax Sep 28, 2024

fonsdant Oct 12, 2024

mjsax Sep 28, 2024

mjsax Sep 28, 2024

fonsdant Nov 5, 2024 •

edited

Loading

mjsax Sep 28, 2024

fonsdant Nov 5, 2024 •

edited

Loading

mjsax Sep 28, 2024

fonsdant commented Oct 8, 2024

fonsdant commented Oct 14, 2024

fonsdant commented Oct 15, 2024

mjsax left a comment

mjsax commented Oct 18, 2024

fonsdant commented Oct 19, 2024

mjsax commented Oct 22, 2024

mjsax commented Oct 30, 2024

fonsdant commented Oct 30, 2024

fonsdant commented Oct 30, 2024

mjsax commented Oct 31, 2024

mjsax Nov 4, 2024

fonsdant Nov 5, 2024

fonsdant commented Nov 5, 2024 •

edited

Loading

mjsax left a comment

mjsax Nov 5, 2024

mjsax Nov 5, 2024

mjsax Nov 5, 2024

KAFKA-16339: [1/4 KStream#transform] Remove Deprecated "transformer" methods and classes #17198

Are you sure you want to change the base?

KAFKA-16339: [1/4 KStream#transform] Remove Deprecated "transformer" methods and classes #17198

Conversation

fonsdant commented Sep 15, 2024

fonsdant commented Sep 16, 2024

fonsdant commented Sep 17, 2024

fonsdant commented Sep 17, 2024

fonsdant commented Sep 20, 2024

fonsdant commented Sep 20, 2024

fonsdant commented Sep 23, 2024

mjsax left a comment • edited Loading

Choose a reason for hiding this comment

mjsax Sep 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fonsdant Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fonsdant Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fonsdant commented Oct 8, 2024

fonsdant commented Oct 14, 2024

fonsdant commented Oct 15, 2024

mjsax left a comment

Choose a reason for hiding this comment

mjsax commented Oct 18, 2024

fonsdant commented Oct 19, 2024

mjsax commented Oct 22, 2024

mjsax commented Oct 30, 2024

fonsdant commented Oct 30, 2024

fonsdant commented Oct 30, 2024

mjsax commented Oct 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fonsdant commented Nov 5, 2024 • edited Loading

mjsax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax left a comment •

edited

Loading

mjsax Sep 28, 2024 •

edited

Loading

fonsdant Nov 5, 2024 •

edited

Loading

fonsdant Nov 5, 2024 •

edited

Loading

fonsdant commented Nov 5, 2024 •

edited

Loading