Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Routine load from Kafka issue with Unix timestamp #52480

Open
dorsegal opened this issue Oct 30, 2024 · 1 comment
Open

Routine load from Kafka issue with Unix timestamp #52480

dorsegal opened this issue Oct 30, 2024 · 1 comment
Labels
type/bug Something isn't working

Comments

@dorsegal
Copy link

Steps to reproduce the behavior (Required)

Follow this - https://docs.starrocks.io/docs/loading/RoutineLoad/#submit-a-routine-load-job-1

I am doing the same but my timestamp is 1715553720000 which throws error and from logs the value is NULL.
If I do 1715553720 it works as expected.

StarRocks version (Required)

3.3.5

@dorsegal dorsegal added the type/bug Something isn't working label Oct 30, 2024
@dorsegal
Copy link
Author

dorsegal commented Oct 30, 2024

I have managed to do a workaroound.

COLUMNS(
temp_timestamp,
timestamp = from_unixtime(cast(temp_timestamp as bigint)/1000)
)

But now I am facing something else I see tasks submitted and messages are being consumed but only for several minutes. after then that it stopped.

TaskId                              |TxnId |TxnStatus|JobId |CreateTime         |LastScheduledTime  |ExecuteStartTime   |Timeout|BeId  |DataSourceProperties                                                                                                                                                                                                                |Message                  |Warehouse        |
------------------------------------+------+---------+------+-------------------+-------------------+-------------------+-------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+-----------------+
49fefa24-18c6-411a-a0fd-7d078dfa549e|157835|COMMITTED|373030|2024-10-30 17:27:45|2024-10-30 17:27:55|2024-10-30 17:27:55|60     |373236|Progress:{"0":263060,"33":191049,"3":137429,"36":216789,"6":143667,"39":151216,"9":123804,"42":213118,"12":123741,"45":174647,"15":147513,"48":153462,"18":103136,"21":133956,"24":151699,"27":124039,"30":254126},LatestOffset:null|task submitted to execute|default_warehouse|
bbda35ff-77ef-4984-ad0c-6d4c18a6c598|157836|COMMITTED|373030|2024-10-30 17:27:45|2024-10-30 17:27:55|2024-10-30 17:27:55|60     |371107|Progress:{"1":227331,"34":138045,"4":110393,"37":89110,"7":145714,"40":164551,"10":87523,"43":233408,"13":113599,"46":143718,"16":148728,"49":116000,"19":96077,"22":103162,"25":144535,"28":61957,"31":172068},LatestOffset:null   |task submitted to execute|default_warehouse|
6ecccbcb-b192-443e-beee-62460c59e985|157837|COMMITTED|373030|2024-10-30 17:27:45|2024-10-30 17:27:55|2024-10-30 17:27:55|60     |372851|Progress:{"32":216134,"2":309174,"35":388152,"5":226989,"38":171677,"8":219902,"41":247702,"11":235816,"44":209313,"14":223932,"47":164389,"17":242534,"20":164987,"23":324165,"26":213792,"29":185581},LatestOffset:null           |task submitted to execute|default_warehouse|

did more digging and saw many of those
WARN (leaderCheckpointer|27260) [RoutineLoadJob.unprotectUpdateState():1321] routine load job 373030-data_from_kafka changed to PAUSED with reason: null

Also Kafka consumer group is empty
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant