Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Initial pass at removing qs logging in view of the PlayerEventLogging System #4542

Draft
wants to merge 26 commits into
base: master
Choose a base branch
from
Draft

[Feature] Initial pass at removing qs logging in view of the PlayerEventLogging System #4542

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
ece6eee
First pass of player_event_loot_items
neckkola Oct 15, 2024
36b499c
Second pass of player_event_loot_items
neckkola Oct 16, 2024
f710ada
Third pass of player_event_loot_items
neckkola Oct 16, 2024
04298e3
Example without RecordDetailEvent template
neckkola Oct 16, 2024
d74a4bd
Cleanup the removal of the template
neckkola Oct 17, 2024
130bf2f
Fourth Pass
neckkola Oct 18, 2024
2b05a12
Reposition to reduce db tasks
neckkola Oct 18, 2024
e3c99a1
Refactor etl processing for easier additions
neckkola Oct 20, 2024
436f280
Add merchant purchase event
neckkola Oct 20, 2024
11a2c56
Fix PlayerEventMerchantPurchase in client_packet.cpp
neckkola Oct 20, 2024
7efcaa2
WIP - Handin
neckkola Oct 20, 2024
dd5093f
Handin Event added
neckkola Oct 20, 2024
534a9a0
Cleanup
neckkola Oct 20, 2024
3f4755e
All a rentention period of 0 days which deletes all current records.
neckkola Oct 20, 2024
57825ed
Updates
neckkola Oct 21, 2024
134e408
Cleanup and Formatting
neckkola Oct 22, 2024
d6c1022
Add etl for
neckkola Oct 24, 2024
75f89fe
Add etl for
neckkola Oct 24, 2024
ab82afa
Add etl for Playerevent::AA_purchase
neckkola Nov 5, 2024
cfc777b
Set etl for speech to off by default
neckkola Nov 15, 2024
6605809
Cleanup before PR
neckkola Nov 15, 2024
7ca379d
Review comment updates.
neckkola Nov 16, 2024
8f69f73
Add world cli etl:settings to output a json on all player event details.
neckkola Nov 17, 2024
8e4aaee
Add reserve for all etl_queues
neckkola Nov 17, 2024
ab7f8d5
Potential solution for a dedicated database connection for player eve…
neckkola Nov 17, 2024
78a538e
Simple thread for player_events. Likely there is a better way to do …
neckkola Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 30 additions & 10 deletions common/events/player_event_logs.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,25 @@ void PlayerEventLogs::ProcessBatchQueue()
return;
}

std::map<uint32, uint32> counter{};
for (auto const& e: m_record_batch_queue) {
counter[e.event_type_id]++;
}


BenchTimer benchmark;

EtlQueues etl_queues{};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at all of this in ProcessBatchQueue and in our deletions, I think we should wrap these functions in thread calls so we don't tie up the main thread.

This also means that we should create a separate database connection just for processing player events because what happens is even if we had all of this in another thread its going to be lock contended with database locks on the main thread.

World needs to be kept as lightweight as possible in the main thread and this certainly adds weight if anything starts to take more than 500ms which it most certainly could here.

Same would go for using QS if folks use QS to process.

I may end up taking this one on.

etl_queues.trade.reserve(counter[PlayerEvent::TRADE] ? counter[PlayerEvent::TRADE] : 0);
etl_queues.speech.reserve(counter[PlayerEvent::SPEECH] ? counter[PlayerEvent::SPEECH] : 0);
etl_queues.loot_items.reserve(counter[PlayerEvent::LOOT_ITEM] ? counter[PlayerEvent::LOOT_ITEM] : 0);
etl_queues.killed_npc.reserve(counter[PlayerEvent::KILLED_NPC] ? counter[PlayerEvent::KILLED_NPC] : 0);
etl_queues.npc_handin.reserve(counter[PlayerEvent::NPC_HANDIN] ? counter[PlayerEvent::NPC_HANDIN] : 0);
etl_queues.aa_purchase.reserve(counter[PlayerEvent::AA_PURCHASE] ? counter[PlayerEvent::AA_PURCHASE] : 0);
etl_queues.merchant_sell.reserve(counter[PlayerEvent::MERCHANT_SELL] ? counter[PlayerEvent::MERCHANT_SELL] : 0);
etl_queues.killed_raid_npc.reserve(counter[PlayerEvent::KILLED_RAID_NPC] ? counter[PlayerEvent::KILLED_RAID_NPC] : 0);
etl_queues.killed_named_npc.reserve(counter[PlayerEvent::KILLED_NAMED_NPC] ? counter[PlayerEvent::KILLED_NAMED_NPC] : 0);
etl_queues.merchant_purchase.reserve(counter[PlayerEvent::MERCHANT_PURCHASE] ? counter[PlayerEvent::MERCHANT_PURCHASE] : 0);

for (auto &r:m_record_batch_queue) {
if (m_settings[r.event_type_id].etl_enabled) {
Expand Down Expand Up @@ -259,6 +275,7 @@ void PlayerEventLogs::ProcessBatchQueue()
}

if (!in.handin_items.empty()) {
etl_queues.npc_handin_entries.reserve(etl_queues.npc_handin_entries.size() + in.handin_items.size());
for (auto const &i: in.handin_items) {
out_entries.charges = i.charges;
out_entries.evolve_amount = 0;
Expand All @@ -285,6 +302,7 @@ void PlayerEventLogs::ProcessBatchQueue()
}

if (!in.return_items.empty()) {
etl_queues.npc_handin_entries.reserve(etl_queues.npc_handin_entries.size() + in.return_items.size());
for (auto const &i: in.handin_items) {
out_entries.charges = i.charges;
out_entries.evolve_amount = 0;
Expand Down Expand Up @@ -343,6 +361,7 @@ void PlayerEventLogs::ProcessBatchQueue()
}

if (!in.character_1_give_items.empty()) {
etl_queues.trade_entries.reserve(etl_queues.trade_entries.size() + in.character_1_give_items.size());
for (auto const &i: in.character_1_give_items) {
out_entries.char_id = in.character_1_id;
out_entries.charges = i.charges;
Expand All @@ -363,6 +382,7 @@ void PlayerEventLogs::ProcessBatchQueue()
}

if (!in.character_2_give_items.empty()) {
etl_queues.trade_entries.reserve(etl_queues.trade_entries.size() + in.character_2_give_items.size());
for (auto const &i: in.character_2_give_items) {
out_entries.char_id = in.character_2_id;
out_entries.charges = i.charges;
Expand Down Expand Up @@ -1294,79 +1314,79 @@ void PlayerEventLogs::LoadEtlIds()
{
.enabled = e(PlayerEvent::LOOT_ITEM),
.table_name = "player_event_loot_items",
.next_id = PlayerEventLootItemsRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventLootItemsRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::MERCHANT_SELL,
{
.enabled = e(PlayerEvent::MERCHANT_SELL),
.table_name = "player_event_merchant_sell",
.next_id = BasePlayerEventMerchantSellRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventMerchantSellRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::MERCHANT_PURCHASE,
{
.enabled = e(PlayerEvent::MERCHANT_PURCHASE),
.table_name = "player_event_merchant_purchase",
.next_id = PlayerEventMerchantPurchaseRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventMerchantPurchaseRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::NPC_HANDIN,
{
.enabled = e(PlayerEvent::NPC_HANDIN),
.table_name = "player_event_npc_handin",
.next_id = PlayerEventNpcHandinRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventNpcHandinRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::TRADE,
{
.enabled = e(PlayerEvent::TRADE),
.table_name = "player_event_trade",
.next_id = PlayerEventTradeRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventTradeRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::SPEECH,
{
.enabled = e(PlayerEvent::SPEECH),
.table_name = "player_event_speech",
.next_id = PlayerEventSpeechRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventSpeechRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::KILLED_NPC,
{
.enabled = e(PlayerEvent::KILLED_NPC),
.table_name = "player_event_killed_npc",
.next_id = PlayerEventKilledNpcRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventKilledNpcRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::KILLED_NAMED_NPC,
{
.enabled = e(PlayerEvent::KILLED_NAMED_NPC),
.table_name = "player_event_killed_named_npc",
.next_id = PlayerEventKilledNamedNpcRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventKilledNamedNpcRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::KILLED_RAID_NPC,
{
.enabled = e(PlayerEvent::KILLED_RAID_NPC),
.table_name = "player_event_killed_raid_npc",
.next_id = PlayerEventKilledRaidNpcRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventKilledRaidNpcRepository::GetNextAutoIncrementId(*m_database)
}
},
{
PlayerEvent::AA_PURCHASE,
{
.enabled = e(PlayerEvent::AA_PURCHASE),
.table_name = "player_event_aa_purchase",
.next_id = PlayerEventAaPurchaseRepository::GetMaxId(*m_database) + 1
.next_id = PlayerEventAaPurchaseRepository::GetNextAutoIncrementId(*m_database)
}
}
};
Expand Down
1 change: 0 additions & 1 deletion common/events/player_event_logs.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
#ifndef EQEMU_PLAYER_EVENT_LOGS_H
#define EQEMU_PLAYER_EVENT_LOGS_H

#include <any>
#include <cereal/archives/json.hpp>
#include <mutex>
#include "../json/json_archive_single_line.h"
Expand Down
10 changes: 10 additions & 0 deletions common/repositories/player_event_aa_purchase_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventAaPurchaseRepository: public BasePlayerEventAaPurchaseRepositor
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_AA_PURCHASE_REPOSITORY_H
10 changes: 10 additions & 0 deletions common/repositories/player_event_killed_named_npc_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventKilledNamedNpcRepository: public BasePlayerEventKilledNamedNpcR
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_KILLED_NAMED_NPC_REPOSITORY_H
10 changes: 10 additions & 0 deletions common/repositories/player_event_killed_npc_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventKilledNpcRepository: public BasePlayerEventKilledNpcRepository
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_KILLED_NPC_REPOSITORY_H
10 changes: 10 additions & 0 deletions common/repositories/player_event_killed_raid_npc_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventKilledRaidNpcRepository: public BasePlayerEventKilledRaidNpcRep
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_KILLED_RAID_NPC_REPOSITORY_H
11 changes: 11 additions & 0 deletions common/repositories/player_event_loot_items_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,17 @@ class PlayerEventLootItemsRepository: public BasePlayerEventLootItemsRepository
* method that can be re-used easily elsewhere especially if it can use a base repository
* method and encapsulate filters there
*/
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_LOOT_ITEMS_REPOSITORY_H
11 changes: 11 additions & 0 deletions common/repositories/player_event_merchant_purchase_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,17 @@ class PlayerEventMerchantPurchaseRepository: public BasePlayerEventMerchantPurch
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_MERCHANT_PURCHASE_REPOSITORY_H
11 changes: 11 additions & 0 deletions common/repositories/player_event_merchant_sell_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,17 @@ class PlayerEventMerchantSellRepository: public BasePlayerEventMerchantSellRepos
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_MERCHANT_SELL_REPOSITORY_H
10 changes: 10 additions & 0 deletions common/repositories/player_event_npc_handin_entries_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventNpcHandinEntriesRepository: public BasePlayerEventNpcHandinEntr
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_NPC_HANDIN_ENTRIES_REPOSITORY_H
10 changes: 10 additions & 0 deletions common/repositories/player_event_npc_handin_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventNpcHandinRepository: public BasePlayerEventNpcHandinRepository
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_NPC_HANDIN_REPOSITORY_H
10 changes: 10 additions & 0 deletions common/repositories/player_event_speech_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventSpeechRepository: public BasePlayerEventSpeechRepository {
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should assume that the server user can't directly query information_schema is there a reason we leaned into this versus max + 1 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I ran into a couple of test cases that failed. If I empty an etl table, the max + 1 will return 1, when in fact the auto_increment may be 100. Further, once the retention period is hit, the next id via max + 1 will again be different from the auto_increment.

TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_SPEECH_REPOSITORY_H
10 changes: 10 additions & 0 deletions common/repositories/player_event_trade_entries_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventTradeEntriesRepository: public BasePlayerEventTradeEntriesRepos
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_TRADE_ENTRIES_REPOSITORY_H
10 changes: 10 additions & 0 deletions common/repositories/player_event_trade_repository.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,17 @@ class PlayerEventTradeRepository: public BasePlayerEventTradeRepository {
*/

// Custom extended repository methods here
static int64 GetNextAutoIncrementId(Database& db)
{
auto results = db.QueryDatabase(
fmt::format(
"SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = '{}';",
TableName()
)
);

return (results.Success() && results.begin()[0] ? strtoll(results.begin()[0], nullptr, 10) : 0);
}
};

#endif //EQEMU_PLAYER_EVENT_TRADE_REPOSITORY_H