How to dedupe replays from same game?

[ohpyupi]

Replay files can be duplicate since they are generated per player. After parsing replay files, is there anyway we can dedupe files?

[icza]

That would require to identify the game the replays belong to. StarCraft Brood War replays do not contain the unique game ID, so the best you can do is designate a set of properties that together define a game, or in other words, they will (most likely) be unique for each game. A good candidate would be the game start time, game title and player properties in the header (like name, race, type, team etc).

[ohpyupi]

I see. And another question is that the package doesn't really verify the checksum of each section of the replay files (link). But I assume this package is being used for the app (https://repmastered.icza.net/), then how do you ensure the integrity of uploaded replay files in the app? Can users manipulate their profile in the app by manipulating the replay files, or does the app have the protection layer somewhere else?

[icza]

Yes, this lib is used in RepMastered. I don't ensure integrity. What do you get by altering replays? Nothing.

Also note that player name spoofing is a thing, you can't ensure the recorded player names are valid. You can't even be sure the names you see on battle.net are the real player names. You can also play locally, choosing / using any player name. It's just not possible, but it's not cruitial.

[remastrTV]

To efficiently dedupe replays from the same game, I generate a unique key for each replay by combining:

• Start time (StartTime)
• Player summary with race, team, and position (PlayersSummary)
• Map name (MapDetails)
• Replay title (Title)

This helps me detect if multiple files belong to the same exact game.

Then I keep only one based on:

• The longest duration
• If duration is the same, the largest file size
• The rest are deleted

For performance, I use the command: screp.exe ... -overview capture_output=True, text=True, encoding="utf-8"

It only reads the header of the replay, which is much faster than parsing the full file.

This is what I currently use, and it works well for my needs — but it's not a definitive solution. It's a practical approach that others can adapt or improve.

[ohpyupi]

Yes, this lib is used in RepMastered. I don't ensure integrity. What do you get by altering replays? Nothing.

Also note that player name spoofing is a thing, you can't ensure the recorded player names are valid. You can't even be sure the names you see on battle.net are the real player names. You can also play locally, choosing / using any player name. It's just not possible, but it's not cruitial.

My original concern was mostly around the security concern such as bad users manipulate someone else's profile data such as APM or win rate, which can result in bad user experience for the victim users.

But, now I'm thinking that the checksum integrity check doesn't really prevent the such issues. The bad actors can still tamper the data and generate its checksum on their own. So I think there is not explicit way to prevent this issue.

[ohpyupi]

To efficiently dedupe replays from the same game, I generate a unique key for each replay by combining:

• Start time (StartTime)
• Player summary with race, team, and position (PlayersSummary)
• Map name (MapDetails)
• Replay title (Title)

This helps me detect if multiple files belong to the same exact game.

Then I keep only one based on:

• The longest duration
• If duration is the same, the largest file size
• The rest are deleted

For performance, I use the command: screp.exe ... -overview capture_output=True, text=True, encoding="utf-8"

It only reads the header of the replay, which is much faster than parsing the full file.

This is what I currently use, and it works well for my needs — but it's not a definitive solution. It's a practical approach that others can adapt or improve.

Thanks for the details explanation! I'd probably start with your suggestion.