Infrequent missing final character on MAC address in CSV file

[pejacoby]

Looking at various CSV files, once in a while I find a line or two that generate an extra column when the CSV is opened in Excel. I noticed that the lines in question always are missing one character in the MAC address. Note the last octet in the examples below is always one character instead of two.

All have this header (from 1.2.0b4 firmware) -- since they are all seen in b4 so far, perhaps this was addressed in #138 for beta 5. I haven't had enough time with b5 installed yet to have many files to explore.

WigleWifi-1.4,appRelease=1.2.0b4,model=wardriver.uk rev3,release=1.0.0,device=wardriver.uk rev3,display=i2c LCD,board=wardriver.uk rev3,brand=JHewitt
MAC,SSID,AuthMode,FirstSeen,Channel,RSSI,CurrentLatitude,CurrentLongitude,AltitudeMeters,AccuracyMeters,Type

6A:B9:D3:93:0F:8,,LEISURE470-6A5237,[WPA2_PSK],2023-11-19 03:55:37,1,-79,44.9734192,-93.2478638,154.20,7.75,WIFI

AE:97:CD:74:A9:2,,,[WPA2_PSK],2023-11-08 21:24:08,6,-86,44.8654404,-93.3223038,268.20,2.50,WIFI

7E:E3:0E:CB:CC:0,,,[WPA_WPA2_PSK],2023-10-29 02:28:40,6,-87,44.9680443,-93.2472763,261.10,2.50,WIFI

8E:EF:16:AF:32:0,,,[WPA2_PSK],2023-10-28 12:54:04,1,-79,44.9812584,-93.2410126,262.00,2.50,WIFI
4A:4B:D4:59:FE:7,,xaayow,[WPA2_PSK],2023-10-28 12:58:17,6,-84,44.9829674,-93.2486572,252.90,2.50,WIFI

I tried looking all these up on the Wireshare OUI tool and none worked. I tried adding a leading '0' and still none lookup. I don't see a pattern in the glitch, but will look at bit more.

[JosephHewitt]

Hi,

I looked through some of my captured files and seem to be affected by the same issue.

For me, the problematic line always contains a \r (which would explain why Excel is showing additonal lines for you) and that is the only pattern I found. This is odd because only \n has been used since #16. I've also seen other odd behavior on these broken lines including MAC addresses with extra octets.

Carriage return characters (the \r) could potentially make it into the file from side B which sends them for certain messages (specifically all of the Serial1.println() calls), however these characters should not be making it into the file unless something corrupts a message. I did find one possible cause of corrupt messages and opened issue #143 for it, however this is likely not what is causing the issue for us since it only affects the 5GHz modded devices and it was also affecting my 2.4GHz only devices.

I will investigate this some more, and expect it is related to communication from side B.

If you want to find other affected lines (or if anybody else reading this does), Notepad++ can be used. Search for \r and ensure "extended" search mode is selected. The \r character should not be found in files generated by firmware v1.0.1 or higher.

[pejacoby]

Sorry my examples above are a little misleading -- they are from four different files, the spaces between lines were just my separations from each file, not actual \r's.

I'm only seeing \n for line endings in the affected files. od -c for the last line above from the full file:

4615200    9   0   ,   2   .   5   0   ,   W   I   F   I  \n   4   A   :
4615220    4   B   :   D   4   :   5   9   :   F   E   :   7   ,   ,   x
4615240    a   a   y   o   w   ,   [   W   P   A   2   _   P   S   K   ]
4615260    ,   2   0   2   3   -   1   0   -   2   8       1   2   :   5
4615300    8   :   1   7   ,   6   ,   -   8   4   ,   4   4   .   9   8
4615320    2   9   6   7   4   ,   -   9   3   .   2   4   8   6   5   7
4615340    2   ,   2   5   2   .   9   0   ,   2   .   5   0   ,   W   I
4615360    F   I  \n   4   2   :   7   5   :   C   3   :   F   F   :   9

I'll dig around other files and see if I find any patterns.

[pejacoby]

One other curious pattern - in one file from 10/28, the same SSID and MAC appear 5 times, but only ONE instance has the broken MAC address.
Line breaks here are just separating the sightings of the same SSID. I also included a couple of adjacent lines where there were APs spotted with almost exactly the same MAC in the same second, only difference being the last character. So maybe MAC randomization and a high volume of input causing occasional glitches?

4A:4B:D4:59:FE:7C,xaayow,[WPA2_PSK],2023-10-28 12:55:24,6,-88,44.9829025,-93.2481537,244.00,2.50,WIFI

4A:4B:D4:59:FE:7C,xaayow,[WPA2_PSK],2023-10-28 12:58:11,6,-86,44.9830360,-93.2488556,253.80,2.50,WIFI
4A:4B:D4:59:FE:7F,,[WPA2_PSK],2023-10-28 12:58:11,6,-87,44.9830360,-93.2488556,253.80,2.50,WIFI

4A:4B:D4:59:FE:7,,xaayow,[WPA2_PSK],2023-10-28 12:58:17,6,-84,44.9829674,-93.2486572,252.90,2.50,WIFI

4A:4B:D4:59:FE:7F,,[WPA2_PSK],2023-10-28 13:10:23,6,-71,44.9832726,-93.2484207,234.10,2.50,WIFI
4A:4B:D4:59:FE:7C,xaayow,[WPA2_PSK],2023-10-28 13:10:23,6,-70,44.9832726,-93.2484207,234.10,2.50,WIFI

4A:4B:D4:59:FE:79,,[WPA2],2023-10-28 14:25:27,6,-80,44.9820671,-93.2492371,241.30,2.50,WIFI
4A:4B:D4:59:FE:7C,xaayow,[WPA2_PSK],2023-10-28 14:25:27,6,-81,44.9820671,-93.2492371,241.30,2.50,WIFI
4A:4B:D4:59:FE:7F,,[WPA2_PSK],2023-10-28 14:25:27,6,-81,44.9820671,-93.2492371,241.30,2.50,WIFI

I have not found this pattern elsewhere yet.

[pejacoby]

Last bit of inspection for today - where there are issues with the MAC, they are consistently seen on channels 1, 6, or 11 which would tie into Side B being the source. I found four other files back to August with anomalies on WIFI lines.

I found one instance of an octet other than the last one being truncated, but it's usually the last character being dropped.

70:03:7E:D:74:72,,,[WPA2_PSK],2023-09-09 00:41:40,1,-88,44.9380455,-92.9432068,308.20,3.25,WIFI

[hansjoerg0]

I also observed the missing final character in the MAC, but I see additional issues. Auth-mode is [UNDEFINED] and the RSSI is set to something which looks like it belongs into auth-mode.

Redacted examples with XXX for privacy:

Last character of MAC missing:
XX:XX:XX:XX:EC:D,,XXX-SSID-XXX,[UNDEFINED],2024-11-04 14:06:20,36,WPA2 AES,XXX-LAT-XXX,XXX-LONG-XXX,309.70,2.50,WIFI
Last character of MAC is a space:
XX:XX:XX:XX:B4:E ,,XXX-SSID-XXX,[UNDEFINED],2024-11-02 16:58:35,5,WPA/WPA2 AES,XXX-LAT-XXX,XXX-LONG-XXX,267.20,4.50,WIFI

MAC ok, but auth-mode and RSSI broken:

XX:XX:XX:XX:03:48,XXX-SSID-XXX,[UNDEFINED],2024-11-02 17:04:53,44,WPA2 AES,XXX-LAT-XXX,XXX-LONG-XXX,257.70,3.50,WIFI

XX:XX:XX:XX:B2:7F,XXX-SSID-XXX,[UNDEFINED],2024-11-02 17:14:58,48,WPA2 AES,XXX-LAT-XXX,XXX-LONG-XXX,273.50,3.75,WIFI

XX:XX:XX:XX:3F:92,,[UNDEFINED],2024-11-04 14:01:17,48,WPA2 AES,XXX-LAT-XXX,XXX-LONG-XXX,319.10,2.50,WIFI

I am running a locally build v1.2.1 (git commit 296645c) with the 5ghz mod

EDIT: Another strange one. With two unexpected line breaks (\n) which leads to a broken up entry with an empty line. Can SSIDs contain line breaks or is it something else?

XX:XX:XX:XX:E5:E2,Galaxy S24 Ultra

,[WPA2_PSK],2024-11-05 11:50:48,1,-91,XXX-LAT-XXX,XXX-LONG-XXX,2.50,WIFI

EDIT2:
Also have seen occurrences of two entries in the same line with an empty line afterwards.