Decoding smart WiFi meter data

[juancrrn]

Hello,

I'm trying to retrieve the data from a Smart WiFi meter and switch. I've successfully connected to the device using TuyAPI, but I don't get to understand the data I'm getting.

Using a simple script based in the asynchronous one in README.md, I get the following information:

Connected to device!
Data from device: { dps: { '1': 5407, '10': 0, '16': true, '18': '111111111111' } }
Data from device: { dps: { '1': 5408 }, t: 1613651565 }
Data from device: { dps: { '6': 'CNEAJ5AACOw=' }, t: 1613651573 }
Data from device: { dps: { '1': 5409 }, t: 1613651581 }
Data from device: { dps: { '6': 'CM8AJ3YACOQ=' }, t: 1613651588 }
Data from device: { dps: { '1': 5410 }, t: 1613651597 }
Disconnected from device.

Data from dps 1 (absolute current consumption), 10, 16 and 18 is irrelevant. I'm guessing that the string in dps[6] contains the rest of the data I was expecting (current, active power and voltage), but I'm not sure. I've tried to decode it using the data shown in the app (decoding each string to binary and then grouping to see if I could find the numbers), but this has been unhelpful. Also, I'm not sure if there are any other dps not being shown (2, 3, 4, 5, 7, 8, 9...).

I also haven't found any specific information about the device online.

Any suggestions?

Thank you.

[codetheweb]

It looks like the data in dps[6] is base-64 encoded, but I'm not sure what it means. I haven't seen a device respond with base 64 before.

Did you try decoding that segment to hex and playing around with the values listed in the app at that time?

[Chanete]

Same here, in my case, SmartMeter is only answering with 1,10,16 and 18. No cue about voltage and instant consumption..

{u'dps': {u'1': 804, u'10': 0, u'18': u'111111111111', u'16': True}}

Seems like status command is not the proper one to retrieve this info.

Did you make any progress??

[juancrrn]

Hello,

In response to @codetheweb, I tried to Base64-decode it, but nothing showed up.

To update this and also in response to @Chanete, I have created a light PHP app to experiment with the decoding of this DPS.

The app is available at a GitHub repository (juancrrn/tuya-experiment) and there is a live demo version available at sqa1g.juancrrn.io/tuya-experiment/.

There is an extra experiment that searches for decimal patterns in the binary strings. Use sqa1g.juancrrn.io/tuya-experiment/?search=override for this search experiment.

I am still developing it, but it is a good start, as groups of changing bits can already bee seen in the result.

One of the main problems is not knowing the correct function that should be used to decode the character string (Utils::stringToBinaryA7) and if it has to be Base64-reencoded previously, as I think the CLI used decodes it and that is wrong.

Any help is welcome.

[codetheweb]

@juancrrn that app looks interesting, I imagine it could be useful for a lot of other Tuya devices as well.

One of the main problems is not knowing the correct function that should be used to decode the character string

Can't you just use base64_decode?

if it has to be Base64-reencoded previously, as I think the CLI used decodes it and that is wrong.

What do you mean?

[Apollon77]

In fact I implement that too in my project that based on the library, but here I have the schema available. So when you know the schema you can "detect" which field needs to be base64 decoded ... but this lib do not have the schema

[juancrrn]

@codetheweb, answering your questions:

What do you mean?

Can't you just use base64_decode?

Taking a look at the message parser, I see that the message is decrypted and then parsed as UTF-8 JSON.

If I just apply base64_decode to one of the strings, e. g. COEAA8MAAMo=, the result is not helpful (��).

Here I explain one new string-to-binary function, to see if any changes should be made, based in your knowledge:

public static function stringToBinaryC(string $string): string
{
    /**
     * 1. Ensure that the string is UTF-8-encoded.
     * 
     * 2. Convert each character of the string to a number between 0 and
     * 255 (format code 'C' with repeater argument '*'). This is an
     * interpretation of the character as an unsigned integer in that
     * range. Other formats are available at the PHP Manual:
     *
     * https://www.php.net/manual/en/function.pack.php
     * 
     * The result of unpack() is an array containing the representations
     * of all the characters. E. g., for 'COEAA8MAAMo=':
     * 
     * [67, 79, 69, 65, 65, 56, 77, 65, 65, 77, 111, 61]
     */
    $byteArray = unpack('C*', utf8_encode($string));
        var_dump($byteArray);

    /**
     * 3. Prepare a variable to store the concatenated string.
     */
    $result = '';

    /**
     * 4. Loop through each representation.
     */
    foreach ($byteArray as $byte) {
        /**
         * 5. Convert the representation from 0 to 255, to a pure binary
         * representation.
         * 
         * 6. As 8 bits are needed to represent the range 0 to 255, pad
         * the binary representation with leading 0s.
         * 
         * 7. Add the representation to the result string.
         */
        $result .= str_pad(decbin($byte), 8, '0', STR_PAD_LEFT);
    }

    return $result;   
}

Does that seem correct as a way to get the binary bits? And then it comes the pattern recognition to find the data.

Thank you.

[codetheweb]

Taking a look at the message parser, I see that the message is decrypted and then parsed as UTF-8 JSON.

TuyAPI is parsing the payload as UTF8, yes. But the equals sign on the end is extremely indicative of base 64 encoding. I strongly doubt that it's intended to be parsed without first decoding from base 64.

If I just apply base64_decode to one of the strings, e. g. COEAA8MAAMo=, the result is not helpful (��).

It's not helpful in ASCII characters. But try decoding it directly to binary and looking for patterns there:

00001000 11100001 00000000 00000011 11000011 00000000 00000000 11001010

It decodes to exactly 8 bytes, so it seems like this is the right route to go.

[juancrrn]

Thank you, @codetheweb.

I have modified the function to work with your decoding and the Base64 advice. I will be working with that and get back to this issue when i have news.

I just have one doubt:

00001000 11100001 00000000 00000011 11000011 00000000 00000000 11001010

This representation was obtained by Base64-decoding the data as an ASCII string. The decoding as a UTF-8 string results in a different representation.

Is this correct and deliberate?

[juancrrn]

In fact I implement that too in my project that based on the library, but here I have the schema available. So when you know the schema you can "detect" which field needs to be base64 decoded ... but this lib do not have the schema

@Apollon77, could you please explain your answer further?

[codetheweb]

This representation was obtained by Base64-decoding the data as an ASCII string. The decoding as a UTF-8 string results in a different representation.

Sorry, not quite sure what you're saying. When going from base 64 to hex / binary / whatever it shouldn't matter if the base 64 string is ascii or utf.

[juancrrn]

Sorry, not quite sure what you're saying. When going from base 64 to hex / binary / whatever it shouldn't matter if the base 64 string is ascii or utf.

It seems like it actually matters. For me, using the default PHP functions, the result was different. As an example, take a look at www.base64decode.org/, there is a select input for "Source character set".

[codetheweb]

It should only be relevant if you're decoding from base64 and then trying to display as ascii or utf. The underlying bytes don't change depending on how they're viewed.

Could you show an example of PHP interpreting it differently?

[juancrrn]

I was copying the code for the examples but you were right, there is no change depending on the source character set. I was confused because of that website.

Thank you.

FYI, the current function that I am using is this one:

public static function stringToBinaryD(string $string): string
{
    $byteArray = unpack('C*', base64_decode($string));

    $result = '';

    foreach ($byteArray as $byte) {
        $result .= str_pad(decbin($byte), 8, '0', STR_PAD_LEFT) . ' ';
    }

    return $result;   
}

For the string, COEAA8MAAMo=, the result is 00001000 11100001 00000000 00000011 11000011 00000000 00000000 11001010, just as you said.

[codetheweb]

You could also try decrypting the decoded base 64 data with your local key; if the byte patterns seem random that's what I would try next (i.e. data -> decode using base 64 -> decrypt using local key).

[juancrrn]

If you take a look at my experiment app (tuya-experiment, 1 to 7 are low load and 8 to 14 are high load), you'll see that bit patterns are clearly visible, so I wouldn't say that the string is cyphered.

[juancrrn]

Good news!

I decoded the string. Using the function from my reply above, the data is stored as follows:

• Voltage: bits 0 to 15 (length 16 bits, represented as unsigned integer, V * 10).
• Electric current: bits 16 to 39 (length 24 bits, represented as unsigned integer, A * 1000).
• Active power: bits 40 to 63 (length 24 bits, represented as unsigned integer, W * 1).

Thanks for all the help!

[codetheweb]

Glad to hear it!

[Chanete]

Hi @juancrrn glad to see you solve it!! but I'm stucked in an earlier starge, My meter only answers to the 'status' command with these DPS:

{'dps': {'1': 18053, '10': 0, '16': True, '18': '111111111111'}}
Idx: 1 Valor: 18053
Idx: 10 Valor: 0
Idx: 16 Valor: True
Idx: 18 Valor: 111111111111

Are you using Status command or is it any other command??

Thx in advance for your help

[juancrrn]

Hi @Chanete. I'm using the API with the example in the README.md file. As you see in the first comment, sometimes I get a data event with the data you mention, and then the device sends the data that is useful for me.

@codetheweb, maybe this should be moved to a different/new issue?

[codetheweb]

Converted to discussion.

[pety12321]

I have ZMAi-90 powermeter with WB3S (instead of TYWE3S) module. I see the data (with "dp-refresh" example from README.md):
Connected to device!
DATA from device: { dps: { '1': 45, '10': 0, '16': true, '18': '111111111111' } }
DP_REFRESH data from device: { dps: { '6': 'CJ8AAAAAAAA=' }, t: 1617797843 }
DP_REFRESH data from device: { dps: { '6': 'CKIAAAAAAAA=' }, t: 1617797858 }
DP_REFRESH data from device: { dps: { '6': 'CJcAAAAAAAA=' }, t: 1617797873 }
DP_REFRESH data from device: { dps: { '6': 'CKwAAAAAAAA=' }, t: 1617797888 }
Disconnected from device.

I set timeout to 60 seconds, and device report this dps 6 value.
Juancrrn, thank you so much for your decoding algorithm, its work fine for me!

[juancrrn]

I'm very glad it helped 😄.

[veskobx]

How to use this function with Home Assistant?

[Apollon77]

maybe you should ask that in the respectice pythion library used by Homemassistant?

[Troon]

I have just written up a solution on the HA forum using info from this thread:

https://community.home-assistant.io/t/convert-this-data-to-an-understandable-format/731424/2

[SteanlyCZ]

perfect working, is it still possible to decode the frequency (Hz)?

[juancrrn]

Hi @SteanlyCZ, the frequency is not present in the encoded string and, as far as I know, it has not been included.