Replies: 3 comments 1 reply
-
|
Removing the database connection would be a shame as I have been using that for a while. Is the plan to have a pure mqqt architecture, does it make sense to try and put together another project to just take mqqt traffic and write to a database ? |
Beta Was this translation helpful? Give feedback.
-
|
My suspicion is very few people use this database integration, a few more than that use InfluxDB, and the vast majority use MQTT indirectly (to feed to HA). When I originally started the project I was mostly interested in MQTT and InfluxDB and later added the database integration just to play with it, but it's by far the hardest integration to maintain. So for the amount of people that I suspect use it, it's probably not worth it. The rust sqlx library isn't the easiest to work with, it's very comprehensive and supports a lot of things but is probably complete overkill for what I need really. We're also using a slightly older version, 0.6 - I tried upgrading sqlx a few months back and gave up, maybe it'd be better to try writing the database integration again from scratch using 0.7. Maybe a simpler solution using another crate could work but I'm not too familiar with Rust ORMs generally. Getting a bit broader, this might turn into more of a "where to go next with this app" type post. [Edit after writing the below]: I've started this now so I'll finish but this might want breaking out into another discussion. None of the below is now really relevant to the database question so feel free to skip all this, it was more for my benefit to organise my thinking ;) There's a more general problem brewing in that as the app gets more complicated and supports more inverters, previous assumptions are no longer holding true. In particular the 3 inputs packets that I work around in the app (inputs/1, inputs/2, and inputs/3) are showing their age and newer inverters can either send them all at once (which was fine, I bodged around that) or apparently send 4 packets instead. Some support has recently been added for this but it's a bit broken and I'm trying to think how to solve it. The app currently collects the 3 packets and collates them into one structure which is sent out over MQTT/to InfluxDB/to databases. This breaks when there's a 4th packet coming and we don't really know whether the inverter will send it or not until we get it. So we have a choice, wait to see if the 4th packet turns up (introduce an artificial delay, ugh), or send the data without the 4th packet like we did before, which loses that data. I've floated yet another workaround elsewhere which is to tell the config file whether your inverter sends 3 packets or 4, which would do for now. It wouldn't be much work and would fix "most" problems (when thinking of this I was tempted to ignore the database storage, whiich prompted this discussion) A better solution would be to decouple the packets the inverter sends from the MQTT we publish. The 3 or 4 packets from the inverter should be an internal implementation detail and MQTT subscribers really shouldn't need to care about it. My initial approach to this was So my next attempt at this (and this has kind of already been started) would be to split the data up instead and send one MQTT packet per inverter register. This has always been the case for The two problems with this approach are InfluxDB and the databases. Those integrations really need a full set of data to write a "row" at a time. This is what's currently keeping me up at night ;) Dropping the database integration solves one problem (basically just punting the problem away to an external script), but InfluxDB I think is popular enough to warrant keeping but unfortunately has the same issue, because it uses one measurement for all the values. If you insert data piecemeal then you end up with lots of null values interspersed, which makes for a terrible graphing experience. So are we back to building up a single datastore then sending it a few seconds after we stop receiving inputs packets, and resetting? ugh. Yeah I'm gonna stop now, thats enough ;) |
Beta Was this translation helpful? Give feedback.
-
|
I'd be happy for database integration to be dropped. Its outside of 'core'
functionality for me. I take what I need from MQTT along with any
calculations and store them according to my needs.
Thanks for the support!
Kind Regards
John
…On Sun, 10 Mar 2024 at 07:42, Chris Elsworth ***@***.***> wrote:
My suspicion is very few people use this database integration, a few more
than that use InfluxDB, and the vast majority use MQTT indirectly (to feed
to HA).
When I originally started the project I was mostly interested in MQTT and
InfluxDB and later added the database integration just to play with it, but
it's by far the hardest integration to maintain. So for the amount of
people that I suspect use it, it's probably not worth it.
The rust sqlx library isn't the easiest to work with, it's very
comprehensive and supports a lot of things but is probably complete
overkill for what I need really. We're also using a slightly older version,
0.6 - I tried upgrading sqlx a few months back and gave up, maybe it'd be
better to try writing the database integration again from scratch using
0.7. Maybe a simpler solution using another crate could work but I'm not
too familiar with Rust ORMs generally.
Getting a bit broader, this might turn into more of a "where to go next
with this app" type post. [Edit after writing the below]: I've started this
now so I'll finish but this might want breaking out into another
discussion. None of the below is now really relevant to the database
question so feel free to skip all this, it was more for my benefit to
organise my thinking ;)
There's a more general problem brewing in that as the app gets more
complicated and supports more inverters, previous assumptions are no longer
holding true. In particular the 3 inputs packets that I work around in the
app (inputs/1, inputs/2, and inputs/3) are showing their age and newer
inverters can either send them all at once (which was fine, I bodged around
that) or apparently send 4 packets instead. Some support has recently been
added for this but it's a bit broken and I'm trying to think how to solve
it.
The app currently collects the 3 packets and collates them into one
structure which is sent out over MQTT/to InfluxDB/to databases. This breaks
when there's a 4th packet coming and we don't really know whether the
inverter will send it or not until we get it. So we have a choice, wait to
see if the 4th packet turns up (introduce an artificial delay, ugh), or
send the data without the 4th packet like we did before, which loses that
data.
I've floated yet another workaround elsewhere which is to tell the config
file whether your inverter sends 3 packets or 4, which would do for now. It
wouldn't be much work and would fix "most" problems (when thinking of this
I was tempted to ignore the database storage, whiich prompted this
discussion)
A better solution would be to decouple the packets the inverter sends from
the MQTT we publish. The 3 or 4 packets from the inverter should be an
internal implementation detail and MQTT subscribers really shouldn't need
to care about it.
My initial approach to this was inputs/all but for reasons detailed above
this is a disaster. We don't know when all is complete!
So my next attempt at this (and this has kind of already been started)
would be to split the data up instead and send one MQTT packet per inverter
register. This has always been the case for hold registers and individual
input register messages were added not too long ago (but disabled by
default). I think these should become the primary way to communicate
outwards to HA (so it no longer relies on inputs/all - this is pretty
easy and invisible to users).
The two problems with this approach are InfluxDB and the databases. Those
integrations really need a full set of data to write a "row" at a time.
This is what's currently keeping me up at night ;) Dropping the database
integration solves one problem (basically just punting the problem away to
an external script), but InfluxDB I think is popular enough to warrant
keeping but unfortunately has the same issue, because it uses one
measurement for all the values. If you insert data piecemeal then you end
up with lots of null values interspersed, which makes for a terrible
graphing experience. So are we back to building up a single datastore then
sending it a few seconds after we stop receiving inputs packets, and
resetting? ugh.
Yeah I'm gonna stop now, thats enough ;)
—
Reply to this email directly, view it on GitHub
<#241 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADHTIDKJCQFNUXUJ6QWYY4DYXQFGZAVCNFSM6AAAAABEIS4CAOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DOMZUGEZDM>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
I was thinking about how the inverter capture and current state of the
inverter could be achieved when some inverters are not always sending a
full set of data. One possible way forward is to extend the publish
subscribe model within the mqqt side and make the database components be
subscribers as well and make them figure out when there is a full set of
data.
John
…On Sun, Mar 10, 2024, 09:30 John ***@***.***> wrote:
I'd be happy for database integration to be dropped. Its outside of 'core'
functionality for me. I take what I need from MQTT along with any
calculations and store them according to my needs.
Thanks for the support!
Kind Regards
John
On Sun, 10 Mar 2024 at 07:42, Chris Elsworth ***@***.***>
wrote:
> My suspicion is very few people use this database integration, a few
more
> than that use InfluxDB, and the vast majority use MQTT indirectly (to
feed
> to HA).
>
> When I originally started the project I was mostly interested in MQTT
and
> InfluxDB and later added the database integration just to play with it,
but
> it's by far the hardest integration to maintain. So for the amount of
> people that I suspect use it, it's probably not worth it.
>
> The rust sqlx library isn't the easiest to work with, it's very
> comprehensive and supports a lot of things but is probably complete
> overkill for what I need really. We're also using a slightly older
version,
> 0.6 - I tried upgrading sqlx a few months back and gave up, maybe it'd
be
> better to try writing the database integration again from scratch using
> 0.7. Maybe a simpler solution using another crate could work but I'm not
> too familiar with Rust ORMs generally.
>
> Getting a bit broader, this might turn into more of a "where to go next
> with this app" type post. [Edit after writing the below]: I've started
this
> now so I'll finish but this might want breaking out into another
> discussion. None of the below is now really relevant to the database
> question so feel free to skip all this, it was more for my benefit to
> organise my thinking ;)
>
> There's a more general problem brewing in that as the app gets more
> complicated and supports more inverters, previous assumptions are no
longer
> holding true. In particular the 3 inputs packets that I work around in
the
> app (inputs/1, inputs/2, and inputs/3) are showing their age and newer
> inverters can either send them all at once (which was fine, I bodged
around
> that) or apparently send 4 packets instead. Some support has recently
been
> added for this but it's a bit broken and I'm trying to think how to
solve
> it.
>
> The app currently collects the 3 packets and collates them into one
> structure which is sent out over MQTT/to InfluxDB/to databases. This
breaks
> when there's a 4th packet coming and we don't really know whether the
> inverter will send it or not until we get it. So we have a choice, wait
to
> see if the 4th packet turns up (introduce an artificial delay, ugh), or
> send the data without the 4th packet like we did before, which loses
that
> data.
>
> I've floated yet another workaround elsewhere which is to tell the
config
> file whether your inverter sends 3 packets or 4, which would do for now.
It
> wouldn't be much work and would fix "most" problems (when thinking of
this
> I was tempted to ignore the database storage, whiich prompted this
> discussion)
>
> A better solution would be to decouple the packets the inverter sends
from
> the MQTT we publish. The 3 or 4 packets from the inverter should be an
> internal implementation detail and MQTT subscribers really shouldn't
need
> to care about it.
>
> My initial approach to this was inputs/all but for reasons detailed
above
> this is a disaster. We don't know when all is complete!
>
> So my next attempt at this (and this has kind of already been started)
> would be to split the data up instead and send one MQTT packet per
inverter
> register. This has always been the case for hold registers and
individual
> input register messages were added not too long ago (but disabled by
> default). I think these should become the primary way to communicate
> outwards to HA (so it no longer relies on inputs/all - this is pretty
> easy and invisible to users).
>
> The two problems with this approach are InfluxDB and the databases.
Those
> integrations really need a full set of data to write a "row" at a time.
> This is what's currently keeping me up at night ;) Dropping the database
> integration solves one problem (basically just punting the problem away
to
> an external script), but InfluxDB I think is popular enough to warrant
> keeping but unfortunately has the same issue, because it uses one
> measurement for all the values. If you insert data piecemeal then you
end
> up with lots of null values interspersed, which makes for a terrible
> graphing experience. So are we back to building up a single datastore
then
> sending it a few seconds after we stop receiving inputs packets, and
> resetting? ugh.
>
> Yeah I'm gonna stop now, thats enough ;)
>
> —
> Reply to this email directly, view it on GitHub
> <
#241 (reply in thread)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/ADHTIDKJCQFNUXUJ6QWYY4DYXQFGZAVCNFSM6AAAAABEIS4CAOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DOMZUGEZDM>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***
> com>
>
—
Reply to this email directly, view it on GitHub
<#241 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AN6SNQM5I7SERSM4ZK5WT5LYXQR2XAVCNFSM6AAAAABEIS4CAOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DOMZUGYYDS>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
As a headsup, I am considering removing the database integration (storing values into MySQL/Postgres/SQlite), or at the very least simplifying it down to one database (probably Postgres) - mainly because sqlx is a real pain to deal with, especially when using a database-agnostic adapter to get support for all 3 databases. I kind of wish I hadn't added it in the first place!
This won't necessarily be any time soon, but I likely won't be updating the database support either to add newer columns like the EG4 Generator stuff thats been added this week.
So before I commit to it, please shout if you're using this integration and couldn't bear to lose it. Maybe we can find alternatives.
Beta Was this translation helpful? Give feedback.
All reactions