Adding valetudo obstacles images to the camera map #296
Replies: 64 comments 22 replies
-
|
@borgqueenx yes I followed the discussion you started with @PiotrMachowski and I also got documented on this topic. The x10 actually provide in the json data the following: {
"__class": "PointMapEntity",
"metaData": {
"label": "Pedestal (89%)",
"id": "33911310-ff43-529c-862b-4765035ecd34",
"image": "/data/record/1.jpg"
},
"points": [
3325,
3560
],
"type": "obstacle"
}and so far I understood from the spec. should be possible to get the image of the obstacle by query to the api of the vacuum. As you mentioned, develop this can require a little of time and evaluation, and to be honest I can't invest on an X10 just to try it. But I think if we cooperate we could maybe get it working. In the first idea I had for this improvement I thought to add to the camera configuration an "user storage folder path" to eventually store the image data. This is actually required as per the www (local) that should be also used from the card actually isn't a private folder.. I mean for who publish the config on GitHub for example. The camera as you say is currently providing a dot at the coordinate of the obstacle, I could probably also implement in a more easy way actually a link (composed with the info from the json) to open the vacuum api url (data that are actually available by default). Indeed I do really like this as per there will be no data actually export to HA and we use the Valetudo spec. as it should be instead. I will check with Piotr how I can provide this link to the card later on, as the link should be in the point info (label). |
Beta Was this translation helpful? Give feedback.
-
|
that last idea sounds easier to implement without too much coding and time, nice :) Will it then open the url in a browser or can we show the image inside of home assistant? If you need any data from my robot or want me to test stuff, just let me know, happy to help with this in whatever way i can. |
Beta Was this translation helpful? Give feedback.
-
|
Well I do not know how this will look in the end but already in the first beta of the next version probably we will have the url, then we can try, what can best fit for all 😃.. I already start to work on it I will open a PR as soon I will finish some test and write the code for this.. I guess anyhow it will work in local. |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx step 1 completed: |
Beta Was this translation helpful? Give feedback.
-
awesome, im glad this has you excited, i think it's awesome that this functionality that normally requires cloud/internet access, now works locally with the help of valetudo. |
Beta Was this translation helpful? Give feedback.
-
If possible by clicking the link (that I will need to discuss with Piotr) I can try to open the image and reproduce it as camera image, but for this we need an event from the card (so some card edit will be required for it). This would be the ideal as it would cover the cloud option. Anyhow with the link it should open the image in the browser when in local mode. |
Beta Was this translation helpful? Give feedback.
-
|
opening the link in the browser is ofcourse not ideal as it opens another app, that we have to exit and go back to home assistant, to then click another object that will then again open the browser. Opening in the camera image would probably be best. a simple click to bring back to map.....just thinking out loud |
Beta Was this translation helpful? Give feedback.
-
|
yep.. I would gladly go for the second option but.. anyhow first let's try to get to the image and possibly open it, then we will figure out how to "make it works" in the most natural way.. this is what I'm trying to propose here. The final spec. is clear to me and align with yours. |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx can you please confirm this urls if they work for you? I mean you need to replace the fields here but is just to compose the url in the proper way, I can give a shot in the beta 0 of the next version of the camera, in parallel I'm trying somehow to test.. if I can switch images. The problem so far is that I need to request the API to get the image. http://<robot_ip>/api/v2/robot/camera_obstacle_imagesThe specific Images: http://<robot_ip>/api/v2/robot/camera_obstacle_images/<image_filename>If those aren't correct I can you please post the correct api url? |
Beta Was this translation helpful? Give feedback.
-
|
How do you want me to test to grab those images? In my browser and get a http not found page. The robot is also working right now with many obstacles found. |
Beta Was this translation helpful? Give feedback.
-
|
You need to open the api of the robot then add the path with the correct file name. |
Beta Was this translation helpful? Give feedback.
-
Can you please ommitting your vacuum name, provide the full url when the image open? |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx if you are running the last valetudo vesion so far I understood as soon you enter in the vacuum web interface it should be possible to click on the obstacle that the vacuum GUI show in the map and from there on your browser should open the image (according to what I see also on the image if you use 192.168.1.162 should open the Valetudo GUI then on the map that should be similar to what I see (with the dots) you can click there.. I need the end point (copy past of the url your vacuum use to open the image). |
Beta Was this translation helpful? Give feedback.
-
|
http://192.168.1.162/api/v2/robot/capabilities/ObstacleImagesCapability/img/195ee1da-be5c-5837-b8c1-1d2ac8cf08ae I also tried to view the directory by removing the image name from the link and removing /img but both are not working either. Edit: i got that link as you suggested, through valetudo's web interface, then get link to image. |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx yep |
Beta Was this translation helpful? Give feedback.
-
|
i am indeed able to select a obstacle relative close and click play, and it will show me the obstacle near. Take your time building, no rush :) Already super appriciate it! |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx just to understand better this phenomena.. is the vacuum working while you check the obstacles? I tested only the scenario when the vacuum is "docked" in my tests, probably this is one of the possible scenario I need to "emulate" in my test? |
Beta Was this translation helpful? Give feedback.
-
|
No, it is docked. |
Beta Was this translation helpful? Give feedback.
-
|
today i had the issue again, but also a run without the issue. |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx when you click on one obstacle and the camera do not switch to obstacle_view can be because of the fact that:
I think I can work a little on this, today and probably later will release a beta or patch version. The issue with the view of the wrong obstacle is in my opinion more important to solve, also because it looks the view. About this new one (as above) it could be there was one of the above, in this case the behaviour is correct.. the "Success" in the card indicates only that the service did run without error. I can raise some service errors to highlight when the camera fails to retrieve the image. |
Beta Was this translation helpful? Give feedback.
-
|
Where does the IP address/hostname from which HA downloads the image come from? Because mine says: HA logs: My workaround in HA currently looks like this: |
Beta Was this translation helpful? Give feedback.
-
|
@marvinwankersteen thanks for reporting this.. The IP is normally in the wifiCapability of the vacuum, I will implement a function that will use the vacuum domain for the api call YOURVACUUMNAME.loca in the case the IP isn't found.. by the way. in the logs should be possible to see if we catch the IP correctly from the payload of the vacuum. l didn't know that this capability might me missing or somehow this value could be somewhere else.. anyhow I guess this should fix permanently this. I might transfer this to the discussion and open some new issues for tracking are we okay with that @borgqueenx? |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx and @marvinwankersteen |
Beta Was this translation helpful? Give feedback.
-
|
Realised the 2024.12.1 the issue with the IP address for the links is now fixed, 🙈 I forgot initially to return the IP if there's just IPv4.. I did also remove the thread during the download and add some more logs to show the camera_mode. Still to implement the errors in case the service will fail somehow.. (the camera raises the warnings) a validation error would give more feedback when something goes wrong. Indeed the most important is that the camera works as intended. I'm considering also to implement the scenario that when there are no coordinates (the play button is pushed without selection) the camera switches to map_view. When we are finally finished with this new functionality will be working on a huge refactoring.. Hopefully we will get it right soon. Thanks for working with me on this. |
Beta Was this translation helpful? Give feedback.
-
|
When i press the play button without a selection it gives a error. I still have the same issue. First obstacle image i clicked opens fine. Then i can not get back to map view. I click somewhere on map then press play and it says "success" but i keep seeing the obstacle image. |
Beta Was this translation helpful? Give feedback.
-
|
Okay got it... will check and let you know.. soon |
Beta Was this translation helpful? Give feedback.
-
In my tests using your map, it works, I will although change the code to make sure "it goes" well the next time. Unfortunately do not have a real vacuum with this ObstacleImages so I thank you really much for your time, cooperation and understanding. Edit: I just changed it so that we use only memory and not access to the disk = performance wise isn't that different. Just noticed that sometimes it needs a little longer to come back to camera_view but.. select + push the button ... wait 3 to 9 seconds and then all works. The refresh is every 3 seconds since the poling is active. So if you do not wait till it switch this can probably create problems. |
Beta Was this translation helpful? Give feedback.
-
|
I always gave it some seconds but it did not help, around 3 till 10. |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx Changes documented also in the PR #299:
I know this is not usual but.. if you download this zip file and with samba or vscode copy the contents in it in the custom_componets /mqtt_vacuum_camera/ folder basically you apply the modifications as HACS would do.
Waiting for your kind feedback, thanks once again for the cooperation here. |
Beta Was this translation helpful? Give feedback.
-
|
unfortunately did not work, just tested it. gave it over 10 seconds to change back to camera view but refuses to do so. always at the first obstacle image i open. The first obstacle image i open works 100%, by the way. I tried again to use the option to create a log in /www but no file was created. I only see the snapshot in that folder of the camera view. If you want, you can send me a private message for teamviewer access, to see the issue for yourself, if that helps? And yes, star trek is great :D |
Beta Was this translation helpful? Give feedback.
-
|
Well I think I found the cause... |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx It takes almost 1 minute to open 2.5Gb immage and switch back to map. I acknowledged that is a little beat frustrating.. I mean you do not see directly in the UI what the hell the camera is doing.. I'm checking if we can trace the camera_mode as a "sensor" or state change with the card and this should be giving us an idea of what happens and what we should expect.
|
Beta Was this translation helpful? Give feedback.
-
|
mmmm this is really strange for instance I can't reproduce it but I noticed something strange as I added a new attribute with camera_mode and I a tile to the card that grab the status.. I think as per it is using the shared value is going just in map_view or obstacle_view and if it stacks it is possible that the second event isn't produced (but from the log you passed me it was not clear). |
Beta Was this translation helpful? Give feedback.
-
|
you can always use teamviewer to debug if you want to see my home assistant instance. |
Beta Was this translation helpful? Give feedback.
-
|
This could be nice, I will try to download it and install it.. |
Beta Was this translation helpful? Give feedback.
-
|
if you have discord, please DM me your username. lets chat and i can provide you live access, to see whats going on. |
Beta Was this translation helpful? Give feedback.
-
|
[redacted] is my discord.. I will be waiting for you there. I think anyhow we can fix this for sure. |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx
I'm currently checking if I can redact from the logs the IP address (privacy wise correct). Thanks a lot for your time and cooperation, very appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
@borgqueenx 2024.12.2b0 is released with fix. It might solve the issue when the vacuum is docked 😉 if you have time tomorrow please let me know if it works as expected 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I wanted to thank and congratulate you for this really great work! After a lot of trouble, I managed to kind of make obstacle view work for me. The main issue I have is that it is not compatible with the Basic Auth feature Valetudo has. If it is enabled, picture download yields a 401 error (as you would expect). I don't know what would be the best way to make it work (store the credentials somewhere locally ?) or if you would even want to make it work. My valetudo is exposed on the internet and "protected" by Basic Auth so I either need to keep it, or just stop exposing it and disable the auth. Anyway that was my two cents, thank you again for your work! |
Beta Was this translation helpful? Give feedback.
-
@T0ytoy This is a really good point.. I didn't consider this scenario as mainly I never thought about to expose directly the vacuum on internet.. as it is c connected via HA and feel there's no need as HA is exposed. I will consider this although and see if we can found add an option and store those credentials in the config... can't actually work on it immediately. |
Beta Was this translation helpful? Give feedback.
-
|
There is no rush obviously ;) Maybe the best way would be to not expose the robot to the Internet, and only access it from a VPN if I really need to change a setting when I'm not at home. Anyway if you decide to work on this, I will be available to test, if you need it ;) |
Beta Was this translation helpful? Give feedback.
-
|
Cool stuff, thanks! It would be nice if it defaulted to the live camera view if no obstacle is selected. |
Beta Was this translation helpful? Give feedback.
-
|
@kevinhaas did not catch exactly what do you mean with: "defaulted to the live camera view if no obstacle is select".. the map_view or obstacle_view? I mean by now of course you need to select the obstacle to see it.. and then click on the image to switch back to map_view. I guess you would like to see the obstacle as soon is detected and automatically switch to map_view after a set time.. is my understanding correct? If I didn't implement it like so is because it's a little complicated to introduces delays in Home Assistant.. and as well not everyone stay in front of the display watching what the vacuum do or found on it's way. As well I think from privacy wise as it is implemented is better... for instance a human need to select "intuitionally" the detected obstacle. Anyhow hope you agree with what stated.. and enjoy this function ;) don't forget to start this project if so :) |
Beta Was this translation helpful? Give feedback.
-
I took it this meant the live camera feed, and not just obstacle images. I meant it would be awesome if the "live view" was an option. Maybe it is and I missed it? |
Beta Was this translation helpful? Give feedback.
-
|
@kevinhaas "in live view" this means that is possible to get the obstacle image while it is detected (if you select it with the modified service call from the card). Actually I'm working out a library.. but probably to feet your request what about this.. adding an option to see the the obstacles.. I can implement a function to show them automatically for let's say 3 seconds.. if this what you mean. Sorry if I replay that late but I was't notified ;) |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I'm not talking about obstacles. I have a Dreame L10S Ultra and it allows you to view the live camera feed and control the robot via the Dreame app. I'm not sure if valetudo exposes that live feed. They do have "manual control", though. |
Beta Was this translation helpful? Give feedback.
-
|
@kevinhaas using valetudo on L10S there isn't so far I know a function to see the camera field of the vacuum on live streaming and I guess the problem is my bad english.. it is possible to see the photos of the obstacles detected during the cycle. What I meant with "live view" was that you can actually load the obstacle images in HA while the vacuum is cleaning, and switch back to map view whenever you want.. if your system is enough power full for that. About the "manual control", yep I guess you can move the robot in manual mode but.. this project is to use the vacuums for what are meant to do.. cleaning, is there a particular future you are interested in that could be useful for everybody I can help with? |
Beta Was this translation helpful? Give feedback.
-
Description
I realise that this feature request is likely quite a challenge, but i have seen that your integration already provides red dots on the map to where obstacles have been detected.
I already asked PiotrMachowski if he could have the time and motivation to build something like this, but not in the forseeable future. But i thought that perhaps this integration could help? I realise that we probably cant make things clickable as it's a kind of video stream of the map, right? But maybe there is a way to click a button we can setup as a tile in xiaomi's map card and then see images of the robot in the map.
I have already figured out where valetudo stores the images, see the thread here: PiotrMachowski/lovelace-xiaomi-vacuum-map-card#774
However there needs to be a way for the user to grab those images to home assistant.
As i have written in that other thread, we could use ssh, but maybe the integration could provde a easy way to grab the images from the robot.
As i think about it, integration it perfectly (clicking obstacles etc) probably requires both some work on the map card and on this integration at the same time. But maybe a easier way can be developped where we can simply check images, without having to click on the map.
Solution
Adding obstacle images to the map of mqtt vacuum camera.
Alternatives
No response
Context
No response
Beta Was this translation helpful? Give feedback.
All reactions