[WIP] improve perception and motion for SVM+VLM eval in sim #455
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- add median filter into voxel-map confguration setup - fix issues with setting up the motion planning to particular instances; still need some improvements when filtering - add a derivative filter to see if we can't remove some of those floating points (not quite enough; need conv filter i think)
… fixes, and RRT for pick and place in cluttered scenes (#435) * Fixed the timestamp format * do not publish for now; update UI code * add some documetnation or something idk * add some more documentation i guess * handle limited obs * finish everything * add some comments * Added the replacement of Instance id: with image in chat UI * add rpc * update demo * update to not rewrite images * added modals and new layout * did some future-proofing for modals when there are multiple crops * demo chat class component * time to remove hardocding * install updates * hide bullet * Sasha's changes on the laptop * poll_delay * publish limited obs * limited obs working * add utils * Added checks for fields tha are not available. * WORKING DEMO NIGHT BEFORE V0 * WORKING * dd some comment and switch to trace * update configuration + some minor debugging + update motion planner setup * update configuration for exploring * added aws ip address comment * update configuration and config * remove start streaming * update config - motion planning setup * remove start streaming * quick status update * Improved the text of user reqeust * param flag to finish before sweeping * move arm to a custom pose * added find_only * added find_only * sleep in * fix comment * switch from 90 to 75 * self missing * Added fix to view modal instance * refactor * cleanup * refactoring and remove publish_obs * update habitat lab * update habitat-lab * update habitat-lab to main * update * update ovmm * Removing GOAT code * removing GOAT stuff * Moving OvmmPerception to its own folder * refactoring the perception wrapper code * remove mock agent * refactor navigation client code (tools for move and place) * remove unused code * refactor * fix some issues with ik * refactored ui to use class components * fixing * refactored ui to use class components * rm vlm-planning related code * remove GOAT stuff - that can be a separate release * binit's suggested changes from the readme * update config + launch file * delete vlm_demo from real world oovmm * update grasping and refactor some things * fix bug * tweak setup and make sure we are getting semantic sensor from env * fix some issues to make it easier to just pass in a name into grasping * update grasping and make it work * remove minigpt4 submodules * clean up docs and make sure place computation works. Next step: generate path to this point * remove goat docs * improve placement code * remove minigpt4 folders * added modalities and instance 2d seg to scannet datasets * fix broken import * added modalities and instance 2d seg to scannet datasets * patch for torch 1.x indexing * fix a few scannet eval hydra configs * update spot installation * revert GOAT planner, get ovmm sim running * update zerodepth test script * add comment to script * update and break out the parameters to make it a bit easier * add parameters file to make this easier * update demo data script * Instance ids and saving data * update readme and documentation * update agent * update isort * move spot demo agent into a new file -- not tested * isort fixed * update parameters * clean up spot ipython notebook * update configs for refactor * update env and clean up other code * libboost in circleci * install deps for boost * add conda install for pin (optional) * update install with zerodepth * fix control mode; delete more Spot stuff * add atomicwrites + mamba->conda * ok keep mamba because conda is so slow * mamba -> conda and see if circleCI still works * fixes from the lab * added refexp * fixes from the lab * typos * number of episodes not found * add reset function * execute trajectory is cleaner, improving setup, add loguru * add rosnumpy * update instructions, add deprecation * update real-world ovvm scripts to make sure they work properly * remove visualization flag * remove the map rotation inside semantic module * keep obstacles in frontier * remove some code; make sure heuristic policy runs * remove import statement * remove viz flag * default config path * add extract scalars function * remove unused languagenav_agent * imports * remove dependency for sep repo * match explored radius to main * set goal ids correctly in sim * remove goat languagenav files * add docstring * remove hab_info dep * add config * typo * spot code fixes * remove minigpt4 directory * fix linter issues * remove some deps * Ovmm sim demo integration (#439) * added emacs gitignore * first pass at ovmm sim * able to run frontier exploration * added make video to see what's happening * commented out some unused code * added task goals and attempts at grasping * fixing installation issues for simulation + make sure rendering test works * update script * fix some install issues and add habsim * Update install stuff, fix issue with pillow version * dd test script * save computed point cloud * save some more data * add voxel code * Script to read information from voxel map and remove unused import in __init__ * fixing some issues * tart exploring again * try-except to make sure we save data even if things are incorrec * try rotating in place; add support for this * move client and fix inputs * trying to get the robot setup * rotating in place * moving around and exploring * modify agent to improve exploration; still seems to get stuck * nter * linter * first attempt at execute vlm plan --------- Co-authored-by: Chris Paxton <[email protected]> Co-authored-by: Chris Paxton <[email protected]> * update the agent code to fix linter problems * Replace conda with mamba in install_deps.sh * Conceptfusion (post-demo) (#441) * add scannet dataset creation * moved scannet to home_robot/datasets and added licenses * adding init.py * Add referring expressions to scannet dataset * SparseVoxelMap and InstanceMemory use tensors and work on gpu * added instance visualization and some misc documentation * nbstriput to not commit notebook outputs * nbstriput to not commit notebook outputs * remove gitattributes -- either we can use git or pre-commit. dont need both. * strip output for circleci * add metadata strip * add jupyter nbconvert for metadata strip * return None for RGB cluster when no RGB provided * return None for RGB cluster when no RGB provided * strip output from nb * Make changes to installation to (hopefully) make build process easier (#351) * strip output from nb * add object detection eval helpers for scannet * scannet build_3d_map and eval example * scannet build_3d_map and eval example * referring expression evals and grounded sam on scannet * added concept fusion demo * adding eval script for full scannet * handle nans and small bounding boxes * add instance association within category * added bbox postprocessing NMS and box-tightening * removed extra code * Add the DBScan clustering * new project, moved major functionality to conceptfusion * added visualizations, tuned parameters * incorporate changes from Sasha * Try Sasha's idea for quering * added config * semmap updatae * float instead of double * cleaning up merge * added a little documentation * added a litle documentation --------- Co-authored-by: Sasha Sax <[email protected]> Co-authored-by: Chris Paxton <[email protected]> Co-authored-by: Karmesh Yadav <[email protected]> * add network option? * command line arguments for install script * remove centernet * update robot model * minor changes to set up code * remove hardcoded keys * update challenge.md * add keys in hssd_demo.yaml * fix instructions for docker build * rviz image bug fix * example * minor bug fixes and helpful logging * delete logger file -- not needed * testing rpc stub * updates to ovmm sim * task_rpc_env_pb2_grpc should be imported after src.rpc is imported * grasping fixes * new fixes * new fixes * Update challenge.md * Update challenge.md * minor fixes + new timestamp for datadump * fixes for sim * camera pose fix + refactor * update hab-lab commit * framework for sample strategy * slight fix * . * remove print * make habitat camera pose work for SVM in sim * Update spot.md * Update spot.md * Update spot.md * add clip queries to svm * minor tweaks for gpt4v agent * fixes for evaluations * adding nr3d only * adding nr3d only * optimize image saving * make padding a param * improving some parameters in stretch demo code * add in-place rotation for stretch demo * rotations work in home * update robot demo code for exploration * fix some issues with Stretch script * formatting * update visualizer * update this code so that we can see whats going on * evaluator * linter errors (isort) * delete and add h5py version * dependency fixes * cleanup deps * roll back hssd-hab commit change; keep using ovmm branch with navmeshes * updating robot code * fixing simple heuristic example * switch to opengl camera pose via config, remove unused code * rename opengl to opencv * clean up debug statements --------- Co-authored-by: Oleksandr Maksymets <[email protected]> Co-authored-by: Sasha Sax <[email protected]> Co-authored-by: PrieureDeSion <[email protected]> Co-authored-by: Jay Vakil <[email protected]> Co-authored-by: pmcvay <[email protected]> Co-authored-by: keke-220 <[email protected]> Co-authored-by: Sriram Yenamandra <[email protected]> Co-authored-by: jdvakil <[email protected]> Co-authored-by: Sasha Sax <[email protected]> Co-authored-by: Karmesh Yadav <[email protected]>
facebook-github-bot
added
the
CLA Signed
cpaxton
reviewed
Dec 21, 2023
cpaxton
reviewed
Dec 21, 2023
cpaxton
reviewed
Dec 21, 2023
cpaxton
reviewed
Dec 21, 2023
…-robot into xiaohan/svm_vlm_sim
…-robot into xiaohan/svm_vlm_sim
…aohan/svm_vlm_sim
7 tasks
…e-robot into xiaohan/svm_vlm_sim
frontier map though.
…e-robot into xiaohan/svm_vlm_sim
…e-robot into xiaohan/svm_vlm_sim
…e-robot into xiaohan/svm_vlm_sim
…e-robot into xiaohan/svm_vlm_sim
based on clip embeddings
visualization options
added 9 commits
March 24, 2024 15:15
…e-robot into xiaohan/svm_vlm_sim
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Changelog
How Has This Been Tested
Types of changes
Checklist:
tests/hw_manual_test.pyon hardware and observed that robot behavior looks normal.