-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP Celery Code Impact #97
Draft
DaanZ
wants to merge
138
commits into
ci-group:celery
Choose a base branch
from
SamFerwerda:master
base: celery
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
138 commits
Select commit
Hold shift + click to select a range
d24bd8d
added celery basics
SamFerwerda 0fbcf7f
celery works
SamFerwerda d31d70d
changed readme file
SamFerwerda 54e9152
Added a standard robot simulation to celery
SamFerwerda 5a62a7f
Added functionality for robot queues and insertion
SamFerwerda 66d3bec
CeleryController added
SamFerwerda aed0ac6
Added some extra dependencies, that are needed for celery + cpp
SamFerwerda 4e6b037
added a simulator queue for celery and changed the manager
SamFerwerda ca8dc87
Evaluate_robot (task) now handles multiple robots at a time, by waiti…
SamFerwerda 8f3f8f0
finished evaluate_robot and adjusted population class to fit celery
SamFerwerda e881ebd
Backup before major changes in tasks.py file
SamFerwerda ad3d620
removed evaluate_robot and made simulator_worker, added a put_in_queu…
SamFerwerda 80e550a
Comments added to all functions.
SamFerwerda e223718
limited celery workers to one task at the time, added evaluate_robot …
SamFerwerda c23601c
Every celery worker will now also analyze the robot, and added speed …
SamFerwerda 0d9968a
Implemented the basics of message consuming in C++.
SamFerwerda 54bfa8c
Simple celery c++ task is set up
SamFerwerda e8388f5
globalized celery channel and consumer tag
SamFerwerda a8a7b6a
Bug was caused by printing the sdf string, also made some variables g…
SamFerwerda 37915a1
c++ part is now handling actual robots, however just returning 1, Non…
SamFerwerda 99cfdc0
Celery is now fully implemented on python and cpp side. However, curr…
SamFerwerda 399e9ad
Some comments and error handling.
SamFerwerda cf71c19
Starting implementing contacts as result
SamFerwerda 5a1498b
Basic locking system added and standard value of 10 added to every co…
SamFerwerda dc035f4
real contacts are send through celery know. However, contacts are cal…
SamFerwerda b82f052
celery messages are converting to robot_manager with the function upd…
SamFerwerda 15a6da3
Adding a function to restart analyzer simulator. Currently in test ph…
SamFerwerda f1b1f15
Added a timeout on the get request of a celery message. Such that if …
SamFerwerda 029a4ec
Added a way to reset celery and gazebo if a get request fails or 10 g…
SamFerwerda 1ed7ec1
Changed the readme description
SamFerwerda 0aa4489
Added changed files and problems with celery
SamFerwerda 876ed5d
Old revolve is now accessible again with the normal world controller.…
SamFerwerda 5ef39d7
Update manager_pop.py
SamFerwerda 85f67d8
Make changes required for running on ripper machines
DaanZ 7850549
celery_pool_asyncio no longer needs patch
SamFerwerda d2d6c23
Update manager.py
SamFerwerda 606de95
Update celery.py
SamFerwerda 10c65fc
Adding logging so we can find some errors or bugs
SamFerwerda 4ecff5c
Commented pkill because not sure about effects on other ripper users.
SamFerwerda a69db25
This file isnt needed.
SamFerwerda 2994824
Old comments and imports deleted.
SamFerwerda 48e0ecc
Update readme.txt
SamFerwerda e77dd5e
Rename readme.txt to README.md
SamFerwerda 5a84078
Updated the readme file for better reader experience.
SamFerwerda 6b06ff9
Update README.md
SamFerwerda 3d6e87f
Update README.md
SamFerwerda 33065d4
Update README.md
SamFerwerda 84e9c2b
Update README.md
SamFerwerda f40267d
Update README.md
SamFerwerda 3a69a95
Update README.md
SamFerwerda 7936037
Update README.md
SamFerwerda 116ca43
Update README.md
SamFerwerda 55b5fad
Update README.md
SamFerwerda a85758a
Update celery.py
SamFerwerda 9ebd6d6
Update CMakeLists.txt
SamFerwerda a903b1b
Got fatal error that there is no such file json/json.h
SamFerwerda 81de1d2
Last commit didnt fix issue
SamFerwerda 923dae3
changed cmake file to use pkgconfig for simpleamqp, fixed jsoncpp
SamFerwerda cf4945d
Currently failing due to not finding libSimpleAmqpClient.so.2
SamFerwerda 7352b4f
Added celery and celery pool asyncio to the python requirements.
DaanZ bd4205c
Changed Cmake to add library to build/lib folder.
SamFerwerda 0a1f6cd
Merge branch 'ripper' of https://github.com/SamFerwerda/revolve into …
SamFerwerda dc7d807
Works on local machine, not yet on Ripper
SamFerwerda e925771
Changed pkg to find_library, works locally, testing on ripper
SamFerwerda 82cf6be
Celery runs with logs
DaanZ 5bf11f2
Added speedtesting requirements.
SamFerwerda 2d200ab
added speedup checks to analyze speedup
SamFerwerda 572b886
Added timing points to find bottleneck of revolve
SamFerwerda e6a49a3
Fixed wrong position of lists
SamFerwerda 2c8a842
reversed location of list again
SamFerwerda f9b0f72
fixes
SamFerwerda 5f08887
speed data
SamFerwerda bfca4c7
speed data
SamFerwerda 4e25224
changed speeddata
SamFerwerda 1a7c630
added data
SamFerwerda 299dade
added data
SamFerwerda a5cd902
changes in manager_pop
SamFerwerda 0a83f4a
Update manager for writing data.
SamFerwerda 56c47cb
Commit the revolve log for data
SamFerwerda 09214b0
Speed data
SamFerwerda 30c82d2
Changed location of time measure.
SamFerwerda 31947ba
data
SamFerwerda 2031c0e
Data for 100 gen
SamFerwerda a832b50
Added extra data writing points.
SamFerwerda 16cef67
some celery data
SamFerwerda c041889
export is now done parallel to evaluating
SamFerwerda 2c4723e
Merge branch 'speedup' of https://github.com/SamFerwerda/revolve into…
SamFerwerda dc68210
possible fix to nonetype error
SamFerwerda 27f6539
speed data
SamFerwerda 4b5b2e9
Implemented speed checks to analyze revolve
SamFerwerda 1273bb1
speeddata
SamFerwerda c829e4e
added start_experiment.py for quick parallel experiments
SamFerwerda 93d0e3a
Changed layout of speedlogging
SamFerwerda 4d2aa47
Speed data of 15 normal runs
SamFerwerda 1b29d56
deleted revolve.log
SamFerwerda 845ac27
changes in start_experiment and manager
SamFerwerda 44e40ba
Merge branch 'speedup' of https://github.com/SamFerwerda/revolve into…
SamFerwerda bf4e329
Deleted the timeout of 120 seconds
SamFerwerda fc82bb5
small changes in manager file
SamFerwerda c92c4bb
speed data of all 30 old revolve
SamFerwerda a6166d8
delete data
SamFerwerda 83aa70e
Queues are now unique for different experiments
SamFerwerda 151caa2
fixed wrong implemententation in population.py
SamFerwerda db57b35
Test for multi experiments with celery
SamFerwerda 0ab9832
Backup
SamFerwerda 3d86b58
Unique queues for different experiments
SamFerwerda 2b0450c
Merge branch 'master' into multiCelery
SamFerwerda 82e514e
added script for consecutive runs
SamFerwerda ac94e40
extra data 32 cores
SamFerwerda 1f5063f
removed worker pids
SamFerwerda e46fd3c
changes for consecutive runs
SamFerwerda b4e3364
data for 32 cores
SamFerwerda 830f63e
Unnecesarry file
SamFerwerda 2384e1e
Data for 16 cores
SamFerwerda 0a806b1
data for 16 cores
SamFerwerda eb6bd87
Merge branch 'master' of https://github.com/SamFerwerda/revolve
SamFerwerda e9a1cd5
all data for 16 cores
SamFerwerda ab8d232
speeddata upload and script for parallel experiments
SamFerwerda 764a26d
Revert "data for 16 cores"
SamFerwerda dc5a03f
added world to start_experiments
SamFerwerda 0c32d0a
Added rotation
SamFerwerda d4f774d
deleted print statement
SamFerwerda 1d1fe80
changed plugin
SamFerwerda 8533057
data
SamFerwerda d30684e
merge
SamFerwerda 40f3c54
fixed abnormal tabs.
SamFerwerda 55e6507
Deleted duplication
SamFerwerda ad6238b
data
SamFerwerda e98081b
Testing possible problems
SamFerwerda 5736985
merge
SamFerwerda df62dfe
merge
SamFerwerda 5d139c6
Increased waiting time
SamFerwerda 5909bf3
Adding other way to parallelize experiments
SamFerwerda d4dec49
speed data
SamFerwerda a4de28b
cleaned data file
SamFerwerda 1018892
Data of 16 and 8 cores
SamFerwerda f8451fb
Data 4 cores
SamFerwerda 750a2e3
Merge pull request #1 from ci-group/master
SamFerwerda File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,9 +2,10 @@ | |
# _____________________________________________________________________________ | ||
|
||
# CMake flag to help local projects find the build dir | ||
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build/lib") | ||
set(CMAKE_RARY_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build/lib") | ||
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build/lib") | ||
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build") | ||
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build/lib") | ||
|
||
include_directories(${CMAKE_SOURCE_DIR}/cpprevolve) | ||
|
||
|
@@ -57,6 +58,13 @@ pkg_check_modules(NLOpt REQUIRED nlopt>=2.4) | |
include_directories(${NLOpt_INCLUDE_DIRS}) | ||
link_directories(${NLOpt_LIBRARY_DIRS}) | ||
|
||
# Find SimpleAmqpClient | ||
find_library(SimpleAmqpClient REQUIRED libSimpleAmqpClient) | ||
include_directories(${SimpleAmqpClient_INCLUDE_DIRS}) | ||
|
||
# Find json | ||
find_package(jsoncpp CONFIG REQUIRED) | ||
|
||
# Find Limbo - LIbrary for Model-Based Optimization | ||
set(LIMBO_DIR ${CMAKE_SOURCE_DIR}/thirdparty/limbo) | ||
set(LIMBO_DEFINES USE_NLOPT) | ||
|
@@ -78,7 +86,7 @@ if (LOCAL_GAZEBO_DIR) | |
NO_DEFAULT_PATH) | ||
message(WARNING "Using local Gazebo @ ${gazebo_DIR}") | ||
else() | ||
find_package(gazebo 9 REQUIRED) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Keep 9 regular, but add the possibility to use gazebo 10 |
||
find_package(gazebo 10 REQUIRED) | ||
endif() | ||
include_directories(${GAZEBO_INCLUDE_DIRS}) | ||
link_directories(${GAZEBO_LIBRARY_DIRS}) | ||
|
@@ -205,6 +213,8 @@ target_link_libraries( | |
${GSL_LIBRARIES} | ||
${YAML_CPP_LIBRARIES} | ||
${NLOpt_LIBRARIES} | ||
jsoncpp_lib | ||
SimpleAmqpClient | ||
) | ||
|
||
target_compile_definitions( | ||
|
@@ -225,6 +235,20 @@ target_link_libraries( | |
${GAZEBO_LIBRARIES} | ||
) | ||
|
||
# Create Celery World plugin | ||
add_library( | ||
CeleryWorldControlPlugin SHARED | ||
plugin/CeleryWorldController.cpp | ||
plugin/register_celeryworld_plugin.cpp | ||
) | ||
|
||
target_link_libraries( | ||
CeleryWorldControlPlugin | ||
revolve-gazebo | ||
revolve-proto | ||
${GAZEBO_LIBRARIES} | ||
) | ||
|
||
# Create Analyzer plugin | ||
add_library( | ||
AnalyzerPlugin SHARED | ||
|
@@ -243,6 +267,7 @@ add_library(TorusWorldPlugin SHARED | |
plugin/TorusWorld.cpp | ||
plugin/register_torus_world_plugin.cpp | ||
) | ||
|
||
target_link_libraries(TorusWorldPlugin | ||
revolve-gazebo | ||
${GAZEBO_LIBRARIES} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you repeat this later then you can remove this or fix the mistake.