Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP Celery Code Impact #97

Draft
wants to merge 138 commits into
base: celery
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
138 commits
Select commit Hold shift + click to select a range
d24bd8d
added celery basics
SamFerwerda Dec 16, 2019
0fbcf7f
celery works
SamFerwerda Dec 17, 2019
d31d70d
changed readme file
SamFerwerda Dec 17, 2019
54e9152
Added a standard robot simulation to celery
SamFerwerda Dec 17, 2019
5a62a7f
Added functionality for robot queues and insertion
SamFerwerda Dec 29, 2019
66d3bec
CeleryController added
SamFerwerda Jan 3, 2020
aed0ac6
Added some extra dependencies, that are needed for celery + cpp
SamFerwerda Jan 27, 2020
4e6b037
added a simulator queue for celery and changed the manager
SamFerwerda Jan 29, 2020
ca8dc87
Evaluate_robot (task) now handles multiple robots at a time, by waiti…
SamFerwerda Jan 30, 2020
8f3f8f0
finished evaluate_robot and adjusted population class to fit celery
SamFerwerda Feb 1, 2020
e881ebd
Backup before major changes in tasks.py file
SamFerwerda Feb 1, 2020
ad3d620
removed evaluate_robot and made simulator_worker, added a put_in_queu…
SamFerwerda Feb 2, 2020
80e550a
Comments added to all functions.
SamFerwerda Feb 2, 2020
e223718
limited celery workers to one task at the time, added evaluate_robot …
SamFerwerda Feb 3, 2020
c23601c
Every celery worker will now also analyze the robot, and added speed …
SamFerwerda Feb 7, 2020
0d9968a
Implemented the basics of message consuming in C++.
SamFerwerda Feb 11, 2020
54bfa8c
Simple celery c++ task is set up
SamFerwerda Feb 16, 2020
e8388f5
globalized celery channel and consumer tag
SamFerwerda Feb 16, 2020
a8a7b6a
Bug was caused by printing the sdf string, also made some variables g…
SamFerwerda Feb 18, 2020
37915a1
c++ part is now handling actual robots, however just returning 1, Non…
SamFerwerda Feb 20, 2020
99cfdc0
Celery is now fully implemented on python and cpp side. However, curr…
SamFerwerda Feb 22, 2020
399e9ad
Some comments and error handling.
SamFerwerda Feb 22, 2020
cf71c19
Starting implementing contacts as result
SamFerwerda Mar 4, 2020
5a1498b
Basic locking system added and standard value of 10 added to every co…
SamFerwerda Mar 5, 2020
dc035f4
real contacts are send through celery know. However, contacts are cal…
SamFerwerda Mar 5, 2020
b82f052
celery messages are converting to robot_manager with the function upd…
SamFerwerda Mar 5, 2020
15a6da3
Adding a function to restart analyzer simulator. Currently in test ph…
SamFerwerda Mar 13, 2020
f1b1f15
Added a timeout on the get request of a celery message. Such that if …
SamFerwerda Mar 17, 2020
029a4ec
Added a way to reset celery and gazebo if a get request fails or 10 g…
SamFerwerda Mar 20, 2020
1ed7ec1
Changed the readme description
SamFerwerda Mar 20, 2020
0aa4489
Added changed files and problems with celery
SamFerwerda Mar 20, 2020
876ed5d
Old revolve is now accessible again with the normal world controller.…
SamFerwerda Mar 23, 2020
5ef39d7
Update manager_pop.py
SamFerwerda Mar 27, 2020
85f67d8
Make changes required for running on ripper machines
DaanZ Apr 21, 2020
7850549
celery_pool_asyncio no longer needs patch
SamFerwerda Apr 21, 2020
d2d6c23
Update manager.py
SamFerwerda Apr 21, 2020
606de95
Update celery.py
SamFerwerda Apr 21, 2020
10c65fc
Adding logging so we can find some errors or bugs
SamFerwerda Apr 21, 2020
4ecff5c
Commented pkill because not sure about effects on other ripper users.
SamFerwerda Apr 23, 2020
a69db25
This file isnt needed.
SamFerwerda Apr 23, 2020
2994824
Old comments and imports deleted.
SamFerwerda Apr 23, 2020
48e0ecc
Update readme.txt
SamFerwerda Apr 24, 2020
e77dd5e
Rename readme.txt to README.md
SamFerwerda Apr 24, 2020
5a84078
Updated the readme file for better reader experience.
SamFerwerda Apr 24, 2020
6b06ff9
Update README.md
SamFerwerda Apr 24, 2020
3d6e87f
Update README.md
SamFerwerda Apr 24, 2020
33065d4
Update README.md
SamFerwerda Apr 24, 2020
84e9c2b
Update README.md
SamFerwerda Apr 24, 2020
f40267d
Update README.md
SamFerwerda Apr 24, 2020
3a69a95
Update README.md
SamFerwerda Apr 24, 2020
7936037
Update README.md
SamFerwerda Apr 24, 2020
116ca43
Update README.md
SamFerwerda Apr 24, 2020
55b5fad
Update README.md
SamFerwerda Apr 24, 2020
a85758a
Update celery.py
SamFerwerda Apr 24, 2020
9ebd6d6
Update CMakeLists.txt
SamFerwerda Apr 24, 2020
a903b1b
Got fatal error that there is no such file json/json.h
SamFerwerda Apr 24, 2020
81de1d2
Last commit didnt fix issue
SamFerwerda Apr 24, 2020
923dae3
changed cmake file to use pkgconfig for simpleamqp, fixed jsoncpp
SamFerwerda Apr 28, 2020
cf4945d
Currently failing due to not finding libSimpleAmqpClient.so.2
SamFerwerda Apr 29, 2020
7352b4f
Added celery and celery pool asyncio to the python requirements.
DaanZ Apr 29, 2020
bd4205c
Changed Cmake to add library to build/lib folder.
SamFerwerda May 1, 2020
0a1f6cd
Merge branch 'ripper' of https://github.com/SamFerwerda/revolve into …
SamFerwerda May 1, 2020
dc7d807
Works on local machine, not yet on Ripper
SamFerwerda May 4, 2020
e925771
Changed pkg to find_library, works locally, testing on ripper
SamFerwerda May 4, 2020
82cf6be
Celery runs with logs
DaanZ May 5, 2020
5bf11f2
Added speedtesting requirements.
SamFerwerda May 6, 2020
2d200ab
added speedup checks to analyze speedup
SamFerwerda May 8, 2020
572b886
Added timing points to find bottleneck of revolve
SamFerwerda May 11, 2020
e6a49a3
Fixed wrong position of lists
SamFerwerda May 11, 2020
2c8a842
reversed location of list again
SamFerwerda May 11, 2020
f9b0f72
fixes
SamFerwerda May 11, 2020
5f08887
speed data
SamFerwerda May 11, 2020
bfca4c7
speed data
SamFerwerda May 11, 2020
4e25224
changed speeddata
SamFerwerda May 11, 2020
1a7c630
added data
SamFerwerda May 11, 2020
299dade
added data
SamFerwerda May 11, 2020
a5cd902
changes in manager_pop
SamFerwerda May 12, 2020
0a83f4a
Update manager for writing data.
SamFerwerda May 12, 2020
56c47cb
Commit the revolve log for data
SamFerwerda May 12, 2020
09214b0
Speed data
SamFerwerda May 14, 2020
30c82d2
Changed location of time measure.
SamFerwerda May 14, 2020
31947ba
data
SamFerwerda May 15, 2020
2031c0e
Data for 100 gen
SamFerwerda May 18, 2020
a832b50
Added extra data writing points.
SamFerwerda May 18, 2020
16cef67
some celery data
SamFerwerda May 21, 2020
c041889
export is now done parallel to evaluating
SamFerwerda May 22, 2020
2c4723e
Merge branch 'speedup' of https://github.com/SamFerwerda/revolve into…
SamFerwerda May 22, 2020
dc68210
possible fix to nonetype error
SamFerwerda May 25, 2020
27f6539
speed data
SamFerwerda May 26, 2020
4b5b2e9
Implemented speed checks to analyze revolve
SamFerwerda May 27, 2020
1273bb1
speeddata
SamFerwerda May 28, 2020
c829e4e
added start_experiment.py for quick parallel experiments
SamFerwerda May 28, 2020
93d0e3a
Changed layout of speedlogging
SamFerwerda May 28, 2020
4d2aa47
Speed data of 15 normal runs
SamFerwerda May 29, 2020
1b29d56
deleted revolve.log
SamFerwerda May 29, 2020
845ac27
changes in start_experiment and manager
SamFerwerda May 29, 2020
44e40ba
Merge branch 'speedup' of https://github.com/SamFerwerda/revolve into…
SamFerwerda May 29, 2020
bf4e329
Deleted the timeout of 120 seconds
SamFerwerda May 29, 2020
fc82bb5
small changes in manager file
SamFerwerda May 29, 2020
c92c4bb
speed data of all 30 old revolve
SamFerwerda May 30, 2020
a6166d8
delete data
SamFerwerda May 30, 2020
83aa70e
Queues are now unique for different experiments
SamFerwerda May 31, 2020
151caa2
fixed wrong implemententation in population.py
SamFerwerda May 31, 2020
db57b35
Test for multi experiments with celery
SamFerwerda Jun 1, 2020
0ab9832
Backup
SamFerwerda Jun 1, 2020
3d86b58
Unique queues for different experiments
SamFerwerda Jun 1, 2020
2b0450c
Merge branch 'master' into multiCelery
SamFerwerda Jun 1, 2020
82e514e
added script for consecutive runs
SamFerwerda Jun 1, 2020
ac94e40
extra data 32 cores
SamFerwerda Jun 3, 2020
1f5063f
removed worker pids
SamFerwerda Jun 3, 2020
e46fd3c
changes for consecutive runs
SamFerwerda Jun 3, 2020
b4e3364
data for 32 cores
SamFerwerda Jun 3, 2020
830f63e
Unnecesarry file
SamFerwerda Jun 8, 2020
2384e1e
Data for 16 cores
SamFerwerda Jun 8, 2020
0a806b1
data for 16 cores
SamFerwerda Jun 10, 2020
eb6bd87
Merge branch 'master' of https://github.com/SamFerwerda/revolve
SamFerwerda Jun 10, 2020
e9a1cd5
all data for 16 cores
SamFerwerda Jun 10, 2020
ab8d232
speeddata upload and script for parallel experiments
SamFerwerda Jun 10, 2020
764a26d
Revert "data for 16 cores"
SamFerwerda Jun 10, 2020
dc5a03f
added world to start_experiments
SamFerwerda Jun 12, 2020
0c32d0a
Added rotation
SamFerwerda Jun 12, 2020
d4f774d
deleted print statement
SamFerwerda Jun 12, 2020
1d1fe80
changed plugin
SamFerwerda Jun 13, 2020
8533057
data
SamFerwerda Jun 13, 2020
d30684e
merge
SamFerwerda Jun 13, 2020
40f3c54
fixed abnormal tabs.
SamFerwerda Jun 13, 2020
55e6507
Deleted duplication
SamFerwerda Jun 13, 2020
ad6238b
data
SamFerwerda Jun 15, 2020
e98081b
Testing possible problems
SamFerwerda Jun 15, 2020
5736985
merge
SamFerwerda Jun 15, 2020
df62dfe
merge
SamFerwerda Jun 15, 2020
5d139c6
Increased waiting time
SamFerwerda Jun 15, 2020
5909bf3
Adding other way to parallelize experiments
SamFerwerda Jun 15, 2020
d4dec49
speed data
SamFerwerda Jun 16, 2020
a4de28b
cleaned data file
SamFerwerda Jun 16, 2020
1018892
Data of 16 and 8 cores
SamFerwerda Jun 18, 2020
f8451fb
Data 4 cores
SamFerwerda Jun 19, 2020
750a2e3
Merge pull request #1 from ci-group/master
SamFerwerda Jun 22, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions cpprevolve/revolve/gazebo/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@
# _____________________________________________________________________________

# CMake flag to help local projects find the build dir
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build/lib")
set(CMAKE_RARY_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build/lib")
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you repeat this later then you can remove this or fix the mistake.

set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build/lib")
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build")
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/build/lib")

include_directories(${CMAKE_SOURCE_DIR}/cpprevolve)

Expand Down Expand Up @@ -57,6 +58,13 @@ pkg_check_modules(NLOpt REQUIRED nlopt>=2.4)
include_directories(${NLOpt_INCLUDE_DIRS})
link_directories(${NLOpt_LIBRARY_DIRS})

# Find SimpleAmqpClient
find_library(SimpleAmqpClient REQUIRED libSimpleAmqpClient)
include_directories(${SimpleAmqpClient_INCLUDE_DIRS})

# Find json
find_package(jsoncpp CONFIG REQUIRED)

# Find Limbo - LIbrary for Model-Based Optimization
set(LIMBO_DIR ${CMAKE_SOURCE_DIR}/thirdparty/limbo)
set(LIMBO_DEFINES USE_NLOPT)
Expand All @@ -78,7 +86,7 @@ if (LOCAL_GAZEBO_DIR)
NO_DEFAULT_PATH)
message(WARNING "Using local Gazebo @ ${gazebo_DIR}")
else()
find_package(gazebo 9 REQUIRED)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep 9 regular, but add the possibility to use gazebo 10

find_package(gazebo 10 REQUIRED)
endif()
include_directories(${GAZEBO_INCLUDE_DIRS})
link_directories(${GAZEBO_LIBRARY_DIRS})
Expand Down Expand Up @@ -205,6 +213,8 @@ target_link_libraries(
${GSL_LIBRARIES}
${YAML_CPP_LIBRARIES}
${NLOpt_LIBRARIES}
jsoncpp_lib
SimpleAmqpClient
)

target_compile_definitions(
Expand All @@ -225,6 +235,20 @@ target_link_libraries(
${GAZEBO_LIBRARIES}
)

# Create Celery World plugin
add_library(
CeleryWorldControlPlugin SHARED
plugin/CeleryWorldController.cpp
plugin/register_celeryworld_plugin.cpp
)

target_link_libraries(
CeleryWorldControlPlugin
revolve-gazebo
revolve-proto
${GAZEBO_LIBRARIES}
)

# Create Analyzer plugin
add_library(
AnalyzerPlugin SHARED
Expand All @@ -243,6 +267,7 @@ add_library(TorusWorldPlugin SHARED
plugin/TorusWorld.cpp
plugin/register_torus_world_plugin.cpp
)

target_link_libraries(TorusWorldPlugin
revolve-gazebo
${GAZEBO_LIBRARIES}
Expand Down
2 changes: 1 addition & 1 deletion cpprevolve/revolve/gazebo/plugin/BodyAnalyzer.h
Original file line number Diff line number Diff line change
Expand Up @@ -123,4 +123,4 @@ class BodyAnalyzer: public ::gazebo::WorldPlugin
}
}

#endif // REVOLVE_BODYANALYZER_H
#endif // REVOLVE_BODYANALYZER_H
Loading