Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sprint3 physfs #63

Open
wants to merge 68 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
e2df1ab
Integrate seperate thread loading
FloreauLuca Feb 24, 2020
b289363
Created a simple LogManager able to handle multiple type of messages …
CanasSimon Feb 24, 2020
061eb93
Added multithreading to the LogManager:
CanasSimon Feb 25, 2020
7e9a385
Added some prettier formatting for the date + time
CanasSimon Feb 26, 2020
01f784d
Removed a pthread line
CanasSimon Feb 26, 2020
c72f193
Fixed MSVC compiling errors
ClangPan Mar 1, 2020
74bcfcc
Create resource Manager
FloreauLuca Mar 2, 2020
ab4d09c
Updated the LogManager to better support multithreading:
ClangPan Mar 2, 2020
4d30818
Added logger executable
Mar 2, 2020
791320c
Merged last changes
Mar 2, 2020
cb7ce33
Added logger folder
Mar 2, 2020
bf0be8a
Added UI
Mar 3, 2020
fc54df7
Updated the logger interface / test program
ClangPan Mar 3, 2020
8e07b77
Added max log option in UI
Mar 3, 2020
5e41c5a
Changed the interface
Mar 3, 2020
c53c479
Code correction:
CanasSimon Mar 4, 2020
4cd3b01
Added a refresh button
Mar 5, 2020
1d37172
Merged develop into sprint2_logger and moved logger_program to comp_g…
Mar 5, 2020
f01a6e7
Correction code review
FloreauLuca Mar 5, 2020
61fb021
Fixed some copiler specific code
CanasSimon Mar 8, 2020
8194413
Fixed some compiler specific code
CanasSimon Mar 8, 2020
ef05836
Merge branch 'sprint2_logger' of https://github.com/EliasFarhan/NekoE…
CanasSimon Mar 8, 2020
5aca2a9
Merge branch 'sprint2_logger' into sprint3_physfs
CanasSimon Mar 9, 2020
447c1d7
Linked PhysFS library + added a test + created a physfs_utility
ClangPan Mar 10, 2020
fd4bb5e
Merge branch 'sprint2_resource' into sprint3_physfs
FloreauLuca Mar 10, 2020
dccea9a
Meta integration and test future
FloreauLuca Mar 10, 2020
ed0ca2b
Added several methods to physfs_utility
ClangPan Mar 11, 2020
8d627c6
Merge branch 'sprint3_physfs' of https://github.com/EliasFarhan/NekoE…
CanasSimon Mar 11, 2020
e7380ae
Added a wrapper for PhysFS
CanasSimon Mar 12, 2020
a459027
Custom Promise for loading
FloreauLuca Mar 17, 2020
1d4c5d4
Created jobsystem files
LoshkinOleg Feb 27, 2020
febb707
Made a working generic Job class.
LoshkinOleg Mar 1, 2020
643a4d5
Implementing workers
LoshkinOleg Mar 2, 2020
a47073f
Changed GrabJob() to be a static function
LoshkinOleg Mar 2, 2020
d3787bb
Got a basic job system working.
LoshkinOleg Mar 5, 2020
9c01f5b
Oleg: jobsystem split between .h/.cpp; Jobsystem TEST added; JobSyste…
LoshkinOleg Mar 6, 2020
7c2f3b1
Adding simple tests for custom allocators and correct prevPos on Stac…
EliasFarhan Mar 6, 2020
9fe98dc
Correct error with test_jobsystem by atomizing the counter and resize…
EliasFarhan Mar 6, 2020
57d55d7
Correct bug with Stack Allocator with NEKO_ASSERt disabled and correc…
EliasFarhan Mar 6, 2020
e817314
Correct renderer notifying the app thread, moving sample browser wind…
EliasFarhan Mar 7, 2020
9cd7096
Correct sync without the double while for resizing window
EliasFarhan Mar 7, 2020
a6f2218
Correct the render loop, move resize window to render thread without …
EliasFarhan Mar 8, 2020
ef3ae8c
Correct bug with multithread compilation with python script
EliasFarhan Mar 8, 2020
21b588d
Adding material generator from mtl and some functions for asset valid…
EliasFarhan Mar 8, 2020
cb8ae88
Making Neko compile again on WebGL2
EliasFarhan Mar 9, 2020
872b00e
Correctly MSVC error on custom allocator from keyword and
EliasFarhan Mar 9, 2020
6e9ca37
Starting to work on the Android port
EliasFarhan Mar 14, 2020
30102dc
Correct errors with file reading on Android
EliasFarhan Mar 15, 2020
52d1bdf
Making Adnroid port work and added NEKO_SAMETHREAD flag. Added Buffer…
EliasFarhan Mar 16, 2020
2821866
Correct SameTHread flag for Emscripten
EliasFarhan Mar 16, 2020
4398f8d
Merger corrections
ClangPan Mar 17, 2020
5365a22
Removed unecessary reference
ClangPan Mar 17, 2020
3fe16d1
Resolved linker bug
ClangPan Mar 17, 2020
693da97
Added some EXPECT to the tests
ClangPan Mar 18, 2020
77aed17
Small corrections
ClangPan Mar 19, 2020
7e980fe
Switch to BufferFile in ResourceManager
FloreauLuca Mar 26, 2020
cef9e29
NekoAssert integration
FloreauLuca Mar 30, 2020
ef2e3f7
EasyProfiler and unlock Loading
FloreauLuca Mar 30, 2020
c944b60
CMake corrections
ClangPan Apr 1, 2020
baf1e64
More profiling
FloreauLuca Apr 1, 2020
7718116
Fixed intrinsic functions (and incidentally made ReflectIntrinsics() …
ClangPan Apr 5, 2020
55d1fc0
Merge branch 'develop' into sprint3_physfs
ClangPan Apr 5, 2020
e652cb6
Optimization and profiling
FloreauLuca Apr 7, 2020
5e33042
Merge branch 'sprint3_physfs' of https://github.com/EliasFarhan/NekoE…
FloreauLuca Apr 7, 2020
cd0675c
Some test and benchmark modifications
FloreauLuca Apr 7, 2020
616b1eb
Corrected physfs_test
ClangPan Apr 27, 2020
753b5d5
Integration PhysFS in ResourceManager
FloreauLuca Apr 27, 2020
5bb229f
Check if Neko_PhysFS is defined
FloreauLuca Apr 27, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ set(Neko_Profile OFF CACHE BOOL "Activate Profiling with Easy Profile")
set(Neko_GLES3 ON CACHE BOOL "Activate OpenGL ES 3.0")
set(Neko_SDL2 ON CACHE BOOL "Activate SDL2")
set(Neko_Box2D OFF CACHE BOOL "Activate Box2D")
set(Neko_PhysFS ON CACHE BOOL "Activate PhysicsFS")
set(Neko_Test ON CACHE BOOL "Activate Test")
set(Neko_Assert ON CACHE BOOL "Activate Neko Assertion")
set(Neko_Benchmark ON CACHE BOOL "Activate Benchmark")
Expand All @@ -27,6 +28,8 @@ if ("${CMAKE_SYSTEM_NAME}" STREQUAL "Emscripten")
set(CMAKE_STATIC_LIBRARY_SUFFIX ".a")
set(Neko_SameThread ON CACHE BOOL "Used for Emscripten and other system when render and engine has to be in the same thread")
mark_as_advanced(Neko_SameThread)
set(Neko_PhysFS OFF CACHE BOOL "Disable PhysFS for Emscripten.")
mark_as_advanced(Neko_PhysFS)
endif()

if(UNIX)
Expand Down Expand Up @@ -59,6 +62,18 @@ if(Neko_Profile)
add_subdirectory(${EASY_PROFILE_DIR})
endif()

if(Neko_PhysFS)
#physfs
add_compile_definitions("NEKO_PHYSFS=1")
set(PHYSFS_VERSION "3.0.2" CACHE INTERNAL "")
set(PHYSFS_ROOT "${EXTERNAL_DIR}/physfs-${PHYSFS_VERSION}" CACHE INTERNAL "")
add_subdirectory("${PHYSFS_ROOT}")
include_directories(${PHYSFS_ROOT}/src)

add_subdirectory("common/physfs_wrapper")

endif()

include("cmake/data.cmake")
set(NEKO_CORE_DIR "${CMAKE_SOURCE_DIR}/core/" CACHE INTERNAL "")
add_subdirectory(${NEKO_CORE_DIR})
Expand Down Expand Up @@ -115,7 +130,7 @@ if(Neko_Test)
add_subdirectory("test/")
endif()

if(Neko_Benchmark)
if(Neko_Benchmark AND NOT Emscripten)
#Google benchmark
set(GOOGLE_BENCH_DIR "${EXTERNAL_DIR}/benchmark-1.5.0" CACHE STRING "")

Expand All @@ -126,6 +141,7 @@ if(Neko_Benchmark)

endif()


set(main_project_dir "main/")
file(GLOB main_projects "${main_project_dir}/*")

Expand Down
138 changes: 138 additions & 0 deletions benchmark/bench_resource.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
#include <iostream>
#include <random>
#include "random_fill.h"
#include <benchmark/benchmark.h>


#include "engine/assert.h"
#include "engine/engine.h"
#include "engine/resource.h"
#include "utilities/file_utility.h"

const int smallFromRange = 4;
const int smallToRange = 512;

const int bigFromRange = 2;
const int bigToRange = 64;


static void BM_SmallResourceThread(benchmark::State& state)
{
neko::Configuration config;
std::string path = config.dataRootPath + "data/test/smallFile.txt";
neko::BufferFile fileContent;
fileContent.Load(path);
neko::ResourceManager resourceManager;
resourceManager.Init();
const size_t size = state.range(0);
std::vector<neko::ResourceId> resourcesId;
std::map<neko::ResourceId, neko::BufferFile> resources;
for (int i = 0; i < size; i++)
{
resourcesId.push_back(resourceManager.LoadResource(path));
}
for (auto _ : state)
{
while (resources.size()<resourcesId.size()) {
for (int n = 0; n < size; n++)
{
if (resources.find(resourcesId[n]) != resources.end())
{
continue;
}
if (resourceManager.IsResourceReady(resourcesId[n]))
{
resources[resourcesId[n]] = resourceManager.GetResource(resourcesId[n]);
neko_assert(fileContent == resources[resourcesId[n]], "Error Loading");
}
}
}
resources.clear();
}
resourceManager.Destroy();
}

static void BM_BigResourceThread(benchmark::State& state)
{
neko::Configuration config;
std::string path = config.dataRootPath + "data/test/bigFile.json";
neko::BufferFile fileContent;
fileContent.Load(path);
neko::ResourceManager resourceManager;
resourceManager.Init();
const size_t size = state.range(0);
std::vector<neko::ResourceId> resourcesId;
std::map<neko::ResourceId, neko::BufferFile> resources;
for (int i = 0; i < size; i++)
{
resourcesId.push_back(resourceManager.LoadResource(path));
}
for (auto _ : state)
{
while (resources.size() < resourcesId.size()) {
for (int n = 0; n < size; n++)
{
if (resources.find(resourcesId[n]) != resources.end())
{
continue;
}
if (resourceManager.IsResourceReady(resourcesId[n]))
{
resources[resourcesId[n]] = resourceManager.GetResource(resourcesId[n]);
neko_assert(fileContent == resources[resourcesId[n]], "Error Loading");
}
}
}
resources.clear();

}
resourceManager.Destroy();
}

static void BM_SmallResourceNonThread(benchmark::State& state)
{
neko::Configuration config;
std::string path = config.dataRootPath + "data/test/smallFile.txt";
neko::BufferFile fileContent;
fileContent.Load(path);
const size_t size = state.range(0);
std::vector<neko::BufferFile> resources;
for (auto _ : state)
{
resources.clear();
for (int i = 0; i < size; i++)
{
neko::BufferFile loadedFile;
loadedFile.Load(path);
resources.push_back(loadedFile);
neko_assert(fileContent == resources.back(), "Error Loading");
}
}
}

static void BM_BigResourceNonThread(benchmark::State& state)
{
neko::Configuration config;
std::string path = config.dataRootPath + "data/test/bigFile.json";
neko::BufferFile fileContent;
fileContent.Load(path);
const size_t size = state.range(0);
std::vector<neko::BufferFile> resources;
for (auto _ : state)
{
resources.clear();
for (int i = 0; i < size; i++)
{
neko::BufferFile loadedFile;
loadedFile.Load(path);
resources.push_back(loadedFile);
neko_assert(fileContent == resources.back(), "Error Loading");
}
}
}

BENCHMARK(BM_SmallResourceThread)->Range(smallFromRange, smallToRange)->UseRealTime();
BENCHMARK(BM_BigResourceThread)->Range(bigFromRange, bigToRange)->UseRealTime();

BENCHMARK(BM_SmallResourceNonThread)->Range(smallFromRange, smallToRange)->UseRealTime();
BENCHMARK(BM_BigResourceNonThread)->Range(bigFromRange, bigToRange)->UseRealTime();
Loading