Performance limits for continuous streaming of data using typed arrays (JS)

[atmgrifter00]

We're currently investigating the possibility of leveraging Perspective to serve as the host for both static and live sets of data. The live data could have a form of something akin to having 2 typed arrays (an Int32 index buffer, and a Float32 sample buffer) being packaged into an Apache Arrow table. Ultimately Perspective is being given an ArrayBuffer.

My initial findings is that, in a variety of different live data sample sizes and update rates, that Perspective seems to be able to process roughly 640K-680K rows per second (each row has two values). This is through using table.replace(...) to update the table for each update, and basically all of this time is being captured inside the WASM boundary (within Perspective).

Does anyone know if Perspective should be able to handle higher update rates, or am I hitting a hard boundary? Our goal is to handle 1M samples per second across 4 traces at roughly 30 fps.

Note: I'll put together a StackBlitz example in a bit to help demonstrate what I'm doing, and link it here.

[texodus]

I'm coincidentally working on some benchmarks for this now. A few questions:

• What version of Perspective?
• What platform and data model (I presume node.js or browser 'cuz TypedArray)?
• How big are update batches?

[atmgrifter00]

Just an update. I've been trying to move over to a Rust server application that uses perspective 3.0.1, but when I run cargo build, I get the following error:

     libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$new_proto_server referenced in function _ZN18perspective_server3ffi12ffi_internal16new_proto_server17h0bbececcb629f32dE
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$new_session referenced in function _ZN18perspective_server3ffi12ffi_internal11new_session17hd23df467325006feE
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$close_session referenced in function _ZN18perspective_server3ffi12ffi_internal13close_session17h877f18c3b0162c1dE
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$handle_request referenced in function _ZN18perspective_server3ffi12ffi_internal14handle_request17hf7e477435dcea019E
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$poll referenced in function _ZN18perspective_server3ffi12ffi_internal4poll17he69ada2c67c50ffcE
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$unique_ptr$ProtoApiServer$null referenced in function _ZN106_$LT$perspective_server..ffi..ffi_internal..ProtoApiServer$u20$as$u20$cxx..unique_ptr..UniquePtrTarget$GT$6__null17h699c0b6a2e526985E
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$unique_ptr$ProtoApiServer$raw referenced in function _ZN106_$LT$perspective_server..ffi..ffi_internal..ProtoApiServer$u20$as$u20$cxx..unique_ptr..UniquePtrTarget$GT$5__raw17h1b60b13275974faaE
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$unique_ptr$ProtoApiServer$get referenced in function _ZN106_$LT$perspective_server..ffi..ffi_internal..ProtoApiServer$u20$as$u20$cxx..unique_ptr..UniquePtrTarget$GT$5__get17hb616f3f0ed8f3a69E
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$unique_ptr$ProtoApiServer$release referenced in function _ZN106_$LT$perspective_server..ffi..ffi_internal..ProtoApiServer$u20$as$u20$cxx..unique_ptr..UniquePtrTarget$GT$9__release17h9869ce4b29a8df06E
      libperspective_server-9cd620a81dc6b854.rlib(perspective_server-9cd620a81dc6b854.perspective_server.8ded57c838308bd0-cgu.0.rcgu.o) : error LNK2019: unresolved external symbol cxxbridge1$unique_ptr$ProtoApiServer$drop referenced in function _ZN106_$LT$perspective_server..ffi..ffi_internal..ProtoApiServer$u20$as$u20$cxx..unique_ptr..UniquePtrTarget$GT$6__drop17hba57192c8c66247aE

Thoughts on what I can do to resolve this? I copied the dependencies from the current rust-axum example in this repo.

My setup is as follows:

• Windows 10
• rustc 1.80.1 (3f5fd8dd4 2024-08-06)
• have the VS dependencies installed as specified in https://rust-lang.github.io/rustup/installation/windows-msvc.html#installing-only-the-required-components-optional

Cargo.toml:
[package]
name = "rust-server"
version = "0.1.0"
edition = "2021"

[dependencies]
perspective = "3.0.1"
axum = { version = ">=0.7,<2", features = ["ws"] }
futures = "0.3"
tokio = { version = "1.0", features = ["full"] }
tower-http = { version = "0.5.0", features = ["fs", "trace"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
arrow = "52.2.0"

[texodus]

Please try this with v3.0.3.

[atmgrifter00]

Just an update after trying with v3.0.0. So I've gotten...further. I've done two things beyond just using v3.0.3 to get where I'm at. I've made an explicit install of boost (v1.82.0), as the developer docs suggest is necessary, and I've also made a change to the perspective CMakeLists.txt file to remove the systems COMPONENTS part located here. This seems like a function of the CMake version I am using (v3.30.3). This seems to have gotten me past the issue of building/linking boost, but now I seem to be hitting issues with protobuf and potentially more specifically its third-party dependency abseil.

The first build error output is the following:

C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(382,5): error MSB3491: Could not write lines to file "flags_commandlineflag_internal.dir\Debug\flags_co.23275C60.tlog\flags_commandlineflag_internal.lastbuildstate". Path: flags_commandlineflag_internal.dir\Debug\flags_co.23275C60.tlog\flags_commandlineflag_internal.lastbuildstate exceeds the OS max path limit. The fully qualified file name must be less than 260 characters. [C:\Dev\github\mraj-perspective-perf\rust-server\target\debug\build\perspective-server-27372263ecb22b36\out\build\protobuf-build\third_party\abseil-cpp\absl\flags\flags_commandlineflag_internal.vcxproj]

It's possible this first error is related to the next error I hit:

C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(1599,5): error MSB6003: The specified task executable "Lib.exe" could not be run. System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Dev\github\mraj-perspective-perf\rust-server\target\debug\build\perspective-server-27372263ecb22b36\out\build\protobuf-build\third_party\abseil-cpp\absl\profiling\exponential_biased.dir\Debug\exponent.82F60CFB.tlog'. [C:\Dev\github\mraj-perspective-perf\rust-server\target\debug\build\perspective-server-27372263ecb22b36\out\build\protobuf-build\third_party\abseil-cpp\absl\profiling\exponential_biased.vcxproj]

[atmgrifter00]

Well those errors were solved by updating my registry to support long paths (🤦‍♂️), however I'm now hitting new errors that I may follow up with if I can't figure it out.

Update:
It's now linker errors. The first interesting information seems to be the following:

= note: libperspective_server-b775bcedcd7958db.rlib(utils.obj) : error LNK2005: "bool __cdecl perspective::parse_all_date_time(struct tm &,class std::basic_string_view<char,struct std::char_traits >)" (?parse_all_date_time@perspective@@YA_NAEAUtm@@v?$basic_string_view@DU?$char_traits@D@std@@@std@@@z) already defined in libperspective_server-b775bcedcd7958db.rlib(server.obj)
Creating library C:\Dev\github\mraj-perspective-perf\rust-server\target\debug\deps\perspective-7494f58b18033605.dll.lib and object C:\Dev\github\mraj-perspective-perf\rust-server\target\debug\deps\perspective-7494f58b18033605.dll.exp
LINK : warning LNK4098: defaultlib 'LIBCMTD' conflicts with use of other libs; use /NODEFAULTLIB:library
libcpmtd.lib(wlocale.obj) : error LNK2001: unresolved external symbol _invalid_parameter

[atmgrifter00]

Well, I've tried all I could think of, but still can't get an install of perspective to work in Rust. I'm still hitting linker issues after the following changes to CMakeLists.txt:

• line 330: find_package(Boost 1.82.0) (At the very least the build process seems to succeed at this point as we can make it to the linking stage)
• lines 58 & 587: changed MT to MTd (I was hoping that the compile was in debug mode and linking with the MT flag was causing the linker issues, but that doesn't seem to be the case).

Interestingly, the linker issue I noted above isn't what I'm seeing now (which is great because that one didn't make a lot of sense). Now I have the following as the first error:

  = note:    Creating library C:\Dev\rust-server\target\debug\deps\perspective-7494f58b18033605.dll.lib and object C:\Dev\rust-server\target\debug\deps\perspective-7494f58b18033605.dll.exp
          LINK : warning LNK4098: defaultlib 'LIBCMTD' conflicts with use of other libs; use /NODEFAULTLIB:library
          libcpmtd.lib(locale.obj) : error LNK2019: unresolved external symbol _invalid_parameter referenced in function "void __cdecl std::_Verify_range<char>(char const * const,char const * const)" (??$_Verify_range@D@std@@YAXQEBD0@Z)

I'm still interested in seeing if perspective can hit the performance goal for us to integrate it. Seems like maybe the Python version on Windows might be a little easier to experiment with since it has precompiled binaries so I'll give that a shot.
If the perf seems reasonable I'd still be interested in the Rust builds longer term for interop and maximum performance with our Windows App. So would still appreciate help on the Windows build with the Rust crates.