Release 0.12.0 (#600)

See CHANGELOG.rst for details!

Author: twslankard

CHANGELOG.rst

@@ -2,6 +2,84 @@
 Changelog
 =========
 
+[20240702] [0.12.0]
+===================
+
+**Important: ouster-sdk installed from pypi now requires glibc >= 2.28.**
+
+ouster_client/Python SDK
+------------------------
+
+* Add support for adding custom fields to ``LidarScan`` s with ``add_field`` and ``del_field``
+* Added per-request timeout arguments to ``SensorHttp``
+* Added sensor user_data to ``sensor_info/SensorInfo`` and metadata files
+* Removed ``updated_metadata_string()`` and ``original_string()`` from ``sensor_info``
+* Added ``to_json_string()`` to ``sensor_info`` to convert a ``sensor_info`` to a non-legacy
+  metadata JSON string
+* Unified Python and C++ ``Packet`` and ``PacketFormat`` classes
+* Added ``validate`` function to ``LidarPacket`` and ``ImuPacket`` to check for ID and size mismatches
+* [BREAKING] LidarScan's width and height have been switched to size_t from ptrdiff_t in C++
+* Refactor metadata parsing
+* Add ``get_version`` to ``sensor_info/SensorInfo`` to retrieve parsed version information
+* Add ``get_product_info`` to ``sensor_info/SensorInfo`` to retrieve parsed lidar model information
+* Raise an exception rather than throw an unrelated error when multiple viable metadata files are found for a given PCAP
+* Add ability to slice a scan source, returning a new sliced ScanSource
+
+* [BREAKING] Removed ``hostname`` in Python ``SensorInfo`` and ``name`` from C++ ``sensor_info``
+* [BREAKING] Removed ``udp_port_lidar``, ``udp_port_imu`` and ``mode`` from C++ ``sensor_info``
+* [BREAKING] Deprecated ``udp_port_lidar``, ``udp_port_imu`` and ``mode`` in Python ``SensorInfo``.
+  These fields now point to the equivalent fields inside of ``SensorInfo::config``.
+* [BREAKING] Removed ``cols`` and ``frequency`` from ``LidarMode`` in Python
+* [BREAKING] Deprecated ```data``` and ``capture_timestamp`` from Python ``Packet``
+* [BREAKING] Removed methods from Python ``ImuPacket`` and ``LidarPacket`` classes that simply wrapped ``PacketFormat``
+* [BREAKING] Removed ``begin()`` and ``end()`` iterators of ``LidarScan`` in C++
+* [BREAKING] Remove deprecated package stubs added in previous 0.11 release.
+* [BREAKING] Replaced integer backed ``ChanField`` enumerations with strings.
+* [BREAKING] Removed ``CUSTOM0`` through ``CUSTOM9`` ChanField enumerations.
+* [BREAKING] Extra fields in sensor metadata are now ignored and discarded if saved from the resulting ``sensor_info/SensorInfo``
+
+* [BUGFIX] Prevent last scan from being emitted twice for PCAP 
+* [BUGFIX] Fix corrupted packets due to poor handling of fragmented packet drop in PCAPs
+* [BUGFIX] Fix possible crash when working with custom UDPProfileLidars
+
+ouster_viz
+----------
+* Support viewing custom ``LidarScan`` fields in viz
+* Support viewing custom ``LidarScan`` 3 channel fields in viz as RGB
+
+* [BUGFIX] Prevent OpenBLAS from using high amounts of CPU spin waiting
+
+ouster_osf
+----------
+
+* Support saving custom ``LidarScan`` fields to OSF files
+
+* [BREAKING] OsfWriter now takes in an optional list of fields to save rather than a list of fields and ChanFieldTypes to cast to
+
+ouster-cli
+----------
+
+* Added support for slicing using time to ``ouster-cli source ... slice`` 
+* Add sensor ``ouster-cli source ... userdata`` command to set and retrieve userdata on a sensor
+* Add chainable ``ouster-cli source ... stats`` command
+* Add chainable ``ouster-cli source ... clip`` command to discard points outside a provided range
+* Add ``--rate max`` option to ``ouster-cli source ... viz```
+* Improve argument naming and descriptions for ``ouster-cli source ... viz`` map and accum options
+
+* [BUGFIX] Prevent dropped frames from live sensors by consuming scans as fast as they come in rather than sleeping
+
+mapping
+-------
+
+* Move mapping into the sdk as ``ouster.sdk.mapping``
+* Better handle looping while mapping
+* Improve automatic downsample voxel size calculation
+
+other
+-----
+
+* Updated VCPKG libraries to 2024.04.26
+
 [20240510] [0.11.1]
 ===================
 
@@ -15,6 +93,10 @@ Python SDK
 
 * Updated the ``open_source`` documentation.
 * Fixed an issue that caused the viz to redraw when the mouse cursor is moved.
+* [BREAKING] The python slice ``[::]`` operator now returns a ``MultiScanSource`` / ``ScanSource``
+  instead of a ``List``. Thus, invoking a ``scan_source[x:x+n]`` yields a fully functional scan source
+  that is scoped to the range ``[x, x+n]``.
+* [BREAKING] The python slice ``[::]`` operator no longer support negative step
 
 ouster_client
 -------------

CMakeLists.txt

@@ -16,7 +16,7 @@ include(VcpkgEnv)
 project(ouster_example VERSION 20231031)
 
 # generate version header
-set(OusterSDK_VERSION_STRING 0.11.1)
+set(OusterSDK_VERSION_STRING 0.12.0)
 include(VersionGen)
 
 # ==== Options ====

cmake/FindCURL.cmake

@@ -1,4 +1,4 @@
-# Define forwards-compatible imported target for old platforms (Ubuntu 18.04)
+# Define forwards-compatible imported target for old platforms
 # Note: curl from VCPKG appears to completely ignore curl find modules despite
 # CMAKE_MODULE_PATH settings
 

cmake/Findglfw3.cmake

@@ -5,7 +5,7 @@ include(FindPackageHandleStandardArgs)
 
 find_package(glfw3 CONFIG REQUIRED)
 
-# Package on >=18.04 sets a target
+# Package on >=20.04 sets a target
 if(TARGET glfw)
   get_target_property(GLFW3_LIBRARIES glfw LOCATION)
   get_target_property(GLFW3_INCLUDE_DIRS glfw INTERFACE_INCLUDE_DIRECTORIES)

conanfile.py

@@ -1,12 +1,14 @@
 import os
 import re
-from conans import ConanFile, CMake, tools
+from conan import ConanFile
+from conan.tools.cmake import CMakeToolchain, CMake, cmake_layout, CMakeDeps
+from conan.tools.files import collect_libs, load
+from conan.tools.scm import Git
 
-from pprint import pformat
 
-
-class OusterSDKConan(ConanFile):
+class ousterSdkRecipe(ConanFile):
     name = "ouster_sdk"
+    package_type = "library"
     license = "BSD 3-Clause License"
     author = "Ouster, Inc."
     url = "https://github.com/ouster-lidar/ouster_example"
@@ -33,7 +35,6 @@ class OusterSDKConan(ConanFile):
         "eigen_max_align_bytes": False,
     }
 
-    generators = "cmake_paths", "cmake_find_package"
     exports_sources = [
         "cmake/*",
         "conan/*",
@@ -49,27 +50,34 @@ class OusterSDKConan(ConanFile):
         "README.rst"
     ]
 
-    # https://docs.conan.io/en/1.51/howtos/capture_version.html#how-to-capture-package-version-from-text-or-build-files
+    # https://docs.conan.io/2/reference/conanfile/methods/set_version.html
     def set_version(self):
-        content = tools.load(os.path.join(self.recipe_folder, "CMakeLists.txt"))
-        version = re.search(r"set\(OusterSDK_VERSION_STRING (.*)\)", content).group(1)
+        content = load(self, os.path.join(self.recipe_folder, "CMakeLists.txt"))
+        version = re.search("set\(OusterSDK_VERSION_STRING (.*)\)", content).group(1)
         self.version = version.strip()
 
     def config_options(self):
         if self.settings.os == "Windows":
-            del self.options.fPIC
+            self.options.rm_safe("fPIC")
+
+    def configure(self):
+        if self.options.shared:
+            self.options.rm_safe("fPIC")
 
     def requirements(self):
-         # not required directly here but because boost and openssl pulling
-         # slightly different versions of zlib we need to set it
-         # here explicitly
+        # not required directly here but because boost and openssl pulling
+        # slightly different versions of zlib we need to set it
+        # here explicitly
         self.requires("zlib/1.3")
 
-        self.requires("eigen/3.4.0")
+        # Since Eigen is a header only library, and the SDK includes Eigen
+        # headers in its headers, we must set transitive_headers=True so that
+        # packages consuming the SDK will also have access to the Eigen headers.
+        self.requires("eigen/3.4.0", transitive_headers=True)
         self.requires("jsoncpp/1.9.5")
-        self.requires("spdlog/1.11.0")
-        self.requires("fmt/9.1.0")
-        self.requires("libcurl/7.84.0")
+        self.requires("spdlog/1.12.0")
+        self.requires("fmt/9.1.0", override=True)
+        self.requires("libcurl/7.86.0")
 
         if self.options.build_pcap:
             self.requires("libtins/4.3")
@@ -86,41 +94,45 @@ def requirements(self):
             # maybe needed for cpp examples, but not for the lib
             # self.requires("tclap/1.2.4")
 
-    def configure_cmake(self):
-        cmake = CMake(self)
-        cmake.definitions["BUILD_VIZ"] = self.options.build_viz
-        cmake.definitions["BUILD_PCAP"] = self.options.build_pcap
-        cmake.definitions["BUILD_OSF"] = self.options.build_osf
-        cmake.definitions["OUSTER_USE_EIGEN_MAX_ALIGN_BYTES_32"] = self.options.eigen_max_align_bytes
-        # alt way, but we use CMAKE_TOOLCHAIN_FILE in other pipeline so avoid overwrite
-        # cmake.definitions["CMAKE_TOOLCHAIN_FILE"] = os.path.join(self.build_folder, "conan_paths.cmake")
-        cmake.definitions[
-            "CMAKE_PROJECT_ouster_example_INCLUDE"] = os.path.join(
-                self.build_folder, "conan_paths.cmake")
-        cmake.definitions["BUILD_SHARED_LIBS"] = True if self.options.shared else False
-        cmake.definitions["CMAKE_POSITION_INDEPENDENT_CODE"] = (
+    def build_requirements(self):
+        if self.options.build_osf:
+            self.build_requires("flatbuffers/<host_version>")
+
+    def layout(self):
+        cmake_layout(self)
+
+    def generate(self):
+        deps = CMakeDeps(self)
+        deps.generate()
+        tc = CMakeToolchain(self)
+        tc.variables["BUILD_VIZ"] = self.options.build_viz
+        tc.variables["BUILD_PCAP"] = self.options.build_pcap
+        tc.variables["BUILD_OSF"] = self.options.build_osf
+        tc.variables[
+            "OUSTER_USE_EIGEN_MAX_ALIGN_BYTES_32"
+        ] = self.options.eigen_max_align_bytes
+        tc.variables["BUILD_SHARED_LIBS"] = True if self.options.shared else False
+        tc.variables["CMAKE_POSITION_INDEPENDENT_CODE"] = (
             True if "fPIC" in self.options and self.options.fPIC else False
         )
 
         # we use this option until we remove nonstd::optional from SDK codebase (soon)
         if self.options.ensure_cpp17:
-            cmake.definitions["CMAKE_CXX_STANDARD"] = 17
+            tc.variables["CMAKE_CXX_STANDARD"] = 17
 
-        self.output.info("Cmake definitions: ")
-        self.output.info(pformat(cmake.definitions))
-        cmake.configure()
-        return cmake
+        tc.generate()
 
     def build(self):
-        cmake = self.configure_cmake()
+        cmake = CMake(self)
+        cmake.configure()
         cmake.build()
 
     def package(self):
-        cmake = self.configure_cmake()
+        cmake = CMake(self)
         cmake.install()
 
     def package_info(self):
-        self.cpp_info.libs = tools.collect_libs(self)
+        self.cpp_info.libs = collect_libs(self)
         self.cpp_info.includedirs = [
             "include",
             "include/optional-lite"
@@ -129,8 +141,14 @@ def package_info(self):
             "lib/cmake/OusterSDK/OusterSDKConfig.cmake"
         )
 
+        self.cpp_info.set_property(
+            "cmake_build_modules",
+            [os.path.join("lib", "cmake", "OusterSDK", "OusterSDKConfig.cmake")],
+        )
         self.cpp_info.set_property("cmake_file_name", "OusterSDK")
+        self.cpp_info.set_property("cmake_target_name", "OusterSDK")
 
+        # TODO: to remove in conan v2 once cmake_find_package* generators removed
         self.cpp_info.filenames["cmake_find_package"] = "OusterSDK"
         self.cpp_info.filenames["cmake_find_package_multi"] = "OusterSDK"
         self.cpp_info.names["cmake_find_package"] = "OusterSDK"

docs/Doxyfile

@@ -934,7 +934,9 @@ RECURSIVE              = YES
 # Note that relative paths are relative to the directory from which doxygen is
 # run.
 
-EXCLUDE                = ../ouster_client/include/optional-lite
+EXCLUDE                = ../ouster_client/include/optional-lite \
+                         ../ouster_pcap/src/ip_reassembler.h \
+                         ../ouster_pcap/src/ip_reassembler.cpp
 
 # The EXCLUDE_SYMLINKS tag can be used to select whether or not files or
 # directories that are symbolic links (a Unix file system feature) are excluded

docs/cli/clip-sessions.rst

@@ -0,0 +1,74 @@
+Clip PointCloud in the Ouster-CLI
+=================================
+
+.. _ouster-cli-clip:
+
+
+Clip Command
+------------
+
+The clip command clips the lidar scan by the given range and streams down the modified lidar
+scan to the subsequent commands (like slam, viz, save, etc.). The position of the clip command
+in the ouster-cli command chain makes a difference as it only affects the commands that follow it.
+
+To explore the parameters you can use with the clip command, you can use the --help flag:
+
+.. code:: bash
+
+        ouster-cli source <SENSOR_HOSTNAME> / <FILENAME> clip --help
+
+
+Example Usage
+-------------
+
+To keep the points within the 20 m to 50 m range and save the modified lidar scan into a PCAP file
+with the visualizer on, run the following command:
+
+.. code:: bash
+
+        ouster-cli source <SENSOR_HOSTNAME> / <FILENAME> clip --min-range 20 --max-range 50 viz save clipped.pcap
+
+
+Remember, the clip command only affects the commands after it. In the following example, the
+viz command runs before the clip command, which means the point cloud modification won't be reflected
+in the visualizer but will affect the subsequent save command and the saved PCAP file:
+
+.. code:: bash
+
+        ouster-cli source <SENSOR_HOSTNAME> / <FILENAME> viz clip --min-range 20 --max-range 50 save clipped_2.pcap
+
+In addition to the ``min-range`` and ``max-range`` parameters, the ``clip`` command also includes the
+``percent-range`` parameter. This parameter discards points with ranges greater than a specified percentile
+in each lidar scan, helping to filter out noise points.
+
+.. code:: bash
+
+        ouster-cli source <SENSOR_HOSTNAME> / <FILENAME> clip --percent-range 99 viz
+
+Combined with SLAM Command
+--------------------------
+
+The ``slam`` command also has ``min-range`` and ``max-range`` parameters. When the clip command is
+used after the ``slam`` command, the ``clip`` command will, by default, use the range settings specified
+in the slam command. However, you can explicitly pass in the range settings to the ``clip`` command to 
+apply different ranges to the clip operation.
+
+Note that the range settings in the ``slam`` command only affect the point cloud within the SLAM algorithm.
+The slam range settings will not modify the lidarscan and will not affect the other following commands.
+
+
+Example Usage
+-------------
+
+Experiment with the following commands using a pre-recorded PCAP or OSF file:
+
+.. code:: bash
+
+        ouster-cli source <FILENAME> slam clip --min-range 20 --max-range 50 viz save clipped_3.ply
+        ouster-cli source <FILENAME> slam --min-range 10 --max-range 100 clip --min-range 20 --max-range 50 viz save clipped_4.ply
+
+You can view the output PLY files using the open source software `CloudCompare`_
+For more details about the slam command, refer to the :ref:`SLAM Command <ouster-cli-mapping>`
+
+
+.. _CloudCompare: https://www.cloudcompare.org/