Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocess bitcoin #7

Open
wants to merge 23 commits into
base: bitcoin-fresheyes-master-10102
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
0be3913
serialization: Support for multiple parameters
ryanofsky Nov 22, 2023
422bd2d
test: add ipc test to test multiprocess type conversion code
ryanofsky Nov 20, 2023
de13c17
serialization: Drop unnecessary ParamsStream references
ryanofsky Nov 22, 2023
c9dad8d
multiprocess: Add type conversion code for serializable types
ryanofsky Nov 20, 2023
9e9dfce
serialization: Reverse ParamsStream constructor order
ryanofsky Nov 22, 2023
afce182
multiprocess: Add type conversion code for UniValue types
ryanofsky Nov 20, 2023
ffc1e9f
serialization: Accept multiple parameters in ParamsStream constructor
ryanofsky Nov 22, 2023
ca82b45
Merge remote-tracking branch 'origin/pull/28921/head'
ryanofsky Jan 11, 2024
d27c337
Merge remote-tracking branch 'origin/pull/28929/head'
ryanofsky Jan 11, 2024
ee4b913
doc: multiprocess documentation improvements
ryanofsky Jan 11, 2024
df27e30
Increase feature_block.py and feature_taproot.py timeouts
ryanofsky Dec 5, 2017
cbc5e36
Add util::Result workaround to be compatible with libmultiprocess
ryanofsky Aug 24, 2022
3427e7e
Add capnp serialization code for bitcoin types
ryanofsky Dec 5, 2017
b6516c4
Add capnp wrapper for Handler interface
ryanofsky Dec 5, 2017
5cf30e5
Add capnp wrapper for Chain interface
ryanofsky Dec 5, 2017
b5fd425
Add capnp wrapper for Wallet interface
ryanofsky Nov 28, 2023
329b34c
Add capnp wrapper for Node interface
ryanofsky Dec 5, 2017
dde2a22
Make bitcoin-gui spawn a bitcoin-node process
ryanofsky Dec 5, 2017
f3349a2
Make bitcoin-node spawn a bitcoin-wallet process
ryanofsky Dec 5, 2017
6ea638c
multiprocess: Add debug.log .wallet/.gui suffixes
ryanofsky Aug 30, 2021
622acd0
doc: Multiprocess misc doc and comment updates
ryanofsky Dec 5, 2017
755ca62
combine_logs: Handle multiprocess wallet log files
ryanofsky Dec 5, 2017
56ef459
Update libmultiprocess library
ryanofsky Jan 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
serialization: Support for multiple parameters
This commit makes a minimal change to the ParamsStream class to let it retrieve
multiple parameters. Followup commits after this commit clean up code using
ParamsStream and make it easier to set multiple parameters.

Currently it is only possible to attach one serialization parameter to a stream
at a time. For example, it is not possible to set a parameter controlling the
transaction format and a parameter controlling the address format at the same
time because one parameter will override the other.

This limitation is inconvenient for multiprocess code since it is not possible
to create just one type of stream and serialize any object to it. Instead it is
necessary to create different streams for different object types, which
requires extra boilerplate and makes using the new parameter fields a lot more
awkward than the older version and type fields.

Fix this problem by allowing an unlimited number of serialization stream
parameters to be set, and allowing them to be requested by type. Later
parameters will still override earlier parameters, but only if they have the
same type.

This change requires replacing the stream.GetParams() method with a
stream.GetParams<T>() method in order for serialization code to retreive the
desired parameters. This change is more verbose, but probably a good thing for
readability because previously it could be difficult to know what type the
GetParams() method would return, and now it is more obvious.
  • Loading branch information
ryanofsky committed Jan 11, 2024
commit 0be39139f2d866ffd088476c4a0e575f19aa64dc
4 changes: 2 additions & 2 deletions src/netaddress.h
Original file line number Diff line number Diff line change
@@ -241,7 +241,7 @@ class CNetAddr
template <typename Stream>
void Serialize(Stream& s) const
{
if (s.GetParams().enc == Encoding::V2) {
if (s.template GetParams<SerParams>().enc == Encoding::V2) {
SerializeV2Stream(s);
} else {
SerializeV1Stream(s);
@@ -254,7 +254,7 @@ class CNetAddr
template <typename Stream>
void Unserialize(Stream& s)
{
if (s.GetParams().enc == Encoding::V2) {
if (s.template GetParams<SerParams>().enc == Encoding::V2) {
UnserializeV2Stream(s);
} else {
UnserializeV1Stream(s);
6 changes: 3 additions & 3 deletions src/primitives/transaction.h
Original file line number Diff line number Diff line change
@@ -326,7 +326,7 @@ class CTransaction

template <typename Stream>
inline void Serialize(Stream& s) const {
SerializeTransaction(*this, s, s.GetParams());
SerializeTransaction(*this, s, s.template GetParams<TransactionSerParams>());
}

/** This deserializing constructor is provided instead of an Unserialize method.
@@ -386,12 +386,12 @@ struct CMutableTransaction

template <typename Stream>
inline void Serialize(Stream& s) const {
SerializeTransaction(*this, s, s.GetParams());
SerializeTransaction(*this, s, s.template GetParams<TransactionSerParams>());
}

template <typename Stream>
inline void Unserialize(Stream& s) {
UnserializeTransaction(*this, s, s.GetParams());
UnserializeTransaction(*this, s, s.template GetParams<TransactionSerParams>());
}

template <typename Stream>
3 changes: 2 additions & 1 deletion src/protocol.h
Original file line number Diff line number Diff line change
@@ -406,9 +406,10 @@ class CAddress : public CService
static constexpr SerParams V1_DISK{{CNetAddr::Encoding::V1}, Format::Disk};
static constexpr SerParams V2_DISK{{CNetAddr::Encoding::V2}, Format::Disk};

SERIALIZE_METHODS_PARAMS(CAddress, obj, SerParams, params)
SERIALIZE_METHODS(CAddress, obj)
{
bool use_v2;
auto& params = SER_PARAMS(SerParams);
if (params.fmt == Format::Disk) {
// In the disk serialization format, the encoding (v1 or v2) is determined by a flag version
// that's part of the serialization itself. ADDRV2_FORMAT in the stream version only determines
37 changes: 15 additions & 22 deletions src/serialize.h
Original file line number Diff line number Diff line change
@@ -181,17 +181,17 @@ const Out& AsBase(const In& x)
static void SerializationOps(Type& obj, Stream& s, Operation ser_action)

/**
* Variant of FORMATTER_METHODS that supports a declared parameter type.
*
* If a formatter has a declared parameter type, it must be invoked directly or
* Formatter methods can retrieve parameters attached to a stream using the
* SER_PARAMS(type) macro as long as the stream is created directly or
* indirectly with a parameter of that type. This permits making serialization
* depend on run-time context in a type-safe way.
*
* Example use:
* struct BarParameter { bool fancy; ... };
* struct Bar { ... };
* struct FooFormatter {
* FORMATTER_METHODS(Bar, obj, BarParameter, param) {
* FORMATTER_METHODS(Bar, obj) {
* auto& param = SER_PARAMS(BarParameter);
* if (param.fancy) {
* READWRITE(VARINT(obj.value));
* } else {
@@ -213,13 +213,7 @@ const Out& AsBase(const In& x)
* Compilation will fail in any context where serialization is invoked but
* no parameter of a type convertible to BarParameter is provided.
*/
#define FORMATTER_METHODS_PARAMS(cls, obj, paramcls, paramobj) \
template <typename Stream> \
static void Ser(Stream& s, const cls& obj) { SerializationOps(obj, s, ActionSerialize{}, s.GetParams()); } \
template <typename Stream> \
static void Unser(Stream& s, cls& obj) { SerializationOps(obj, s, ActionUnserialize{}, s.GetParams()); } \
template <typename Stream, typename Type, typename Operation> \
static void SerializationOps(Type& obj, Stream& s, Operation ser_action, const paramcls& paramobj)
#define SER_PARAMS(type) (s.template GetParams<type>())

#define BASE_SERIALIZE_METHODS(cls) \
template <typename Stream> \
@@ -246,15 +240,6 @@ const Out& AsBase(const In& x)
BASE_SERIALIZE_METHODS(cls) \
FORMATTER_METHODS(cls, obj)

/**
* Variant of SERIALIZE_METHODS that supports a declared parameter type.
*
* See FORMATTER_METHODS_PARAMS for more information on parameters.
*/
#define SERIALIZE_METHODS_PARAMS(cls, obj, paramcls, paramobj) \
BASE_SERIALIZE_METHODS(cls) \
FORMATTER_METHODS_PARAMS(cls, obj, paramcls, paramobj)

// Templates for serializing to anything that looks like a stream,
// i.e. anything that supports .read(Span<std::byte>) and .write(Span<const std::byte>)
//
@@ -1134,7 +1119,15 @@ class ParamsStream
void ignore(size_t num) { m_substream.ignore(num); }
bool eof() const { return m_substream.eof(); }
size_t size() const { return m_substream.size(); }
const Params& GetParams() const { return m_params; }
template<typename P>
const auto& GetParams() const
{
if constexpr (std::is_convertible_v<Params, P>) {
return m_params;
} else {
return m_substream.template GetParams<P>();
}
}
int GetVersion() = delete; // Deprecated with Params usage
int GetType() = delete; // Deprecated with Params usage
};
@@ -1176,7 +1169,7 @@ class ParamsWrapper
/** \
* Return a wrapper around t that (de)serializes it with specified parameter params. \
* \
* See FORMATTER_METHODS_PARAMS for more information on serialization parameters. \
* See SER_PARAMS for more information on serialization parameters. \
*/ \
template <typename T> \
auto operator()(T&& t) const \
7 changes: 4 additions & 3 deletions src/test/serialize_tests.cpp
Original file line number Diff line number Diff line change
@@ -289,7 +289,7 @@ class Base
template <typename Stream>
void Serialize(Stream& s) const
{
if (s.GetParams().m_base_format == BaseFormat::RAW) {
if (s.template GetParams<BaseFormat>().m_base_format == BaseFormat::RAW) {
s << m_base_data;
} else {
s << Span{HexStr(Span{&m_base_data, 1})};
@@ -299,7 +299,7 @@ class Base
template <typename Stream>
void Unserialize(Stream& s)
{
if (s.GetParams().m_base_format == BaseFormat::RAW) {
if (s.template GetParams<BaseFormat>().m_base_format == BaseFormat::RAW) {
s >> m_base_data;
} else {
std::string hex{"aa"};
@@ -327,8 +327,9 @@ class Derived : public Base
public:
std::string m_derived_data;

SERIALIZE_METHODS_PARAMS(Derived, obj, DerivedAndBaseFormat, fmt)
SERIALIZE_METHODS(Derived, obj)
{
auto& fmt = SER_PARAMS(DerivedAndBaseFormat);
READWRITE(fmt.m_base_format(AsBase<Base>(obj)));

if (ser_action.ForRead()) {