[Docs] Improve Security Considerations Documentation - include pointer to validation functions for validating IPC streams.

[rustyconover]

Describe the bug, including details regarding any error messages, version, and platform.

Hi Arrow Friends,

I was pointed to: https://arrow.apache.org/docs/dev/format/Security.html#ipc-format on the Arrow Community call today.

This is a great document.

In this section:

Advice for users

Arrow libraries will typically ensure IPC streams are structurally valid but may not also validate the underlying Array data. It is extremely recommended that you use the appropriate APIs to validate the Arrow data read from an untrusted IPC stream.

As a reasonably experienced Arrow C++/PyArrow user, I didn't know what APIs were referenced here.

It seems like this text is talking about these methods:

https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html#pyarrow.RecordBatch.validate
https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.validate

Are those sufficient for the validation?

Would it be a good idea to add an always_validate flag to the IpcReadOptions when dealing with untrusted data sources?

https://arrow.apache.org/docs/python/generated/pyarrow.ipc.IpcReadOptions.html#pyarrow.ipc.IpcReadOptions

Thank for your consideration,

Rusty

Component(s)

Documentation

[shashbha14]

take

[shashbha14]

I’ve opened a PR that adds a “Security considerations for untrusted IPC data”
section to the Python IPC docs, pointing to RecordBatch.validate() and
Table.validate() and including small usage examples.

[raulcd]

Thanks for the PR @shashbha14 . As shared on the review I think there's still a discussion to be had about the question that @rustyconover is opening:

Would it be a good idea to add an always_validate flag to the IpcReadOptions when dealing with untrusted data sources?

Also, the guidelines should probably go to the Security Considerations page and not to the IPC page.

[shashbha14]

Thanks for the feedback.I’m happy to adjust the docs once there’s agreement on whether we want an
always_validate-style option and what exactly should go into the Security
Considerations page. For now I’ll wait for that discussion and can update or
open a new PR to move the guidance to the Security docs when you’re ready.

[rustyconover]

I prototyped this new feature for a validation option of IpcReadOptions in PyArrow on this branch:

https://github.com/rustyconover/arrow/pull/new/feat-validation-option-python-ipc-reader

[raulcd]

Personally, I think an option like this could makes sense. As per the prototype, I think it would make more sense to be directly embedded as an option on arrow::ipc::IpcReadOptions instead of a wrapper on top of it only on pyarrow with this it could be used by other bindings too.

The only problem I foresee with this is that we are moving the responsibility from the user to the IPC Reader implementation, with the implications that this has. One of the key ones is around security. The user will trust we are validating, as we should with this new contract. If we then miss a code path for example because a new read method has been added, we could end up with unvalidated data that the user trusts as validated. That becomes a potential security issue (CVE) rather than just a bug.

I am not saying that we shouldn't add it just that we should be clear that the responsibility is shifting.

cc @pitrou