AAPL extensions/context

[tucari]

Hi!

Thanks so much for ksmbd!

I wonder if there is any interest or ongoing work in implementing AAPL SMB2 extensions (as seen in vfs_fruit for Samba)?
Main reason (in my case) would be for the bulk readdirattr calls which -hugely- improve directory traversal speed on macOS.

I've managed to get the context handling and capability advertising working in ksmbd but not yet the readdirattr call.

Thanks
Jon

[namjaejeon]

Thanks for suggestion:)
I don't have Mac book and can't test it although readdirattr is implemented.
Is there any specification of AAPL SMB2 extensions ?

[tucari]

Hi Namjae,

Hope you're well.

Not found a solid spec but been working off the following:
https://www.snia.org/sites/default/files/SDC/2016/presentations/smb/RafalSzczesniak_Sailing_Unchartered_Waters_Apple_OSX_Extensions_SMB2_Wide.pdf

As well as the vfs_fruit module from Samba, specifically:
https://github.com/samba-team/samba/blob/master/source3/modules/vfs_fruit.c

static NTSTATUS check_aapl(vfs_handle_struct *handle,
			   struct smb_request *req,
			   const struct smb2_create_blobs *in_context_blobs,
			   struct smb2_create_blobs *out_context_blobs)
{
	struct fruit_config_data *config;
	NTSTATUS status;
	struct smb2_create_blob *aapl = NULL;
	uint32_t cmd;
	bool ok;
	uint8_t p[16];
	DATA_BLOB blob = data_blob_talloc(req, NULL, 0);
	uint64_t req_bitmap, client_caps;
	uint64_t server_caps = SMB2_CRTCTX_AAPL_UNIX_BASED;
	smb_ucs2_t *model;
	size_t modellen;

	SMB_VFS_HANDLE_GET_DATA(handle, config, struct fruit_config_data,
				return NT_STATUS_UNSUCCESSFUL);

	if (!config->use_aapl
	    || in_context_blobs == NULL
	    || out_context_blobs == NULL) {
		return NT_STATUS_OK;
	}

	aapl = smb2_create_blob_find(in_context_blobs,
				     SMB2_CREATE_TAG_AAPL);
	if (aapl == NULL) {
		return NT_STATUS_OK;
	}

	if (aapl->data.length != 24) {
		DEBUG(1, ("unexpected AAPL ctxt length: %ju\n",
			  (uintmax_t)aapl->data.length));
		return NT_STATUS_INVALID_PARAMETER;
	}

	cmd = IVAL(aapl->data.data, 0);
	if (cmd != SMB2_CRTCTX_AAPL_SERVER_QUERY) {
		DEBUG(1, ("unsupported AAPL cmd: %d\n", cmd));
		return NT_STATUS_INVALID_PARAMETER;
	}

	req_bitmap = BVAL(aapl->data.data, 8);
	client_caps = BVAL(aapl->data.data, 16);

	SIVAL(p, 0, SMB2_CRTCTX_AAPL_SERVER_QUERY);
	SIVAL(p, 4, 0);
	SBVAL(p, 8, req_bitmap);
	ok = data_blob_append(req, &blob, p, 16);
	if (!ok) {
		return NT_STATUS_UNSUCCESSFUL;
	}

	if (req_bitmap & SMB2_CRTCTX_AAPL_SERVER_CAPS) {
		if ((client_caps & SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR) &&
		    (handle->conn->fs_capabilities & FILE_NAMED_STREAMS)) {
			server_caps |= SMB2_CRTCTX_AAPL_SUPPORTS_READ_DIR_ATTR;
			config->readdir_attr_enabled = true;
		}

		if (config->use_copyfile) {
			server_caps |= SMB2_CRTCTX_AAPL_SUPPORTS_OSX_COPYFILE;
			config->copyfile_enabled = true;
		}

		/*
		 * The client doesn't set the flag, so we can't check
		 * for it and just set it unconditionally
		 */
		if (config->unix_info_enabled) {
			server_caps |= SMB2_CRTCTX_AAPL_SUPPORTS_NFS_ACE;
		}

		SBVAL(p, 0, server_caps);
		ok = data_blob_append(req, &blob, p, 8);
		if (!ok) {
			return NT_STATUS_UNSUCCESSFUL;
		}
	}

	if (req_bitmap & SMB2_CRTCTX_AAPL_VOLUME_CAPS) {
		int val = lp_case_sensitive(SNUM(handle->conn));
		uint64_t caps = 0;

		switch (val) {
		case Auto:
			break;

		case True:
			caps |= SMB2_CRTCTX_AAPL_CASE_SENSITIVE;
			break;

		default:
			break;
		}

		if (config->time_machine) {
			caps |= SMB2_CRTCTX_AAPL_FULL_SYNC;
		}

		SBVAL(p, 0, caps);

		ok = data_blob_append(req, &blob, p, 8);
		if (!ok) {
			return NT_STATUS_UNSUCCESSFUL;
		}
	}

	if (req_bitmap & SMB2_CRTCTX_AAPL_MODEL_INFO) {
		ok = convert_string_talloc(req,
					   CH_UNIX, CH_UTF16LE,
					   config->model, strlen(config->model),
					   &model, &modellen);
		if (!ok) {
			return NT_STATUS_UNSUCCESSFUL;
		}

		SIVAL(p, 0, 0);
		SIVAL(p + 4, 0, modellen);
		ok = data_blob_append(req, &blob, p, 8);
		if (!ok) {
			talloc_free(model);
			return NT_STATUS_UNSUCCESSFUL;
		}

		ok = data_blob_append(req, &blob, model, modellen);
		talloc_free(model);
		if (!ok) {
			return NT_STATUS_UNSUCCESSFUL;
		}
	}

	status = smb2_create_blob_add(out_context_blobs,
				      out_context_blobs,
				      SMB2_CREATE_TAG_AAPL,
				      blob);
	if (NT_STATUS_IS_OK(status)) {
		global_fruit_config.nego_aapl = true;
	}

	return status;
}

Converted to ksmbd context, this then shows the share as type: SMB (OSX) in macOS and tries to use their readdirattr extension (i.e stuffing a whole folder into another structure inside the response). This stops macOS from reading the listing then statting each file for attributes and, testing with Apple's SMB server:

Traversing folder structure of 400 folder and 18000 files:
Without AAPL: 28 seconds
With AAPL: 2 seconds

I should have a working PR for this soon if you're interested.

(different issue, I'm currently working from Commit 888b780 rather than master, as master causing a hang and 100% cpu load of ksmbd-io on kernel 6.13, unsure if that's something specific to my config)

Thanks!
Jon

[namjaejeon]

I should have a working PR for this soon if you're interested.

Okay, I am very interested in this! Please send a PR to us:)

I'm currently working from Commit 888b780 rather than master, as master causing a hang and 100% cpu load of ksmbd-io on kernel 6.13, unsure if that's something specific to my config)

Sound great:) Let me know if you have question while checking this issue.

[mmakassikis]

(different issue, I'm currently working from Commit 888b780 rather than master, as master causing a hang and 100% cpu load of ksmbd-io on kernel 6.13, unsure if that's something specific to my config)

is that what the last commit on master is supposed to fix ("ksmbd: fix infinite loop when client send search pattern") ?

AAPL create context is also required for Time Machine backups on SMB shares. However, there are a few more things (durable handles v2 for example). I found this page which may be of interest:

https://developer.apple.com/library/archive/releasenotes/NetworkingInternetWeb/Time_Machine_SMB_Spec/

[namjaejeon]

@mmakassikis

is that what the last commit on master is supposed to fix ("ksmbd: fix infinite loop when client send search pattern") ?

Right. I think so. He should check it with the latest master version.

AAPL create context is also required for Time Machine backups on SMB shares. However, there are a few more things (durable handles v2 for example). I found this page which may be of interest:

Thanks for sharing it.

[tucari]

Hi Both,

Working from latest version now.

Work got a bit busy so still working on this, AAPL context is done as is AAPL Readdirattr (which stuffs unix perms and finderinfo into reserved field and easize). I think Time Machine requires the full sync flag to be considered for use, haven't yet looked at that in the samba vfs_fruit code.

Should have a PR soon for the context and basic ops.

Best
Jon

P.s thanks for the extra Time machine info!
Has been hard to find much info on AAPL extensions so I've mostly wiresharked it from a osx server.

[namjaejeon]

@tucari Sound great and thanks for your work.

[Nmishin]

It would be great if I could use ksmbd as a Time Machine server instead of Samba!

[namjaejeon]

@tucari Is there any news for your AAPL Readdirattr work ?

[jeanphilippe-mh]

Hello 👋🏻 @tucari and @namjaejeon, since macOS Tahoe 26 (only warning for now) and his successor macOS 27 will totally depreciated AFP, KSMBD is the only GPLv2 modern tools capable of bringing this compatibility instead of Samba, will it be achievable to have all Apple extensions (like Fruits for Samba) to make it works great with Avahi ? 🙏🏻

In summary 👍 :

• It will give Time Machine compatibility for our NAS starting macOS 26+. ✅
• It will bring back iOS R/W compatibility for SMB (for example : for now, if you want add it in File.app this is only Readonly since iOS18 because Apple had enforced restrictions with these AAPL advanced metadata). ✅

Thanks for your help!

[namjaejeon]

@jeanphilippe-mh
I really hope this feature is included in ksmbd. I hope developers who want this feature and have the time will contribute. Otherwise, there's a lot of work to be done on ksmbd, and I hope a company will provide funding or hire ksmbd(SMB) developers.

[jeanphilippe-mh]

Hello @namjaejeon,

Thanks for your valuable feedback 😊, I hope @mmakassikis (Freebox team) will be able to help you on it ! 🙏🏻

Kind regards,
Jean-Philippe

[LvanProosdij]

Thanks @jeanphilippe-mh for bringing this here 🙏, Freebox/SMB/KSMBD/Apple Time Machine.

[Cong827]

To use the time machine function, not only does one need to support the AAPL feature.I also attempted to forge AAPL messages on the ksmbd, but still failed to successfully use the time machine. The packet capture results show that the mdssvc protocol needs to be supported. Could you please provide the technical details of the relevant protocol?

[namjaejeon]

@Cong827 I asked Ralph, who implemented mdssvc in Samba, and he said that mdssvc has nothing to do with the time machine functionality. and he reverse engineered it and implemented it in Samba. I think the details for that will be in the samba code.