Random errors occurs for custom file system

[vitonsky]

My use case

I use https://pglite.dev/ and want to keep FS in RAM, but dump this FS as ArrayBuffer once changes occurs.

PGLite have method https://pglite.dev/docs/api#dumpdatadir that even with no compression takes 100ms and this is too long time, since data dump may be called frequently.

Since PGLite uses a MEMFS i think the problem is they iterate whole FS content to build archive every time we call dump.

As potential solution I see a FS that will store data in single ArrayBuffer and will let us dump this buffer immediately (only time to copy buffer).

I found package @zenfs/core that implements exactly I need

import { configureSingle, fs, SingleBuffer } from "@zenfs/core";

const sharedArrayBuffer = new ArrayBuffer(100000);

await configureSingle({
	backend: SingleBuffer,
	buffer: sharedArrayBuffer,
});

fs.writeFileSync('/test.txt', 'You can do this anywhere, including browsers!');

The problem

Now I trying to wrap this FS via pglite custom FS API

pglite/packages/pglite/src/fs/base.ts

Lines 75 to 79 in a5d8a21

	/**
	* Abstract base class for all custom virtual filesystems.
	* Each custom filesystem needs to implement an interface similar to the NodeJS FS API.
	*/
	export abstract class BaseFilesystem implements Filesystem {

It does not work for me and throws random errors and behave randomly too.

My code with debug prints is:

/* eslint-disable spellcheck/spell-checker */
import { PGlite } from '@electric-sql/pglite';
import { BaseFilesystem } from '@electric-sql/pglite/dist/fs/base';
import { x } from '@electric-sql/pglite/dist/pglite-CyDq4d4K';
import { configure, fs, SingleBuffer } from '@zenfs/core';

class MyFS extends BaseFilesystem {
	protected readonly dataDir = '/';
	chmod(path: string, mode: number): void {
		fs.chmodSync(path, mode);
	}
	close(fd: number): void {
		fs.closeSync(fd);
	}
	fstat(fd: number): x {
		console.log('fstat', fd);
		const stats = fs.fstatSync(fd);

		return {
			blocks: stats.blocks,
			atime: stats.atimeMs,
			mtime: stats.mtimeMs,
			ctime: stats.ctimeMs,
			dev: stats.dev,
			ino: stats.ino,
			mode: stats.mode,
			nlink: stats.nlink,
			uid: stats.uid,
			gid: stats.gid,
			rdev: stats.rdev,
			size: stats.size,
			blksize: stats.blksize,
		};
	}
	lstat(path: string): x {
		console.log('lstat', path);

		try {
			const stats = fs.lstatSync(path);
			const result = {
				blocks: stats.blocks,
				atime: stats.atimeMs,
				mtime: stats.mtimeMs,
				ctime: stats.ctimeMs,
				dev: stats.dev,
				ino: stats.ino,
				mode: stats.mode,
				nlink: stats.nlink,
				uid: stats.uid,
				gid: stats.gid,
				rdev: stats.rdev,
				size: stats.size,
				blksize: stats.blksize,
			};

			return result;
		} catch (error) {
			throw error;
		}
	}
	mkdir(
		path: string,
		options?:
			| { recursive?: boolean | undefined; mode?: number | undefined }
			| undefined,
	): void {
		console.log('mkdir', path);
		fs.mkdirSync(path, options);
	}
	open(path: string, flags?: string | undefined, mode?: number | undefined): number {
		console.log('open', path);
		return fs.openSync(path, flags ?? 'rw', mode);
	}
	readdir(path: string): string[] {
		console.log('readdir', path);
		return fs.readdirSync(path);
	}
	read(
		fd: number,
		buffer: Uint8Array,
		offset: number,
		length: number,
		position: number,
	): number {
		console.log('>> read', fd);
		return fs.readSync(fd, buffer, offset, length, position);
	}
	rename(oldPath: string, newPath: string): void {
		fs.renameSync(oldPath, newPath);
	}
	rmdir(path: string): void {
		fs.rmdirSync(path);
	}
	truncate(path: string, len: number): void {
		fs.truncateSync(path, len);
	}
	unlink(path: string): void {
		fs.unlinkSync(path);
	}
	utimes(path: string, atime: number, mtime: number): void {
		fs.utimesSync(path, atime, mtime);
	}
	writeFile(
		path: string,
		data: string | Uint8Array,
		options?:
			| {
				encoding?: string | undefined;
				mode?: number | undefined;
				flag?: string | undefined;
			}
			| undefined,
	): void {
		console.log('WRITE FILE', path);
		const { encoding, ...restOptions } = options ?? {};
		fs.writeFileSync(path, data, {
			...restOptions,
			encoding: encoding as BufferEncoding,
		});
	}
	write(
		fd: number,
		buffer: Uint8Array,
		offset: number,
		length: number,
		position: number,
	): number {
		console.log('>> write');
		return fs.writeSync(fd, buffer, offset, length, position);
	}
}


function proxyMethods(obj: any, onCall: any) {
	return new Proxy(obj, {
		get(target, prop, receiver) {
			const value = Reflect.get(target, prop, receiver);
			if (typeof value !== "function") return value;
			return function (...args) {
				const original = value.bind(this);
				onCall(target, prop, args, original);
				return original(...args);
			};
		}
	});
}

test('debug', async () => {
	const sharedArrayBuffer = new ArrayBuffer(52_428_800);

	await configure({
		mounts: {
			'/': {
				backend: SingleBuffer,
				buffer: sharedArrayBuffer,
			},
		},
	});

	const pg = new PGlite({
		fs: proxyMethods(new MyFS(), (_target, prop, args) => {
			console.log("Call for", prop, args);
		})
	});
	await pg.waitReady;

	console.log(fs.readdirSync('/', { recursive: true }));
});

The output looks weird, it looks code calls lstat with empty path '' and never tries to create file, then promise waitReady rejects with message "unreachable":

stdout | src/core/storage/database/pglite/zenfs.test.ts > debug
Call for initialSyncFs []

stdout | src/core/storage/database/pglite/zenfs.test.ts > debug
Call for lstat [ '/PG_VERSION' ]
lstat /PG_VERSION
Call for lstat [ '/PG_VERSION' ]
lstat /PG_VERSION

stdout | src/core/storage/database/pglite/zenfs.test.ts > debug
Call for lstat [ '/PG_VERSION' ]
lstat /PG_VERSION
Call for lstat [ '' ]
lstat 
Call for lstat [ '/postgresql.conf' ]
lstat /postgresql.conf
Call for lstat [ '' ]
lstat 

 ❯ src/core/storage/database/pglite/zenfs.test.ts (1 test | 1 failed) 129ms
   × debug 129ms
     → unreachable

The request

It is not clear about current status of custom FS in pglite, so let's figure it out first. Does pglite provides any API to implement custom FS on user side?

If so, we need in docs and maybe trivial example how to implement custom FS.

I found issue #533 where @tdrz said

We are looking into making it easier to add custom filesystems but it will take a while...

It was almost year ago. I no need in API to make custom FS easier, I want to implement custom FS in any way, to address real use case.

With no custom FS pglite have no sense in my case, because the use case is privacy focused app that can't keep data in FS/IDB/etc with no encryption. That's why we use in-memory DB. But persistence and performance still does matter.

In case I can't implement custom FS, the only way is to dump data as tar archive, but this way show very poor performance for frequent syncs.

[vitonsky]

@tdrz is there any updates on this?

I've tried to debug this problem today, but i can't debug WASM so it is unclear for me why fs checks for '' path and what i can do to move forward. I'm stuck on unreachable error

[tdrz]

@vitonsky Sorry but we cannot afford to invest time in your particular implementation at the moment.

We still have the custom filesystems on our radar and we will address it at some moment.

[vitonsky]

@tdrz the problem not in specific FS implementation. The problem is any custom files storage cannot be implemented due to problems mentioned in #810 (comment)

• FS tries to access entities with path '' (empty string)
• It is not possible to debug any problems with custom FS, because PGLite throw unreachable error with no context

Factually your response means the PGLite currently do not support any custom FS and PGlite are not ready even to join discussion in issue to debug a problem.

That's totally fine you're not ready to implement custom FS yet. However, this approach with ignore a real use cases even in GitHub discussions about bugs that introduced by PGLite (not even a Postgres) looks disappointing and do not build a credibility to a project and its future.

Again, that's ok you will not write code for specific use cases. Because at least it's obviously requires a time to write a code.

But discussions are async communication method, so you could share your knowledge about how to debug your product, and maybe even propose a debugging and development vector. This approach would make PGLite move faster, because it would have external contributors.

Currently it looks like a huge project with one man gang contributor. Of course that sounds totally not scalable.

For me as a developer who choose a PGLite because of great potential, this way of project management looks dangerous. I would like to PGLite would grown. This is why I ask you take a few minutes and think about how to communicate with community efficiently, to make them contribute into PGLite.

There are at least 2 options

• Hire some community manager who are engineer who would share knowledge with a community and help them contribute to a PGLite. (Current team could work on project core meanwhile)
• If you are the one who write code on this project and you have no resource to hire new people or there are low community activity yet, you could manage it yourself if you change your mind and will see issues as a tool of async communication that means you can respond in threads like this once per week when you have time

I'm serious, think about it. PGlite have great potential, but a really great things cannot be done alone

[tdrz]

Thank you @vitonsky !

the problem not in specific FS implementation.

Sorry, I might have misread this: "I found package @zenfs/core that implements exactly I need"

and maybe even propose a debugging and development vector.

Our debugging section is a good starting point for any determined contributor.