Add kqueue backend to avoid event types weirdness from hfsevents on OS X

[sethfowler]

OS X's FSEvents API has some odd behavior which I still don't entirely understand. It often provides several flags for each event. For example (output produced using trace from hfsevents), if I do this:

$ touch myfile
$ mv myfile myfilex
$ mv myfilex myfile
$ rm myfile

I get this log output:

id:    18147469115959082481
path:  /Users/mfowler/Code/test/hfsevents/test/myfile
flags: ItemCreated ItemIsFile

id:    18147469115959083733
path:  /Users/mfowler/Code/test/hfsevents/test/myfile
flags: ItemCreated ItemRenamed ItemIsFile

id:    18147469115959083734
path:  /Users/mfowler/Code/test/hfsevents/test/myfilex
flags: ItemRenamed ItemIsFile

id:    18147469115959088855
path:  /Users/mfowler/Code/test/hfsevents/test/myfilex
flags: ItemRenamed ItemIsFile

id:    18147469115959088856
path:  /Users/mfowler/Code/test/hfsevents/test/myfile
flags: ItemRenamed ItemIsFile

id:    18147469115959090377
path:  /Users/mfowler/Code/test/hfsevents/test/myfile
flags: ItemRemoved ItemIsFile

Take a look at the flags for the second entry.

Consider also this command:

$ touch x && echo test > x && rm x

Which yields this log:

id:    18147469115959120388
path:  /Users/mfowler/Code/test/hfsevents/test/x
flags: ItemCreated ItemInodeMetaMod ItemModified ItemIsFile

id:    18147469115959120391
path:  /Users/mfowler/Code/test/hfsevents/test/x
flags: ItemCreated ItemRemoved ItemInodeMetaMod ItemModified ItemIsFile

Notice the fact that each event contains flags from other recent events. This explains the anomalies I've seen regarding the types of events generated by System.FSEvents on OS X. (I ended up just ignoring the event type totally for now.) There definitely needs to be a change to how System.FSEvents decides what type of event to generate. It seems impossible to do this correctly in a stateless fashion, which is unfortunate. (Though one option is just to generate duplicate events and give the user of the library the responsibility to deduplicate them if they need to.)

[gregwebs]

Do you want to open this issue up at: https://github.com/luite/hfsevents
@luite may already know about this

[sethfowler]

Well, I'm pretty sure @luite is faithfully reproducing the information that the FSEvents API gives. I looked at the code and it's just yanking this info right out of the data structure the OS provides. The question is just how to better map it to the abstractions in this library.

[gregwebs]

ahh, ok. I think there is already some trickiness with mapping what happens in Windows.

[luite]

yep i don't think this is an hfsevents problem. I compared the output from trace to the simplest other tracer i could find: https://github.com/andreyvit/fseventsmon.git (change the last parameter of FSEventStreamCreate to kFSEventStreamCreateFlagFileEvents to get file-level events) and i got exactly the same, sometimes weird, results

[UnkindPartition]

I offered a bounty for a similar issue at StackOverflow, but no-one seems to know what to do about it.

So my plan is the this: use fsevents only as a trigger, and use the logic already used in the polling to compute the actual events. That logic will be factored out of S.F.Polling into a new module.

What do others think?

[gregwebs]

seems like it is worth a shot :) One thing we can always look to is what the libraries in other languages do. Some of those are also buggy though :)

[UnkindPartition]

Any specific suggestions? I'm not familiar with the Mac OS X ecosystem.

[UnkindPartition]

rb-fsevent doesn't support file events at all

[gregwebs]

Seems that you are referring to this, which has a lot of info on OS X limitations:
guard/rb-fsevent#14

So yeah, we should not try to listen to files, only directories.

[UnkindPartition]

I was referring to their README:

This and issues of a similar nature have prevented me from adding the option to the ruby code despite the fsevent_watch binary supporting file level events for quite some time now.

[UnkindPartition]

It's been suggested that we use kqueue instead.

This is interesting both because it probably avoids this bug, but also because we get support for BSDs this way, too. There's already bindings for Haskell by @hesselink http://hackage.haskell.org/package/kqueue

[gregwebs]

That is a good suggestion, although kqueue has its own issues. It is probably best to keep both backends and make it possible for users to choose one. I usually just monitor directories with infrequent changes so fsevents works for me.

[UnkindPartition]

What issues does kqueue have?

This issue has nothing to do with frequency of changes. The reported events are simply wrong.

[gregwebs]

In general my understanding is that kqueue works well to monitor a file, but there are issues to figure out with renames and watching directories. Perhaps a hybrid approach is actually best: node apparently switched to using fsevents for watching directories: http://blog.nodejs.org/2012/09/17/node-v0-9-2-unstable/
http://stackoverflow.com/questions/10762630/nodejs-fs-watch-on-directory-only-fires-when-changed-by-editor-but-not-shell-or
nodejs/node-v0.x-archive#2062

[UnkindPartition]

In general my understanding is that kqueue works well to monitor a file, but there are issues to figure out with renames and watching directories.

From your links it looks like the opposite: watching directories (for renames, creation and removal) works fine, but it doesn't detect actual file changes inside the directory. (Well, watching individual files probably works too.)

Still, you're right in that this doesn't do what we need, sadly.

[hesselink]

In the kqueue package I have a module System.KQueue.HighLevel which tracks a directory and all files in it, by watching the directory, and when a new file is added, starting to watch that.

[UnkindPartition]

@hesselink do you know if it scales/performs well?

[hesselink]

No , sadly I don't. I think that some system limits like the maximum number of open files could become a problem, but that should probably be tested.

[UnkindPartition]

@hesselink

In the kqueue package I have a module System.KQueue.HighLevel which tracks a directory and all files in it

Did you mean to export monitorChangesIO (or a higher-level wrapper around it) from that module? Or should I use that code just as an example?

[hesselink]

@feuerbach It seems I misremembered. The HighLevel module watches a file if it exists, otherwise it will start watching it when it comes into existing. I wrote it for monitoring a configuration file. Still, the components could probably be used to watch a directory. I could take a better look next week when I'm at work again, or you can send a pull request if you want.