Improvements to VQL reference documentation and minor fixes

docs/references/vql.yaml

@@ -43,7 +43,14 @@
   - windows_386_cgo
   - windows_amd64_cgo
 - name: alert
-  description: Generate an alert message.
+  description: |
+    Generate an alert message.
+
+    ### See also
+
+    - [log](/vql_reference/basic/log/): alerts and log messages are similar in
+      concept and use the same deduplication mechanism which is explained with
+      examples for the `log()` function.
   type: Function
   version: 2
   args:
@@ -1785,8 +1792,8 @@
   category: windows
 - name: ebpf_events
   description: |
-    Dump information about potential ebpf_events that can be used by the
-    watch_ebpf() plugin.
+    Dumps information about potential ebpf_events that can be used by the
+    [watch_ebpf](/vql_reference/misc/watch_ebpf/) plugin.
   type: Plugin
   platforms:
   - linux_amd64_cgo
@@ -2697,7 +2704,7 @@
 
     ### Examples
 
-    ```sql
+    ```vql
     LET csv <= '''John,ate,banana
     Mary,had,little lamb'''
     LET my_words <= SELECT * FROM parse_csv(accessor="data", filename=csv, auto_headers=True)
@@ -2707,7 +2714,7 @@
     `Sentence: John ate a banana.`
     `Sentence: Mary had a little lamb.`
 
-    ```sql
+    ```vql
     LET T <= timestamp(epoch="2024-02-02T04:42:00Z")
     SELECT format(format="%d-%02d-%02dT%02d:%02d:%06.3fZ", args=[
       T.Year, T.Month, T.Day, T.Hour, T.Minute, T.Nanosecond / 1000000000 ])
@@ -2720,16 +2727,12 @@
 
     ### See also
 
-    - [log](vql_reference/basic/log/)
-      - a function which uses the same string formatting as the `format` function.
-    - [serialize](/vql_reference/basic/serialize/)
-      - a function that provides serialized (stringified) representations of
-        compound data types such as JSON and YAML objects.
-    - [timestamp_format](/vql_reference/misc/timestamp_format/)
-      - a function that simplifies the string formatting of timestamp objects.
-    - [typeof](/vql_reference/basic/format/)
-      - a dedicated function equivalent to the special use case:
-      `format(format="%T",args=x)`
+    - [log](vql_reference/basic/log/): a function which uses the same string
+      formatting as the `format` function.
+    - [timestamp_format](/vql_reference/misc/timestamp_format/): a function
+      that simplifies the string formatting of timestamp objects.
+    - [typeof](/vql_reference/basic/format/): a dedicated function equivalent
+      to the special use case: `format(format="%T",args=x)`
   type: Function
   args:
   - name: format
@@ -4455,7 +4458,7 @@
 
     ### Example
 
-    ```sql
+    ```vql
     // Setting this in a notebook will tell the GUI to treat this
     // column as URL.
     LET ColumnTypes <= dict(HuntLink="url_internal")
@@ -4506,15 +4509,14 @@
   - windows_amd64_cgo
 - name: log
   description: |
-    Log the message and return TRUE.
+    Log a message to the query log stream. Always returns TRUE.
 
-    The message will be logged into the query log stream (Viewable in
-    the Logs tab). The `message` parameter represents a format string
+    The `message` parameter represents a format string
     that will be expanded using the `args` parameter list if needed.
 
     Since `log()` always returns TRUE it is easy to use in a WHERE
-    clause as a form of debugging (It is basically equivalent to the
-    print statement of other languages).
+    clause as a form of debugging. It is basically equivalent to the
+    print statement of other languages.
 
     ```vql
     SELECT * FROM glob(...)
@@ -4524,7 +4526,7 @@
     ### Deduplication
 
     Log messages will be deduped according to the `dedup`
-    parameter - each distinct format string will not be emitted more
+    parameter - each **distinct format string** will not be emitted more
     frequently than the `dedup` parameter (by default 60 seconds).
 
     This makes it safe to use `log()` frequently without overflowing
@@ -4545,6 +4547,30 @@
     SELECT * FROM glob(...)
     WHERE log(message="Processing file %v", args=OSPath)
     ```
+
+    ### Example
+
+    In this more complex example the query will produce 10 rows, at a rate of
+    1 row every 5 seconds. However the log messages will be limited to 1 every
+    15 seconds.
+
+    ```vql
+    SELECT count() AS Count, String AS EventTime FROM clock(period=5)
+    WHERE log(message="Logging #%v at %v", args=[Count, EventTime], level="INFO", dedup=15)
+    LIMIT 10
+    ```
+
+    Thus the log message will be emitted for the 1st, 4th, 7th, and 10th rows.
+    To observe the deduplication behaviour in real time you can run this query
+    in a notebook cell and tweak the arguments to understand their impacts.
+
+    ### See also
+
+    - [format](/vql_reference/basic/format/): a function that uses the same
+      string formatting syntax.
+    - [alert](/vql_reference/misc/alert/): alerts are a special type of log
+      message that are added to a server alerts queue, which can be monitored.
+
   type: Function
   version: 2
   args:
@@ -5411,7 +5437,39 @@
   - windows_386_cgo
   - windows_amd64_cgo
 - name: now
-  description: Returns current time in seconds since epoch.
+  description: |
+    Returns the current time in seconds since epoch.
+
+    Note that an integer value is returned, not a timestamp.
+
+    Typically this function is used together with the `timestamp` function to
+    create a timestamp object which can be used in datetime comparisons.
+
+    ### Examples
+
+    Creating a timestamp representing the current time:
+
+    `timestamp(epoch=now())`
+
+    Creating a timestamp representing an hour ago:
+
+    `timestamp(epoch=now() - 3600)`
+
+    Using `now()` in timestamp arithmetic:
+
+    ```vql
+    SELECT * FROM flows(client_id="C.8cfee3cef5dc6915")
+    WHERE state =~ "FINISHED" AND timestamp(epoch=active_time) > now() - 60 * 60 * 24
+    ```
+
+    Note that the above time comparison works even though `now() - 60 * 60 * 24`
+    results in an integer. This is because one of the operands is a timestamp
+    object, so VQL will convert the int to a timestamp for purposes of the
+    comparison.
+
+    ### See also
+
+    - [timestamp](/vql_reference/basic/timestamp/)
   type: Function
   category: basic
   platforms:
@@ -9384,12 +9442,12 @@
 
     ### Examples
 
-    ```sql
+    ```vql
     SELECT typeof(x=1) AS Type FROM scope()
     ```
     returns: `Type: int64`
 
-    ```sql
+    ```vql
     LET my_time <= timestamp(epoch="2024-03-26T06:53:37Z")
     SELECT typeof(thing=my_time) AS Type FROM scope()
     ```
@@ -9400,14 +9458,14 @@
     The `typeof` function is a more concise alternative to using the more
     flexible and more powerful `format` function:
 
-    ```sql
+    ```vql
     SELECT format(format="%T", args=x) AS Type FROM scope()
     ```
 
     The `typeof` function is often used to inspect the data type of returned
     values when writing and testing VQL.
 
-    It is also useful as a row filter by including it in the `WHERE` clause of
+    It is also useful as a row filter by including it in the WHERE clause of
     a query to ensure that a specific column does not contain values of an
     unexpected data type.
 
@@ -10353,6 +10411,11 @@
     This plugin uses the integrated tracee eBPF engine to stream events.
 
     See https://github.com/Velocidex/tracee_velociraptor for more details.
+
+    ### See also
+
+    - [ebpf_events](/vql_reference/misc/ebpf_events/): Dumps information about
+      potential ebpf_events that can be used by the watch_ebpf() plugin.
   type: Plugin
   args:
   - name: events