Skip to content

Latest commit

 

History

History
631 lines (485 loc) · 24.2 KB

DOCUMENTATION.md

File metadata and controls

631 lines (485 loc) · 24.2 KB
Raw

Directus Sync

Directus 11.2.2

Important

Latest version of directus-sync introduces breaking changes and is not compatible with Directus 10.x.x. If you are using Directus 10.x.x, please run npx [email protected]

Note

Help us improve Directus Sync by sharing your feedback! Take a quick survey about your usage here: https://forms.gle/LnaB89uVkZCDqRfGA

The directus-sync command-line interface (CLI) provides a set of tools for managing and synchronizing the schema and collections within Directus across different environments.

By leveraging Directus's REST API, it aligns closely with the native actions performed within the application, ensuring a high fidelity of operation.

Updates are granular, focusing on differential data changes rather than blunt table overwrites, which means only the necessary changes are applied, preserving the integrity and history of your data.

Moreover, directus-sync organizes backups into multiple files, significantly improving readability and making it easier to track and review changes. This thoughtful separation facilitates a smoother version control process, allowing for targeted updates and clearer oversight of your Directus configurations.

Table of Contents

Requirements

Usage

The CLI is available using the npx command.

npx directus-sync <command> [options]

Here's how to use each command in the CLI:

Commands

Pull

npx directus-sync pull

Retrieves the current schema and collections from Directus and stores them locally. This command does not modify the database.

It also retrieves the specifications (GraphQL & OpenAPI) and stores them locally. It gets specifications from the /server/specs/* endpoints:

Diff

npx directus-sync diff

Analyzes and describes the difference (diff) between your local schema and collections and the state of the Directus instance. This command is non-destructive and does not apply any changes to the database.

Push

npx directus-sync push

Applies the changes from your local environment to the Directus instance. This command pushes your local schema and collection configurations to Directus, updating the instance to reflect your local state.

Available options

Options are merged from the following sources, in order of precedence:

  1. CLI arguments
  2. Environment variables
  3. Configuration file
  4. Default values

CLI and environment variables

These options can be used with any command to configure the operation of directus-sync:

  • -c, --config-path <configPath>
    Change the path to the config file. Default paths are: ./directus-sync.config.js, ./directus-sync.config.cjs or ./directus-sync.config.json.

  • -d, --debug
    Display additional logging information. Useful for debugging or verifying what directus-sync is doing under the hood.

  • -u, --directus-url <directusUrl>
    Specify the Directus instance URL. Alternatively, set the DIRECTUS_URL environment variable.

  • -t, --directus-token <directusToken>
    Provide the Directus access token. Alternatively, set the DIRECTUS_TOKEN environment variable. If provided, the directus-email and directus-password options are ignored.

  • -e, --directus-email <directusEmail>
    Provide the Directus email. Alternatively, set the DIRECTUS_ADMIN_EMAIL environment variable.

  • -p, --directus-password <directusPassword>
    Provide the Directus password. Alternatively, set the DIRECTUS_ADMIN_PASSWORD environment variable.

  • --dump-path <dumpPath>
    Set the base path for the dump. This must be an absolute path. The default is "./directus-config".

  • --collections-path <collectionPath>
    Specify the path for the collections dump, relative to the dump path. The default is "collections".

  • -o, --only-collections <onlyCollections>
    Comma-separated list of directus collections to include during pull push or diff process.

  • -x, --exclude-collections <excludeCollections>
    Comma-separated list of directus collections to exclude during pull push or diff. Can be used along with only-collections.

  • --preserve-ids <preserveIds>
    Comma-separated list of directus collections to preserve the original ids during the pull or push process.
    Possible collections are: dashboards, operations, panels, policies, roles and translations.
    flows and folders ids are always preserved.
    The value can be * or all to preserve ids of all collections, when applicable.

  • --snapshot-path <snapshotPath>
    Specify the path for the schema snapshot dump, relative to the dump path. The default is "snapshot".

  • --no-snapshot
    Do not pull and push the Directus schema. By default, the schema is pulled and pushed.

  • --no-split
    Indicates whether the schema snapshot should be split into multiple files. By default, snapshots are split.

  • --specs-path <specsPath>
    Specify the path for the specifications dump (GraphQL & OpenAPI), relative to the dump path. The default is "specs".

  • --no-specs
    Do not dump the specifications (GraphQL & OpenAPI). By default, specifications are dumped.

  • -f, --force
    Force the diff of schema, even if the Directus version is different. The default is false.

  • -h, --help
    Display help information for the directus-sync commands.

Configuration file

The directus-sync CLI also supports a configuration file. This file is optional. If it is not provided, the CLI will use the default values for the options.

The default paths for the configuration file are ./directus-sync.config.js, ./directus-sync.config.cjs or ./directus-sync.config.json. You can change this path using the --config-path option.

The configuration file can extend another configuration file using the extends property.

This is an example of a configuration file:

// ./directus-sync.config.js
module.exports = {
  extends: ['./directus-sync.config.base.js'],
  debug: true,
  directusUrl: 'https://directus.example.com',
  directusToken: 'my-directus-token',
  directusEmail: '[email protected]', // ignored if directusToken is provided
  directusPassword: 'my-directus-password', // ignored if directusToken is provided
  directusConfig: {
    clientOptions: {},  // see https://docs.directus.io/guides/sdk/getting-started.html#polyfilling
    restConfig: {}, // see https://docs.directus.io/packages/@directus/sdk/rest/interfaces/RestConfig.html
  },
  dumpPath: './directus-config',
  collectionsPath: 'collections',
  onlyCollections: ['roles', 'policies', 'permissions', 'settings'],
  excludeCollections: ['settings'],
  preserveIds: ['roles', 'panels'], // can be '*' or 'all' to preserve all ids, or an array of collections
  snapshotPath: 'snapshot',
  snapshot: true,
  split: true,
  specsPath: 'specs',
  specs: true,
};

Collections hooks

In addition to the CLI commands, directus-sync also supports hooks. Hooks are JavaScript functions that are executed at specific points during the synchronization process. They can be used to transform the data coming from Directus or going to Directus.

Hooks are defined in the configuration file using the hooks property. Under this property, you can define the collection name and the hook function to be executed. Available collection names are: dashboards, flows, folders, operations, panels, permissions, policies, presets, roles, settings and translations.

For each collection, available hook functions are: onQuery, onLoad, onSave, and onDump. These can be asynchronous functions.

During the pull command:

  • onQuery is executed just before the query is sent to Directus for get elements. It receives the query object as parameter and must return the query object. The second parameter is the Directus client.
  • onDump is executed just after the data is retrieved from Directus and before it is saved to the dump files. The data is the raw data received from Directus. The second parameter is the Directus client. It must return the data to be saved to the dump files.
  • onSave is executed just before the cleaned data is saved to the dump files. The "cleaned" data is the data without the columns that are ignored by directus-sync (such as user_updated) and with the relations replaced by the SyncIDs. The first parameter is the cleaned data and the second parameter is the Directus client. It must return the data to be saved to the dump files.

During the push command:

  • onLoad is executed just after the data is loaded from the dump files. The data is the cleaned data, as described above. The first parameter is the data coming from the JSON file and the second parameter is the Directus client. It must return the data.
Simple example

Here is an example of a configuration file with hooks:

// ./directus-sync.config.js
module.exports = {
  hooks: {
    flows: {
      onDump: (flows) => {
        return flows.map((flow) => {
          flow.name = `🧊 ${flow.name}`;
          return flow;
        });
      },
      onSave: (flows) => {
        return flows.map((flow) => {
          flow.name = `🔥 ${flow.name}`;
          return flow;
        });
      },
      onLoad: (flows) => {
        return flows.map((flow) => {
          flow.name = flow.name.replace('🔥 ', '');
          return flow;
        });
      },
    },
  },
};

Warning

The dump hook is called after the mapping of the SyncIDs. This means that the data received by the hook is already tracked. If you filter out some elements, they will be deleted during the push command.

Filtering out elements

You can use onQuery hook to filter out elements. This hook is executed just before the query is sent to Directus, during the pull command.

In the example below, the flows and operations whose name starts with Test: are filtered out and will not be tracked.

// ./directus-sync.config.js
const testPrefix = 'Test:';

module.exports = {
  hooks: {
    flows: {
      onQuery: (query, client) => {
        query.filter = {
          ...query.filter,
          name: { _nstarts_with: testPrefix },
        };
        return query;
      },
    },
    operations: {
      onQuery: (query, client) => {
        query.filter = {
          ...query.filter,
          flow: { name: { _nstarts_with: testPrefix } },
        };
        return query;
      },
    },
  },
};

Warning

Directus-Sync may alter the query after this hook.

Using the Directus client

The example below shows how to disable the flows whose name starts with Test: and add the flow name to the operation.

const { readFlow } = require('@directus/sdk');

const testPrefix = 'Test:';

module.exports = {
  hooks: {
    flows: {
      onDump: (flows) => {
        return flows.map((flow) => {
          flow.status = flow.name.startsWith(testPrefix)
            ? 'inactive'
            : 'active';
        });
      },
    },
    operations: {
      onDump: async (operations, client) => {
        for (const operation of operations) {
          const flow = await client.request(readFlow(operation.flow));
          if (flow) {
            operation.name = `${flow.name}: ${operation.name}`;
          }
        }
        return operations;
      },
    },
  },
};

Snapshot hooks

Like the collections hooks, the snapshot hooks are defined in the configuration file using the hooks.snapshot property. Under this property, you can define the hook functions to be executed.

Available hook functions are: onLoad, onSave:

  • onLoad is executed during the push and diff processes, just after the data is loaded from the files, and before it is sent to Directus.
  • onSave is executed during the pull process, just before the data is saved to the files.

Note

This function can be asynchronous. It receives the snapshot object and the Directus client as parameters and must return the snapshot object.

Here is an example of a configuration file that exclude some fields when loading the snapshot. This will be similar for the onSave hook.

// ./directus-sync.config.js
module.exports = {
  hooks: {
    snapshot: {
      /**
       * @param {Snapshot} snapshot
       * @param {DirectusClient} client
       */
      onLoad: async (snapshot, client) => {
        // Remove some fields from the snapshot
        const fieldsToExclude = {
          my_model: ['date_created', 'user_created'],
        };
        const collections = Object.keys(fieldsToExclude);
        const nodeFilter = (node) => {
          const { collection } = node;
          return !(collections.includes(collection) && fieldsToExclude[collection].includes(node.field));
        }
        snapshot.fields = snapshot.fields.filter(nodeFilter);
        snapshot.relations = snapshot.relations.filter(nodeFilter);

        return snapshot;
      },
    },
  },
};

Note

For more information about the snapshot object, see the Snapshot interface.

Helpers

Untrack

npx directus-sync helpers untrack --collection <collection> --id <id>

Removes tracking from an element within Directus. You must specify the collection and the ID of the element you wish to stop tracking.

Remove permission duplicates

Permissions should be unique regarding the role, collection, and action. Unfortunately, Directus does not enforce this uniqueness. This can lead to unexpected behavior, such as missing ids or other permissions fields. More details can be found in the Directus issue #21965.

If you have permission duplicates, you can use the following command to remove them.

npx directus-sync helpers remove-permission-duplicates --keep <keep>
  • --keep <keep>: The permission's position to keep, first or last. The default is last.

This command will keep the last (or first) permission found and remove the others for each duplicated group role, collection, and action.

Lifecycle & hooks

Pull command

flowchart
subgraph Pull[Get elements - for each collection]
direction TB
B[Create query for all elements]
-->|onQuery hook|C[Add collection-specific filters]
--> D[Get elements from Directus]
--> E[Get or create SyncId for each element. Start tracking]
--> F[Remove original Id of each element]
-->|onDump hook|G[Keep elements in memory]
end
subgraph Post[Link elements - for each collection]
direction TB
H[Get all elements from memory]
--> I[Replace relations ids by SyncIds]
--> J[Remove ignore fields]
--> K[Sort elements]
-->|onSave hook|L[Save to JSON file]
end
A[Pull command] --> Pull --> Post --> Z[End]
Loading

Diff command

Coming soon

Push command

Coming soon

Tracked Elements

directus-sync tracks the following Directus collections:

  • dashboards
  • flows
  • folders
  • operations
  • panels
  • permissions
  • policies
  • presets
  • roles
  • settings
  • translations

For these collections, data changes are committed to the code, allowing for replication on other Directus instances. A mapping table links Directus instance IDs with SyncIDs, managed by the directus-extension-sync.

Roles & Policies

Roles and policies are tracked.
By default, Directus creates a default administrator role and two default policies: admin and public.

  1. To avoid recreating the default admin role, directus-sync uses the CLI user's role as the default admin role.
  2. To avoid recreating the default public policy, directus-sync uses the first policy found with role = null.
  3. To avoid recreating the default admin policy, directus-sync uses the first policy linked to the admin role with admin_access = true.

Presets

Global and role based presets are tracked (even the administrator role based presets). However, the users' presets are not tracked. This is because the users are not tracked and any relation with the users will cause conflicts.

Dependency: directus-extension-sync

To utilize the directus-sync tool, it is imperative to install the directus-extension-sync extension on your Directus instance. This extension acts as a bridge between directus-sync and Directus, managing the crucial mapping table that correlates the SyncIDs with Directus's internal IDs.

Installation

The directus-extension-sync must be added to each Directus instance involved in the synchronization process, whether as a source or a destination. Follow the installation instructions provided in the directus-extension-sync repository to add this extension to your Directus setup.

How It Works

directus-sync operates on a tagging system similar to Terraform, where each trackable element within Directus is assigned a unique synchronization identifier (SyncID). This system is key to enabling version control for the configurations and schema within Directus. Here is a step-by-step explanation of how directus-sync functions:

Tagging and Tracking

Upon execution of the pull command, directus-sync will:

  1. Scan the specified Directus collections, which include dashboards, flows, folders, operations, panels, permissions, policies, presets, roles, settings and translations.
  2. Assign a SyncID to each element within these collections if it doesn't already have one.
  3. Commit the data of these collections into code, allowing for versioning and tracking of configuration changes.

This SyncID tagging facilitates the replication of configurations across different instances of Directus while maintaining the integrity and links between different entities.

Note

The original IDs of the flows are preserved to maintain the URLs of the webhook type flows. The original IDs of the folders are preserved to maintain the associations with fields of the file and image types.

Tip

You can use the --preserve-ids option to preserve the original ids of some collections. Eligible collections are collections using UUID: dashboards, operations, panels, policies, roles and translations. If you have already used the pull command, you may use the untrack helper to remove the id tracking of an element before using this option.

Mapping Table

Since it's not possible to add tags directly to entities within Directus, directus-sync uses a mapping table that correlates the SyncIDs with the internal IDs used by Directus. This mapping is essential for the synchronization process, as it ensures that each element can be accurately identified and updated across different environments.

Synchronization Process

The synchronization process is split into two main commands:

  • diff: This command performs a comparison between the local JSON files generated by pull and the current state of a Directus instance. It outlines the elements that need to be created, updated, or deleted to achieve synchronization.

  • push: This command executes the actual synchronization plan, applying the necessary changes to the Directus instance. It handles dependencies and circular dependencies carefully by potentially running the synchronization process multiple times until the Directus instance is fully in sync with the JSON definitions.

Schema Management

The Directus schema, which defines the data modeling and user interface, is managed by the Directus API. However, for better code repository management, directus-sync stores the schema elements in separate files organized within a clear directory structure. This separation allows developers to easily track changes to the schema and apply version control principles to the database structure.

Non-Tracked Elements and Ignored Fields

Elements that are not meant to be tracked, such as user activities and logs, are not affected by the synchronization process. Certain fields are specifically ignored during synchronization because they are not relevant for version control purposes, such as creation dates and the identity of the user who created an entity.

Strengths of directus-sync

The strength of directus-sync lies in its ability to maintain consistent and reproducible configurations across multiple environments. It ensures that only the necessary changes are made, avoiding unnecessary recreation of configurations and maintaining the relationships between tracked and non-tracked entities. This selective updating is what makes directus-sync a robust tool for managing Directus instances in a team or multi-environment setup.

By following these mechanisms, directus-sync streamlines the development workflow, allowing for local changes to be efficiently deployed to various environments, all while keeping the Directus instances synchronized and version-controlled.

Directus upgrades

When upgrading Directus, it is important to update the configurations pulled by directus-sync. Here is a general guide to upgrading Directus using directus-sync:

  1. Run Directus on Version A: Start with your current Directus setup that is actively running on version A.
  2. Push Configuration (if needed): Use npx directus-sync push to push the latest configurations to your Directus instance. This step ensures that all your current settings are up-to-date on the server.
  3. Stop the Directus Server: Shut down your Directus server to prepare for the upgrade.
  4. Update to Version B: Change the Directus version to B in your configuration files or update scripts ( e.g., package.json, docker-compose.yml or Dockerfile).
  5. Restart and Migrate: Start the Directus server to initiate the upgrade. Directus will automatically run migration scripts necessary to update the database and apply new system configurations.
  6. Pull New Configuration: Once Directus is stable on version B, execute npx directus-sync pull to download the latest configuration snapshot. This action captures any changes or migrations that occurred during the upgrade from version A to B.

Use Cases

Troubleshooting