3.0

[wangmm0220]

Description

Issue(s)

• Close/close/Fix/fix/Resolve/resolve: Issue Link

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[gemini-code-assist]

Summary of Changes

Hello @wangmm0220, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant enhancements across the system, focusing on improved data security through encryption migration for core components, expanded data ingestion capabilities with Parquet file support, and refined query processing logic. The changes also streamline monitoring and user management features, contributing to a more robust and feature-rich platform.

Highlights

• Documentation Updates: Updated documentation across multiple files to reflect new features and improved explanations. This includes adding td.connect.token for TMQ consumers, updating taospy version to 2.8.8 with authentication support, detailing taosX Parquet file import, and enhancing CREATE/ALTER USER syntax with expanded IP/time range options. The SHOW XNODE commands now support WHERE clauses.
• Encryption Enhancements and Migration: Implemented encryption migration logic for SDB (mnode's internal database), BSE (vnode's block storage engine) tables, and stream checkpoints. This ensures that existing unencrypted data can be seamlessly migrated to an encrypted format upon restart when encryption keys are configured. Encryption key handling and buffer sizes were also standardized.
• Query Planning and Optimization Improvements: Refactored query planning for subqueries, introducing new node types (QUERY_NODE_REMOTE_VALUE_LIST) and improving how subquery results are handled. Optimized window function processing by removing WIN_OPTIMIZE_BATCH and WIN_OPTIMIZE_SINGLE hints and streamlining internal logic for virtual table window scans. Enhancements were made to join optimization and error handling for various query constructs.
• Slow Log Monitoring Refinements: The client-side slow log monitoring mechanism has been refactored for better file management, data sending, and queue handling. This includes new slow log queue types and improved error logging for monitor-related operations.
• Scalar Function and Operator Logic: Updated scalar function logic to handle NULL values more robustly in comparison operators and IN/NOT IN clauses. Improved type conversion mechanisms within vector operations and refined error messages for function usage.
• Xnode Management and Configuration: Added support for WHERE clauses in DROP XNODE JOB and SHOW XNODE commands. Improved xnoded executable location logic and environment variable passing for better management and debugging.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/new-framework-test.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This is a large pull request that introduces several new features, includes major refactorings, and fixes numerous bugs. Key feature additions include support for Parquet file import, token-based authentication for TMQ and Python connectors, and enhancements to SQL syntax like SHOW ... WHERE and subqueries in IN clauses. There are significant refactorings in configuration loading, slow log monitoring, and the query execution planner, particularly for virtual tables and window functions. A major focus is on enhancing data encryption across different components, including checkpoint files and block storage, with data migration paths for existing unencrypted data. The PR also contains many critical bug fixes, addressing memory leaks, logic errors, and potential crashes in the transport layer and management nodes. My review highlights a couple of potential issues in the new code concerning logging severity and IP whitelist logic. Overall, the changes are substantial and improve the system's functionality, robustness, and security.

[gemini-code-assist]

This logic appears to be incorrect based on the documentation for HOST/NOT_ALLOW_HOST. If a user has no IP whitelist/blacklist rules configured, taosHashGet will return NULL, and this function will return false, denying access. However, the documentation states: 'If neither HOST nor NOT_ALLOW_HOST is set, the user is allowed to log in from any address.' The implementation should be changed to return true in this case to align with the documented behavior.

  SWhiteUserList** ppList = taosHashGet(pWhiteList, user, strlen(user));
  if (ppList == NULL || *ppList == NULL) {
    return true;
  }

[gemini-code-assist]

Downgrading this log from uError to uWarn may cause developers to overlook potential bugs. An out-of-range array access is typically a programming error that should be treated as an error, not a warning. It's recommended to revert this to an error-level log to ensure such issues are highly visible and addressed promptly.

    uError("index is out of range, current:%" PRIzu " max:%" PRIzu, index, pArray->size);