[pull] master from mage-ai:master #11
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Fix performance issue of fetching variables. Avoid repeatedly calling `get_variables_dir` and read/parse the metadta.yaml. Set the default value of MEMORY_MANAGER_VERSION to 1 since it causes the slowdown of the variables API. # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally. Setting to MEMORY_MANAGER_VERSION reduces the variables API time from 27s to 1s. Reducing repeat get_variables_dir calls reduces the time from 1s to 250ms. # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
- Adapt dynamic blocks to use optionally use generators - Adapt to use new reader and writer for PyArrow - Fix a lot of UI - Variable meta info reading caching to optimize app performance - Memory tracking - Show block run runtime in the table view for Tim - A lot more --------- Co-authored-by: mager <[email protected]>
# Description - When editing a backfill, if user entered a value for "Max concurrent runs" but then deleted it and didn't enter anything else, there would be a `ValueError: invalid literal for int() with base 10: ''` error with the scheduler, causing the backfill to be stuck in the `initial` status. This PR fixes this. # How Has This Been Tested? - Locally # Checklist - [X] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [X] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
- Tests for dynamic block functions --------- Co-authored-by: mager <[email protected]>
# Description Problem: Runtime crashes occurred due to improper escaping of backslashes in raw string patterns used in code blocks, exemplified by: ``` foo = re.match(r'[A-Z]+(\d+)', segments[2]) ``` This resulted in errors such as: ``` server-1 | KeyError: '\\d' ``` The root cause was traced back to insufficient backslash escaping in the escaped_code variable during dynamic string operations. # Key Changes: - Enhanced Backslash Escaping: Updated the escaping mechanism to ensure correct handling of backslashes in dynamically evaluated strings. This change doubles backslashes in escaped_code, preventing runtime errors during regex operations. The adjustment specifically affects the __interpolate_code_content method, where re.sub() is utilized for string replacements. # How to Test: - Reproduce the Error: Before applying the changes, insert a raw string with an unescaped backslash in any Mage code block and execute it to observe the crash. - Verify the Fix: After implementing the changes, rerun the same block. The absence of the previous KeyError confirms the correction. # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: @wangxiaoyou1993 @dy46 @tommydangerous
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Fic circular dependency of importing from variable_manager file. <img width="971" alt="image" src="https://github.com/mage-ai/mage-ai/assets/80284865/b4b756fe-ff44-49f4-b3a9-920953f872e4"> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> * Speed up pipeline update endpoint by removing unnecessary pipeline file updates * Consolidate print output for scratchpad block * Speed up variables write (not call get_variables_dir duplicately) * Speed up yaml load with CLoader # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Reduce pipeline update time with same payload from 400ms to 100ms # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
- Dynamic blocks can run and process upstream data as they are created --------- Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Fix custom template creation. <img width="881" alt="image" src="https://github.com/mage-ai/mage-ai/assets/80284865/5da1f003-fa55-454c-8998-a123a544cf99"> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> * Catch exception in variables API * Let BaseEnum to also inherit from str so that the comparison between an enum value and a string is not always false. # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
) # Description 1) Motivation: It is helpful to be able to interpolate `PipelineSchedule.description` attribute in notification messages 2) Problem: Currently there is no way to interpolate `PipelineSchedule.description` attribute 3) Solution: Include a `pipeline_schedule_description` var that interpolates from `PipelineSchedule.description` # How Has This Been Tested? Locally Ran `scripts/dev.sh` with the following notification config in project metadata.yaml (send notifications to slack) ```yaml notification_config: alert_on: - trigger_success - trigger_failure - trigger_passed_sla slack_config: <<YOUR_SLACK_URL>> message_templates: success: details: | > Run Status: `Success ✅` > Environment: `{{ env_var('ENV') }}` > Pipeline UUID: `{pipeline_uuid}` > Pipeline Schedule ID: `{pipeline_schedule_id}` > Pipeline Schedule Name: `{pipeline_schedule_name}` > Pipeline Schedule Description: `{pipeline_schedule_description}` > Pipeline Run URL: `{pipeline_run_url}` > Execution Time: `{execution_time}` failure: details: | > Run Status: `Failure ❌` > Environment: `{{ env_var('ENV') }}` > Pipeline UUID: `{pipeline_uuid}` > Pipeline Schedule ID: `{pipeline_schedule_id}` > Pipeline Schedule Name: `{pipeline_schedule_name}` > Pipeline Schedule Description: `{pipeline_schedule_description}` > Pipeline Run URL: `{pipeline_run_url}?status=failed` > Execution Time: `{execution_time}` passed_sla: details: | > Run Status: `SLA Exceeded ⌛` > Environment: `{{ env_var('ENV') }}` > Pipeline UUID: `{pipeline_uuid}` > Pipeline Schedule ID: `{pipeline_schedule_id}` > Pipeline Schedule Name: `{pipeline_schedule_name}` > Pipeline Schedule Description: `{pipeline_schedule_description}` > Pipeline Run URL: `{pipeline_run_url}` > Execution Time: `{execution_time}` ``` Results: <img width="704" alt="Screenshot 2024-05-27 at 17 48 00" src="https://github.com/mage-ai/mage-ai/assets/37792010/60beff60-020e-439a-98ee-f1a9481637b4"> # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [X] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [X] I have made corresponding changes to the documentation cc: @wangxiaoyou1993 --------- Co-authored-by: hjhdaniel <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Disable autoreload in non-dev environments to reduce CPU usage. Profiled webserver CPU usage with library yappi. <img width="1356" alt="image" src="https://github.com/mage-ai/mage-ai/assets/80284865/31933907-b88d-414b-a58b-ae01950d0a45"> Columns in Profiling Output * ncalls: The number of calls to the function. If this shows two numbers separated by a slash (e.g., 6155/6794), it means the function was called recursively. The first number is the number of primitive calls, and the second number is the total calls. * tottime: The total time spent in the given function (excluding time made in calls to sub-functions). * percall (tottime): The total time divided by the number of calls (the first number in ncalls). This represents the average time spent in the function per call. * cumtime: The cumulative time spent in this function and all sub-functions (from invocation till exit). It is the sum of the tottime and the cumulative time of all sub-functions. * percall (cumtime): The cumulative time divided by the number of calls. This represents the average time spent in the function and its sub-functions per call. autoreload has a relatively high tottime and cumtime, suggesting it could be a good candidate for optimization. # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally. After disable autoreloading. The CPU usage went down from 5% to 0.15% when no requests are made. # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Reduce CPU usage in scheduler by reducing unnecessary yaml parsing <img width="1253" alt="image" src="https://github.com/mage-ai/mage-ai/assets/80284865/8ea24399-dca5-49b4-ab07-1a5fb511c3ee"> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Reduce max scheduler CPU from 70%+ to 12%+ # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> --------- Co-authored-by: mager <[email protected]>
 Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Bump version to 0.9.71 <!-- Release notes generated using configuration in .github/release.yml at release/0.9.71 --> ## What's Changed ### 🎉 Exciting New Features * [xh]Create streaming oracle destination by @matrixstone in #4896 ### 🐛 Bug Fixes * [xy] Exclude dropped columns from the postgres schema discovery query. by @wangxiaoyou1993 in #5002 * [xy] Fix None Bigquery database error. by @wangxiaoyou1993 in #5005 * [xy] Fix extracting table name from INSERT statement in SQL block. by @wangxiaoyou1993 in #5008 * [xy] Catch yaml interpolation error. by @wangxiaoyou1993 in #5012 * [xy] Not run insert query for empty dataframe. by @wangxiaoyou1993 in #5017 * [dy] Fix git and seed issues by @dy46 in #5025 * [xy] Fix custom template issue on server start. by @wangxiaoyou1993 in #5038 * [xy] Comment out flaky unit tests by @wangxiaoyou1993 in #5046 * [jk] Allow files to be selected for pipeline zip import by @johnson-mage in #5042 * [dy] Update git email field by @dy46 in #5045 * [dy] Fix circular dependency by @dy46 in #5050 * [td] Update charts and make a ton of bug fixes by @tommydangerous in #5075 * [td] Fix IDE outputs by @tommydangerous in #5088 * [td] Fix git issue by @tommydangerous in #5089 * Td fix git2 by @tommydangerous in #5090 * [td] Upgrade deltalake because of Docker image build error by @tommydangerous in #5091 * [xy] Fix websocket authentication. by @wangxiaoyou1993 in #5094 * [td] Fix interpolating dictionaries when adding custom code to ide execution by @tommydangerous in #5095 * [xy] Fix switching active kernel by @wangxiaoyou1993 in #5098 * [xy] Fix retry in streaming pipeline. by @wangxiaoyou1993 in #5099 * [xy] Fix performance issue of fetching variables. by @wangxiaoyou1993 in #5108 * [jk] Fix max concurrent runs value for backfill by @johnson-mage in #5115 * [xy] Fix circular dependency. by @wangxiaoyou1993 in #5122 * [td] Don’t raise error if can’t load by @tommydangerous in #5132 * [xy] Fix custom template creation. by @wangxiaoyou1993 in #5134 * [xy] Fix variables API and BaseEnum by @wangxiaoyou1993 in #5135 * [td] Fix after on pipeline dashboard by @tommydangerous in #5142 * [td] Fix after panel showing up everywhere by @tommydangerous in #5143 ### 💅 Enhancements & Polish * [xy] Support configuring logs_dir_path in environment variable. by @wangxiaoyou1993 in #5018 * [dy] Add prune option for fetch by @dy46 in #5014 * [xy] Include server and scheduler process id in status response. by @wangxiaoyou1993 in #5027 * [xy] Add pipeline_run_id to kwargs. by @wangxiaoyou1993 in #5030 * [xy] Allow configuring authentication mode in mssql config. by @wangxiaoyou1993 in #5051 * Add configuration parameters to V1JobSpec in k8s executor by @artche in #5044 * Reload libraries when running blocks from edit pipeline by @hugabora in #4953 * [xy] Support configuring concurrency config via env vars. by @wangxiaoyou1993 in #5057 * [xy] Some improvements on multi project by @wangxiaoyou1993 in #5063 * Add OIDC_ROLES_MAPPING support by @mihaivint in #5053 * [jk] Improve visibility of variable/secret names by @johnson-mage in #5072 * [xy] Cache project platform settings data in API calls. by @wangxiaoyou1993 in #5079 * [td] Memory upgrades by @tommydangerous in #5092 * [td] Track resource usage and run block execution function in a context by @tommydangerous in #5105 * [td] Dynamic blocks use the new memory enhancements by @tommydangerous in #5106 * [xy] Speed up pipeline update endpoint. by @wangxiaoyou1993 in #5125 * [td] Dynamic block stream mode by @tommydangerous in #5121 * [xy] Disable autoreload in non-dev environments. by @wangxiaoyou1993 in #5136 * [xy] Reduce CPU usage in scheduler. by @wangxiaoyou1993 in #5137 ### Other Changes * relax pytz version requirements by @alex-hunsaker in #5006 * [xy] Not show locals in CLI exception. by @wangxiaoyou1993 in #5009 * [dy] Remove error pop up by @dy46 in #5015 * Add support for user_roles when using OIDC auth by @mihaivint in #4899 * upgrade deltalake for tableMerger by @oonyoontong in #5020 * [dy] Update multi project platform to support active project per user by @dy46 in #4865 * docs: typo in architecture.mdx by @stefaan1o in #5034 * Update setup-dbt.mdx by @richardlhughes in #5033 * [tc] Update releases.mdx by @thomaschung408 in #5056 * [td] Fix charts when using Spark by @tommydangerous in #5028 * [td] Fix interactions, triggers, dynamic blocks, charts, and more by @tommydangerous in #5064 * [td] Remove PYTHONPATH from docker-compose by @tommydangerous in #5073 * [td] Turn off unoptimized images by @tommydangerous in #5082 * Grammar corrections to doc "Update non-root-user.mdx" by @CLHdevOps in #5077 * [td] Fix infer variables by @tommydangerous in #5102 * [td] Dynamic block resilience by @tommydangerous in #5113 * Escaping Backslashes when interpolating code content by @jimilp7 in #5112 * Add pipeline_schedule_description variable to notification sender by @hjhdaniel in #5118 # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Comment tests locally - [x] Performance tests # Checklist N/A cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Update pypi github action. * Set the version of the package instead of master branch * Set verbose to true * Set skip-existing to true # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested in github action # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
…t feature (#5144) # Description In multi-project the get_repo_path() in the settings lib returns the active project repo path. This makes it hard to read config files using generic code. Having the repo path in kwargs makes it easy to use that in blocks. Having it in os.environ local to python subprocesses, allows libraries to access this. This works with local executors. A different solution may be appropriate for spark and cloud executors. # How Has This Been Tested? Tested in our local multi-project codebase. # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [X] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [X] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: @wangxiaoyou1993
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Fix delete workspace. `self.current_user` is not defined in the resource class. # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested in workspace management page # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
#5131) POST `/api/code_executions` - Execute client provided code - As code executes, output is incrementally added to a queue - Results are dequeued and broadcasted to subscribers who are reading text stream data from `/event-streams/[uuid]` --------- Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> --------- Co-authored-by: mager <[email protected]>
# Description - Change `ENV_TEST` value back to `test` due to issue with triggers in code not loading because `test_mage` was not considered a valid environment. - Keep `ENV` value as `test_mage` when running unit tests, but update code in various places to make it backwards compatible with `test` environments. # How Has This Been Tested? - Confirmed that triggers in code load and are created as expected with various `envs` values (e.g. A trigger in code with `envs` value of `['dev']` will be created in `dev` environment but NOT `test` environment.) - Confirmed that `test` environments can still use the Postgres metadata db, while `test_mage` environments use a `test.db` sqlite metadata db for running unit tests. # Checklist - [X] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [X] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
Fixed typo # Description differnt > different
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: mager <[email protected]>
# Description - Add docs for overriding project/pipeline configurations based on environment. # How Has This Been Tested? - Tested locally.   # Checklist - [X] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [X] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [X] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description Address the issue reported in #5556. # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [X] Checked the configuration settings against the fields in its class definition # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> @wangxiaoyou1993
# Description # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Adds an information box to the Data Integrations>Batch Pipelines page detailing the dependency of the `Data integrations in batch pipeline` feature on `Add new block v2`. This closes #5583 # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> Locally rendered and checked. # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] ~I have added unit tests that prove my fix is effective or that my feature works~ - [ ] ~I have commented my code, particularly in hard-to-understand areas~ - [x] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
This PR updates the frontend build artifacts. Co-authored-by: GitHub Actions <[email protected]>
# Description this include the implementation of airtable destination # How Has This Been Tested? - [x] Test Destination UI # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation cc: @wangxiaoyou1993 --------- Co-authored-by: wangxiaoyou1993 <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [ ] Test A - [ ] Test B # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [ ] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Bump up version to 0.9.75 <!-- Release notes generated using configuration in .github/release.yml at master --> ## What's Changed ### 🎉 Exciting New Features * Airtable Destination by @TalaatHasanin in #5454 * [hw] Support Mage in Python 3.11 and 3.12 by @csharplus in #5393 * Add postgres client package by @jx2lee in #5486 * [enhancement] Update Elasticsearch + verify_certs connection option by @syepes in #5462 * [enhancement] adds support of _op_type for bulk operations of destinations by @syepes in #5482 * [enhancement] Elasticsearch support of _op_type for bulk operations by @syepes in #5471 * Add support to multiples webhook endpoints for Microsoft Teams notification service by @messerzen in #5508 * [hw] Update Redshift connector to enable merge load and correct row count by @csharplus in #5522 ### 🐛 Bug Fixes * [xh]Pass OpenAI API key to OpenAI Library by @matrixstone in #5430 * [jk] Fix error detail log parsing by @johnson-mage in #5443 * [jk] Render json objects in block outputs as string instead of nested table by @johnson-mage in #5450 * [jk] Block output table cell overflow by @johnson-mage in #5451 * [Bug] Updated SalesForce source to handle multiple date formats by @tolson17 in #5493 * [jk] Handle encoded page_block_layouts and block_outputs routes by @johnson-mage in #5544 * [jk] Revert name of test environment by @johnson-mage in #5478 * [hw] Correct the SSL settings in the nats configuration template by @csharplus in #5579 * Fix encoding in GCS Integration by @TalaatHasanin in #5447 * [enhancement] Elasticsearch - Align the doc publishing with the Standard (batch) by @syepes in #5510 * [Enhancement] io/base.py: add Excel support by @LucasGrugru in #5542 * [hw] Update deltalake to a recent version `0.20.2` by @csharplus in #5541 * bugfix/#5562 python data exporter failed insert 2d array to postgres by @sugimiyanto in #5563 * Fix unable to pass extra parameters to psycopg2 in Postgres connector by @kanenorman in #5449 * [hw] Avoid throwing exceptions when `block_type` is None by @csharplus in #5439 * Fix load data in GCS integration by @TalaatHasanin in #5467 ### 💅 Enhancements & Polish * [xh] Update python version to fix vulnerabilities by @matrixstone in #5523 * [jk] Display ID in block runs table by @johnson-mage in #5457 * [Oracle DB] Modify inefficient code that converts dataframe to list of tuples by @farmboy-dev in #5502 * [jk] Allow user to set number of lines displayed for block output sample preview by @johnson-mage in #5485 * [hw] Use `self.logger.debug` to replace `print` to make code clean by @csharplus in #5494 * Updated Facebook Ads SDK to 20.0.2 by @jonatansthlmstratlab in #5437 ### Other Changes * Update documentation for Git in Mage in the Getting Started section by @Lennardvb in #5459 * [hw] Use the built-in IntEnum instead of creating a new class by @csharplus in #5433 * [hw] Add time delay to avoid getting the same file timestamp for multiple changes by @csharplus in #5499 * Update text in documentation about enabling HTTPS in AWS by @Lennardvb in #5517 * Fix typos in compute-resource.mdx by @mvillaizan in #5519 * Update alerting-teams.mdx for multiples webhooks. by @messerzen in #5511 * Update README.md by @neubert-analytics in #5441 * Typo fix ai-client.mdx by @MageKai in #5582 * Add information regarding feature dependency to docs by @oscarlofwenhamn in #5584 ## New Contributors * @jonatansthlmstratlab made their first contribution in #5437 * @Lennardvb made their first contribution in #5459 * @syepes made their first contribution in #5471 * @jx2lee made their first contribution in #5486 * @mvillaizan made their first contribution in #5519 * @farmboy-dev made their first contribution in #5502 * @tolson17 made their first contribution in #5493 * @neubert-analytics made their first contribution in #5441 * @kanenorman made their first contribution in #5449 * @LucasGrugru made their first contribution in #5542 * @sugimiyanto made their first contribution in #5563 * @oscarlofwenhamn made their first contribution in #5584 **Full Changelog**: 0.9.74...0.9.75 # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Basic tests locally # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
# Description I deleted the destination redirect json item for streaming sources and destination links not working # How Has This Been Tested? All the links are working on mintlify dev env, see photos of some sources     - [ ] Test A - [ ] Test B # Checklist - [X] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [X] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [X] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> Co-authored-by: Cole Freeman <[email protected]>
# Description Address the upgrade request in #5525. # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested with Github builds and tests # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> @wangxiaoyou1993
# Description Added destinations back to redirects for streaming links # How Has This Been Tested? Tested in mitlify dev env - [ ] Test A - [ ] Test B # Checklist - [X] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [X] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [X <img width="403" alt="Screenshot 2024-12-13 at 4 12 31 PM" src="https://github.com/user-attachments/assets/0afcaf7e-1375-4e44-94c0-3f72528d106f" /> <img width="1430" alt="Screenshot 2024-12-13 at 4 14 45 PM" src="https://github.com/user-attachments/assets/f372353c-7130-428b-96d4-bca49fca2067" /> <img width="1613" alt="Screenshot 2024-12-13 at 4 15 04 PM" src="https://github.com/user-attachments/assets/b47b52e7-0d40-4d7b-b6cd-37eecdd5d839" /> ] I have made corresponding changes to the documentation cc: @johnson-mage --------- Co-authored-by: Cole Freeman <[email protected]>
# Description Use a recent version of nkeys as the current version is too old, and causes build issues in Python 3.12. # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Github Builds and Tests # Checklist - [ ] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> @wangxiaoyou1993
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Move help improve_mage check outside of asyncio.run so that scheduler isn't stuck in asyncio.run Close: #5621 # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: @tommydangerous @johnson-mage <!-- Optionally mention someone to let them know about this pull request -->
# Description Update links in docs Co-authored-by: mager <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Support hiding and rotating API trigger token. * It can be enabled with env var `HIDE_API_TRIGGER_TOKEN=true` * Once it's enabled, any API calls to pipeline schedule API won't return the token in response. API trigger token won't be shown in the UI <img width="638" alt="image" src="https://github.com/user-attachments/assets/d88416d7-bf92-42fd-929f-b02359702132" /> * API trigger token will only be returned once on PipelineSchedule UPDATE request with `"rotate_token": true" in the paylod * How to rotate token ``` PUT /api/pipeline_schedules/[pipeline_schedule_id] { "pipeline_schedule": { "rotate_token": true } } ``` # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally <img width="446" alt="image" src="https://github.com/user-attachments/assets/6b1b6ba3-99d7-4eba-94f6-2c491d88439e" /> # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: @tommydangerous @johnson-mage <!-- Optionally mention someone to let them know about this pull request -->
# Description Added Mage Pro documentation to the getting started page # How Has This Been Tested? This was tested in mintlify dev. env. - [ ] Test A - [ ] Test B # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation cc: @johnson-mage <img width="1765" alt="Screenshot 2025-01-03 at 1 46 12 PM" src="https://github.com/user-attachments/assets/a7ccfd70-f508-48ce-acc1-d3f7e4e0b8e2" /> <img width="1784" alt="Screenshot 2025-01-03 at 1 46 32 PM" src="https://github.com/user-attachments/assets/45278c87-a1c0-4f56-abeb-6fd789eb6b2f" /> <img width="1787" alt="Screenshot 2025-01-03 at 1 46 53 PM" src="https://github.com/user-attachments/assets/bf89aa2b-d192-4261-97b8-6898b4f6fa5c" /> New screenshots of UI to follow Co-authored-by: Cole Freeman <[email protected]>
# Description This PR updates the documentation for mage pro environment variables and the getting started page. # How Has This Been Tested? The documentation is working on mintlify dev env. - [ ] Test A - [ ] Test B # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request --> <img width="1155" alt="Screenshot 2025-01-06 at 4 02 23 PM" src="https://github.com/user-attachments/assets/2e80e26a-0e4f-478a-a09d-a84253b709a2" /> <img width="840" alt="Screenshot 2025-01-06 at 4 02 53 PM" src="https://github.com/user-attachments/assets/3278d889-6d92-4b7f-ad71-250ba3835609" /> <img width="773" alt="Screenshot 2025-01-06 at 4 03 05 PM" src="https://github.com/user-attachments/assets/eff29abe-3f45-4b44-be7f-9b3d45887705" /> Co-authored-by: Cole Freeman <[email protected]>
# Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Support configuring Redis default timeout. * Set `REDIS_LOCK_DEFAULT_TIMEOUT` env var to number of seconds for the lock timeout Close: #5619 # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
) # Description <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> Support passing chunksize and method to mssql export method. Close: #5636 # How Has This Been Tested? <!-- Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. --> - [x] Tested locally # Checklist - [x] The PR is tagged with proper labels (bug, enhancement, feature, documentation) - [x] I have performed a self-review of my own code - [ ] I have added unit tests that prove my fix is effective or that my feature works - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation cc: <!-- Optionally mention someone to let them know about this pull request -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )