Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 [firestore-bigquery-export] fs-bq-import-collection can crash with wildcard notation #2202

Open
dhlanm opened this issue Oct 23, 2024 · 0 comments
Labels
type: bug Something isn't working

Comments

@dhlanm
Copy link

dhlanm commented Oct 23, 2024

[REQUIRED] Step 2: Describe your configuration

  • Extension name: ** firestore-bigquery-export **
  • Extension version: 0.1.51
  • Configuration values (redact info where appropriate):
    --table-name-prefix=mycollectionname
    --source-collection-path=some/{someid}/path/{pathid}/mycollectionname
    --query-collection-group=true
    --non-interactive
    --dataset=firestore_export
    --dataset-location=us_west2
    --project=myproject
    --big-query-project=myproject

[REQUIRED] Step 3: Describe the problem

Say you have the setup described in the docs here, with collections users/user1/pets and users/user2/pets, but also admins/admin1/pets. If you set ${COLLECTION_GROUP_QUERY} to true and provide the collection path as ${users/{uid}/pets}

However, consider that collection admins/admin1/pets is many times larger than the user collections. Then, the code here may get a page of query results with only admin pets, which will be filtered out, resulting in an empty rows variable. This crashes the script with Error importing Collection to BigQuery: Error: You must provide at least 1 row to be inserted.

Steps to reproduce:

Create the situation above, and run the export script.

Expected result

The export should work as normal.

Actual result

The script crashes.

@dhlanm dhlanm added the type: bug Something isn't working label Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant