Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2.0.0 #90

Open
wants to merge 21 commits into
base: master
Choose a base branch
from
Open

v2.0.0 #90

wants to merge 21 commits into from

Conversation

toptobes
Copy link
Collaborator

No description provided.

toptobes and others added 20 commits September 29, 2024 20:26
* remove deprecated vector/vectorize parameters

* remove bulkWrite

* remove deleteAll

* removed namespace terminology

* removed db.collections()

* removed client.db(id, region)

* update api report
* some experimental table typing

* dark magic

* added default type for table

* made InferTableSchema more flexible

* moved table.ts into its own folder (mirrors collections)

* more type errors

* broke up table.ts file in to its proper structure

* more work on tavles idk

* start work on common command impls class

* fixed couple rebasing errors

* moved all collection functions to generic internal CommandsImpls object

* fixed all of the bugs I introduced in the previous commit :)

* more implmeennation work

* added table methods for db as well

* added types (but not impl) for alterTable

* add countRows

* add more missing table functions

* added various datatypes (dates & ip addrs)

* update build report

* update build report

* some datatype tweaks (mostly for InetAddress)

* some type operations when translating cql types
* various cursor tweaks & fixes

* minor typing tweaks

* fixed bug with filter being potentially muitable

* made cursors immutable

* some tests not all idk
* started work overhauling validation logic

* so much validation

* some folder resturcuting

* loggign hierarchy

* basic logging impl

* implement warning events

* tests for logging and such
* documented table-schema file

* tsdoc work

* remove checkExists

* ser/des work

* playground script

* started adding custom inspects and work on datetimes

* made _httpClient fully public

* work on DataAPIVector stuff

* some resturcturing

* made all Promise<true> methods just reutrn Promise<void>

* removed rackstackk hack

* cqlblob type

* additionalHeaders

* started documenting table-related stuff

* formatting for events logging

* documentation for logging

* document collection class

* created CursorError

* set up test suite for table tests

* move dropIndex to db level

* lot of work on bignumbers hack...

* remove CollectionNotFoundError

* added table.deifnition()

* super basic insertOne test

* basic findone tests

* toomanyrowstocount error

* start documenting serdes

* split cumulative errors + some more bignumber serdes work

* changed $PrimaryKeyType to be a string + some test typing fixes

* added sparse data support
* make DataAPIDbAdmin keyspace options no longer extend AdminBlockingOptions

* refactor raw db info into more workable objects

* more unified naming convention of admin interfaces

* bit of name tweaking

* some admin info itnerface tweaking

* light, temp documentation
* split cursor classes

* little bit of cleanup

* moved cursors test to documents/collections

* unit tests for split cursors

* integration tests for split cursors
* timeouts overhaul

* keyspace impl & tests

* more tests & tweaks

* fix timeout/sort bgus

* refined WithTimeout types

* createCollection custom timeout impl

* more docs and stuff
* a bunch of tests work

* couple minor test fixes
* more intuitive naming for events/logging stuff

* formatting + timestamps for log messages

* dropIndex ifExists

* sourceModel

* createindex options resturcuting

* split filter & update types

* remove cql from datatypes names

* timeout for cursor.toArray() & coll.distinct()

* added class names for admin event name sourcse

* remove "spawn" from spawn type names
* updated serdes

* changed tokne provider a bit

* minor linting fix

* fixed couple import/typing issues + tables readme

* add shorthand datatype functions

* update readme w/ shorthand datatype fns

* marked mroe internal values

* removed deeppartial & strict filters/sorts/projs

* tiny bit of renaming
* updated serdes

* add shorthand datatype functions

* update readme w/ shorthand datatype fns

* marked mroe internal values

* start documentation of tables

* added listIndexes

* some altertable fixes

* bump min node version to v18+
* advanced typings for tables/colls; typings for includeSimialrity

* fixed tables typing test file

* update readme
* switch to codec-based ser/des system

* camel snake case interop

* example and many bug fixes and tweaks and stuff idk lol

* added serdes path matching & class-mapping example
* some fixes/work

* some ttyping work

* added tsdoc everyuwhere for the msot part

* reset example astardbts versions

* few more tests

* minor uipdates to examples
* update uuid stuff + start datatypes tests

* reexport bignumber

* some datatype tests
* unficiation of codec types

* snakeCaseInterop => keyTransformer

* some unit tests on ser-des options
@vkarpov15
Copy link
Contributor

Overall looks reasonable, just had a couple of suggestions/thoughts:

  1. I thought the plan was to make httpClient public on client and db?
  2. It would be great to improve error message when mixing up collections and tables. For example, if you try to insert a doc using table logic into a collection, you end up with the following error message:
    TypeError: Cannot convert undefined or null to object
      at Function.entries (<anonymous>)
      at TableSerDes.adaptDesCtx (node_modules/@datastax/astra-db-ts/dist/documents/tables/ser-des/ser-des.js:38:53)
      at TableSerDes.deserializeRecord (node_modules/@datastax/astra-db-ts/dist/lib/api/ser-des/ser-des.js:51:26)
      at CommandImpls.insertOne (node_modules/@datastax/astra-db-ts/dist/documents/commands/command-impls.js:50:38)

@toptobes
Copy link
Collaborator Author

toptobes commented Dec 22, 2024

I thought the plan was to make httpClient public on client and db?

DataAPIClient doesn't actually have an HttpClient, but it is public on Db, Table, Collection, and the three *Admin classes as ._httpClient.

The _ prefix remains to keep it out of autocomplete and to signify that it's really not a common property you should be accessing and working with.

Anyways, you shouldn't need the Feature-Flag hack anymore anyways since tables are enabled by default now (right?). But, if you did need any other Feature-Flag in the future, there's a new dbOptions.additionalHeaders options for specifying that sort of thing in the DataAPIClientOptions (or just additionalHeaders in DbOptions)

For example, if you try to insert a doc using table logic into a collection, you end up with the following error message...

Oh yeah I forgot about this case, I'll add some checks that'll basically say "hey you inserted a collection doc into a table and it succeeded but we failed to parse the response since you're using the table class", or, for finds, "hey you tried to find on a table but it was actually a collection so we failed".

Thanks for pointing this one out 👍

If you have any other thoughts/suggestions, no matter how nitpicky or small, please do share

* start work on more docs md pages

* created colls/tables datatype cheatsheet

* did collections dts

* readme work

* ran npm audit fix

* minor tweaks to DATATYPES.md

* bnunch of tsdoc for create-table related types

* tsdoc for DataAPILoggingDefaults

* update examples to use @next version of astra-db-ts

* documented list-tables
@vkarpov15
Copy link
Contributor

A couple of other issues I ran into:

  1. DataAPITimestamp's SerializeForCollection appears to serialize DataAPITimestamps as { $date: string } rather than { $date: number }, which leads to DataAPIResponseError: Bad JSON Extension value: Date ($date) needs to have NUMBER value, has STRING when trying to save a DataAPITimestamp into a collection.
  2. Why doesn't table serialization automatically handle converting JavaScript dates into timestamps?

@toptobes
Copy link
Collaborator Author

  1. Ope, thanks for pointing that out, definite bug
  2. It uses DataAPITimestamp for symmetry with the other dates/durations; also ensures people realise Datedate. That being said, I suppose I could still allow you to insert as Dates, but you'd still read them back as DataAPITimestamps. Also if you really want to use Dates instead of Timestamps for both r/w, there's a codec for that in TableCodecs, but it's undocumented atm since the ser/des options are still in beta

@vkarpov15
Copy link
Contributor

Re: DataAPITimestamp point (2), I would recommend automatically serializing JS dates to DataAPITimestamp because they're fundamentally the same type: integer containing milliseconds since epoch. Not much reason to make a distinction between the two. Is there even a reason to have a separate DataAPITimestamp type?

Right now sending a JS Date to insertOne() results in a "Error trying to convert to targetCQLType TIMESTAMP from value.class java.util.HashMap, value {}. Root cause: no codec matching value type." error, so I think the two options are (1) serialize Dates automatically, (2) throw a more readable error indicating that Dates are not serialized automatically. The current error is not great because users don't know why dates are getting serialized into hashmaps.

Some more notes:

  1. It doesn't look like I can import TableIndexDescriptor, and without that I can't use listIndexes() because of error Return type of public method from exported class has or is using name 'TableIndexDescriptor' from external module "node_modules/@datastax/astra-db-ts/dist/astra-db-ts" but cannot be named.
  2. There's a createIndex() method on the Table class, why is there no dropIndex()? Looks like there's just a dropTableIndex() on db, but that seems inconsistent.
  3. No BigInt support for tables? Looks like enableBigNumbers is only allowed on collections. Also, why is BigInt support behind a flag?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants