JSON select support for sqlite #536

oscar345 · 2025-12-09T09:56:19Z

Adds support for selecting json with sqlite. To create tests, I have copied the tests from mysql and added them to sqlite test folder. I made some tweaks to the tests, for example using a different test database for sqlite and making small changes to the sql syntax for the asserts. This PR also changes the qrm package for dealing with returned strings instead of bytes for json values in sqlite.

There are two tests I do not get working, and those are:

TestSelectJson_GroupBy: the returned json from sqlite has different values for items in the amount key. Some sums or averages are slightly different because of rounding errrors, but it does mean the test wont pass. For example:

Original from test

{
		"CustomerID": 2,
		"StoreID": 1,
		"FirstName": "PATRICIA",
		"LastName": "JOHNSON",
		"Email": "[email protected]",
		"AddressID": 6,
		"Active": "1",
		"CreateDate": "2006-02-14T22:04:36Z",
		"LastUpdate": "2019-04-11T18:11:49Z",
		"Amount": {
			"Sum": *128.73000000000002*,
			"Avg": *4.767777777777779*,
			"Max": 10.99,
			"Min": 0.99,
			"Count": 27
		}

JSON from sqlite json statement:

{
  "CustomerID": 2,
  "StoreID": 1,
  "FirstName": "PATRICIA",
  "LastName": "JOHNSON",
  "Email": "[email protected]",
  "AddressID": 6,
  "Active": "1",
  "CreateDate": "2006-02-14T22:04:36Z",
  "LastUpdate": "2019-04-11T18:11:49Z",
  "Amount": {
    "Sum": *128.73*,
    "Avg": *4.76777777777778*,
    "Max": 10.99,
    "Min": 0.99,
    "Count": 27
  }
}

The other test that fails needs more work, but i have not been able to solve it. The TestSelectJsonObj_NestedObj returns json but the nested json statements are not returned as json by sqlite but as strings containing json (so it is not a nested object but a string). Because of that the json unmarshal function does not work. To make sqlite return a nested object instead of a string, a JSON() statement should be added that wraps the subquery. So the query that fails is:

SELECT JSON_OBJECT(
          'actorID', actor.actor_id,
          'firstName', actor.first_name,
          'lastName', actor.last_name,
          'lastUpdate', strftime('%Y-%m-%dT%H:%M:%fZ', actor.last_update),
          'LongestFilm', (
               SELECT JSON_OBJECT(
                         'filmID', film.film_id,
                         'title', film.title,
                         'description', film.description,
                         'releaseYear', film.release_year,
                         'languageID', film.language_id,
                         'originalLanguageID', film.original_language_id,
                         'rentalDuration', film.rental_duration,
                         'rentalRate', film.rental_rate,
                         'length', film.length,
                         'replacementCost', film.replacement_cost,
                         'rating', film.rating,
                         'specialFeatures', film.special_features,
                         'lastUpdate', strftime('%Y-%m-%dT%H:%M:%fZ', film.last_update)
                    ) AS "json"
               FROM film_actor
                    INNER JOIN film ON (film.film_id = film_actor.film_id)
               WHERE actor.actor_id = film_actor.actor_id
               ORDER BY film.length DESC
               LIMIT ?
               OFFSET ?
          )
     ) AS "json"
FROM actor
WHERE actor.actor_id = ?;

But to make it work, the query should look like this:

SELECT JSON_OBJECT(
          'actorID', actor.actor_id,
          'firstName', actor.first_name,
          'lastName', actor.last_name,
          'lastUpdate', strftime('%Y-%m-%dT%H:%M:%fZ', actor.last_update),
          'LongestFilm', JSON((  -- <- notice the json function here
               SELECT JSON_OBJECT(
                         'filmID', film.film_id,
                         'title', film.title,
                         'description', film.description,
                         'releaseYear', film.release_year,
                         'languageID', film.language_id,
                         'originalLanguageID', film.original_language_id,
                         'rentalDuration', film.rental_duration,
                         'rentalRate', film.rental_rate,
                         'length', film.length,
                         'replacementCost', film.replacement_cost,
                         'rating', film.rating,
                         'specialFeatures', film.special_features,
                         'lastUpdate', strftime('%Y-%m-%dT%H:%M:%fZ', film.last_update)
                    ) AS "json"
               FROM film_actor
                    INNER JOIN film ON (film.film_id = film_actor.film_id)
               WHERE actor.actor_id = film_actor.actor_id
               ORDER BY film.length DESC
               LIMIT ?
               OFFSET ?
          ))
     ) AS "json"
FROM actor
WHERE actor.actor_id = ?;

copied and added to sqlite folder and altered to use a different database and sql syntax. This also changes the qrm package for dealing with returned strings instead of bytes for json values in sqlite.

oscar345 · 2025-12-10T13:39:19Z

I forgot to change the date and time format statement from mysql syntax to sqlite syntax

go-jet · 2025-12-11T19:11:45Z

@oscar345 Nice work

TestSelectJson_GroupBy: the returned json from sqlite has different values for items in the amount key. Some sums or averages are slightly different because of rounding errrors, but it does mean the test wont pass.

Not much we can do about it. It seems that json real numbers are serialized differently in sqlite json compared to sqlite driver.

To pass the test, we can unmarshal test data from customer_payment_sum.json into new test dest object and then compare objects for equality with float rounding.

The other test that fails needs more work, but i have not been able to solve it. The TestSelectJsonObj_NestedObj returns json but the nested json statements are not returned as json by sqlite but as strings containing json (so it is not a nested object but a string).

Since every nested json object has to be aliased, we can intercept alias call by implementing .AS() method on selectJsonStatement. If statement is of type JSON_OBJ we will wrap statement with JSON function and then call alias, otherwise we will just forward AS call to *selectStatementImpl.

go-jet · 2025-12-11T19:14:51Z

qrm/qrm.go

+					}
+
+					value = []byte(valueStr)
 				}


I don't think this code is ever called in your tests. This case is active only when SELECT_JSON appears as a column inside regular SELECT statement - wiki.

Hmm, now that I check, it seems that this test is missing for mysql.

It is needed though since I was using the select_json inside a regular select in my own code and otherwise there was convert error. I can copy the tests from postgres into sqlite to see check if all cases work correctly. The tests for mysql should probably be in another PR?

Yeah, that makes sense.

go-jet · 2025-12-11T19:17:00Z

qrm/qrm.go

+					}
+
+					value = []byte(valueStr)
 				}


Hmm, now that I check, it seems that this test is missing for mysql.

go-jet · 2025-12-11T19:20:21Z

sqlite/dialect.go

+				return CustomExpression(Token("strftime('%Y-%m-%dT00:00:00Z', "), e, Token(")"))
+			case BoolExpression:
+				return CustomExpression(Token("CASE"), e, Token("WHEN 1 THEN json('true') ELSE json('false') END"))
+			}


All this types needs to be covered by tests as well. Check TestAllTypesJSON test for mysql.

Okay I will take a look at the mysql tests and add them to the sqlite tests as well

oscar345 · 2025-12-11T21:46:52Z

Not much we can do about it. It seems that json real numbers are serialized differently in sqlite json compared to sqlite driver.

To pass the test, we can unmarshal test data from customer_payment_sum.json into new test dest object and then compare objects for equality with float rounding.

Should I just create a second file called json_customer_payment_sum.json with different rounding values?

Since every nested json object has to be aliased, we can intercept alias call by implementing .AS() method on selectJsonStatement. If statement is of type JSON_OBJ we will wrap statement with JSON function and then call alias, otherwise we will just forward AS call to *selectStatementImpl.

If I looking at the other stuff I will try to make this working but currently I am not totally sure if I can solve the problem. I will give an update about it, but maybe you can take a look at it afterwards @go-jet

go-jet · 2025-12-12T19:02:59Z

Should I just create a second file called json_customer_payment_sum.json with different rounding values?

That would give us two almost identical file, but we still wouldn't be sure they are truly identical.

If I looking at the other stuff I will try to make this working but currently I am not totally sure if I can solve the problem. I will give an update about it, but maybe you can take a look at it afterwards @go-jet

It shouldn't be that complicated. Something like:

func (s *selectJsonStatement) AS(alias string) Projection {
	if statementType == SelectJsonObjStatementType {
		return Func("JSON", s.selectStatementImpl).AS(alias)
	}

	return s.selectStatementImpl.AS(alias)
}

oscar345 · 2025-12-20T19:55:08Z

Small update: I have not been working on it this week, but I think I will finish it next week.

make the date 0000-01-01 when selecting a time.

json. However, since sqlite does not natively have a base64 function, a custom function is registered inside the main sqlite test.

sqlite requires that the json object is wrapped inside a json function, otherwise sqlite returns a json string, instead of an actual json object.

function for json objects, to deal with different rounding values between returned values from sqlite and json from sqlite, and some tests from postgres.

function was available. Also added json test data inside the test_sample.db.

oscar345 · 2025-12-28T22:20:59Z

When i made changes to the test data for sqlite (added a json and json_ptr column to the all types database), I think i have to make a PR for that seperate repo? That is the reason the tests are still failing.

The PR is not entirely done yet. I still have to clean up the code a bit. And one test is currently commented out since I could not get the same SQL with whitespacing between the actual sql and expected.

oscar345 added 2 commits December 9, 2025 10:34

Adds support for select json with sqlite. The tests from mysql are

934c02b

copied and added to sqlite folder and altered to use a different database and sql syntax. This also changes the qrm package for dealing with returned strings instead of bytes for json values in sqlite.

Update time and date expression from mysql syntax to sqlite syntax

42a0d9c

go-jet reviewed Dec 11, 2025

View reviewed changes

oscar345 added 5 commits December 28, 2025 22:56

change dialect file for sqlite, to make boolean values nullable and to

386f742

make the date 0000-01-01 when selecting a time.

Add a base64 encoding function to sqlite to deal with blobs inside of

1c89a01

json. However, since sqlite does not natively have a base64 function, a custom function is registered inside the main sqlite test.

When a select statement is a json object select inside another select,

b390687

sqlite requires that the json object is wrapped inside a json function, otherwise sqlite returns a json string, instead of an actual json object.

update tests for json_select in sqlite to adapt to the added json wrap

02a42f4

function for json objects, to deal with different rounding values between returned values from sqlite and json from sqlite, and some tests from postgres.

All datatypes inside json should be tested. This required that a base64

32e9a07

function was available. Also added json test data inside the test_sample.db.

Uh oh!

JSON select support for sqlite #536

Are you sure you want to change the base?

JSON select support for sqlite #536

Uh oh!

Conversation

oscar345 commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oscar345 commented Dec 10, 2025

Uh oh!

go-jet commented Dec 11, 2025

Uh oh!

go-jet Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

go-jet Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

oscar345 Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

go-jet Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

go-jet Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

go-jet Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

oscar345 Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

oscar345 commented Dec 11, 2025

Uh oh!

go-jet commented Dec 12, 2025

Uh oh!

oscar345 commented Dec 20, 2025

Uh oh!

oscar345 commented Dec 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

oscar345 commented Dec 9, 2025 •

edited

Loading

oscar345 Dec 11, 2025 •

edited

Loading