90%: Add computed fields #40

rsinger · 2015-02-24T10:52:32Z

This is a spike to throw up a proof of concept to allow generation of dynamic table values based on the values elsewhere in the generated table row document.

It also introduces the notion of ‘temp’ fields that aren’t stored in the collection: these fieldNames have a "temporary": true property.

Example of a table spec with this

{
    "_id": "t_wibble",
    "type": "owl:Thing",
    "from": "CBD_foobar",
    "fields" : [
        {
            {
                "fieldName": "foobarLink",
                "value": "link"
            },   
            {
                "fieldName": "mainType",
                "predicates": ["rdf:type"],
                "temporary": true
            },
            {
                "fieldName": "lastModMain",
                "predicates": ["dct:modified"],
                "temporary": true
            },
            {
                "fieldName": "mainTitle",
                "predicates": ["dct:title"],
                "temporary": true
            }
        }
    ],
    "joins": {
        "dct:hasPart" : {
            "fields" : [
                {
                    "fieldName": "partTitle",
                    "predicates": ["dct:title"],
                    "temporary": true
                },
                {
                    "fieldName": "partType",
                    "predicates": ["rdf:type"],
                    "temporary": true
                },
                {
                    "fieldName": "lastModPart",
                    "predicates": ["dct:modified"],
                    "temporary": true
                }
            ]
        },
        "dct:isPartOf" : {
            "fields" : [
                {
                    "fieldName": "partOfTitle",
                    "predicates": ["dct:title"],
                    "temporary": true
                },
                {
                    "fieldName": "partOfType",
                    "predicates": ["rdf:type"],
                    "temporary": true
                },
                {
                    "fieldName": "lastModPartOf",
                    "predicates": ["dct:modified"],
                    "temporary": true
                }
            ]
        },        
    },
    "counts": [
        {
            "fieldName": "referenceCount",
            "property": "dct:references"
        },
        {
            "fieldName": "referencedCount",
            "property": "dct:isReferencedBy"
        }
    ],
    "computed_fields": [
        {
            "fieldName": "title",
            "value": {
                "conditional": {
                    "if":[["bibo:Article", "bibo:Chapter"], "contains", "$mainType"],
                    "then":"$mainTitle",
                    "else": {
                        "conditional":{
                            "if":[["bibo:Article", "bibo:Chapter"], "contains", "$partType"],
                            "then":"$partTitle",
                            "else":{"conditional":{
                                "if":[["bibo:Article", "bibo:Chapter"], "contains", "$partOfType"],
                                "then":"$partOfTitle",
                                "else":"Unknown!!"
                            }}
                        }
                    }
                }
            }
        },
        {
            "fieldName": "type",
            "value": {
                "replace": {
                    "search":"bibo:",
                    "replace":"",
                    "subject":"$mainType"
                }
            }
        },
        {
            "fieldName": "referenceDiff",
            "value": {
                "conditional": {
                    "if":["$referenceCount",">","$referencedCount"],
                    "then":{
                        "arithmetic":["$referenceCount","-","$referencedCount"]
                    },
                    "else":{
                        "arithmetic":["$referencedCount","-","$referenceCount"]
                    }
                }
            }
        }
    ]            
}

Example of document outputted above

{
    "foobarLink": "http://example.com/1234",
    "referenceCount": 32,
    "referencedCount": 14,
    "title": "I was the isPartOf title",
    "type": "Book",
    "referenceDiff": 18
}

malcyL · 2015-02-25T11:57:25Z

src/mongo/delegates/MongoTripodTables.class.php

+ break;
+ default:
+ $value = null;
+ }


Arithmetic operators contains: "+", "-", "", "/", "%", "*"

Did we miss "**" from this switch?

No - it should have been removed from the valid operators: exponents were introduced in PHP 5.6, so I don't think we should support them here.

malcyL · 2015-02-25T12:19:21Z

Reading through this, I was thinking there is a lot of parsing code - is there some library in PHP that might do this for us? A PHP yacc/bison? I guess we are already parsing the tablespec like this, and this is just extending what we have.

So - other than my question above on "**" - 👍

rsinger · 2015-02-25T12:25:44Z

I was also concerned about the size of the validation code (which I think is what you mean about parsing), especially given that it needs to be done on every request. I sort of wonder about keeping them, but making the actual bit that runs on MongoTripod::loadConfig() much smaller and put a more comprehensive one as a script.

rsinger · 2015-02-25T12:26:14Z

The only downside to this is that if your tripod config is generated on the fly (like it is in TARL).

lordtatty · 2015-02-25T12:44:49Z

My first thoughts on this is that we seem to have an element of "code is data, and data is code" hovering around here, and this is a problem which has already been solved by languages such as LISP. I have found a couple of sites where people are discussing embedding conditionals within JSON, like we are trying to do, and lisp-like s-expressions seem to be approaches which are favoured, as they directly translate to this problem. http://alandipert.tumblr.com/post/7193534410/discover-lisp-in-your-web-browser-with-javascript - part way down this page the author proposes a JSONscript language (I googled - it doesn't actually exist yet), but I wonder if it might provide some interesting ideas for us here. Perhaps we could write a simple parser for JSONscript and open source it. There is a stack overflow post here where the top answer is proposing a similar design: http://stackoverflow.com/questions/20737045/representing-logic-as-data-in-json

Having said that, there is nothing inherently wrong with this approach if we are too far down the rabbit hole to change it now - I was just thinking in terms of standing on the shoulders of giants, and such. It would also help with the above thoughts on validation by separating out that code from tripod itself.

rsinger · 2015-02-25T13:22:02Z

@lordtatty Are you proposing that the "value" value for the fieldName "type" should look something like:

["_replace_",["bibo","","$mainType"]]

and the "value" value for fieldName "referenceDiff" should look like:

[
    "_conditional_",
    [
        "if",
        [
            ">",
            ["$referenceCount", "$referencedCount"]
        ],
        "then",
        [
            "_arithmetic_",
            [
                "-",
                ["$referenceCount","$referencedCount"]
            ]
        ],
        "else",
        [
            "_arithmetic_",
            [
                "-",
                ["$referencedCount","$referenceCount"]
            ]
        ]
    ]
]

lordtatty · 2015-02-25T13:38:10Z

It's been a little while since I touched something like lisp myself, but for the latter I was thinking something along the lines of:

        {
            "fieldName": "referenceDiff",
            "value": {
                "_evaluated_": [
                    "-",
                    ["MAX", "$referenceCount", "$referencedCount"],        
                    ["MIN", "$referenceCount", "$referencedCount"],               
                ]
            }
        }

Maybe could make it easier a function like MAXMIN so we don't need to do two separate MAX and MIN calls, but in terms of raw functionality, something like that.

…/counts

rsinger · 2015-03-24T13:22:05Z

I think all of the comments here have been addressed:

Docs updated to include computed field functions and temporary fields
_function_ has been removed in favor of just 'conditional', 'replace', etc. 'link' has been deprecated in favor of 'link' and now logs a warning. Also log warning was added
Functions (predicate/computed field) need to be consolidated and modularized #44
The naming convention for temporary fields has been replaced with adding a 'temporary': true property on the fieldspec

rgubby · 2015-03-25T09:46:02Z

docs/primers/tables.md

+ {
+ "fieldName": "type",
+ "value": {
+ "_replace_" : { // _replace_ is a function name


Should the underscores be removed?

rgubby · 2015-03-25T09:53:41Z

src/mongo/MongoTripodConfig.class.php

@@ -296,6 +318,14 @@ protected function loadConfig(Array $config)
 {
 throw new MongoTripodConfigException("View spec does not contain " . _ID_KEY);
 }
+ if(!isset($spec['from']) || !in_array($spec['from'], $this->getPods($storeName)))


You've got a validateTableSpec function to validate table specs, but validation of view specs (and search specs) are done inline. Worth moving to a function to keep it consistent?

Originally, I had done this, but, actually, the amount of validation done for search and view specs is the same as we do inline for table specs.

Really, it's just that we do a lot more validation for table specs.

scaleupcto · 2015-03-25T10:07:39Z

👍

rgubby · 2015-03-25T10:43:49Z

docs/primers/tables.md

+"then" and "else" values may be of any type, including another function. To replicate 'else if', you can use another
+ conditional function as the value of 'else'.
+
+ Arithmetic functions


Do you want something in here to tell you what arithmetic functions are available?

rgubby · 2015-03-25T10:55:23Z

docs/primers/tables.md

+ "replace": {
+ "search": "bibo:",
+ "replace": "",
+ "subject": "$x"


I noticed in the PR that uses this functionality that you have something like:
"subject": "$!concatenatedType" - what does that do?

I think that "!concatenatedType" is the name of the field - just confused me a bit as it looked like it could be an operator..

That was based on an earlier version of this branch.

READ THE COMMENTS, @rgubby!

rgubby · 2015-03-25T11:59:28Z

looks good to me! 👍

…-data 90%: Add computed fields

rsinger added 5 commits February 24, 2015 10:50

Add computed fields

abbeead

Add a replaceValue function

f38dc4d

Added arithmetic function; ability to cast field value type

178bfbf

Left out a case break

550ca6b

Missing docblock

b9b2ef1

rsinger mentioned this pull request Feb 24, 2015

Tablespecs should allow row values to be set dynamically based on compiled fields #22

Closed

rsinger added 3 commits February 25, 2015 08:27

Added more sophisticated validation for table specs

25cd693

C'mon mate, that's an endless loop

1a6b57e

Test configs weren't actually valid!

f1b0d18

malcyL reviewed Feb 25, 2015
View reviewed changes

Remove "**" from arithmetic operators

818f479

Formatting/docs

1c67610

Be consistent in computed field spec

c3bb79d

rsinger added 10 commits February 25, 2015 14:26

Wrong signature sent to doComputedFields

ad2df2e

Wrong logic to determine intersection

0cef3b5

Another stab at that logic

db23b52

Trial and error

c607826

Need to return null if variable is not set

2328028

Unit tests for computed field config validation

4de05ae

Not on 5.4, yet, rsinger

7cc80f3

Added tests for computed fields

e815b17

Simple config validation script

e8bb327

Nicer handling of exceptions

0858889

rsinger changed the title ~~30%: Add computed fields~~ 90%: Add computed fields Mar 2, 2015

rsinger added 4 commits March 23, 2015 16:10

remove underscores from functions; add 'temporary' property to fields…

bea23ea

…/counts

Docs

309871b

Docs; allow nested functions throughout functions

7014145

By reference malarkey

09b54e1

rsinger mentioned this pull request Mar 24, 2015

Functions (predicate/computed field) need to be consolidated and modularized #44

Open

Remove literal 'value' from search document specifications

c1b8726

rgubby reviewed Mar 25, 2015
View reviewed changes

s/_//

d5087a0

rgubby reviewed Mar 25, 2015
View reviewed changes

Clean up warningLog; doc

f95654a

rgubby reviewed Mar 25, 2015
View reviewed changes

Add arithmetic operators to docs

7cd1aa1

rgubby reviewed Mar 25, 2015
View reviewed changes

Update old config in performance test; add regex operators

a9e5d09

rsinger added a commit that referenced this pull request Mar 25, 2015

Merge pull request #40 from talis/issue/22/save-conditional-table-row…

b5e00c3

…-data 90%: Add computed fields

rsinger merged commit b5e00c3 into master Mar 25, 2015

rsinger deleted the issue/22/save-conditional-table-row-data branch March 25, 2015 12:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

90%: Add computed fields #40

90%: Add computed fields #40

rsinger commented Feb 24, 2015

malcyL Feb 25, 2015

rsinger Feb 25, 2015

malcyL commented Feb 25, 2015

rsinger commented Feb 25, 2015

rsinger commented Feb 25, 2015

lordtatty commented Feb 25, 2015

rsinger commented Feb 25, 2015

lordtatty commented Feb 25, 2015

rsinger commented Mar 24, 2015

rgubby Mar 25, 2015

rgubby Mar 25, 2015

rsinger Mar 25, 2015

scaleupcto commented Mar 25, 2015

rgubby Mar 25, 2015

rgubby Mar 25, 2015

rgubby Mar 25, 2015

rsinger Mar 25, 2015

rsinger Mar 25, 2015

rgubby commented Mar 25, 2015

90%: Add computed fields #40

90%: Add computed fields #40

Conversation

rsinger commented Feb 24, 2015

Example of a table spec with this

Example of document outputted above

Choose a reason for hiding this comment

Choose a reason for hiding this comment

malcyL commented Feb 25, 2015

rsinger commented Feb 25, 2015

rsinger commented Feb 25, 2015

lordtatty commented Feb 25, 2015

rsinger commented Feb 25, 2015

lordtatty commented Feb 25, 2015

rsinger commented Mar 24, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scaleupcto commented Mar 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgubby commented Mar 25, 2015