Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partial context updates #93

Open
wants to merge 392 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
392 commits
Select commit Hold shift + click to select a range
39d0da7
file-based
pseusys Jul 19, 2023
1ca66ed
with_stem removed
pseusys Jul 19, 2023
9bb3eb7
ydb ??? again??
pseusys Jul 19, 2023
a8c6497
len and prune
pseusys Jul 21, 2023
2bbf6e4
redis delete number of args changed
pseusys Jul 21, 2023
7aefa5b
Update community.rst, revert some changes
pseusys Jul 21, 2023
6fa0542
one line reverted
pseusys Jul 21, 2023
fa9359f
double serialization removed
pseusys Jul 21, 2023
9fdf5bd
no_dependencies_tests_fixed
pseusys Jul 21, 2023
c70157a
serializer changed
pseusys Jul 21, 2023
05f0d94
serializer unchanged (example)
pseusys Jul 21, 2023
95ba296
partial tutorials started
pseusys Jul 30, 2023
cd020c9
context storages made async
pseusys Jul 31, 2023
687ba7e
tutorials added
pseusys Jul 31, 2023
425a744
example context storage removed
pseusys Jul 31, 2023
2403aed
docs added
pseusys Aug 1, 2023
b4546b4
storages docs updated
pseusys Aug 1, 2023
bdda5ff
reviewed problems fixed
pseusys Aug 2, 2023
414e4a0
file-based dbs made sync
pseusys Aug 3, 2023
e5357fc
quickle removed
pseusys Aug 3, 2023
edeb376
Excessive description removed
pseusys Aug 4, 2023
895011c
Merge branch 'dev' into feat/partial_context_updates
pseusys Aug 4, 2023
78d2ccc
migrated to pydantic 2.0
pseusys Aug 4, 2023
d4bff86
Documentation building fixes (#186)
pseusys Aug 4, 2023
821713c
add patch for json context storage
ruthenian8 Aug 4, 2023
fca7c42
json storage fixed
pseusys Aug 4, 2023
33dabc8
Merge branch 'feat/partial_context_updates' of https://github.com/dee…
pseusys Aug 4, 2023
c5ad6d5
test pickle save and load with logging
pseusys Aug 4, 2023
8deaabd
timestamp conversion test for windows
pseusys Aug 7, 2023
cbe7c70
time in nanoseconds for windows
pseusys Aug 7, 2023
b14239e
ok ok windows take this
pseusys Aug 7, 2023
ed888d4
some other idea to trick windows
pseusys Aug 7, 2023
998fb2c
excessive logging removed
pseusys Aug 8, 2023
12f938e
config dicts fixed + module docstrings added
pseusys Aug 9, 2023
dbc8928
linting and formatting fixed
pseusys Aug 9, 2023
ab43a98
s's removed from docstrings
pseusys Aug 10, 2023
cc18acc
Merge branch 'dev' into feat/partial_context_updates
pseusys Aug 10, 2023
7f850ee
type defined
pseusys Aug 11, 2023
9fe28c9
property docstring added
pseusys Aug 14, 2023
6e4eb75
Merge branch 'dev' into feat/partial_context_updates
pseusys Aug 30, 2023
c25c48d
dff installation cell added to tutorial 8
pseusys Aug 30, 2023
6856ee5
shelve improved
pseusys Sep 5, 2023
5314e31
partial review reaction
pseusys Sep 19, 2023
7f1835e
more documentation added
pseusys Sep 20, 2023
cd76105
finished review
pseusys Sep 24, 2023
4238f9b
Merge branch 'dev' into feat/partial_context_updates
RLKRo Mar 22, 2024
50cda47
put benchmark tutorial after partial updates one
RLKRo Mar 22, 2024
4cc055a
Merge branch 'dev' into feat/partial_context_updates
pseusys Jul 4, 2024
7f77c8f
context storages updated
pseusys Jul 4, 2024
2617255
old naming reset
pseusys Jul 4, 2024
4fb8f67
context merge fixed
pseusys Jul 4, 2024
1230d16
context ids removed
pseusys Jul 4, 2024
c3d82da
context equality tested
pseusys Jul 4, 2024
0bd6347
framework data comparison removed
pseusys Jul 4, 2024
4a15bf0
context id removed from everywhere
pseusys Jul 4, 2024
9b3dd80
lint applied
pseusys Jul 4, 2024
4f0562a
documentation building fixed
pseusys Jul 4, 2024
ef0a9ee
RST syntax fixed
pseusys Jul 4, 2024
3d364bc
context dict added
pseusys Jul 29, 2024
e7ad269
async + pydantic
pseusys Jul 30, 2024
be34714
fixes
pseusys Jul 31, 2024
b8701a0
hashes manipulation only on `write_full_diff`
pseusys Jul 31, 2024
a58eace
ctx_dict + ctx updated
pseusys Aug 5, 2024
33f2823
setting removed
pseusys Aug 5, 2024
c4f9fce
sets added
pseusys Aug 6, 2024
e892a52
serialization added, sample context storage class created
pseusys Aug 6, 2024
1b8aa0d
iterative async access made synchronous
pseusys Aug 6, 2024
173b1fe
sql prototype
pseusys Aug 6, 2024
9665038
context API updated proposal
pseusys Aug 7, 2024
3468af5
context schema and serializer removed
pseusys Aug 7, 2024
71bd9f3
context API updated once again
pseusys Aug 7, 2024
2e6b334
review notes fixed
pseusys Aug 8, 2024
830ea40
ContextDictView made mutable
pseusys Aug 8, 2024
5d3dd95
context dict file split
pseusys Aug 8, 2024
f00ba02
turn introduction reverted
pseusys Aug 8, 2024
1af24db
turns separated (again)
pseusys Aug 13, 2024
3616ac0
key deletion now nullifies value
pseusys Aug 13, 2024
81ce7ba
memory storage
pseusys Aug 16, 2024
1f9e653
ctx_dict tests done
pseusys Aug 17, 2024
c981cc5
general context storages tests created
pseusys Aug 27, 2024
5002dda
ctx_dict updated not to use serializer
pseusys Sep 18, 2024
3e6a8f4
merge dev
RLKRo Sep 18, 2024
6991fb6
Merge branch 'refs/heads/dev' into feat/partial_context_updates
RLKRo Sep 18, 2024
5b80818
fix imports in newly added files
RLKRo Sep 19, 2024
96af9bc
hide circular imports behind type checking
RLKRo Sep 19, 2024
000fb0d
fix imports in test files
RLKRo Sep 19, 2024
2c2ab9d
merge context.init into context.connected
RLKRo Sep 19, 2024
2eb5a2c
remove get_last_index imports
RLKRo Sep 19, 2024
06d54b9
update pipeline.context_storage type
RLKRo Sep 19, 2024
f80e6a3
fix bug with setting sequence type values under a single key
RLKRo Sep 19, 2024
c5311f6
revert primary_id renaming
RLKRo Sep 19, 2024
d43752a
memory test (almost!) finished
pseusys Sep 23, 2024
1ae3e4f
ctx_dict tests fixed
pseusys Sep 23, 2024
85315a6
add overload for getitem
RLKRo Sep 23, 2024
351a43e
split typevar definitions
RLKRo Sep 23, 2024
e9eb2fb
remove asyncio mark
RLKRo Sep 23, 2024
6d93399
allow using negative indexes for context dict
RLKRo Sep 23, 2024
e2053dc
add validation on setitem for context dict
RLKRo Sep 24, 2024
acdcd3c
fixes
RLKRo Sep 24, 2024
16a3d77
allow non-str context ids
RLKRo Sep 24, 2024
9a76ae3
add current_turn_id
RLKRo Sep 24, 2024
5e37651
fix tests
RLKRo Sep 24, 2024
d376e49
update doc
RLKRo Sep 24, 2024
256e296
integer keysreversed
pseusys Sep 24, 2024
e2ffa0a
sql storage update function fix
pseusys Sep 24, 2024
9043dca
move context factory and pipeline fixtures to global conftest
RLKRo Sep 24, 2024
d58ce7c
unbound V from BaseModel
RLKRo Sep 24, 2024
6905bcd
remove default marker; return None by default
RLKRo Sep 24, 2024
0ac3c1e
fix key slicing
RLKRo Sep 24, 2024
3956348
use current_turn_id in check_happy_path
RLKRo Sep 24, 2024
d37c4e2
use context_factory to initialize context in non-core tests
RLKRo Sep 24, 2024
2bf82f9
fix: await misc get
RLKRo Sep 24, 2024
8a4d8be
update pipeline tutorials
RLKRo Sep 24, 2024
6404eb4
allow initializing MemoryContextStoraeg via context_storage_factory
RLKRo Sep 25, 2024
240cded
move all db tests into a single parametrized test class
RLKRo Sep 25, 2024
535d524
SQL testing fixed
pseusys Sep 27, 2024
6e0a103
Merge branch 'feat/partial_context_updates' of https://github.com/dee…
pseusys Sep 27, 2024
862e7d3
test_dbs fixed
pseusys Sep 27, 2024
e82d086
file context storages implemented
pseusys Sep 27, 2024
59f91c1
file and sql fixed
pseusys Sep 28, 2024
1c97303
async file dependency removed
pseusys Sep 30, 2024
f5ceb2f
rename delete_main_info to delete_context
RLKRo Sep 30, 2024
cf27afa
fix load_field_items typing
RLKRo Oct 1, 2024
c1a24ee
rewrite db tests
RLKRo Oct 1, 2024
f2ec013
Merge branch 'refs/heads/dev' into feat/partial_context_updates
RLKRo Oct 1, 2024
cb22d12
small None checking update
pseusys Oct 3, 2024
8ba5aed
Merge branch 'feat/partial_context_updates' of https://github.com/dee…
pseusys Oct 3, 2024
d9b95f6
tests updated
pseusys Oct 3, 2024
7277bf9
mongo done
pseusys Oct 3, 2024
e1cb50d
redis done
pseusys Oct 4, 2024
782bf66
ydb finished
pseusys Oct 4, 2024
0fb487b
raise error in abstract method
RLKRo Oct 4, 2024
ff70324
update service tests
RLKRo Oct 7, 2024
b59cf95
Merge remote-tracking branch 'origin/feat/partial_context_updates' in…
RLKRo Oct 7, 2024
d3af3b2
update lock file
RLKRo Oct 7, 2024
e38e2d4
fieldconfig removed
pseusys Oct 10, 2024
de739f2
update benchmark utils
RLKRo Oct 11, 2024
eaa8a87
aiofile reverted
pseusys Oct 13, 2024
53bf877
misc tables removed
pseusys Oct 13, 2024
7629fbc
Merge branch 'feat/partial_context_updates' of https://github.com/dee…
pseusys Oct 13, 2024
757fe48
denchmark awaiting removed
pseusys Oct 17, 2024
a001c27
Merge branch 'refs/heads/dev' into feat/partial_context_updates
RLKRo Oct 18, 2024
96d05dc
update lock file
RLKRo Oct 18, 2024
1430544
fix context size calculation
RLKRo Oct 18, 2024
403e2e1
change model_dump mode
RLKRo Oct 18, 2024
5340256
key filter implementation
pseusys Oct 21, 2024
9aad1bb
Merge branch 'feat/partial_context_updates' of https://github.com/dee…
pseusys Oct 21, 2024
b32b367
ctx_dict hashes update added
pseusys Oct 24, 2024
edc85bd
added and removed sets cleared upon storage
pseusys Oct 24, 2024
e61b1b7
Revert "key filter implementation"
RLKRo Oct 24, 2024
d114d42
sql and file logging added
pseusys Oct 28, 2024
3619125
Merge branch 'feat/partial_context_updates' of https://github.com/dee…
pseusys Oct 28, 2024
5618484
debug logging added
pseusys Oct 28, 2024
5e6e223
use standard logging practices
RLKRo Oct 30, 2024
4323871
make logging more uniform across the methods and collapse long lists
RLKRo Oct 31, 2024
93144df
fix potential error in prefix parsing
RLKRo Oct 31, 2024
83c7b33
Merge branch 'refs/heads/dev' into feat/partial_context_updates
RLKRo Oct 31, 2024
b763f21
create tmp file only for file dbs
RLKRo Nov 1, 2024
69d1520
add test for load_field_items
RLKRo Nov 2, 2024
291396f
test fix: misc no longer context dict
RLKRo Nov 2, 2024
c3d8c73
test fix: load_field_items no longer returns dict
RLKRo Nov 2, 2024
4bb6ca7
test fix: field config was removed
RLKRo Nov 2, 2024
dbbbb28
remove debug artefact
RLKRo Nov 2, 2024
710554c
all user input escapedin ydb
pseusys Nov 6, 2024
20b6b5f
ctx_dict moved
pseusys Nov 8, 2024
2b6eebf
async lock introduced
pseusys Nov 8, 2024
6c458c6
codestyle fixed
pseusys Nov 14, 2024
46e0112
Merge branch 'dev' into feat/partial_context_updates
pseusys Nov 14, 2024
e263fa1
SOME of the errors FIXED!!!
pseusys Nov 20, 2024
1f96f6d
rebuild script updated
pseusys Nov 22, 2024
ce6c8b6
turns added, empty ctx_dict method also added
pseusys Nov 22, 2024
9e7cf47
context creation field set removed
pseusys Nov 22, 2024
c34f8e7
contex storage class splitted
pseusys Nov 22, 2024
1d3859c
rebuild was cleaned (once again)
pseusys Nov 22, 2024
5514c7b
turns added and tested
pseusys Nov 25, 2024
2b9b947
splitted database methods + locks and validations
pseusys Nov 27, 2024
86d745c
insert limit removed
pseusys Nov 27, 2024
214fb92
_locks removed from subclasses
pseusys Nov 27, 2024
5a8d0d5
lazy connection
pseusys Nov 27, 2024
abbd920
uuid length and name changed
pseusys Nov 28, 2024
b9a0680
logs location changed
pseusys Nov 28, 2024
0115b83
none and empty subscript forbidden
pseusys Nov 28, 2024
0587881
names extracted to a special class
pseusys Nov 28, 2024
e756f75
set strings removed
pseusys Nov 28, 2024
61619e3
configuration name changed
pseusys Nov 28, 2024
aad2c49
literal keys instead of strings
pseusys Nov 28, 2024
539005d
loggers from SQL removed
pseusys Nov 29, 2024
2feb094
connect before load in file
pseusys Nov 29, 2024
2ac91a2
logging moved to commect
pseusys Nov 29, 2024
f4e5f33
context dict made abstract
pseusys Nov 29, 2024
68a1c5f
connect moved to pipeline.run
pseusys Nov 29, 2024
8671233
ctx_dict overloads fixed
pseusys Nov 29, 2024
48b6444
configuration renamed
pseusys Nov 29, 2024
e40786c
context_info dataclass added
pseusys Nov 29, 2024
a54df18
test-time comparison fixed
pseusys Nov 29, 2024
49d3bff
lock staticmethod extracted
pseusys Nov 29, 2024
6fd0e1a
initial locking system fixed
pseusys Nov 29, 2024
47edbda
codestyle
pseusys Nov 29, 2024
567fdb1
id_length moved to init arguments
Jan 10, 2025
5d5879d
tutorial updated
Jan 13, 2025
c19947f
Merge branch 'refs/heads/dev' into feat/partial_context_updates
RLKRo Jan 16, 2025
95b7cbe
fix: use context factory in slot partial extraction tests
RLKRo Jan 16, 2025
fcf3739
revert misc await
RLKRo Jan 16, 2025
cdca730
another documentation update
Jan 17, 2025
3d73555
Merge remote-tracking branch 'origin/feat/partial_context_updates' in…
RLKRo Jan 17, 2025
683f23d
lint fixed
Jan 20, 2025
78c7d7c
tests fixed
Jan 20, 2025
59b0aa0
mini-fix for context copy :)
Jan 23, 2025
28e2d25
synchronization attempt!
Jan 23, 2025
b5eb543
explicit connect made mandatory
Jan 23, 2025
9317cb1
context storage initialization rules relaxed
Jan 23, 2025
1441814
doc fixed and one silly error also fixed
Jan 23, 2025
52fd530
generic removed at all :(
Jan 24, 2025
7ba5b39
last generic remains removed
pseusys Jan 28, 2025
1f04c35
overloaded methods added
pseusys Jan 28, 2025
ba602b3
serializer and validator replaced
pseusys Jan 28, 2025
da1f4bc
super validate and serialize model methods made static
pseusys Jan 28, 2025
7d9bdb6
serializer and validator added to context dict generic class
pseusys Jan 28, 2025
a540fbc
naming conflict resolved
pseusys Jan 28, 2025
b1f9e68
validators names changed
pseusys Jan 28, 2025
e3e24f2
child class validator and serializer removed
pseusys Jan 28, 2025
bd37d13
potential warnings fixed
pseusys Jan 28, 2025
60231e5
subscript set test added
pseusys Jan 28, 2025
c85ef7e
clear all test added and also small YDB set fix
pseusys Jan 28, 2025
0cfebbe
lint fixed
pseusys Jan 28, 2025
e01e739
coverage increased further
pseusys Jan 31, 2025
0ef6d97
context guide tutorial update prototype
Feb 3, 2025
10ae3b1
formatting fixed
Feb 3, 2025
6ae396a
context storing API updated
Feb 7, 2025
c67dd72
legacy API removed from tests
Feb 10, 2025
817cb67
context updating fixture updated
Feb 10, 2025
1e529b7
old API remains removed
Feb 10, 2025
502bb15
some more tests updated
Feb 10, 2025
e110c06
field validation returned
Feb 10, 2025
7baf4a9
field name fixed
Feb 10, 2025
68b2d1e
sql length updated
Feb 10, 2025
aa02ef4
mongo session introduced
Feb 10, 2025
25675be
motor asyncio transaction
Feb 10, 2025
10619a4
mongo and postgres updated AGAIN
Feb 11, 2025
50b0ce5
mongo transacrion made optional
Feb 11, 2025
0820450
import fixed
Feb 11, 2025
87ec2a0
few more tests fixed
Feb 11, 2025
3646271
one more small redis update
Feb 11, 2025
191303b
SQL column binary defaults added
Feb 11, 2025
25a0b7d
empty SQL queries not sent
Feb 11, 2025
1a6efa7
different field updates collapsed
Feb 11, 2025
f4eb249
assignment changed
Feb 11, 2025
9677882
lint applied
Feb 11, 2025
ff6661c
sql updates separated
Feb 11, 2025
9148970
docs updated and type updated
Feb 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
dist/
venv/
build/
dbs/
docs/source/apiref
docs/source/_misc
docs/source/release_notes.rst
Expand Down
4 changes: 4 additions & 0 deletions chatsky/__rebuild_pydantic_models__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,13 @@
from chatsky.core.script import Node
from chatsky.core.pipeline import Pipeline
from chatsky.slots.slots import SlotManager
from chatsky.context_storages import DBContextStorage, ContextInfo
from chatsky.core.ctx_dict import ContextDict
from chatsky.core.context import FrameworkData, ServiceState
from chatsky.core.service import PipelineComponent

ContextInfo.model_rebuild()
ContextDict.model_rebuild()
PipelineComponent.model_rebuild()
Pipeline.model_rebuild()
Script.model_rebuild()
Expand Down
2 changes: 1 addition & 1 deletion chatsky/conditions/standard.py
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ def __init__(
super().__init__(flow_labels=flow_labels, labels=labels, last_n_indices=last_n_indices)

async def call(self, ctx: Context) -> bool:
labels = list(ctx.labels.values())[-self.last_n_indices :] # noqa: E203
labels = await ctx.labels[-self.last_n_indices :] # noqa: E203
for label in labels:
if label.flow_name in self.flow_labels or label in self.labels:
return True
Expand Down
7 changes: 3 additions & 4 deletions chatsky/context_storages/__init__.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
# -*- coding: utf-8 -*-

from .database import DBContextStorage, threadsafe_method, context_storage_factory
from .json import JSONContextStorage, json_available
from .pickle import PickleContextStorage, pickle_available
from .database import DBContextStorage, ContextInfo, context_storage_factory
from .file import JSONContextStorage, PickleContextStorage, ShelveContextStorage, json_available, pickle_available
from .sql import SQLContextStorage, postgres_available, mysql_available, sqlite_available, sqlalchemy_available
from .ydb import YDBContextStorage, ydb_available
from .redis import RedisContextStorage, redis_available
from .memory import MemoryContextStorage
from .mongo import MongoContextStorage, mongo_available
from .shelve import ShelveContextStorage
from .protocol import PROTOCOLS, get_protocol_install_suggestion
Loading
Loading