Migration to Directory Elements (DEs)

[Urmel]

In order to have discussion for this in one place, let's have it here...

@nemethviktor wrote in #56:

Ultimately I think we should try to move away from the current 3-tier datatables (DT1, DT2, DT3) and have the DEs instead. I think we'd need to keep the 3-tiers unless we come up with a better way of doing it but as in the current version, the 3 tiers are identical in structure.
Since the DE stuff is ultimately your baby I think you know more about it than I do but if we tried to transition something (as in point at a random current function and transition it) from DT3 to DE we could probably replicate that comparatively easily afterwards.

At the moment the DEs support two "views" on a property: initial value and changed value.

There are 3 tiers of data tables (all use Filename + Setting-ID as primary key):

• DtFileDataToWriteStage1PreQueue
• DtFileDataToWriteStage2QueuePendingSave
• DtFileDataToWriteStage3ReadyToWrite

@nemethviktor, could you elaborate a bit on how they are used?
Thanks :)

[nemethviktor]

@Urmel ,

Lol I always struggle in my mind with DtFileDataToWriteStage2QueuePendingSave :D

Anyway. In some obscure order....

• DtFileDataToWriteStage3ReadyToWrite is the write-queue itself as well as an overlay on the listview. Stuff that is sitting in DT3 is to be written to the file unless the user cancels out. It also feeds (in the non-DE world) the changes in listview because stuff gets read in from a file, gets put onto the listview and from then on DT3 takes over - if something exists in DT3 then it gets put onto the listview during a listview-update.
• DtFileDataToWriteStage1PreQueue is generally a temp storage. This is primarily used in frmEdit because in Edit the user can modify bits of data and then either decide to press OK, in which case data gets moved to DT3 or Cancel, in which case the data is discarded.
• DtFileDataToWriteStage2QueuePendingSave is very poorly named but I haven't quite gotten around to giving it a better name. It's used in the Edit form primarily. I actually had to look up wtf it does. I think there is a bit more to it but it's generally triggered on the text/value change of textboxes and generally change-able objects. When the users has a value in listview/DT3 (either) of say 5 for altitude and the user then proceeds to type 0 (to make it 50) we have to account for that change in a temporary fashion. If "old" text != "new" text then the Control gets formatted bold plus a few other things happen. It's a little imperfect because if the user then proceeds to clear the value and type 5, going back to the original, it won't be "unformatted".

Where you'll have "fun" with this whole thing is that certain things (time shift amongst others) aren't exif tags and taken/create datetimes can be moved around, especially when coming to copy-pasting them. (eg you can copy the time-shift between files and they need to be properly accounted for, even with further pastes onto the same files).

The reason for using DTs originally, especially in the format they are (ie. PK = Filename + Setting-ID rather than just having a wide table per filename) is twofold...threefold.

• One is that I'm a SQL person and this is my first C# attempt, ever, so since I never took a course or anything, C# classes (not school-classes but...you know, classes) are a bit alien to me and while it's a staple of OOP, I never got my head around them as my brain is relational and I struggle with disorganised data.) so DTs are a lot more natural to me.
• Also having a narrow table is easier to account for when deleting a value. I (to myself) argued that null values are completely valid/legal in a lot of circumstances, and so are zeros and blanks and literal - values, so if something needs deleting then having a narrow table is a lot easier than enforcing a logic in a wide table and figuring out if something had been missing originally or became a null during the process or what.
• In the earliest version(s) the only non-[exif]tag was coordinates. Since the listview is basically a grid and so is a DT there was simple logic to move data from one to another. Coordinates were a bit of a pain in the backside but I got around the issue in due course. We now have an increasing number of non-tags such as time-shift (again -> copy-paste time-shift) and time-zone and whatever else. I generally expect to see more of them as the app develops. Narrow DTs are easy to add new fake-tags into because it doesn't influence the code itself. A new fake-tag (or a real one for that matter). That level of flexibility would need to be ported to the DE world some way.

Hope this helps a bit?

[Urmel]

Hi @nemethviktor,

Hope this helps a bit?
Yes!!!!
Thank you so much for these details!!

Some thoughts from my side...

DtFileDataToWriteStage3ReadyToWrite --> change notification of DE to list view

Conceptually, this should be covered by the version "Modified" of a property of a DirectoryElement/DE. I would propose to do an update on the DE which in turn has a change notification to the list view that then pulls updates of the changed DEs.

DtFileDataToWriteStage1PreQueue --> multiple list view and modifications

That is an interesing thing :) We could for this, introduce a new version "Temporary Modification", or an even more general concept to have addtional "named" modification of DE properties. Alternatively, we could allow multiple (not classified) modification that are in an ordered list, so that one can remove the latest if needed.
In either case the above described update list view mechanism needs to account for multiple list view clients (edit form vs. main form). We could have a concept in which the DE collection can have different iterators so that we can have sub sets of the full collection as input to a list view.

DtFileDataToWriteStage2QueuePendingSave --> version

If I understand that correctly, we want to either

• mark property that were changed with respect to the currently in the file stored version ("original") --> in the list view
• mark property that were changed with respect to a base version (might also be previously edited and not yet saved) --> in the edit form

The general comparison could be done using above ("DtFileDataToWriteStage1PreQueue") described mechansims to collect a change history or alike. Then the info "change with respect to ..." should easily be accessible.

Time Shift in Copy/Paste

Regarding time shift - I understand that we have an "edit command" (like +2h). If we just have a notion of "taken date" this addtional info "how we got there" is lost. Interesting problem :)
We could add to every version of a DE property a "edit operation". In this, we can document how we got there and thus, redo it. You could even go so far to put every edit in such a command object and just run these on all the affected DEs - thereby fully encapsulating this editing functionality.

In General ...

... I would argue that the abstraction of model (DEs) and View (ListView) is exactly suitable for this situation (tags versus not-actual-tags). We can easily account for "virtual" columns. I.e. columns that are never edited an which data is basically a composition of other columns (like coordinates).

But from what you write, there is quite some logic involved, so the only way to migrate (further) if we want to, is to find an approach to do it step by step...

[nemethviktor]

Just a very quick one as I'm out and about.
Since DEs are ultimately a Class that are part of a Collection then why not substitute the DTs with further DE Collections and just have data move between themselves? Sounds easier.

[nemethviktor]

Heads up that I started working on this overall migration. Should take a bit of time but seems doable. I've spent about 7 hours on it today and managed to get to a stage where I can load data into the Edit Form, change it and then see the result in the listview, then save the file successfully and reload the whole thing. Needs more testing tho'.

[Urmel]

Heads up that I started working on this overall migration. Should take a bit of time but seems doable. I've spent about 7 hours on it today and managed to get to a stage where I can load data into the Edit Form, change it and then see the result in the listview, then save the file successfully and reload the whole thing. Needs more testing tho'.

That sounds cool. Did you also fill the edit form list with the DEs? Did you create clones of the DEs oder used a facade?

[nemethviktor]

I'm using your logic - kinda. Instead of having Original and Modified there is Original, Stage 1/2/3 (so a total of 4) and also a "delete" Boolean flag on each. You'll see it soon, it works quite well but still buggy here and there.

[nemethviktor]

@Urmel if you feel like playing around with the current 'canary' version -> https://drive.google.com/file/d/1BGTTWI-4rIdeM_m1j08AozSRZcM5263b/view?usp=share_link -- i think it has the files you need. (worst case copy dev to a temp folder and overwrite w the files that are in the zip file.)

[nemethviktor]

Referencing this in cd937b4