Some heisig primitives cannot be displayed

[MinmoTech]

These character require a custom font that we can use:

Ideally we would have a design for each of our existing fonts.
Example with all fonts:

[MinmoTech]

[MinmoTech]

This one has a very similar kanji:

[MinmoTech]

[MinmoTech]

[MinmoTech]

http://rtkelements.blogspot.com/

[MinmoTech]

[MinmoTech]

[mjuhanne]

@MinmoTech
I also could be interested in working with this.

The Migaku Kanji Database Excel sheet lists each primitive as a single Unicode character so to use also non-Unicode primitives we would need a way to address them in the database and then map those into custom font for viewing.

I propose to use brackets to enclose a keyword assigned to the primitive. Example:
For character 倍 (double) the primitive list would be '人[muzzle]' instead of '人立口'

I haven't looked yet how the font files are composed and indexed. Do we need a separate custom font file for each new primitive?

[mjuhanne]

@MinmoTech I've made some progress using the great RTK index and copying their list of primitives and associated SVGs, and then filtering out all of those that have Unicode representation. This leaves us a collection of 65 non-Unicode primitives that we have now nice images for.

With few modifications I'm able to now create distinct cards for these missing primitives and reference those from other kanjis that use them. This is possible with the [primitive_name] addressing scheme explained above. If a character has this format, it will be converted to a image link for visual representation.

The SVGs are a bit too small though..

.. and currently show up in wrong color.

I've made a script that modifies kanji.db with external .tsv file to add these missing primitives and then recalculates references from kanjis to primitives. I've manually added few select kanjis (such as for the 'tribe' above), but a lot of manual work has to be done to go through the main list and add all the rest.

I'll need to do some tidying but I'll put the code later into my repository.

[mjuhanne]

Here's the code for testing

MAKE COPY OF kanji.db BEFORE TESTING db_merge_tsv.py !! :)
python tools/db_merge_tsv.py addon/kanji-ext.tsv addon/kanji.db

[MinmoTech]

@mjuhanne Sorry for taking so long to respond, this looks pretty great!

I actually started working on fixing this issue myself
a while ago, but I was simply copying the Private Use Area code of a custom font into the google sheet, your solution is way better!

While working on it I noticed how large the effort that still needs to go in and couldn't justify the time commitment at the time so the work had kinda stalled again.

This is the font I used in case it helps you out: Pangolin_font für Heisig Primitive.zip (not sure if it's the same one you are using)

Are you on our discord server btw? :)
https://discord.gg/JR5NrTTFhF

You can find me on there as "Julius", you can contact me there if you ever want quicker feedback from me about something.

Also do you want to open a PR with your changes?
I'd be very interested in getting those merged 😀

[calculuschild]

Here's the code for testing

MAKE COPY OF kanji.db BEFORE TESTING db_merge_tsv.py !! :)
python tools/db_merge_tsv.py addon/kanji-ext.tsv addon/kanji.db

How would I go about testing this? Are there any good resources on how to build it into an Anki plugin or perhaps a pre-compiled plugin I can install somehow?

[mjuhanne]

Here's the code for testing
MAKE COPY OF kanji.db BEFORE TESTING db_merge_tsv.py !! :)
python tools/db_merge_tsv.py addon/kanji-ext.tsv addon/kanji.db

How would I go about testing this? Are there any good resources on how to build it into an Anki plugin or perhaps a pre-compiled plugin I can install somehow?

@calculuschild
You can just clone my repository and overwrite the 1872210448 directory (in Mac OS it's ~/Library/ApplicationSupport/Anki2/addons21/1872210448)
with contents from 'Migaku-Kanji-Addon/addon'.

IMPORTANT! Do make a backup of the 1872210448 and especially its user_files/ directory. Also it's wise to export all your Anki stacks using File | Export.. before testing, just in case something goes horribly wrong :)

[calculuschild]

@mjuhanne Ok, I finally had a chance to download your repository. I backed up the original elsewhere and then overwrote it with yours. Anki goes through a "refreshing cards" step. In all, the cards already look so much cleaner, especially the scrambled primitive references and missing story entries.

Some observations:

• ✔️ Looking at some of the known problem cards, I see that "舎 cottage" for example is now correct (used to have the story for "lidded crock" instead, and now correctly points to the new SVG "lidded crock" card as its primitive instead of "aerosol can").

• ❓ I also see a handful of my old cards have been renamed just -. For example, my old "chop-seal" card now has two copies, each using a slightly different character. Both cards already show they have been practiced previously, so they likely were already duplicate (I was wondering why "chop-seal" was showing up so often). I assume I can just remove all the - cards now? I don't see an obvious easy way to do that, since the MigakuData field is encoded somehow and hence not searchable.

• ❓ Except for "chop-seal" which was already a duplicate in the database, the new fixed primitives don't seem to be added to my study queue as new cards, and sifting through to find all the changed cards would be a nightmare. Is there a way tell Anki to generate all the cards for any of these new fixed primitives and remove the old - ones? For example I still have the old "scarecrow" card, but the new one isn't showing up in my deck:

• ❓ Is there a way to transfer learning progress from an old primitive to the new, replacement version?

• ❌ The stroke order graphic appears entirely broken on every card:

[mjuhanne]

@calculuschild Thank you for testing!

Those with existing decks will have a bit nuisance because of the orphan cards (renamed '-'). I would be very hesitant adding code to remove cards from users' decks, so it is just best to Suspend the card whenever one comes across it (there are only a handful of those anyway).
It would be possible to implement additional checks for missing cards for the extra primitives in the 'Refresh cards' action, but there's the added complexity. I will think about this.
For the mean time, whenever you see a kanji that references a primitive you haven't got in your deck, just follow the reference link to the aforementioned primitive and click the 'Create recognition/production card' button.

When it comes to the non-Unicode primitives, I don't think KanjiVG database has to stroke order graphic for those. I'll have to research later how to create them. For the other cards, they should work. You aren't seeing this?

You mean the stroke order graphic is broken in EVERY card or just those that use the new primitives?

[calculuschild]

@mjuhanne

...so it is just best to Suspend the card whenever one comes across it (there are only a handful of those anyway).

That's fair.

It would be possible to implement additional checks for missing cards for the extra primitives in the 'Refresh cards' action, but there's the added complexity. I will think about this.

Do you know if the Kanji GOD addon itself has some function to just re-scan for missing cards? I seem to remember if I add new vocab to my vocab deck, Kanji GOD is supposed to detect that and generate new Kanji cards. It just doesn't seem to be happening in this case, or I'm not finding the right button.

You mean the stroke order graphic is broken in EVERY card or just those that use the new primitives?

EVERY card. You can see my screenshot above of the "cottage" card. It doesn't show the stroke order that yours shows. Do I need to install some other dependency?

[mjuhanne]

@calculuschild Ok that is weird, I cannot reproduce your findings.

• I started from scratch and installed Anki 2.1.65 on a different Linux machine (I mainly use MacOS for dev).
• Added vanilla Kanji GOD as addon
• Created Production+Recognition stacks
• Added 'cottage' card, which created it + 4 it's sub-primitives
• Close anki
• Cloned my non_unicode_primitives branc. Make sure it's the right branch (git switch non_unicode_primitives)
• Copied the addon directory by overwriting 1872210448 (recursively with -r option of course)
• I can see the stroke graphics (cottage, soil, mouth, but not umbrella or lidded crock obviously) using Lookup window and study window in both recognition and production cards.

I tried even both Qt5 and Qt6 versions of Anki with same results.

If you really want to get your hands dirty, you could try the AnkiWebView Inspector to see if there's some weird Javascript error that prevents showing the stroke graphics. The normal version supports inspecting only the card windows, but I've made a patch to enable inspecting also the Lookup window

[mjuhanne]

@calculuschild Hold on, I came across the same problem now. Will investigate.

[mjuhanne]

@calculuschild Ok, found out the culprit. I think you're missing the 'addon/kanjivg' directory and its contents (the actual svg files containing the stroke graphics). I don't know how the installation procedure invokes the command to download the separate kanjivg repository but you can do it manually:

tools/kanjivg_download.sh 

[calculuschild]

@mjuhanne OK, that looks like it was probably the issue; I manually installed using your script, then moved it into the \1872210448 folder since I renamed \addon to overwrite the original extension. Seems the SVGs are being recognized now.

One weird thing; not sure if it matters. In the card browser, there is a weird ]> in the field, see below.

Also, while the cards now show the stroke order on my Android and in the card lookup tool on PC, they don't show up when doing normal study on PC. I assume this is not you but some weird Anki caching issue since Android is working fine.

This is honestly really really great. You're doing (Kanji) God's work. Hoping they merge your PR soon.