fix(contentlayer): Improve Topics Processing and Generation

[kouloumos]

This PR fixes #54 and and introduces improvements to the structure and handling of topic data in the content layer.

Key Changes

• Simplify topics data generation while resolving the inconsistencies in topic names on the /categories and /topics pages.
• Transitioned to using our own topics index, replacing reliance on Optech's.
• Renamed JSON outputs for better understanding:
  • tag-data.json → topics-by-category-counts.json
  • topics-data.json → topics-counts.json
• Added enhanced display options for miscellaneous topics.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
registry	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 29, 2024 3:20pm

[IgboPharaoh]

Great work on the pr and addition of our own topic index. That said, I've been able to test the pr both the latest deployment and the code and I think everything works accordingly. I left a comment on what I think might be a value add but everything works as is.

[kouloumos]

Rebased and made a minor change based on @IgboPharaoh comment.

[Emmanuel-Develops]

A couple of comments I made elsewhere, providing them here for context:

the tags on bitcointranscripts do not always correlate to the topics in https://github.com/bitcoinsearch/topics-index/blob/main/topics.json

For example, this transcript
has tags: lightning and p2p the corresponding topics in topics index are P2P Network Protocol

These tags aren't topics in the topics.json from the topic index, neither are they any topic aliases. Therefore they won't be accounted for in category page, in this example P2P Network Protocol

@kouloumos what is the consensus whether the changes will be on bitcoin-transcript or we add them to the respective alias in topics index?

Also this change directly maps to how we select topics in transcripts queuer (switch to utilizing topic index rather than optech tags)

[kouloumos]

In the /categories page they are currently under "Miscellaneous" as they were before this PR. So, we could merge this and care about handling them later.

That said, I just added a configuration in the topics-index that allows for custom category slugs that solves this issue. So we should be good to go!

To give additional context on your questions:

or we add them to the respective alias in topics index?

What you are describing is not how we currently use the aliases field in the topics index. The aliases as taken from their usage in the Optech Index are used for other known names of a topic. For example "Scriptless scripts" for "Adaptor Signatures". So, it doesn't make sense to add p2p or lighting as aliases of the respective Misc topic tags that we are creating for each category.

The issue arises from the fact that those specific tags/slugs don't follow the pattern that we have with the rest of the misc category tags.

Also this change directly maps to how we select topics in transcripts queuer (switch to utilizing topic index rather than optech tags)

good point! We will need to change that also. It will be a simple change, just switching the source of truth JSON. I don't really like the way that we are currently selecting those topics inside the Editor, as we are only using the slugs, with no additional context to help the reviewer decide/understand the tag. We have that information (through the excerpt) so I'll open an issue with clarifications on the improvement that we can do there. That change could be combined with the switch to our own topics-index.

[Emmanuel-Develops]

In the /categories page they are currently under "Miscellaneous" as they were before this PR. So, we could merge this and care about handling them later.

They aren't under miscellaneous anymore in the latest deployment.
here, doesn't have the p2p and lightning tags/topics in miscellaneous which are not accounted for in other categories.
cc: @kouloumos

This PR looks good to go apart from that

[kouloumos]

They aren't under miscellaneous anymore in the latest deployment.

They were at the time of your comment. Then I pushed a change (which I am explaining in the same comment) to deal with the issue that you raised.

So now, you can see a "Miscellaneous" topic under the "P2P Network Protocol" and "Lightning Network" categories, following the same pattern as the rest of the categories.

[Emmanuel-Develops]

They aren't under miscellaneous anymore in the latest deployment.

They were at the time of your comment. Then I pushed a change (which I am explaining in the same comment) to deal with the issue that you raised.

So now, you can see a "Miscellaneous" topic under the "P2P Network Protocol" and "Lightning Network" categories, following the same pattern as the rest of the categories.

okay got it