Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

Latest commit

 

History

History
239 lines (190 loc) · 10.1 KB

notes.org

File metadata and controls

239 lines (190 loc) · 10.1 KB

Tasks

[#A] Unhandled USOCKET:NS-TRY-AGAIN-ERROR

Unhandled USOCKET:NS-TRY-AGAIN-ERROR in thread #<SB-THREAD:THREAD tid=434578 “Cron CACHE-CACHE::UPDATE-ISSUES

Add a readme

[#A] Add a test system

Make stuff thread-safe

See sbcl-concurrent-hash-tables or luckless

Update projects periodically

Get all epics

Get all labels

Could be inferred from the issues

Keep a “recently viewed” list

It would be nice if that list was persistant.

Yack-shaving: use websocket to update the page (without losing

the state) when we update a component.

Use dexador instead of drakma

Save issues to disk

Show the issues’ project

show swank or slynk’s port in the web interface

Once there’s a command line, add proper command line completion

Resources:

Indexing with montezuma

Fuzzy match

https://github.com/vindarel/fuzzy-match

Parenscript

Pagefind API

> MIT Licence - Copyright 2022 CloudCannon

https://pagefind.app/docs/api/

{ * … other result keys … * “url”: ”url-of-the-page”, “excerpt”: “A small snippet of the <mark>static</mark> content, with the search term(s) highlighted in &lt;mark&gt; elements.”, “meta”: { “title”: “The title from the first h1 element on the page”, “image”: “/weka.png” }, “sub_results”: [ { * … other sub_result keys … * “title”: “The title from the first h1 element on the page”, “url”: ”url-of-the-page”, “excerpt”: “A small snippet of the <mark>static</mark> content, with the search term(s) highlighted in &lt;mark&gt; elements”, }, { * … other sub_result keys … * “title”: “Inner text of some heading”, “url”: “/url-of-the-page/#id-of-the-h2”, “excerpt”: “A snippet of the <mark>static</mark> content, scoped between this anchor and the next one”, } ] }

Handling multiple sources of data

> The ultimate yak-shaving

Motivation

“Even more yak-shaving”

I’ve been using this project for a long while to help me find issues and projects in GitLab very quickly. But I want more.

  • First of all, I want to add “epics”, but I don’t want to duplicate

even more code as it is.

  • I would like to add GitHub too
  • I would like to search the contents, not just the title
  • I want actual indexes (see prototype)
  • I want to index local files (e.g. find and grep -R)
  • I want to index local repositories (I should try hound-search before, I haven’t used it since 2017…)
  • I want to support multiple GitLab instances (I currently have at least 3)
  • I want to (optionally) include personal projects.
  • I want to search quicklisp and/or specific references, like clhs, Alexandria, serapeum, uiop, asdf, etc…
  • I want to search guix…
    • other package managers: pypi, nix, npm, Debian’s, Ubuntu’s, docker.io…
  • github’ gists

Plan

  • I’ll use CLOS
  • I want to take advantage of serapeum for once
  • I need to figure out which generic method I need
  • I need to figure out the nomenclature, but “source” is a good start… I could look at how magit-forge, elasticsearch, splunk, and hound-search for inspirations
  • How will I handle the configuration? Json sounds nice, but lisp too :P

Do more than just caching/searching

“The ultimate yak-shaving”

I would like to do more stuff with the things I have cached

  • mark as TODO
  • mark as favourites
  • add links between items
    • especially those that can’t be linked easily, like linking 2 projects
    • Be able to add a note describing the link (you just can’t do that in GitLab)
  • add local-only comments on items
    • put them in an org-mode file!?
  • automated tasks?
    • useful for things that can take a while to index

Terminology Brainstorm

Current terminology

source
items
topics

Use “fetch” instead of “get”

From ElasticSearch/LogStash

Data structure

termdefinition
aliasSecondary name for a group of data streams or indices
documentJSON
indexCollection of documents
datasetA homogenous set collection of data
eventTimestamped data
field referenceA reference to an event field.
fieldKey-value pair in a document OR an event property
filterQuery that doesn’t score
id
index patternglob for index names
data stream~ index for time series? no, not only time series
index template
runtime/scripted field
saved search
computed fieldthat one is from me, not ELK
search sessioncaching search results
search template
snapshot
segment
source field
system index
tagA keyword or label that you assign to Kibana saved objects
term
token
workpad
scratchpadthat one is from me, not ELK

Data processing

termdefinition
actionwhat to do when a rule fires. a rule can have multiple actions.
condition
rule
ingestionprocess of collecting data from data sources into ELK
pipeline“flow of event” input -> filters -> output
codec
broker
message broker
aggregation
monitor
buffer
queue
query
lifecycle
[term] join
watcher
workerthat one is from me, not ELK
mapping

Other

data source
data view
datafeed
endpoint
inputlogstash’s data source?
outputa pipeline’s destination?
flush
merge
jobs
lens
query syntax
namespace
planrefers to the architecture, but I had terraform’s plan in mind
roolup
connector
data tierhot/warm/cold etc

Clustering stuff

cluser
node
shard

References