Add a chatbot feature directly to the app?

[jkomoros]

The pattern of 'export a collection of cards into a Google Doc to then import into Claude as context' works so well for "liquid textbooks."

You could imagine having a "chat with this collection of cards" feature directly in the webapp that automates this.

Building a full-featured chatbot into the application would be a heavy, expensive lift (and also it could only be offered to users who have useAI permission, since it would cost resources). In a perfect world there'd be an automated "Open a new conversation in Claude with this pre-filled context." (Though I imagine such a feature would be a security problem for people sending people subtly malicious contexts.)

[jkomoros]

The more I think about this, the more I think this might be worth doing, given that my pattern of "curated export" is hitting the context limit for Claude.

The overall behavior will be that you start a conversation with a set of cards. We send a list of all card IDs in the current collection up to the server, as well as the collection description and how much space to leave for the conversation in the context, and the first user message in the discussion. It then embeds that first user message and uses it to select the most semantically related cards (judging by embeddings) up to to the target context length, and uses that as the first system message in the conversation. That first message is never actually downloaded to the client, which allows us to give view-only access to a thread to other viewers who the user allows access to without directly revealing the card content.

From then on, the user can add more messages on top of that baseline starter context, and from that point on the conversation is the same.

This setup is nice because it allows users to have deep conversations that appear to be with the entire active collection, which can appear to be quite big, naturally focusing in on the subset that are most related semantically.

The "have a constructive conversation with my notes" is not a bonus feature for a personal knowledge management system, it's a core use case now.

The overall design will have a few new document collections:

//chats/CHAT_ID
//This type is quite small, allowing enumerating them easily.
{
  owner: UID
  //In the future we'll have ACLs for view to allow people without admin to see chats they are ACLd into
  last_updated: timestamp
  created: timestamp,
  //The number of tokens of length to target for the initial background (that is, the card content that is passed in)
  background_length: 8192,
  collection: {
      description: 'serialized collection description path'
      configuration: {
        //A CollectionConfiguration object
      }
  },
  The card ids that are actually included in the system prompt.
  card_ids: string[]
  title: "string"
}

//chat_messages/MESSAGE_ID
{
  chat: CHAT_ID
  //A monotincally increasing message inddex for sorting
  message_index: 0,
  //System is for the innitial snapshot of all of the cards that are selected; we snapshot it at thread creation time so later calls to the API for new messages can simply select all of the messages, in order, with the right chat_id, instead of also having to do an expensive set of fetches for the cards. No user should be able to read back the system prompt object locally, it's just used in the server to concatenate the whole conversation history for api requests.
  role: 'system' | 'assistant' | 'user',
 //True while a response is still streaming. Once the response is complete streaming is set to false. The client has a live updater that queries for streaming:false, to make sure it doesn't constantly get updates as they are written (using the incomding stream tokens) and then gets the final result once it's done.
  streaming: true,
  timestamp: TIMESTAMP
  content: "markdown text"
}

For security rules, only owners or admins should be able to fetch a chat, and chat_messages should only be able to be fetched by admins or owners of the associated chat. No one should be able to write; all writes happen via admin in the cloud functions for creating chats and adding messages to them. Add security rules and tests for those rules.

A few difficulties / notes:

• The server currently doesn't have to check if a given card the client asks for similarity to is allowed to be seen (I don't think), we assume that the worst that can happen is that a rogue client gets the cardID of a card they can't see. That logic will be somewhat annoying to duplicate
• We need to do some work to support Claude (but it's worth it)
• Conversations (especially with the starter context) can get very very big to store in the firestore database. But they have to be fully reified and stored if you want to be able to extend the conversation in the future
• Streaming tokens is necessary for a good user experience but also that implies a long-lived firebase function during the conversation, and also the default method of "as tokens come in store it on the conversation_message firestore document" also gets extremely expensive, constantly resyncing each one as tokens stream in, which would have n^2 work.
• The default UI is a big dialog inside the collection, but if this becomes a major interface then that feels like the wrong URL structure; the card subset is a jumping off point. Maybe a chat/chat-id route that shows a description of the collection the conversation is based on, but is a full-screen chat interface. It should be possible to share the chat transcript with people who are ACLed to see it that you want to share it with.
• We'll need a list of chat thread titles

[jkomoros]

The right hack for doing "streaming" events is to use Firestore. Have a chat_messages_streaming collection that is just {chat: ID, messge_index: 3, index: 1, content: 'token'}. When the message is streaming in, the client hooks a streaming query for tokens with that chatID sorted by index, and concatenates them in the client. Once the streaming is done for the message in question and it's loaded to the client, the streaming responses are thrown out and just the final concatenated message is loaded. Then some time after (or maybe right as the last token comes in) the streaming token messages are all deleted to save space since they'll never be asked for again. In the future if this is too many writes (e.g. multiple users hitting it at once) we can add coalescing so only emitting a concatenation of tokens every second or so.

[jkomoros]

• Upgrade functions openai version
• Clean up src/actions/ai.ts to match the call signature of the new openai api
• Create the anthropic endpoint
• Have claude implement the anthropicCompletion
• Experiment with using claude completions to verify they work
• Swap to using claude completions by default
• Separate out usage of selectUserMayUseAI to also include userMayUseEmbeddings (there is at least one use of it that is used for htat), and uptate that one use.
• selectUserMayUseAI should return true if openai or if anthropic is anabled (once the line above is done)
• Add claude as an api endpoint
• Make the Timestamp type not literally import firestore, and dedupe it on client and server so we can share e.g. tweetinfo and others. See also Share constants and code between cloud functions and client #134 .
• Create the chats collection type and safety rules (see above)
• Create the chat_messages collection and safey rules (see above)
• Get function deployment working
• Create an endpoint to createChat that takes the collectionDescription, initial query, and list of card_ids. For now, simply error if it's more than 20 or so. It should create a message
• createChat on server actually creates the chat object.
• Add a state.chat substate.
• Render markdown in the chats in the client
• Viewing a chat should kick any stalled threads (from assistant API failures)
• Allowing adding a message to an existing thread.
• Update updated timestamp in chat when adding new message
• A slider in UI to select how much of background_percentage to use
• Do all of the TODOS in functions/src/chat.ts
• When viewing the chat itself on first load, don't block on downloading all of the cards too.
• VERIFY that other cloud functions don't also OOM at 256mb due to gpt-tok import. They do OOM unfortunately
• chat-view fetches the chat with the given ID and also all messages
• chat-view renders markdown messages
• markdown content of assistant message for ordered lists restarts counting for each item (it inserts a br)
• visual styling
• Mobile friendly styling for chat
• Add a way in chat view to navigate to precisely the selected cards (this allows a workflow that uses the embedding filtering to export into a google doc). components/reference-block shows this logic. The collection would be the collection description but plus a cards/CARD-IDS filter appended.
• Consider making a card MCP endpoint
• Some way to edit / fork conversations (ideally while leaving in place the messages from other threads)
• When ChatMessage status is 'failed', render the error warning in client.
• Optimize prompt prefix to make it more aligned with how conversational claude is
• Cull any cards that the user is not allowed to view.
• Don't show all card IDs expanded at the top of the chat, especially for long lists
• Scroll automatically to keep the bottom of the chat focused when new content comes in
• When chat is loaded, it should focus textarea
• When viewing /chat/ and there are some chats the user owns, select the most recent one automatically
• Create a correct systemMessagse in createChat based on cards sorted by similarity
• createChat, sendChatMessage should be added to gulp deploy
• Add a sendChatMessage cloud function
• Create a chat-view that is a top-level view in main-view. It looks for the chatID in the URL and then fetches it and any messages in it and renders them.
• The prompt for first message to create a thread should not be in a prompt() but in a nice dialog
• Add a createChatMessage endpoint that adds a new user message to the given chatID and then asks the LLM to respond with a new message.
• Allow the user to pick a different model (only models that there are API keys for)
• createChat should use embedding-based similarity to pick the most related cards that fit into the initial prompt.
• Support streaming (as descirbed in the comment above)
• Consider a feature to allow other viewers to easily do chat by bringing their own openaiAPI token
• Chats can be published, so anyone who has the URL can view them (though not necessarily add to them)
• When viewing /chat/ with no chatID, render all chats.
• Published chats never actually load on the client for non-owners, since the connectLiveChat selects based on the owner.
• Copy button on chat message for convenience
• Show a chat icon in the toolbar for people who can view chats, to make it easier to find them again
• "No chat data available" shouldn't show while loading (it should show 'Loading chat...')
• When creating chat, the UI goes back to normal card view for a second before navigating to the chat, and it's unclear that it's still loading.
• The back button from the chat to the collection that generated it leads to a stalled app
• Add a collection navigation link for the chat collection to navigate back to the cards that generated it (or, a similar collection)
• Show an author chip for the owner of a thread (useful for when it's published) (or maybe we shoudl NOT show whose thread it is)
• Streaming: when finished(), delete all chunks
• Support streaming in openai
• Streaming: investigate why the performance is so slow (likely firestore write / sync throughput)
• Streaming: clientside: once a message is marked complete, remove any chunks for that message in state, to avoid holding it in memory.
• Streaming: in state.chats.chunks, the chunks should be stored by message natively and in sorted order (to avoid ahving to recalc and resort into arrays and then sort by order each time)
• Streaming: on the server, when a message is done streaming, after some wait period, delete all of the streaming chunks
• Streaming: fetch chunks (discard and throw out subscribers when the full message comes in complete)
• Streaming: clean up how streaming is called; only pass a streamer if STREAMING_ENABLED and in a callsite where it's useful, and if not, receive the full message and write it.
• Streaming: enable streaming in the server by default
• newStreaming: delete createChat and postMessageInChat after deploying to production
• newStreaming: the url fragments for /postMessage, etc should be constants shared between server and client
• newStreaming: a way for a user to nudge an assistant message into a retry loop (failed --> retry, via a new callback endpoint)
• newStreaming: stream out tokens word by word at a constant rate instead of in larger bursty chunks
• After retry is hit, the button should gray out to show it's doing something.
• Add Gemini
• Allow deleting a chat (only the owner may delete)
• Add o3 model
• It's possible if the createChat fails to send the entire starter message to the client (e.g. that's what Gemini does)
• Why are only 505 cards ever used at max?