AI agent

[0xApotheosis]

Architecture Overview

The AI agent’s architecture is composed of:

1. A front-end web app (client-side)
2. An agent runtime (ElizaOS) with plugins (client-side)
3. Connections to external services for AI inference and blockchain interactions.

Note

Open question: Do we want to add a backend component here to handle long-running tasks such as a DCA strategy? If yes, do we want to ignore it for now and add it once the initial PoC is built?

Front-end UI web app (Vite?)

Provides chat interface, wallet connect UI, and displays results (including simulation previews and transaction details), ability to sign transactions. Runs in browser.

Wallet connector

Facilitates user wallet connectivity and signing.

Simulation engine

Used to simulate transactions off-chain and predict outcomes before execution. Provides a preview of state changes.

E.g. uses Tenderly Simulation API to show token output of a swap and changes in balances, helping users avoid malicious or failing tx.

A limitation here is that we cannot simulate off-chain or multi-step non-atomic actions, e.g. THORChain trades.

Session Memory Store

Keeps track of conversation context, user preferences (like selected wallet/network, active accounts), and partial results across browser reloads etc.

We'll want to ensure relevant data is passed to Eliza (see the Knowledge Management section of the docs).

Agent

Core (ElizaOS)

Orchestrates agent behavior, conversation management, and plugin usage. Maintains agent persona and session memory.

Uses a Venice or Chutes API Key to access inference. We might want to start with Venice because it's effectively free (for us), and move to Chutes when we hit limitations. The implementation will be inference-provider agnostic, so there is no concern with changing as we go.

Plugins

https://eliza.how/docs/core/plugins

Each plugin registers a set of actions or tools the agent can use. The LLM is prompted with a description of these actions. When the LLM chooses an action, ElizaOS executes the underlying plugin code, then returns the result (if any) back into the model’s context. For instance, the workflow for a swap might be:

1. Model: thinks it should call SwapPlugin.executeSwap(ETH, BTC, 0.1).
2. Agent runtime: calls the Swap plugin code with those parameters.
3. Swap plugin: gathers price quote and crafts a transaction (using other sub-plugins/APIs), then returns a summary like “Prepared swap tx to convert 0.1 ETH to ~0.0023 BTC via THORChain. Gas cost ~0.005 ETH. Awaiting user confirmation.”
4. Agent runtime: presents that summary to the user in the chat.

Example plugins we might use:

• 0X - https://eliza.how/packages/plugins/0x
• crypto prices - https://eliza.how/packages/plugins/coinmarketcap
• evm - https://eliza.how/packages/plugins/evm
  • has LiFi support
• solana - https://eliza.how/packages/plugins/solana
• zapper - https://eliza.how/packages/plugins/zapper
• zerion - https://eliza.how/packages/plugins/zerion
• more here

Character and persona

https://eliza.how/docs/core/project#bio--style

We can configure a specific persona for the agent (helpful, knowledgeable about DeFi, cautious with security, etc.) using ElizaOS’s character definition. This influences how the LLM responds to the user (e.g. the agent might explain steps verbosely or ask for confirmation in a certain style).

LLM Inference Backend

Handles natural language understanding and generation. The agent’s prompts are sent here to produce actions or responses.

Once the user’s message and current context are assembled, the agent sends this to the LLM backend. The agent’s request to the model will include relevant plugin documentation (so the model knows what tools it can use) and the conversation history (for context).

Repo

We should probably use a monorepo for this project to maximise development velocity and shared TypeScript code (types etc).
We might even want to look into TurboRepo or even nx if we want to be really AI native.

.
├── packages/
│   ├── agent/             # ElizaOS agent core (plugins, character, logic)
│   ├── ui/                # Frontend app
│   ├── shared/            # Shared types, utils, constants
├── .env                   # API keys for inference providers etc.
├── turbo.json / nx.json
└── package.json

[giantkin]

Would this be possible to setup that the Agent Runtime(eliza) can be changed out easily? ie: eliza2 is coming out soon, supposed to be better, less ugly to setup, and lighter. (or some other package) maybe selectable by the user? (but definitly by us?)
(again, any testing needs for inference, contact me) i can setup an api key quick)

[0xApotheosis]

We'd start straight on Eliza2, as the docs are already available for it. Switching out agent runtimes is less straight forward than switching out inference providers, though still possible if we ever needed to provided we kept out front end sufficiently agnostic (which we should).

Also appreciate your offer of a key for initial inference ser.

[giantkin]

What about using different agents for different tasks. (from what im learning each one has strength /weaknesses) some are better at using web, while others are better at telling you answers. for different parts of the app. (or maybe let the user choose diff agents!)
let them try eliza, switch to a diff one, and try owo etc.

(just seeing ppl change to different agents alot and ppl debating on their favorite choice for what they want)

[NeOMakinG]

What about using different agents for different tasks. (from what im learning each one has strength /weaknesses) some are better at using web, while others are better at telling you answers. for different parts of the app. (or maybe let the user choose diff agents!) let them try eliza, switch to a diff one, and try owo etc.

(just seeing ppl change to different agents alot and ppl debating on their favorite choice for what they want)

I want to add that by multi agent it doesn't 100% mean using different technologies, different agent focusing on different tasks can be achieved through Eliza without adding another technology on top

[NeOMakinG]

We'd start straight on Eliza2, as the docs are already available for it. Switching out agent runtimes is less straight forward than switching out inference providers, though still possible if we ever needed to provided we kept out front end sufficiently agnostic (which we should).

Also appreciate your offer of a key for initial inference ser.

Did we try to install Eliza 2? Because I lost a lot of time trying without achieving it 2 weeks ago, not sure if it's so stable right now even if they seems like saying that it is, I might be the dumb guy struggling at using it

[gomesalexandre]

Stunning draft @0xApotheosis!

Open question: Do we want to add a backend component here to handle long-running tasks such as a DCA strategy? If yes, do we want to ignore it for now and add it once the initial PoC is built?

My vote would be on no for the sake of simplicity, feels like inference APIs should do the thing and we should just be able to stream it, eliza handles all the heavy lifting AFAIK. Though, I think this only applies to chat UI (to be double-checked) and we may need to deploy Eliza's REST API if we want to integrate it into an app.

Wallet connector
Facilitates user wallet connectivity and signing.

Are we set on whether or not we're EVM only? My understanding is yes, but just double-checking here. If so, we may not even need anything custom which slows things down 10x(cough hdwallet), could just use viem/wagmi connector.

Simulation engine
Used to simulate transactions off-chain and predict outcomes before execution. Provides a preview of state changes.

Would be amazing to have such a UI, and something I was thinking in web for quite some time - both Alchemy and Tenderly alike provide this, and for such an app, that sounds like a requirement more than a nice-to-have indeed.

Agent - we may want to start with Venice

Absolutely, as discussed, think we won't hit limit until there's actual proper usage, which is a later thing (hopefully!)

Session Memory Store

Looks like this one may actually be a good contender for a back-end?

[0xApotheosis]

@gomesalexandre I agree that is a much easier flow for the user, though my concern is then we are then attracting users with no crypto - we'd need to have an onramp flow built in so they could acquire funds (or of course they can send funds from an existing wallet).

This is something we need to align on with @shapeshift/product, but we might instead want to target users with existing wallets (and thus, funds to trade).

[gomesalexandre]

@0xApotheosis we're aligned here! That's another argument for BYOW too, as having an embedded one would mean no funds.

One counter-argument I just thought about is that users may be reluctant to ever connect an existing wallet to an AI-enabled app, thinking it could rug their whole funds, which it technically can (sure, Rabby has simulation UI, MM now does too, and we will, but users won't know that beforehand).

[NeOMakinG]

I agree with the level of complexity BTC adds, as a MVP we would probably want to avoid it for sure

For the login flow, are we sticking with the decentralization nature? Because wallets solutions like Privy would ease the user connection flow so much as they support:

• Email login
• OAuth login with any social providers
• EIP6963
• Fiat ramps, even using apple pay and google pay
  
• Bitcoin support through they own wallet solution (https://docs.privy.io/wallets/using-wallets/bitcoin/using-bitcoin#using-bitcoin)

The drawback would probably be: decentralization concerns and the price of course, over 2500 users by months it costs 300$ a month

[gomesalexandre]

For the login flow, are we sticking with the decentralization nature? Because wallets solutions like Privy would ease the user connection flow so much as they support:

• Email login
• OAuth login with any social providers
• EIP6963

If we go with Privy, we could also not sacrifice on the decentralisation and go with their self-custodial solutions or half-custodial one (pre-generated wallet, which means you have to login), which could potentially allow us to have our cake and eat it, WDYT? Not saying we go with Privy, just noting that.
Definitely sounds like an amazing solution and potentially the best product-wise, though still unsure about the product lift of having to handle integrated wallets vs. bringing your own.

Perhaps a solution would be to indeed use Privy, but using their BYOW solution as an MVP, which could let us move to an integrated wallet solution in the future?

[gomesalexandre]

i.e https://docs.privy.io/wallets/wallets/create/overview

Not sure what client-side generation entails but I'd assume (and hope) there's nothing going to Privy and they just provide an hdwallet-like SDK for wallet gen

[gomesalexandre]

@twblack88 suggested using Privy for wallet infrastructure.

They have proper support for ETH, BTC, and even Solana if we'll want to add that as a stretch.

Looks like it also has UI elements for signing UI which we could use to ease some lift: https://docs.privy.io/wallets/using-wallets/ui-library

Though if we go with EVMs only, perhaps we want to BYOW instead? I think that's mostly a product question, since there are many ramifications to an integrated wallet that BYOW won't have, which makes me think it's the better one for MVP.

[NeOMakinG]

Ah damn, jinxed https://github.com/orgs/shapeshift/discussions/9273#discussioncomment-12767657

I'm using privy since almost 1 year now on one of my project, it works pretty well I like it

What do you mean by BYOW @gomesalexandre ? Because You can bring your own wallet with Privy, they can act as a SIWE provider, they both support their wallet solution + EIP6963 and more

We might have limitation on the BTC side, as it seems we can't import any btc wallet, that would be the BYOW part that I'm missing

And building our own privy integration could make it work with both wallet integration and BYOW https://docs.privy.io/wallets/connectors/usage/connect-or-create

[NeOMakinG]

I could only see Bitcoin support through expo btw, that would be a big limitation (Privy deserves it's own spike anyway)

[twblack88]

aligned on spiking with privy

[twblack88]

I reached out to the founder of get para tonight as well. Apparently they are adding BTC support soon so trying to get us some early alpha access.

[twblack88]

I might have lead us astray on privvy, they do have the snazzy bits for when we want an embedded wallet. But we still gotta research and deliberate on the BYOW and make that good. It's for sure a few discussions and research.

to wit:
a. we're on the discussion board already. discussing things. 🤦 oy vey my bad.
b. neo to the rescue with some more thoughts,

[NeOMakinG]

Open question: Do we want to add a backend component here to handle long-running tasks such as a DCA strategy? If yes, do we want to ignore it for now and add it once the initial PoC is built?

This is an interesting question because Eliza is using a PostegreSQL database, we will need to deploy it as a server and host a database or use a database service

So it means that we might need to think right now the way we want this database, as it would also be consumed by the backend service you are talking about

I like eliza because there are tons of plugins that would enable us to have even more feature such as searching on internet for example (using APIs like SERP)

Have we considered GAME from Virtuals (credit to @gomesalexandre that shared this with me)

It does support any open ai compliant LLM APIs so it would work with Venice

Drawback is that it would require way more works, it has way more less plugins, but maybe we could achieve a POC on client side without even a backend component

(I'm personally interested about the pros and cons if we have considered it)

[twblack88]

I'm not sure GAME is fit for this purpose, but open to checking it out more. they seemed more on the "agents with coins launchpad," maybe they pivoted though.

[twblack88]

Notes from first spike on UI elements, stack, and use cases