-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SIP-019: Notifications for Token Metadata Updates #72
SIP-019: Notifications for Token Metadata Updates #72
Conversation
Really happy to see this! My first thought around this is the concept around having to specify a list of asset id's for the NFT section (define-public (nft-metadata-update-notify (contract <nft-trait>) (token-ids (list 100 uint))) what if we wanted to refresh the megapont robot components (tens of thousands of assets)? I feel like we need an option to do all, or a range, or some other syntax. What do you all think? |
I think this is a great proposal - I like the approach of standardizing My only real feedback is the same as Thomas's - the most likely use case is going to be "update all metadata", in the case of reveals. This should definitely be able to handle that. Secondarily would be ranged updates, which would be nice, but my guess is that it's not a hugely common use case. Another thought - is including the Overall I'd say 👍🏼 |
@hstove I agree that this is a very real and valid use case. A simple solution would be to make the
That's right, including the contract-id is necessary if you want to send notifications for tokens declared by old contracts. We designed it this way so any contract may send notifications on behalf of any other contract. As you say, this could be abused but we decided transaction fees were enough to discourage that. Besides, the side effect of abuse is that contracts will just have their metadata refreshed. |
Thank you for your submission @rafaelcr! I think this SIP looks pretty good. One thing I would add before advancing it would be a section about the API contract this creates for indexers and metadata servers. Because in the legacy case anyone can trigger a refresh event, I think at the very least you'll want to point this out in this SIP that indexers should be prepared for things like gratuitous, delayed, and out-of-order notification delivery (since you don't know the order txs will be mined in, or that they even correspond to an actual metadata change), and all metadata URL queries should be idempotent. |
Couple thoughts in response to the feedback: I agree with @aulneau and @hstove about usefulness of bulk updates for NFT metadata. That said, I'd like to dig more into the scalability implications (e.g. DDoS possibilities) that have been brought up. We should encourage NFT-class owners to only issue update notifications for tokens which actually need updated, as opposed the easy path of "update all", which would likely be (ab)used if available. Perhaps having a token limit per contract-call tx would actually be a helpful limiter here. As an example, a request to refresh metadata for all tokens in a given NFT class results in DDoS-like traffic patterns, where an indexer ends up issuing tens of thousands of http requests. Common hosts like cloudflare or public ipfs-http gateways have high likelihood of rate-limiting / IP-banning the indexer service. We should disallow the ability for anyone to trigger tens or hundreds of thousands of NFT metadata resolution processes with a few relatively cheap contract-calls. This could be done in a way that is backwards-compatible with existing token contracts by requiring a signed message that matches the principal of the contract in the update event payload. After thinking through the above, it's clear that there are significant interface and scale difference between FT and individual-NFT metadata operations. I think it would be reasonable to separate this into two different SIPs: one for FT metadata, another for NFT metadata. In my opinion, this first SIP should be limited to FT metadata updates, and a later followup SIP can be related to NFT metadata updates and tackle issue around scalability, DDoS, and developer-friendly interfaces. That seems like a logical progression:
|
Thanks for your review @jcnelson , I've addressed your observations and I've also added a new section covering considerations for metadata indexers. Looking forward to your comments. |
@zone117x and myself were discussing abuse vectors for NFT metadata updates, here's a brief summary on some conclusions we reached:
A simple solution to this problem could be to modify this SIP so that notifications are treated as valid only when initiated by the same principal who deployed the contract declaring the tokens to be refreshed. For example, if the Megapont developers wish to notify metadata was updated for the entire Making this change will get rid of any worries on third-party attackers constantly issuing "update all" notifications for large NFT collections, i.e. it will remove the ability to perform gratuitous updates. Nevertheless, there are remaining worries around other kind of attacks like a malicious agent deploying an NFT contract, intentionally making metadata refresh calls really heavy (in size and/or computation required for nodes), and then making constant "update all" notifications on their contract. This could potentially still successfully DDoS metadata indexers and some nodes. |
It's in the process of being activated. |
Left a few more comments. Feel free to rename this tip SIP-019 and set its status to Accepted (and add me to the |
Exactly. For example, the reference contract allows a token owner to still emit a notification for their old contract that was deployed before this SIP was proposed. For newer contracts, you have the option to emit the |
Updated meta data via contract call: https://explorer.stacks.co/txid/0xfc81a8c30025d7135d4313ea746831de1c7794478d4e0d23ef76970ee071cf20?chain=mainnet |
Is there any use for these transactions on-chain? If not, the argument against off-chain communication is not convincing. I suggest to investigate the matrix protocol, create an open room, where signed messages indicate updates of token meta data. Indexers can verify the signatures and act upon the message content. No centralized service involved. This SIP could be reduced to the definition of which matrix room to choose for notifications and a process to update the room. |
Hi @friedger, thanks, that is a good question. I really believe these notifications should be on-chain primarily for the following reasons:
To be fair, these advantages would not present themselves entirely if the metadata update was done without the need for a transaction (e.g. updating a JSON file in Amazon S3), however, calling one of the provided reference contracts to emit the notification would make this task simple for developers or creators in this case. After writing this response I realize that these reasons should also be included in the SIP text, so I will fix that. Nevertheless, if you have any other thoughts on this I'll be happy to discuss. Cheers |
Thank you for your thoughts. It makes sense to have a standard when looking at tx like I still believe that we should not promote notifications for off-chain data if the ordering of events do not matter. Maybe that can be clarified in the SIP. Probably, high gas fees will prevent this use case anyway in the long run. Re dependencies, we have already dependencies on http, p2p protocols, gaia,.. Does the Stacks ecosystem have closed borders? |
I now approve :-) |
Conversely, client-side applications now have an extra dependency/cost of running an indexer. Pushing more centralized databases seems counter-productive in the long term. |
@dantrevino what I tried to say is that if there is already any client app that listens to node RPC messages to process token transactions and events, they only need to add an extra check to see if the print events they already receive (in the |
Adoption, so far, is not that big yet: https://github.com/boomcrypto/clarity-deployed-contracts/search?q=token-metadata-update |
Is it just about increasing this SIP's awareness to target audiences like the marketplaces? If so we (or I) can help keep relevant people updated on this |
Hiro's Token Metadata Service could also help increase adoption once it's deployed to production and it gets some user traction. We aim to publish a beta before year end. I'll be hosting a demo this week if you're interested: https://discord.com/channels/621759717756370964/623217767356694547/1049746072840454214 @friedger @Hero-Gamer |
We'll be adding this to Gamma's Create Portal contracts by default within the next week or so. |
Hi @Jamil I wonder if has Gamma added SIP-019 support? |
Adoption of this SIP has picked up considerably in the last few months. We're now at ~1,600 contracts using these notifications: https://github.com/boomcrypto/clarity-deployed-contracts/search?q=token-metadata-update As far as I can tell from these contracts, in terms of indexer support these notifications are emitted by and/or used by:
I think we can now safely consider all activation criteria are met for this SIP 🎉 cc @Hero-Gamer |
Marvin here from the Steering Committee. I think it is safe to say that all activation criteria have been met 😁. You can increment the status to |
Done @MarvinJanssen , thanks! |
🎉 |
hi there, i deployed a token yesterday, but metadata are not yet indexed. there is something I can do to fix it? should i launch a metadata update to see if it works? with the same contract on testnet it tooks about 8hours to index, but now are more than 24h on mainnet and still wrong decimals.. |
Hi @eriqferrari you can try a metadata update. Usually IPFS takes a while to reflect a file across nodes so Hiro's API (and other indexers) could have missed the data when you first deployed the token. However, to keep this conversation about this SIP only, please open an issue in https://github.com/hirosystems/token-metadata-api if you still believe there is a problem. Thanks. |
Preamble
SIP Number: 019
Title: Notifications for Token Metadata Updates
Author: Rafael Cárdenas ([email protected]), Matthew Little ([email protected])
Consideration: Technical
Type: Standard
Status: Activation-in-Progress
Created: 17 May 2022
License: GPL-3.0
Sign-off: Jude Nelson ([email protected]), Aaron Blankstein ([email protected])
Layer: Traits
Abstract
As the use of tokens (fungible and non-fungible) has grown in popularity, Stacks developers have
found novel ways to define and use metadata to describe them. This rich data is commonly cached and
indexed for future use in applications such as marketplaces, statistics aggregators, and developer
tools like the Stacks Blockchain API.
Occasionally, however, this metadata needs to change for a number of reasons: artwork reveals, media
storage migrations, branding updates, etc. As of today, these changes do not have a standardized way
of being propagated through the network for indexers to refresh their cache, so the display of stale
metadata is a very common problem.
This SIP aims to define a simple mechanism for developers to notify the Stacks network when metadata
for a token has changed, so interested parties can refresh their cache and display up-to-date
information in their applications.
Introduction
Smart contracts that declare NFTs, FTs and SFTs conform to a standard set of traits used to describe
each token (see SIP-009,
SIP-010 and
SIP-013).
One of these traits is
get-token-uri
, which should return a URI string that resolves to a token'smetadata usually in the form of a JSON file. There is currently no defined structure for this data,
and it is not considered to be immutable.
To illustrate a common use of
get-token-uri
, we'll look at theSPSCWDV3RKV5ZRN1FQD84YE1NQFEDJ9R1F4DYQ11.newyorkcitycoin-token-v2
contract which declares the NewYorkCityCoin fungible token.
At the time of writing, the value returned by this contract for
get-token-uri
is the string:When this URI is resolved, it returns a JSON file with the following metadata:
Even though the URI string is fixed, this file lives off-chain so it is conceivable that its
contents could change at any point in the future. Additionally, this contract includes a way for its
owners to change this URI via a
var-set
function call:This setup is very flexible for administrators, but it creates a complex problem for metadata
indexers which now need to figure out if (and when) they should re-index token contracts to avoid
displaying stale metadata in their applications.
Metadata staleness
Within the Stacks ecosystem, there are a number of applications that need to index token metadata
and struggle with specific challenges caused by changed metadata. For example:
get-token-uri
on-chainvariable could change, the off-chain JSON file could change, and/or the image served by the URL
could change.
metadata to return account balances correctly.
Critical balance draining is possible when this property is zero at contract launch but updated
later.
For indexing, developers usually run and maintain a background process that listens for new token
contracts deployed to the blockchain so they can immediately call on their metadata to save the
results. This works for new contracts, but it is insufficient for old ones that may change their
metadata after it has been processed.
To avoid staleness, some indexers resort to a cron-like periodic refresh of all tracked contracts,
but while this may work for individual applications, it does not provide a consistent experience for
Stacks users that may interact with different metadata-aware systems with different refresh periods.
This workaround also adds unnecessary network traffic and creates extra strain on public Stacks
nodes due to aggressively polling contract-read RPC endpoints.
Metadata update notifications
To solve this problem reliably, contract administrators need a way to notify the network when they
have made changes to the metadata so any indexers may then perform a refresh just for that contract.
The proposed mechanism for these notifications leverages the
print
Claritylanguage function. When
used, its output is bundled inside an event of type
contract_event
:This event is then attached to a transaction object and broadcasted when the same transaction is
included in a block or microblock.
This SIP proposes a standard message structure (similar to a notification payload) that would be
used through
print
. Existing metadata indexers would receive this event through the Stacks nodeevent-emitter
interface,
parse and validate its contents, and refresh any contracts that were updated.
print
was alsoselected for the following reasons:
print
notifications in the Stacks ecosystem: the BNScontract, for example, uses it to notify the network when a change to a name or its zonefile has
occurred. The PoX-2 contract for Stacks 2.1 will make heavy use of it to record stacking state
changes across addresses. This SIP aims to continue this trend.
would enable, for example, a notification to be clearly displayed in the Stacks Explorer alongside
its transaction.
print
notification to a function's Clarity code also serves as self-explanatorydocumentation.
print
structure and indexers would be quick to adopt these if they need to. See Notificationstructure reusability.
Specification
Notification messages for each token class are specified below. Token metadata update notifications
must be made via a contract call transaction to the deployed reference
contract
or from a call to
print
within any other contract, including the token contract itself.Fungible Tokens
When a contract needs to notify the network that metadata has changed for a Fungible Token, it
shall call
print
with a tuple with the following structure:notification
"token-metadata-update"
payload.token-class
"ft"
payload.contract-id
payload.update-mode
payload.ttl
payload.update-mode: dynamic
Non-Fungible Tokens
When a contract needs to notify the network that metadata has changed for a Non-Fungible Token,
it shall call
print
with a tuple with the following structure:notification
"token-metadata-update"
payload.token-class
"nft"
payload.contract-id
payload.token-ids
payload.update-mode
payload.ttl
payload.update-mode: dynamic
If a notification does not contain a value for
payload.token-ids
, it means it is requesting anupdate for all tokens.
Semi-Fungible Tokens
When a contract needs to notify the network that metadata has changed for a Semi-Fungible Token,
it shall call
print
with a tuple with the following structure:notification
"token-metadata-update"
payload.token-class
"sft"
payload.contract-id
payload.token-ids
payload.update-mode
payload.ttl
payload.update-mode: dynamic
Notifications for SFTs must include a value for
payload.token-ids
.Metadata update modes
Applications may use tokens for very different purposes. Some of these could require none or very
few metadata updates ever (e.g. digital artwork that never changes except for reveals), while others
could need to alter it several times a day (e.g. NFTs for in-game items that are traded and modded
continuously).
This use-case variety also affects how developers decide to host their metadata JSON files. For
example, they could choose to use IPFS for low-frequency updates and finality, versus Amazon S3 for
high-frequency off-chain updates.
In order to allow creators and app developers to specify how token metadata should be treated by
indexers, notifications support an optional
payload.update-mode
key that may contain one of thefollowing values:
standard
: The new metadata will be valid until the next notification comes.This is the default mode if none is specified.
frozen
: This token's metadata will never change again, ever.Indexers should ignore new notifications for this token, even if valid.
dynamic
: The new metadata is expected to change very quickly and many times in the future (evenoff-chain).
Indexers should not expect to receive explicit notifications for each of these changes and
should consider refreshing this token's metadata frequently. Token developers may suggest a
reasonable amount of time between refreshes by adding an estimated value (defined in seconds) to the
payload.ttl
notification property.Considerations for metadata indexers
For a token metadata update notification to be considered valid by metadata indexers, it must meet
the following requirements:
NFT or an SFT.
contract_identifier
field of the contract event must be equal to thepayload.contract-id
(i.e., the event was produced by the contract that owns the metadata) orthe transaction's
tx-sender
principal should match the principal contained in thenotification's
payload.contract-id
(i.e., the STX address that sent the transaction which emitsthe notification should match the owner of the token contract being updated).
Notifications that do not meet these requirements must be ignored.
Other implications
not necessarily have a distinct effect in metadata responses when processed in the present.
updates.
create distinct records in the internal metadata database.
call rate limiting should be implemented locally.
metadata was actually updated.
Given these constraints the notifications this SIP proposes should be taken as hints to metadata
indexers. Metadata indexers are not obliged to follow them.
Notification structure reusability
Even though establishing a generalized smart contract notification standard is out of scope for this
SIP, the proposed
print
message structure was designed for reusability by future SIPs that wish tostandardize other events.
For example, developers could vary the
notification
andpayload
values to notify the networkwhen an NFT collection has been fully minted or another important milestone is reached.
Related work
An alternative considered for token metadata update notifications is for them to be transmitted via
an off-chain notification service that indexer developers may subscribe to, such as:
While these channels would have several advantages like being simpler to update, faster to
propagate, and easier to moderate, they have key disadvantages that make them inadequate for this
SIP's intended use:
unrelated to the Stacks ecosystem. As such, they could modify the channel, its reach, or its
rules at any time while affecting the entire network.
APIs and implementations, defeating the standardization purpose of this SIP. Moving
notifications to the blockchain establishes a canonical way to store and access them.
of friction for developer adoption.
channel will be much more difficult once the Stacks ecosystem has more token applications and
metadata indexers.
Backwards compatibility
Developers who need to emit metadata update notifications for tokens declared in older contracts
(that were deployed before this notification standard was established) could do so by either calling
the contract described in Reference Implementations or by first
deploying a new separate contract containing a public function that prints this notification and
then calling it to have it emitted.
Activation
This SIP will be activated when the following conditions are met:
that print the proposed notification payload.
listening for and reacting to the emitted notifications.
If the Stacks blockchain reaches block height 170000 and the above has not happened, this SIP will
be considered rejected.
Reference implementations
A reference contract has been deployed to mainnet as
SP1H6HY2ZPSFPZF6HBNADAYKQ2FJN75GHVV95YZQ.token-metadata-update-notify
.It demonstrates how to send notifications for each token class and it is available for developers to
use for refreshing any existing or future token contract. If the SIP evolves to require a change to
this contract pre-activation, a new one will be deployed and noted here.
The Stacks Blockchain API will also add
compatibility for this standard while this SIP is being considered to demonstrate how indexers can
listen for and react to these notifications.