This ignores the point I made.
The fastsync is about UTXOs, wallets (and SPV) is about transactions.
You can’t just replace one with the other and pretend you can get SPV proofs that are meant for transactions, for utxos.
If you think you can, please build it. Prove me wrong.
Until then, I don’t think fastsync in consensus makes much sense. Too high risk of changes needed (by hardfork) to make the normal usecase work.
Yes, I was inaccurate, sorry, let’s try again:
Bootstarting historyless SPV-serving nodes:
I agree.
This is in line with Satoshi’s “Nodes will be server farms”.
Of course, the corpos and operators running these farms will not be willing to pay the costs of maintaining all the tree back to genesis for no compensation.
But I also think that it is important that some independent/hobbyist/academic/subsidied by government nodes remain, so it can be verified at any time that somebody is not cheating and supplying clients with fake history for some nefarious/for-profit reason that we cannot imagine today.
Here’s some empirical evidence of there always being some copies of history: people seed Torrents, r/DataHoarder exists.
If Bitcoin Cash grows, having an archival node will come with big bragging rights.
Yes, also this is not going to happen in the next 20-30 years, because technology is still following Moore’s Law.
Hard drive space and Internet connection speed is rising geometrically. Much faster than adoption of BCH can probably proceed (that is, assuming no Black Swan-type scenario happens that causes billions of people to suddenly want to use Crypto).
I think the crux of the problem is the details of this step
A client that just synced the UTXO set could in theory connect to every known Electrum server and issue blockchain.transaction.get and blockchain.transaction.get_merkle for each UTXO to achieve it. (No I don’t think this is a good solution)
Or, since the block height of each UTXO is known, set the appropriate bloom filter and request merkelblock for specific block from nodes with full history. Rinse and repeat for each UTXO.
Or the node gets the data from a side-channel not specified in this spec.
One key point is that those transactions and proofs are not contained within the snapshot and we are still dependent on someone being around and serving it in some way.
To summarize:
For a node to be able to fully validate everything (i.e. being a “full node”) from the point of the UTXO snapshot nothing more is needed. This is the problem solved by this CHIP and a miner validated commitment in a future CHIP could make this fully trust-less.
For a node to able to serve history (transactions and proofs) from the point of the UTXO snapshot additional data needs to fetched and validated. To me this seems like a separate CHIP if we would like a standardized way of doing it.
So,
you just turned the small utxo set download into “download all full transactions that still have at least 1 unspent output in them”.
Maybe good to re-do the math (how big the download is) and the fastsync chip based on this new approach.
I suspect it changes a LOT.
When someone implements that, so we can be certain it actually works (not just theoretically), I’d love to see some discussion on how useful or viable it is for fastsync.
That’s only if you want to run your node as a historyless SPV server.
If you just need a node wallet, you don’t need the SPV stuff since you’ll have obtained the UTXOs from the snapshot and you’ll continue to maintain it as part of normal node activities.
You are not wrong, but for that usecase I suggest you just run an SPV wallet instead. Its massively less download for the same gain.
Also permissionless, trustless and actually works on-chain today.
and fully dependent on altruistic nodes that need to sync all history from genesis, and be ready to support such wallets queries.
yes,
so lets aim to make the fastsync actually capable of doing so. Because if we don’t then the vast majority of people that are indeed going to use SPV wallets can no longer use BitcoinCash as more and more fastsync nodes start popping up.
As I pointed out for some time, there is a logical error in your steps to get from the basic sync as the CHIP now states to actually being able to serve wallets. If you don’t believe me, then prove me wrong by building your suggestion in some proof of concept.
Arguing here makes no sense. This forum software is telling me I’ve commented enough on this thread. And, you know, its right. I’m not going to continue nicely trying to save you from the mistakes I see (for 18 months now) and you are not seeing.
Build it, saves us a lot of talk.
BLISS Presentation: Committing to UTXOs with Calin Culianu
It recently occurred to me that a SPV wallet doesn’t need the transactions and proofs from the timepoint before the UTXO snapshot. The node can just give the wallet the relevant UTXOS and the wallet is able to spend it by signing with SIGHASH_UTXOS.
This will simplify everything significantly since a node serving thin wallets don’t need the additional historical data.
Yeah! In that case, no need for this:
You can safely assume that the UTXOs you were given are legitimate and try spend them with SIGHASH_UTXOS. If the TX fails, it can only be one of the 2 reasons:
TXID:n doesn’t match any UTXO)TXID:n matches some UTXO, but you got the wrong contents for it)It has come to my attention that Kaspa has already implemented UTXO commitments in block header and new nodes sync off headers and UTXO snapshots. I dug out a blog post by Yonatan Sompolinsky:
Accordingly, Kaspa nodes prune block data by default, and new nodes by default do not request historical data, rather, they sync in SPV mode, i.e., by downloading and verifying only block headers. I reiterate that this is not a stronger trust assumption than a history-verifying node, rather a different requirement. The node then requests the UTXO set from untrusted peers in the network, and verifies it against the UTXO commitment embedded inside the latest received header (technically, this is done against the latest pruning point). If those do not match, the node bans the sending peers, requests the UTXO set from new untrusted peers, and repeats the process. If those match, the node verifies that no unexpected inflation occurred by comparing the sum of UTXOs to the specified minting schedule, a comparison for which block headers suffice.
Not only that, but they’re not using EC multiset, but something else. I asked them about it and the said Muhash had better performance, and they were in fact using EC multiset and then replaced it by Muhash, as evident in this PR: Replace ECMH with Muhash by elichai · Pull Request #1624 · kaspanet/kaspad · GitHub
Note I am aware of MuHash and we will evaluate it too. We already added it as a potential hasher to BCHN: src/crypto/muhash.h · master · Bitcoin Cash Node / Bitcoin Cash Node · GitLab
You’re amazing, sir! Always ahead of the game!
TIL BU/Nexa did it another way:
We use a compact bitwise prefix trie rather than a Merkle tree, so that miners commit only to the set of transactions each block. The order that transactions are added to the trie does matter. This trie structure then enables compact proofs that a coin is unspent.
Asked about trade-offs when compared to EC multiset:
EC multiset doesn’t enable compact membership proofs: you CAN prove you have the right set, but you can NOT prove that a given coin is a member of that set in a compact way.
The trade-off is that to get the compact membership proofs, the UTXO set must be stored in a specific structure (e.g., a compact bitwise prefix trie). It would be difficult to shoehorn that trie structure efficiently into LevelDB. So implementing EC multiset is easier.
We’d use a 256-bit commitment (same as Merkle root) so commitment size is small in both cases.
It scales a log(N), so each insert/delete costs a bit more the larger the UTXO set grows. But it never becomes unmanageable: even for very very large UTXO sets it’s less computationally than a single elliptic curve point multiplication.
You can roughly think of it as hashing one branch of the trie, from leaf to root, to update the commitment.
– source
May be worth the trade-off, log(N) for capability to make compact proof about any UTXO.
This would mean that contracts could “read” any UTXO without it having to be spent or referenced by the TX. The spender would provide the UTXO+proof as input script.
Speaking with @cculianu it’s probably NOT worth it, it would tie our hands for future scaling, having an irremovable O(log(N)) operation part of validation critical path. Even if the constants are low so time per UTXO is small even with a 1T set, still, an O(1) algo would outperform that. There’s also the question of parallelization. All those problems, for what benefit?
Exactly. How useful is a UTXO proof? You know that at exactly height N, the moment block N was confirmed, UTXO U was unspent.
You do now know if has since been spent. (At least the proof itself provides you with no guarantees in that regard!! Think about it!).
For applications that truly care about this (I ask which ones care?) – you have no way to know if the UTXO is unspent without additional information and/or things you must do anyway.
For the lite wallet case, UTXO proofs are of limited use and do not really tell you more than you couldn’t have equally figured out with simple SPV proofs of txns which we have now.
A lite wallet also knows which UTXOs it has signed for and which it has not – so for it, receiving a simple SPV txn proof is enough for it to know the UTXO is still unspent – since it knows it never spent it!
So yeah, the UTXO proof is not that useful I think, in the real-world, and it comes with some burden.
Also – there are faster algos for multisets. EC Multiset is just one. Core is flirting with MuHash which is 10-100x faster per add/delete than EC Multiset. Just sayin’.