@mtrycz The specification seems to say all input UTXO’s must be confirmed for a tx to be protected. If that is the case, it isn’t possible to double-spend an unconfirmed ancestor.
Hello Mr Harding!
The spec is somewhat vague in its language. To the best of my understanding what is meant is that only double spends of confirmed outputs are to be considered “protected by DSPs”, but actually, in the current implementation in BCHN (Flowee is same, and I also think BU too) DSPs will be produced form unconfirmed transactions.
The BCHN test suite for DSPs is rather extensive about the various cases.
1 Like
I am thinking the following for the RPC:
getdsproof
(no args) - This would dump all the known DSPs (if any). Returns an array of json objects, each object is of the form:
[
{
"dspid" : "01231879afedfab28cd...", // dspid hex of this dsproof
"txid" : "abcdef012331babbcaef1231..", // txid associated with this dsproof
"outpoint" : "0e9389872837182731abcdeffff012:0" // <prevouthash:N outpoint that is being double-spent>
}, ...
]
Also lookup by txid, dspid, or prevouthash:N:
getdsproof <txid or dspid or prevouthash:N>
If txid/dspid/prevouthash:N has a DSP associated, then returns the dict above – additionally ALSO adding the data blob of the proof as the following:
"proof" : "<hex data>"
otherwise returns null.
Querying orphans: I’m not sure if this is useful but if it were, then we could dump orphans additonally with null as the txid as well.
Optionally, maybe we could also add a key to the getrawmempool verbose=true call for dspid or somesuch… perhaps.
1 Like
Looks reasonable.
Does Flowee have any API for this already?
This is under the subject of “Limitation and risks”, not part of the spec per-see. We moved the spec to the BCH common one as can be found here. The quoted part is about consumers. Consumers should be aware that they won’t get protection from the message alone, they need to do more. For instance subscribe to the entire unconfirmed chain (put all the unconfirmed parents in your bloom filter will do it).
This is actually relevant here because the DSProof message in-and-of-itself isn’t enough for SPV wallets to protect themselves. The RPC calls we are talking about here thus should also be task-oriented and for that we need to think about the usecases.
Pure SPV
An Pure SPV wallet that receives a transaction using its bloom filters should somehow reach the goal of having all unconfirmed inputs in its bloom filter. This is the way to get the proof not just for the tip, but for the entire chain. To do this they need to first actually fetch the transactions that the inputs refer to in order to do this adding recursively. A getdata will do that for you. But you won’t know if these transactions are confirmed or not…
The bottom line here is that the p2p network is too slow and doesn’t have the features to protect anything but the tip.
This is relevant to understand. In order to do proper DSProofs we will need middleware like Fulcrum. Practically all wallets are already leaning that way, so I’m not fighting that.
If people strongly disagree with that path then we need to consider adding dsproof support to the mempool p2p command. Or maybe invent a new p2p command that is for SPV. The merkleblock command is purely for SPV in the same manner.
Flowee invented a new API which is a binary protocol. It exists alongside the RPC (JSON/REST) and the ZMQ services but you can avoid the latter two by just using the former. Much faster too. Most Flowee components use this binary API to communicate between themselves.
In this API I have the address-monitor service. Its a ZMQ style service where the node pushes notifications. A client registers an address (actually its a script-hash like electronx uses) and gets notificatoins on stuf happening on that address. You get a push when a transaction has said address as an output, you get a push when said transaction is mined.
And indeed it has 2 ways to push double spend proofs. One is when the node itself noticed the double spend proof, in which case the node has both transactions.
The other is when the node gets a validated proof from the network, in which case it doesn’t have both transactions, but it does have the proof. (docs).
1:
for (auto hash : match.hashes)
builder.add(Api::AddressMonitor::BitcoinScriptHashed, hash);
for (auto amount : match.amounts)
builder.add(Api::AddressMonitor::Amount, amount);
builder.add(Api::AddressMonitor::TxId, first.createHash());
builder.add(Api::AddressMonitor::TransactionData, duplicate.data());
2:
for (auto hash : match.hashes)
builder.add(Api::AddressMonitor::BitcoinScriptHashed, hash);
for (auto amount : match.amounts)
builder.add(Api::AddressMonitor::Amount, amount);
builder.add(Api::AddressMonitor::TxId, txInMempool.createHash());
builder.addByteArray(Api::AddressMonitor::DoubleSpendProofData,
&serializedProof[0], serializedProof.size());
No RPC API, just the push-apis.
I like this one too. I would add a boolean ‘recursive’ when querying the txid or prevout, which recursively checks prevout transactions until the confirmed ones and returns dsproofs for all.
1 Like
Sounds perfect.
Actually, I ponder whether the recursion should be the default with optional false
2 Likes
The wallet wanting notifications asap would have to keep asking for the chain. Maybe the bloom filter could be updated with the unconfirmed ancestors so that any double-spend notifications along the chain would go out immediately.
Updated my comment at RPC API considerations for DSProof - #3 by freetrader with some links to BU commits for their ZMQ notification options:
-zmqpubhashds=endpoint-zmqpubrawds=endpoint
That is not a bad idea, but it causes more problems. The fact is that your pure-p2p SVP wallet doesn’t have the entire unconfirmed chain, and as such when a DSProof comes in from a node for one of the unconfirmed ancestors, it doesn’t know how to validate it. Which routes back to my earlier point: the p2p layer is just too slow for this. We want a wallet to reach a conclusive “All Ok” within several seconds (say 3).
As practically all wallets in use today depend on some middleware, the best bang for the buck is to make the electrumx protocol more mature and more used as a place for SPV wallets to get their info.
Ok well I implemented RPC support. Here’s the branch where I added it:
There are two new RPC commands:
getdsprooflist - lists all dsproofs (optionally returning verbose info as well as orphans)
getdsproof - gets a specific dsproof using dspid, txid, or coutpoint. May return an orphan.
RPC help for getdsprooflist:
getdsprooflist ( verbosity include_orphans )
List double-spend proofs for transactions in the mempool.
Arguments:
1. verbosity (numeric, optional, default=0) Values 0-3 return progressively more information for each increase in verbosity. This option may also be specified as a boolean where false is the same as verbosity=0 and true is verbosity=2.
2. include_orphans (boolean, optional, default=false) If true, then also include double-spend proofs that we know about but which are for transactions that we don't yet have.
Result (for verbosity = 0 or false):
[ (json array of string)
"dspid" (string) Double-spend proof ID as a hex string.
, ...
]
Result (for verbosity = 1):
[ (json array of object)
{ (json object)
"hex" : "xxx", (string) The raw serialized double-spend proof data.
"txid" : "xxx" (string) The txid of the transaction associated with this double-spend. May be null for "orphan" double-spend proofs.
}, ...
]
Result (for verbosity = 2 or true):
[ (json array of object)
{ (json object)
"dspid" : "xxx", (string) Double-spend proof ID as a hex string.
"txid" : "xxx", (string) The txid of the transaction associated with this double-spend. May be null for "orphan" double-spend proofs.
"outpoint" : (json object) The previous output (coin) that is being double-spent.
{
"txid" : "xxx", (string) The previous output txid.
"vout" : n , (numeric) The previous output index number.
}
}, ...
]
Result (additional keys for verbosity = 3):
...
"spenders" : (json array of object) The conflicting spends.
[
{ (json object)
"txversion" : n (numeric) Transaction version number.
"sequence" : n (numeric) Script sequence number.
"locktime" : n (numeric) Spending tx locktime.
"hashprevoutputs" : "xxx" (string) Hash of the previous outputs.
"hashsequence" : "xxx" (string) Hash of the sequence.
"hashoutputs" : "xxx" (string) Hash of the outputs.
"pushdata" : (json object) Script signature push data.
{
"asm" : "xxx" (string) Script assembly representation.
"hex" : "xxx" (string) Script hex.
}
}, ...
]
Examples:
> bitcoin-cli getdsprooflist 2 false
> bitcoin-cli getdsprooflist false false
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getdsprooflist", "params": [1, true] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/
RPC Help for getdsproof:
getdsproof "dspid_or_txid_or_outpoint" ( verbosity recursive )
Get information for a double-spend proof.
Arguments:
1. dspid_or_txid_or_outpoint (string, required) The dspid, txid, or output point associated with the double-spend proof you wish to retrieve. Outpoints should be specified as a json object containing keys "txid" (string) and "vout" (numeric).
2. verbosity (numeric, optional, default=2) Values 0-3 return progressively more information for each increase in verbosity. This option may also be specified as a boolean where false is the same as verbosity=0 and true is verbosity=2.
3. recursive (boolean, optional, default=true) If doing a lookup by txid, then search for a double-spend proof for all in-mempool ancestors of txid as well. This option is ignored if not searching by txid.
Result (for verbosity = 0, 1, false):
{ (json object)
"hex" : "xxx", (string) The raw serialized double-spend proof data.
"txid" : "xxx" (string) The txid of the transaction associated with this double-spend. May be null for "orphan" double-spend proofs.
}
Result (for verbosity = 2, true):
{ (json object)
"dspid" : "xxx", (string) Double-spend proof ID as a hex string.
"txid" : "xxx", (string) The txid of the transaction associated with this double-spend. May be null for "orphan" double-spend proofs.
"outpoint" : (json object) The previous output (coin) that is being double-spent.
{
"txid" : "xxx", (string) The previous output txid.
"vout" : n , (numeric) The previous output index number.
}
}
Result (additional keys if searching by txid and recursive = true):
...
"ancestors" : (json array of string) Ancestors leading to the double-spend.
[
"txid" : (string) Txid hex, ordered by by child->parent.
, ...
]
Result (additional keys for verbosity = 3 or true):
...
"spenders" : (json array of object) The conflicting spends.
[
{ (json object)
"txversion" : n (numeric) Transaction version number.
"sequence" : n (numeric) Script sequence number.
"locktime" : n (numeric) Spending tx locktime.
"hashprevoutputs" : "xxx" (string) Hash of the previous outputs.
"hashsequence" : "xxx" (string) Hash of the sequence.
"hashoutputs" : "xxx" (string) Hash of the outputs.
"pushdata" : (json object) Script signature push data.
{
"asm" : "xxx" (string) Script assembly representation.
"hex" : "xxx" (string) Script hex.
}
}, ...
]
Examples:
> bitcoin-cli getdsproof d3aac244e46f4bc5e2140a07496a179624b42d12600bfeafc358154ec89a720c false
> bitcoin-cli getdsproof fb5ae5344cb6995e529201fe24247ac38452f4e5ab5669b649e935853a7a180a
> bitcoin-cli getdsproof fb5ae5344cb6995e529201fe24247ac38452f4e5ab5669b649e935853a7a180a true true
> bitcoin-cli getdsproof fb5ae5344cb6995e529201fe24247ac38452f4e5ab5669b649e935853a7a180a 1 false
> bitcoin-cli getdsproof '{"txid": "e66c1848fd3268a7d1cfac833f9164057805cc9b22ea5521d36dc4cf63f5fe83", "vout": 0}' true
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getdsproof", "params": ["fb5ae5344cb6995e529201fe24247ac38452f4e5ab5669b649e935853a7a180a", true, false] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getdsproof", "params": [{"txid": "e66c1848fd3268a7d1cfac833f9164057805cc9b22ea5521d36dc4cf63f5fe83", "vout": 0}, true] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/
Note: Updated to include the recursive paramenter
2 Likes
Hmm. So recursion might be kind of slow but – it would be useful otherwise the wallet has to do it (even slower).
I’ll see what I can do.
I might end up making that a separate call since it doesn’t fit into the “basic” query RPC methods I am adding above (see previous my comment where I outline the basic API I developed).
2 Likes
Looking at ther verbosity flag I’m thinking that the abilty for someone to download the actual proof (hexdata) is a bit overly expensive as you put that in the verbosity=2.
Maybe the hex is what you want to return when verbosity is zero. And you never return the hex for higher verbosity, just the interpreted version. This seems more consistent with other bitcoind RPC calls.
1 Like
@tom - Ok, verbosity flag is changed as per your suggestion. For getdsproof verbosity=false does what you expect, and dumps hex data (same as getblock false).
Note that the getdsprooflist command’s verbosity=false defaults to just a list of dspids. You need to do verbosity=1 to get the hex data in this “listing” mode (this is a command designed for polling… that’s why).
This now works more like getblock. Thanks for the suggestion!
FYI – Merge request for BCHN for this lives here now: RPC methods for the DSProof subsystem (!1026) · Merge Requests · Bitcoin Cash Node / Bitcoin Cash Node · GitLab
2 Likes
@tom The ancestors are relayed to the peer in addition to being added to the filter, sorry if this wasn’t obvious.
Its not a bad idea, but its not the best solution I’d work on today. Imagine your wallet is on mobile and the transaction is 500 deep.
@tom How can an electrum client validate the double-spend of one of those 500 ancestors without receiving them? Does the server say “trust me, this is one of your unconfirmed ancestors, and it has been double-spent”?
This is an excellent question.
As the current level of the stack people are working on is full node (and thus the input to electrumx protocol servers) it is likely that the next step is to talk about the protocol to talk between the electrumx server and its clients.
Everyone that has good ideas about how to do this in a smart way while still making it fast is needed to solve this in the best way possible.
Notifying of an additional RPC call that has been merged into BCHN’s master branch.
https://docs.bitcoincashnode.org/doc/json-rpc/getdsproofscore.html
Whether to release in current form is something that @cculianu has been thinking about, and I would invite others to give feedback on the call.