RPC API considerations for DSProof

Technical
#1

We’ve had some preliminary discussions within BCHN about what might be useful in terms of the RPC API for double spend proofs.

This thread is to consolidate this into some proposals, using input from the node teams, so we can come up with a common API if possible.

I’ll start off by dumping some comments / ideas around this, harvested from BCHN slack channel (#dev-general) from around December 20, 2020.

@cculianu suggested that there could be an API that informs of the DSP status of a txid, in terms of the following:

  1. DSP detected
  2. No DSP detected (yet)
  3. DSP not supported

@BigBlockIfTrue reasoned that

if you did not receive a dsproof, it is important to know whether that is because there is no double-spend attempt, or because no dsproof can be constructed

He also proposed

whatever you do, just make sure the final api clearly distinguishes these three possible situations for each tx:

  1. covered by dsproofs, double-spend attempt detected
  2. covered by dsproofs, no double-spend attempt detected
  3. not covered by dsproofs

covered = if peers behave properly, they are able to detect a double-spend and notify us about it

I think being able to obtain such “coverage” status for a transaction would be very helpful to check, as a first step, whether one has to bother enquiring about the arrival of a DSProof again after a certain time.

For transactions that aren’t covered, one can know in advance to simply skip that follow-up check.

@mtrycz tried to promote a conceptual shift:

Instead of talking of double spent transactions,
Let’s start talking of double spent chains.

Stemming from the possibility of a double spend in an unconfirmed chain not being the leaf/tip of that chain of transactions.

Being able to quickly look up if a transaction has a double-spent ancestor might be useful. Same for a notification received for children when a doublespend of an ancestor is detected.

5 Likes
#2

From private discussions elsewhere, a crude suggestion:

getdsproof <optional txid>

Returns verified dsproof for the specified txid. if txid is not specified, return a list of hashes (or the entirety of the dsproofs themselves?) of all dsproofs with their associated txids.

Just to open things up. :slight_smile:

1 Like
#3

And just from looking at my node’s log, I’d like a ZMQ notification I can subscribe to for DSProofs that either arrive or are generated.

It should inform whether they’ve been validated, or invalid, or resulted in an orphan.
It might be better to have separate notifications that filter for these cases.

In fully validated case, it might be nice to provide both the mempool tx and the doublespend tx in full, as they should still be available at that time.

Oops, not sure if ZMQ is covered under this topic but I’m leaving this here for now.

Update:

I saw BU had set up some ZMQ notifications in their client already, ref.

#4

ZMQ is the first interface that comes to mind because:

  • it’s a fast push interface (vs the pull/poll RPC interface), and it’s good to have DSP notifications asap
  • ZMQ infrastructure is already in place.

To the best of my understanding, tho, the current ZMQ messages only inform that there is a message and the id (txid and blockid) to check it through RPC . If I’m correct I would tiepidly favour maintaining this paradigm. But do DPSs even have an id?

1 Like
#5

Agreed, and yes, DSPs have an id

The hash-ID for the double-spend-proof is a double sha256 over the entire
serialized content of the proof

#6

Yes, adding notifications via ZMQ is not a bad idea and probably would be useful. It might be useful to have the ZMQ message contain the txid as well as the COutPoint (coin) being spent … these could all be message “parts” (ZMQ supports multiple subparts in the same message, IIRC).

#7

It should inform whether they’ve been validated, or invalid, or resulted in an orphan.

I don’t think anybody will care about invalid DSPs. That’s just not something that should bubble up to the top layer.

As for orphans: those may or may not be useful. Perhaps the orphan can have a null txid but still inform of the COutPoint being spent…

IMHO – It’s best to notify when the DSP is actually associated with a mempool tx. That’s the primary usecase.

#8

@mtrycz The specification seems to say all input UTXO’s must be confirmed for a tx to be protected. If that is the case, it isn’t possible to double-spend an unconfirmed ancestor.

#9

Hello Mr Harding!

The spec is somewhat vague in its language. To the best of my understanding what is meant is that only double spends of confirmed outputs are to be considered “protected by DSPs”, but actually, in the current implementation in BCHN (Flowee is same, and I also think BU too) DSPs will be produced form unconfirmed transactions.

The BCHN test suite for DSPs is rather extensive about the various cases.

1 Like
#10

I am thinking the following for the RPC:

getdsproof

(no args) - This would dump all the known DSPs (if any). Returns an array of json objects, each object is of the form:

[
    {
        "dspid" : "01231879afedfab28cd...", // dspid hex of this dsproof
        "txid" : "abcdef012331babbcaef1231..", // txid associated with this dsproof
        "outpoint" : "0e9389872837182731abcdeffff012:0" // <prevouthash:N outpoint that is being double-spent>
    }, ...
]

Also lookup by txid, dspid, or prevouthash:N:

getdsproof <txid or dspid or prevouthash:N>

If txid/dspid/prevouthash:N has a DSP associated, then returns the dict above – additionally ALSO adding the data blob of the proof as the following:

"proof" : "<hex data>"

otherwise returns null.

Querying orphans: I’m not sure if this is useful but if it were, then we could dump orphans additonally with null as the txid as well.

Optionally, maybe we could also add a key to the getrawmempool verbose=true call for dspid or somesuch… perhaps.

1 Like
#11

Looks reasonable.

Does Flowee have any API for this already?

#12

This is under the subject of “Limitation and risks”, not part of the spec per-see. We moved the spec to the BCH common one as can be found here. The quoted part is about consumers. Consumers should be aware that they won’t get protection from the message alone, they need to do more. For instance subscribe to the entire unconfirmed chain (put all the unconfirmed parents in your bloom filter will do it).

This is actually relevant here because the DSProof message in-and-of-itself isn’t enough for SPV wallets to protect themselves. The RPC calls we are talking about here thus should also be task-oriented and for that we need to think about the usecases.

Pure SPV

An Pure SPV wallet that receives a transaction using its bloom filters should somehow reach the goal of having all unconfirmed inputs in its bloom filter. This is the way to get the proof not just for the tip, but for the entire chain. To do this they need to first actually fetch the transactions that the inputs refer to in order to do this adding recursively. A getdata will do that for you. But you won’t know if these transactions are confirmed or not…

The bottom line here is that the p2p network is too slow and doesn’t have the features to protect anything but the tip.

This is relevant to understand. In order to do proper DSProofs we will need middleware like Fulcrum. Practically all wallets are already leaning that way, so I’m not fighting that.
If people strongly disagree with that path then we need to consider adding dsproof support to the mempool p2p command. Or maybe invent a new p2p command that is for SPV. The merkleblock command is purely for SPV in the same manner.

#13

Flowee invented a new API which is a binary protocol. It exists alongside the RPC (JSON/REST) and the ZMQ services but you can avoid the latter two by just using the former. Much faster too. Most Flowee components use this binary API to communicate between themselves.

In this API I have the address-monitor service. Its a ZMQ style service where the node pushes notifications. A client registers an address (actually its a script-hash like electronx uses) and gets notificatoins on stuf happening on that address. You get a push when a transaction has said address as an output, you get a push when said transaction is mined.

And indeed it has 2 ways to push double spend proofs. One is when the node itself noticed the double spend proof, in which case the node has both transactions.
The other is when the node gets a validated proof from the network, in which case it doesn’t have both transactions, but it does have the proof. (docs).

1:
        for (auto hash : match.hashes)
            builder.add(Api::AddressMonitor::BitcoinScriptHashed, hash);
        for (auto amount : match.amounts)
            builder.add(Api::AddressMonitor::Amount, amount);
        builder.add(Api::AddressMonitor::TxId, first.createHash());
        builder.add(Api::AddressMonitor::TransactionData, duplicate.data());

2: 
       for (auto hash : match.hashes)
            builder.add(Api::AddressMonitor::BitcoinScriptHashed, hash);
        for (auto amount : match.amounts)
            builder.add(Api::AddressMonitor::Amount, amount);
        builder.add(Api::AddressMonitor::TxId, txInMempool.createHash());
        builder.addByteArray(Api::AddressMonitor::DoubleSpendProofData, 
                       &serializedProof[0], serializedProof.size());
#14

No RPC API, just the push-apis.

I like this one too. I would add a boolean ‘recursive’ when querying the txid or prevout, which recursively checks prevout transactions until the confirmed ones and returns dsproofs for all.

1 Like
#15

Sounds perfect.

Actually, I ponder whether the recursion should be the default with optional false

2 Likes
#16

The wallet wanting notifications asap would have to keep asking for the chain. Maybe the bloom filter could be updated with the unconfirmed ancestors so that any double-spend notifications along the chain would go out immediately.

#17

Updated my comment at RPC API considerations for DSProof - #3 by freetrader with some links to BU commits for their ZMQ notification options:

  • -zmqpubhashds=endpoint
  • -zmqpubrawds=endpoint
#18

That is not a bad idea, but it causes more problems. The fact is that your pure-p2p SVP wallet doesn’t have the entire unconfirmed chain, and as such when a DSProof comes in from a node for one of the unconfirmed ancestors, it doesn’t know how to validate it. Which routes back to my earlier point: the p2p layer is just too slow for this. We want a wallet to reach a conclusive “All Ok” within several seconds (say 3).

As practically all wallets in use today depend on some middleware, the best bang for the buck is to make the electrumx protocol more mature and more used as a place for SPV wallets to get their info.

#19

Ok well I implemented RPC support. Here’s the branch where I added it:

There are two new RPC commands:

getdsprooflist - lists all dsproofs (optionally returning verbose info as well as orphans)
getdsproof - gets a specific dsproof using dspid, txid, or coutpoint. May return an orphan.

RPC help for getdsprooflist:

getdsprooflist ( verbosity include_orphans )

List double-spend proofs for transactions in the mempool.

Arguments:
1. verbosity          (numeric, optional, default=0) Values 0-3 return progressively more information for each increase in verbosity. This option may also be specified as a boolean where false is the same as verbosity=0 and true is verbosity=2.
2. include_orphans    (boolean, optional, default=false) If true, then also include double-spend proofs that we know about but which are for transactions that we don't yet have.

Result (for verbosity = 0 or false):
[                                  (json array of string)
  "dspid"                          (string) Double-spend proof ID as a hex string.
  , ...
]

Result (for verbosity = 1):
[                                  (json array of object)
  {                                (json object)
    "hex" : "xxx",                 (string) The raw serialized double-spend proof data.
    "txid" : "xxx"                 (string) The txid of the transaction associated with this double-spend. May be null for "orphan" double-spend proofs.
  }, ...
]

Result (for verbosity = 2 or true):
[                                  (json array of object)
  {                                (json object)
    "dspid" : "xxx",               (string) Double-spend proof ID as a hex string.
    "txid" : "xxx",                (string) The txid of the transaction associated with this double-spend. May be null for "orphan" double-spend proofs.
    "outpoint" :                   (json object) The previous output (coin) that is being double-spent.
    {
      "txid" : "xxx",              (string) The previous output txid.
      "vout" : n ,                 (numeric) The previous output index number.
    }
  }, ...
]

Result (additional keys for verbosity = 3):
    ...
    "spenders" :                   (json array of object) The conflicting spends.
    [
      {                            (json object)
        "txversion" : n            (numeric) Transaction version number.
        "sequence" : n             (numeric) Script sequence number.
        "locktime" : n             (numeric) Spending tx locktime.
        "hashprevoutputs" : "xxx"  (string) Hash of the previous outputs.
        "hashsequence" : "xxx"     (string) Hash of the sequence.
        "hashoutputs" : "xxx"      (string) Hash of the outputs.
        "pushdata" :               (json object) Script signature push data.
        {
          "asm" : "xxx"            (string) Script assembly representation.
          "hex" : "xxx"            (string) Script hex.
        }
      }, ...
    ]

Examples:
> bitcoin-cli getdsprooflist 2 false
> bitcoin-cli getdsprooflist false false
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getdsprooflist", "params": [1, true] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/

RPC Help for getdsproof:

getdsproof "dspid_or_txid_or_outpoint" ( verbosity recursive )

Get information for a double-spend proof.

Arguments:
1. dspid_or_txid_or_outpoint    (string, required) The dspid, txid, or output point associated with the double-spend proof you wish to retrieve. Outpoints should be specified as a json object containing keys "txid" (string) and "vout" (numeric).
2. verbosity                    (numeric, optional, default=2) Values 0-3 return progressively more information for each increase in verbosity. This option may also be specified as a boolean where false is the same as verbosity=0 and true is verbosity=2.
3. recursive                    (boolean, optional, default=true) If doing a lookup by txid, then search for a double-spend proof for all in-mempool ancestors of txid as well. This option is ignored if not searching by txid.

Result (for verbosity = 0, 1, false):
{                                (json object)
  "hex" : "xxx",                 (string) The raw serialized double-spend proof data.
  "txid" : "xxx"                 (string) The txid of the transaction associated with this double-spend. May be null for "orphan" double-spend proofs.
}

Result (for verbosity = 2, true):
{                                (json object)
  "dspid" : "xxx",               (string) Double-spend proof ID as a hex string.
  "txid" : "xxx",                (string) The txid of the transaction associated with this double-spend. May be null for "orphan" double-spend proofs.
  "outpoint" :                   (json object) The previous output (coin) that is being double-spent.
  {
    "txid" : "xxx",              (string) The previous output txid.
    "vout" : n ,                 (numeric) The previous output index number.
  }
}

Result (additional keys if searching by txid and recursive = true):
  ...
  "ancestors" :                  (json array of string) Ancestors leading to the double-spend.
  [
    "txid" :                     (string) Txid hex, ordered by by child->parent.
    , ...
  ]

Result (additional keys for verbosity = 3 or true):
  ...
  "spenders" :                   (json array of object) The conflicting spends.
  [
    {                            (json object)
      "txversion" : n            (numeric) Transaction version number.
      "sequence" : n             (numeric) Script sequence number.
      "locktime" : n             (numeric) Spending tx locktime.
      "hashprevoutputs" : "xxx"  (string) Hash of the previous outputs.
      "hashsequence" : "xxx"     (string) Hash of the sequence.
      "hashoutputs" : "xxx"      (string) Hash of the outputs.
      "pushdata" :               (json object) Script signature push data.
      {
        "asm" : "xxx"            (string) Script assembly representation.
        "hex" : "xxx"            (string) Script hex.
      }
    }, ...
  ]

Examples:
> bitcoin-cli getdsproof d3aac244e46f4bc5e2140a07496a179624b42d12600bfeafc358154ec89a720c false
> bitcoin-cli getdsproof fb5ae5344cb6995e529201fe24247ac38452f4e5ab5669b649e935853a7a180a
> bitcoin-cli getdsproof fb5ae5344cb6995e529201fe24247ac38452f4e5ab5669b649e935853a7a180a true true
> bitcoin-cli getdsproof fb5ae5344cb6995e529201fe24247ac38452f4e5ab5669b649e935853a7a180a 1 false
> bitcoin-cli getdsproof '{"txid": "e66c1848fd3268a7d1cfac833f9164057805cc9b22ea5521d36dc4cf63f5fe83", "vout": 0}' true
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getdsproof", "params": ["fb5ae5344cb6995e529201fe24247ac38452f4e5ab5669b649e935853a7a180a", true, false] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getdsproof", "params": [{"txid": "e66c1848fd3268a7d1cfac833f9164057805cc9b22ea5521d36dc4cf63f5fe83", "vout": 0}, true] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/

Note: Updated to include the recursive paramenter

2 Likes
#20

Hmm. So recursion might be kind of slow but – it would be useful otherwise the wallet has to do it (even slower).

I’ll see what I can do.

I might end up making that a separate call since it doesn’t fit into the “basic” query RPC methods I am adding above (see previous my comment where I outline the basic API I developed).

2 Likes