Hi TierNolan.
Just want to say that you nailed it and all you write is correct. Thank you for adding!
Personally I prefer to not talk about “hard fork” and just call it a protocol upgrade. This makes clear that such a change does not cause the chain to split.
Again, thank you for adding your post here!
Correct, but as @markblundeberg points out in the description, it’s more about not having unwanted tx versions be mined after announcement of a proposal.
I was thinking of someone just creating transactions. That is mostly protected by miners using the IsStandard rules. But agreed, even then a non-standard enforcing miner could mine those transactions, if a spammer could get them to the miner.
The only way to actually prevent the spam would be to have a majority of miners do a covert soft fork.
Essentially, 50%+ of miners announce that for the next 60 days, they will reject any blocks with transactions versions other than 1 or 2. During that 60 days, a formal rule could be discussed and put in place.
Is there a list of what versions have already been used?
Something like this would mean that it doesn’t require a hard to add a new version number later.
if (block_height < 750000) {
if (version < 1 || version > 2) {
return false;
}
}Every year, 50k could be added to the threshold unless there is another transaction version required.
The other side of the argument is what happens to people who have non-standard transactions but can’t spend them due to locktimes in the future. Arguably, it is their own fault for using unspecified version numbers.
How big of a problem is this? Those dealing with new not-yet-mined TX-es can parse them using the new rules even if the same ver. appeared in the past, because whatever block they’ll land in, it will certainly be higher than the block where some new TX format was activated.
If you’re doing mined TXes verification, then you need to be aware of block height anyway to know which consensus rules to apply for verification.
I guess the problem would be for non-node software which is fed already verified TX-es and not verifying it by itself, and if it assumes that the TX format will be the same regardless of block height and what’s written in the version field. Do we know how much software has this assumption?
I was talking with @im_uname and @freetrader about this, and we agreed that a decision should be made ASAP whether to commit to breaking this assumption for May 2023 upgrade, and then communicated to stakeholders, so they can start updating their software “today” not to make assumptions which won’t hold in the future.
I agreed to write some kind of technical bulletin that can be circulated among stakeholders, consider this as information gathering in preparation for a “Need to create new tx format for 2023 upgrade?” forum post 
This whole thing leaves a bad taste in my mouth as this was discussed at length 18 months ago with all the relevant stakeholders agreeing. Not blaming anyone, just sad to see that something this low on impact and this high in return would miss the 2022 slot.
The quoted part from Mark is something I agree with; its a technical nightmare for wallets should this not be done WELL in front of any transaction format change.
The timing is the point here, there are thousands of places that parse transactions. Many javascript libraries, tons of websites. Practically all of them just ignore the version, as they would break otherwise. Visitors to a website will call the website broken if it marks as unreadable a transaction that has been included in a block but lied about its version number.
To roll out both the restriction-of-version-number and the actual new tx format at the same time means you need to push the entire ecosystem an update that says “NOW you can trust the version number, and a version N has to be routed to different code”.
That is literally a nightmare scenario.
The alternative, as Mark also said, means that those pieces of code should know more context that they currently don’t need or have, most places that parse transactions are at best SPV and can’t do something like “MTP”.
Both scenarios make the amount of work needed to introduce a new tx format a lot harder and a lot more fragile in the months between release of such software and actual BCH upgrade. The solution to do the version number requirement first means that such software deployments completely eliminate this complexity as they can be certain that a tx of version NEW is going to be in the new format.
A self-contained transaction that without context can be judged to be well-formed is a pretty useful thing for software developers.
If the people in charge would be suggesting now that in 2023 an incompatible transaction format is going to be activated and no version safety exists until that same day, then I fear a lot of libraries will not get written or released as the foundation is too unstable to build on. Its just too hard to do so correctly.
I agree, it’s unfortunate that TX version lock didn’t make it, as it would make it possible to later use not-yet-seen version numbers to signal some consensus rule.
I have drafted the bulletin mentioned above, it’s already been reviewed and approved by freetrader and matricz:
Once merged, it can be a starting point for any public duscussions regarding the TX FORMAT.
Maybe you can include the observations there, that the risk can be lowered massively by staggering the tx-format versioning upgrade and the actual tx-format change upgrade.
Added a subsection:
There would be value in paying off the technical debt by versioning the FORMAT, so any future upgrades that would prefer to change it would not have to take the whole burden of changing the format.
Old software doesn’t expect the version field to have anything to do with the format of the TX it’s being fed.
First step would be locking the version field using consensus, so it could later be used to signal any future consensus rules that would come after the version lock.
Just the version lock wouldn’t break old software so only new, version-aware, software would benefit from that.
It would still help with general preparedness for a future, breaking, change.
From then onward, we could use part of the version field as an upgrade counter and match it 1-to-1 with applicable consensus specification.
As a consequence:
version value seen before the version lock would relate many-to-many with “prehistorical” consensus specifications, therefore be indeterminate.version field would be 1-to-1 with newer consensus specifications.version field relate many-to-1 with consensus specifications.Alongside the counter, we could use a flag to signal whether an upgrade is breaking or non-breaking, e.g. when we increment the counter for a new HF, if it’s non-breaking then we don’t toggle the flag, and if it is, then we toggle the flag.
Old version-aware software would know the old state of the flag so could know whether it can safely process counter+1 TXes even if it knows it can’t fully understand them.
In that scenario, upgraded software could know what rules apply to the TX even without knowing the block height, but it still needs to be upgraded so that it could match the version with applicable consensus rules.
As the consensus upgrade counter part of the version field would be the most important and long-lived feature, the 4-byte* uint should be split into 16 bits for the counter, 1 bit for the breaking flag, and 8 bits for the type of transaction.
The remainder* would be reserved for future use.
(*) Because the version field was never locked, and if we want to enable context-free transaction parsing, we’d have to use only the not-seen-prior-to-lock numbers to encode the newVersion so the new version would have a range smaller than 4 bytes.
Good section, and an interesting proposition. I left some comments on the MR.
It’s unfortunate that we didn’t use the chance to lock the version field but I think it doesn’t have to mean a potential future change would be set back for one upgrade cycle, because it’s possible to roll it out simultaneously and deal with the “attack” of new numbers appearing before activation.
The node would have to build a little database of seen TX versions and keep updating it till’ activation height, and the newVersion post-activation would be deterministically computed based on the database.
If most miners are mining v1/v2 then TX-es having non-standard numbers would be significantly rate-limited, so whatever “attack” would be rate-limited and rendered pointless by the newly-seen-version-aware activation code.
At some point post activation, the code could be shed and the database hard-coded.
To illustrate, imagine the version field is a 4-bit number (0 to 15), but we have mined some “pre-lock” blocks so numbers 0, 1, 2, 3, 5, 6, and 12 are unusable. We’d have only 10 numbers available for the cleanVersion.
We could assign the pre-lock TXes a single version, like cleanVersion==2 and define a function using some simple number skip algorithm:
version |
cleanVersion = dirty2clean(version) |
|---|---|
| 0 | 0 |
| 1 | 0 |
| 2 | 0 |
| 3 | 0 |
| 4 | 1 |
| 5 | 0 |
| 6 | 0 |
| 7 | 2 |
| 8 | 3 |
| 9 | 4 |
| 10 | 5 |
| 11 | 6 |
| 12 | 0 |
| 13 | 7 |
| 14 | 8 |
| 15 | 9 |
It’s not a bijection, so we’d have to pick a number for converting back, e.g. clean2dirty(0) == 0.
Implemented, and edited the post above, too.
The timing issue is still not addressed in your idea. Your conclusion seems overly optimistic.
The timing issie is where deployment requires all libraries, apps, javascript files and wallets to get replaced with the latest code at the time of the protocol upgrade. A nightmare of deployment.
I would say we want to give the ecosystem months or years time to update the software, test it and deploy it. And that is only possible by staggering the version number restriction protocol upgrade and the transaction-format change protocol upgrade.
You speak like its the past tense, the protocol upgrade is still many months in the future.
It would show maturity of the leadership of BCH to either include the (trivial to code and test) this upgrade that already has majority support. Or to accept that, and plan for, the tx-format change to be activatd no earlier than 2024.
Take responsibilty for the decisions made, in short.
If you code them to use the newVersion accessible through a new node API, then they can prepare in advance. Whatever version is added between software release and HF activation would not affect the newVersion value, it would only affect the upper bound of the available range. There’s still the problem of those who’d want to read the version directly from the raw TX, they’d need to keep their list of used up versions updated, and continue to be aware of block height context until the HF activation. So yeah, not impossible but still ugly, and I was overly optimistic.
I know. It would be great to have it locked in 2022. Even if we don’t change the format, context-free parsing is still valuable. Post upgrade, version could relate 1-to-1 to the applicable consensus specification.
Because the nature of the change makes it a soft-fork, maybe it could still be added if people see the value? Even if it missed the November window… with this kind of change, it’s enough that miners upgrade. Edit: SFs are ugly… my thinking was flawed.
We got the data (till’ block 720189) now!
And the data tells us that 100% of the miners are currently mining v1/v2, sticky defaults?
@tom issue tracker is disabled on your repo, there was some talk on telegram regarding moving this forward and @Jonathan_Silverblood wanted to submit an issue when he discovered he can not.
The last commit was 9 months ago, so, we need to know, do you intend to continue to work on this CHIP?
For reference, I think that it would be good to add a clear list of identified stakeholders, make a technical section which shows all possible outcomes and behaviour and I’ve be happy to provide a statement of my own if suitable.
The statement by andrew can further be strengthened now that BU have voted for the change in BUIP170.
In a discussion touching on this subject with regards to native introspection, some research was done on how much abuse there has been over time for the transaction version which would be helpful to have in the chip as well.
@bitcoincashautist or @bitjson can probably provide more detailed information on the usage.
Both Jonathan and Jason have been given full access to the repo, please feel free to do anything with it you want, I know you guys have the same goals I have so no need to ask permission first.
Hope that helps!
Just wanted to leave a comment here that it’s unfortunate we didn’t get this into the May 2022 upgrade. But I strongly support this as a new consensus rule moving forward for any future upgrade. Bump +1
I support this change as a BCHN developer and Knuth lead dev.
I’ve done a bunch of changes and set up an MR, would appreciate review so I won’t mess stuff up: