I think that before we worry about fee estimation we should think about creating an ability for miners to broadcast their minimum fee levels to the rest of the network (i.e., wallets) on-chain. It would be incredibly helpful to future fee estimation to have access to the fee schedules for different miners. At that point, it becomes much easier for a wallet to accurately calculate fees necessary for different levels of transaction processing priority. All they would have to do is look at the minimum sat/byte fee levels posted by miners in recent blocks.
To Tom’s point, I am assuming that miners care only about transaction size here. That may or may not be valid. Perhaps miners do care about coin days destroyed or the ratio of inputs to outputs. If I was a miner, I would personally care only about fee per byte. However, it certainly would make sense to ask. I do believe that coming to a general consensus around the variable(s) of interest to miners when setting fee policies will be extremely helpful to the network in the long-run. Once we figure out what variables they wish to use to target fees, it becomes easier to come up with workable solutions for communicating miner preferences, assuming that’s something others also think would be valuable, and fee estimation for wallets.
Whatever we do, I hope we do not mimic BTC and attempt to calculate fees based on mempool and block inclusion statistics. BCH intends to keep the block size cap above the level of normal transaction volume as long as its sustainable, so such inaccurate modeling of likely fees as what is done on the BTC network should never be necessary. Users should never be (directly) bidding against other users for inclusion in any given block. They should be negotiating a fair price with the miners.
It is a difficult problem, I agree, because it it at the intersection of code and economics, and requires coordination as we know to keep 0-conf working well.
That said, I want to put some of my unrefined thoughts about this out here, since we are pretty much in the brainstorming phase on this matter.
And many, many voices should be heard.
The most reliable data we have, is that of past blocks.
It is very difficult to relate some fiat numbers to anything stable. For example, a currency like USD. The choice is arbitrary, these currencies can hyperinflate, we should not put too much stock in those numbers.
Miners have always and will continue to pick what transactions they include, or even relay, since they do not have to accept any code we propose.
That said, we can only hope that BCH miners are on board with the plan of keeping transactional capacity far above demand, and keeping BCH ultra-competitive when it comes to payment system competition.
I’m hoping can use fee data from past blocks to lower the relay floor, with a built-in protection of a minimum floor (substantially lower than current 1000 sat/kilobyte, maybe lowered initially by a factor of 100)
This introduces some kind of statefulness since the active floor is higher than the ultimate safeguard, and so nodes (and wallets) have to somehow find out what is the active floor. Nodes can (mostly, but not guaranteed) look at enough pasts blocks to work it out. SPV wallets cannot easily, they need to trust other infrastructure, or get some kind of proof via commitments, which we have not thought about. So how to communicate this active floor remains a difficult problem overall. I don’t have a satisfactory solution yet.
We can’t just blindly trust nodes which are vulnerable to sybil attacks.
I think some people have described what we’d be trying to do, as bringing minimum fees into consensus. It’s hard to argue against that point. I basically agree with them. Is is even a smart thing to do? I’m not sure about it, as this could perhaps be a volatile area and most certainly new point of attack.
We (full nodes) almost certainly cannot react quick enough to market changes in coordinating such changes without automation in the protocol. This has basically been demonstrated in BTC (ok, partly wilful but it still took months if not years to fork), ETH, LTC, XMR (remember that long high-fees period?), now DOGE, … so some kind of consensus-level mechanism seems unavoidable.
Any kind of downward fee floor ratcheting mechanism would need, and get, wide discussion, and must be analysed for unintended consequences. We should think also on other conditions, like measured utilitization of block space, when moving this fee floor. And then we need to consider its interaction with other scaling efforts, such as perhaps increasing the max block by several factors etc.
such inaccurate modeling of likely fees as what is done on the BTC network should never be necessary
I agree, I don’t think that would be needed as long as we manage to keep block space supply up enough.
Users should never be (directly) bidding against other users for inclusion in any given block
In the sense that there should not be any losers, yes, but in the sense that miners always can have a cut-off and do, realistically, have a cut-off … we need to accept realities here and there are some that stare us in the face (currently):
Miners on BCH do not make use of the maximum block size of 32MB at all. They are very conservative. Maybe this is due to what @jtoomim has pointed out, that 1 sat/byte is already so near the marginal cost of adding transactions, that they are not incentivized enough. This is one fine line to tread, I’m not sure how any mechanism we come up with here, will manage safely treading this line. Definitely a lot more thoughts and inputs are needed.
I don’t think that consensus is the best way to solve it, since that means we’ll have one fee level for everybody, whereas some people might sell their space far cheaper than others.
I think one of the biggest problems right now is that people don’t even set that setting at all. Maybe we could improve that by setting the default relay/mining fee artificially too low (1sat/mb?), which will force miners to set it. That way maybe we could develop an algorithm that looks at the distribution of previous blocks and fees and gives the user an estimate of how fast (likely) his tx will be mined with a selected level? (i.e. 123 sat/kb will be mined with 100% certainty in the next block, while 1sat/kb has only 5% certainty and will probably require at least 5 blocks, because only 20 out of last 100 blocks included these transactions?)
This could work if we automatically calculate relay TX fee based on First Quartile of X last blocks.
Let’s say if the Q1 (25% lowest) fees from last 100 blocks is 10 sat per MB, then we set relay fee automatically to 50% of that, meaning 5 sat per MB.
Mined blocks should ultimately provide somewhat reliable data about real world environment and it is probably the only kind of data that is not prone to sybil-attacks.
Of course that would be the default settings, miners and nodes could change it in configuration files.
Yes, I had a similar mechanism in mind. Though perhaps not with such a radical lowering, but more gradual.
We would just need to define how often this calculation is done and what the maximum rate is at which it should be able to lower the relay fee rate (e.g. lower up to 50% over the course of a day or 144 blocks)
It is also a problem for pruning nodes which may not have this fee data for all the blocks. I think they might need to be equipped to retrieve the full blocks from other nodes to verify the fees in those blocks in order to do the calculation. Of course when they have done this once for a block, they can cache that information easily.
This still leaves us with the problem of the mining fee minima.
Mining nodes apply their minimum fees they want for mining a block. There should be this option to make that automatically track the calculated relay fees.
If miners set their fees higher, then overly greedy wallets might be putting out transactions that get relayed, but not mined.
We would have created a “dead zone” between what gets mined and the (lower) relay fee. Clearly something we’d want to avoid, right?
The good thing about relay/economic nodes calculating their relay fee rates from past blocks is they could easily advertise that info, and SPV wallets could just have a look at a range of nodes (perhaps with some sanity checks) and calculate a “realistic” fee rate (perhaps double the minimum?) that should get mined.
However, this is all still blue-skies thinking that has not considered possible attacks.
Since I wrote the task about a year ago I have been giving this some thought too.
First and most important part is that fees are an artificial spam-protection invented by Core. Yes, we had fees before, but our main weapon against spam was always days-destroyed. A concept that Core removed.
My thinking about this is that fees make sense in the BTC world because they want high fees. But we don’t. Our goals are very different. Fees exist as a fallback only. Other metrics that can not be be faked make much more sense to use. Now, don’t get me wrong. Fees are still going to be a concept we have. The point is that miners should be able to say “Hey, I want to get paid $10000 per block. The blockreward plus fee-paying transactions make up 5MB, and after that I fill up the block with another X-MB worth of high priority, zero-fee transactions”
The thought process here is:
A miner including 1MB or 10MB of transactions should be no difference in cost whatsoever.
Those numbers go up over time as better tech is rolled out.
Block-rewards and fees pay the cost for the miner. The miner has a stable cost and many competing miners. Any individual miner can demand huge fees, but as long as other miners mine low fee transactions this will have no other effect than simply giving that high-fee demanding miner less income.
Result: Miners can’t artificially push up fees above cost, other miners will outcompete them and steal their income.
As our technology gets better and we can mine bigger blocks, our available block-space will grow faster than the requirement for having a profitable mining business.
Result: miners require a certain amount of fees. The amount of transactions paying them is irrelevant.
Given the fact that a miner got paid after N-MB and they have Y-MB of free space to fill, they want a way to determine the best transactions to include which ensure the highest health of the system. Higher health means more payout from the coins they earn.
When we follow this thinking we get to the point where we have the basics to build a good solution on. And my suggestion is to indeed turn the fees into a secondary point. Fees are just the bottom layer of a miners “Maslow’s Hierarchy of Needs”. As we grow we know that those miners will get paid. So lets start with the top of that pyramid and see what is useful.
We can fight those pesky 1 in 300 out transactions that deposit dust in many addresses by setting a score on a transaction based on its inputs vs its outputs. A tx with 1 → 300 gets a deep negative score which means it now has to pay a large fee to overcome that.
users that combine massive amounts of inputs into one (hello read,cash!) can benefit from that same rule by essentially getting their transaction for free. No fees needed.
People that use BCH for normal shopping will end up with enough UTXOs that are a week or a month old and paying with those can be counted in number of blocks, we can determine that older coins being moved gives people a higher priority to get included.
This means that the core argument from OP where those living on $2 a day will effectively have zero fees. (assuming enough financial activity happens from people that can pay).
Services that have high-transaction throughput and chain transactions in the same block get a negative score on the UTXO-age. And they need to adjust this with either combining lots of inputs, or by paying fees.
This is a balance that leaves those that are going to be using BCH as actual simple money to have fees which are effectively paid for by those that use BCH as a financial instrument. The rich are paying for the security that the poor can use for free.
Edit:
ps. this scheme also means we can drop the dust-limit or at minimum severely lower it. Again making the system more accessible to those less off.
Notice that BCHN inherited this feature from Core who’s intention of having massive fees is distinctly diffrent from ours. The quesion is if you really want to keep the relay-fee in the client.
The design that Satoshi built was that transactions were rated on priority and zero-free transactions were rate-limited.
This requirement falls away when DSProofs start being used in apps receivers use.
Free transactions for most people (you are paying “spam” fees with days destroyed), and fees for high-frequency transactions - that would be pretty freaking amazing! I think that’s also the case for many areas, where high-caliber users basically subsidize all other users (think advertisers paying to keep YouTube free or like a tax system, where high-income earners pay more and people barely scraping by don’t have to pay anything). I think cryptos like Nano, IOTA where users have to pay no fees have a huge selling point.
And again, days-destroyed are protecting us from spam. You can’t easily fake it. Or at least we could think of an algorithm (or recover Satoshi’s one if that was good enough) where you can’t flood the system because you’re spending a scarce resource (days destroyed)
I am not sure about removing relay fees - this seems like a groundbreaking change so it needs more thinking, but I certainly think we should return Days Destroyed to the codebase immediately. It is a great spam discouragement mechanism, best of all probably.
There is just no logical reason why Days Destroyed shouldn’t be used as a spam discouragement mechanism again.
^ Your remark above is what made it seem to me like you were suggesting to remove it from clients.
If it was a general remark, including a suggestion to remove it from Flowee, then I don’t understand why BCHN is being referenced here - other clients have this minrelaytxfee (or similar) feature too… e.g. BCHD.
A healthy network will in future require a large volume of transactions, paying many small fees. Transitioning gradually from block reward to this is the future that Satoshi proposed.
We do NOT want to drive away users or use cases that have a need for more frequent transaction. They would already pay proportionally to their use.
A system where they subsidize the majority of other users seems to me to
(a) be unstable in the long term (what happens when the subsidizing “rich” decide BCH is too expensive and go to some alternative?)
(b) shift the minrelaytxfee question to a “coinage days setting” (or even more complicated machine) configuration question
(c) unintended side effects on peoples’ spending behavior, possibly inhibiting natural growth of the system
Additionally, I think there is a lot of wishful thinking/armchair economics in the comments. Miners will NOT just include a bunch of MB’s of free transactions because of <insert complicated thought process>.
Dictating policy from one or many external oracles is a terrible idea. If we do that we can pretty much fold up any claims of being a permissionless currency and join the ranks of the EOS constitutional council or XRP. It’ll turn the whole thing into a joke. Seriously don’t do it, don’t even mention it. Internal policy of an independent cryptocurrency should never ever depend on reading from an external fiat.
Forcing the miners to float fees is less terrible, but still unequivocally bad. Once you got a jungle of desynchronized fee policies you can say goodbye to any claims to reasonable 0-conf reliability. Yes. I think having a reasonable 0-conf is orders of magnitude more important than crossing some arbitrary red line drawn around one US cent per typical transaction.
Free transactions are pointless. The problem isn’t that they encourage spam - we can arbitrarily limit their existence in any number of ways, coindays and hard size limit being two of them - but rather it cannot be deployed for any application that has any aspiration for reliability, at all.
If such a space strictly caters to unimportant applications that need no reliability, I’m not sure why we should care. If it aims to provide value to some useful application, these applications can and will fail-by-congestion at arbitrary times due to adverse conditions inside the “free space”, and that problem is unsolvable. Damage to BCH’s reputation as an unreliable chain will, in my opinion, be far worse than having each transaction pay two cents till the next upgrade. We’ll necessarily have to heavily discourage its use for anything useful, at which point I hope we’ll be self-aware enough to do some serious reflections.
There is one place where free txs are useful: In case you’re spammed by malicious miners with dust that can’t pay for its own movement, free space allows you to eventually consolidate them. But I doubt that’s a relevant problem these days.
Alright, enough pessimism. Looking at actually plausible solutions:
Some form of algorithm that reads from aggregate fees of past n blocks. It’s easy to see how that’ll work in a congested chain - minimum inclusion fees are fairly useful as proxy to demand, due to users bidding against each other (otoh the UX, as we know, will be absolutely horrible). On a mostly non-congested chain like BCH I’m not sure how that can be done though - in “peacetime” we’ll have the overwhelming majority of tx be at the exact prevailing fee floor, so reading from that aggregate of fees doesn’t yield much useful information. We can’t have the floor constantly lower itself from prevailing fee, since that inevitably ends in a jungle of different policies as miners ditch the unreasonable adjustment algorithm. Again, 0-conf is far more important than crossing some arbitrary line denominated in a constantly devaluing fiat.
There’s probably some clever way to extract more useful information out of the past-transactions aggregate, but I don’t know what it is. If anyone has a new idea here I’m all ears.
It’s a bit puzzling reading the heavy distrust towards miners here - we know they take initiatives on Ethereum to raise blocksizes, so having an adjustment mechanism that depends on miner/pool voting isn’t completely out of the question. In any case, better solutions on this front definitely exist, at least when compared to the very ugly process of lobbying node developers, which is happening right now.
Re:making minfee a consensus rule, that might not be a bad idea… but imo it’s limited in usefulness, since the only difference between that and strictly relay policy is that miners cannot arbitrary lower fees themselves to ruin coordination. Note that they may raise fees and there’s nothing anyone can do about that.
Here are my additional thoughts after reading this thread:
Agree/Reiterate: Do Not Add Fees to Consensus Rules
I agree with others that making fee levels a consensus rule is a bad idea. IMO, fee levels always need to be flexible. What fees really pay for are mining/node costs, and these costs vary in terms of BCH because BCH’s value changes over time. We should also not take away a miner’s ability to mine whatever valid transaction (exclusive of fee) they want to mine. Sure, a miner could “attack” the network by mining a ton of no-fee transactions, but that behavior is unsustainable in the long run and may even be fought off in the short run by other miners orphaning such blocks. I view Bitcoin mining as a cartel. We’ve already seen this in action with off-chain meetings of miners to make decisions and commitments regarding policies they will enforce. Unlike most real-world cartels, however, miners have a pretty slick enforcement mechanism: orphaning blocks. We do not need to try an centrally plan this behavior. It will emerge on its own as necessary since it’s in miners’ self-interest to police this on their own.
TL;DR: I feel like the consensus ruleset should be as minimal as possible (keep it simple, stupid). Transaction fee management does not, IMO, come anywhere near passing the high bar of necessitating consensus enforcement. In fact, I would view such attempts as harmful to the network.
Interesting, but Not Convinced: Different Fee Treatment Based on UTXO Consolidation/Coin Days Destroyed Behavior
I agree that there is value to the network in consolidating UTXOs. I can also see the argument for discouraging “high velocity” usage of the blockchain, though I don’t subscribe to that point of view. Due to the added complexity that these sorts of policies add to the code and the potential for abuse (e.g., spamming the network by creating and then consolidating massive numbers of otherwise un-economic UTXOs becomes possible, and perhaps even cheap for someone with a lot of BCH who can slowly cycle groups of coins to take advantage of both UTXO consolidation and CDD subsidies) I’m skeptical of the need for or wisdom of such policy.
TL;DR: My default position on these is to oppose the added code/system complexity since I don’t yet see the benefits clearly outweighing the risk.
Great User-Configurable Setting: mintxfeerelay
I am all for individual node operators being able to set their own minimum transaction fee relay. I understand that this potentially leads to problems with low-fee transactions not getting relayed and hurts zero-conf security, but I think the tradeoff is worth it and further believe that it should always be a user’s choice in how altruistic they wish to be when running a node. The ability for node operators to manually change this setting on their nodes could also help them and the network avoid a DoS attack scenario where an attacker starts spamming 0-fee, unconfirmed transactions (especially now that there’s no unconfirmed chained tx limit).
The zero-conf security issue is the biggest problem with relaying only transactions with a certain level of fees. However, I agree with freetrader that it is alleviated if not eliminated, by Double Spend Proofs.
TL;DR: I like user-configurable minimum transaction fee relay settings and think that DS Proofs eliminate the best argument for removing this from nodes/user configurability.
Clarification
I assume that when you say “forcing miners to float fees,” you’re referring to the idea of having miners set their own fee schedules and broadcasting them somehow (such as in blocks they mine), but I’m not sure. I don’t see how a jungle of desynchornized fee policies is any worse when it’s explicit (as in the suggestion to broadcast fee schedules) vs. implicit (as is already the case today). We already have mining pools which don’t mine any transactions, and there is absolutely nothing from stopping miners from setting their own minimum transaction fee policies right now. All I advocated for in my first post was creating an ability for miners to communicate policies which they either already have or will likely have in the future.
I also question the idea that there will ever be a “jungle” of desynchronized fees. As I said further up in my comment, I believe that miners will behave more and more like a cartel where it comes to block acceptance as transaction fees replace coin emissions as their primary source of revenue. Transaction fees are just a price like any other, and prices from disparate parties tend to converge even when they are not centrally planned.
I assume that when you say “forcing miners to float fees,” you’re referring to the idea of having miners set their own fee schedules and broadcasting them somehow (such as in blocks they mine), but I’m not sure.
No, I’m referring to the idea of setting default fees so low, the miners have to set their own minrelay fees independently. (Note: local minrelaytxfee is already a config item, has been for a while. Try it!) Miners mostly adhering to the same policy is a good thing, don’t force them into a strictly worse jungle-of-policies situation by taking away the sensible default.
Aha. In that case nevermind that area of my earlier comment. Anyway, I agree that default fee policy should probably reflect some reasonable effort at estimating the cost of a transaction likely to be processed within x blocks.
Ah, the part you quoted was not an advice or suggestion. It was merely an item to earmark for further research. I understand your viewpoint now, good to know!
Interesting point of view. Its obvious that a lot of people would not mind stuff being free, so thats half of your argument gone. Your point that it would somehow not be reliable has been solved already in Thinking about TX propagation.
The concept of “Free transactions” is also missing the bigger picture I tried to explain. This is not just about fees, this is about the wider picture with several metrics. Transactions that have very very low priority need to pay fees. Transactions that have high priority may omit this, but can wait an extra block or two.
This is thus a false assumption, nobody was suggesting a system.
I invite you to look at my post above
where I explain the economics where miners are fully allowed to be greedy, without hurtning the system. No need to trust or distrust.
This statement is shown to be false. Imagine 1 restaurant asking twice the price of all the other ones. The only effect it will have is that they will not get any customers and thus get even less income. Even if 90% of the miners ask for higher fees, the only effect it will have is that people paying lower fees wait for more confirmations.
Your assertion is not supported by basic economics.
Yes. And more to the point, the fee/price should be left to the open market to allow discovery of the optimal. Centrally planned fees, costs, prices etc have been tried and they always cause massive issues. Free market, learn to trust it
Interesting thoughts.
First of all, I don’t think that asking people to pay a minimal fee is equal to discouraging them from participation.
My design is that a mining policy prioritizes transactions for inclusion. And transactions that need to be in the same block as 100 of its parent transactions can get that guarenteed by paying a small fee. As the system grows and we reach 10k tx/sec then the amount of fee per transaction is going to be very low in order or for miner to still get a good income every 10 minutes. So don’t worry about this being anything like a high fee.
Like freetrader said; they are paying more because they use more block-space.
The abuse is a good point to take into account should a mining software implement this idea: the cost of doing 1 → 300 and then doing 300 → 1 should not end up costing the same as a 1->1. I’m not worried about this being too hard, finding some good curves to fine-tune this doesn’t seem a big problem and this is not a consensus part so iterate and test.
Notice that this would not be any complexity in the “system”. This would be a better algorithm that miners use to build a block. Even if a small number of miners use this, the benefit would be that the system as a whole becomes more responsive to economic incentives. None of the other components need more complexity.
I’m not sure whether tom forgot to read most of my actual post before picking out of context quotes to straw-man against, or deliberately chose not to. But I’ll encourage other interested readers to go through my original post.
