Chainstate obfuscation and sharing datadir content

Technical
#1

Hello all,
While in some BCH telegram groups I’ve noticed that one of the difficulties people are facing is initial sync and indexing, it takes time and requires a decent hosting plan with enough storage space and bandwidth.

I’ve learned that:

Each node has a unique block database, and all of the files are highly connected. So if you copy just a few files from one installation’s “blocks” or “chainstate” directories into another installation, this will almost certainly cause the second node to crash or get stuck at some random point in the future. [1]

@andrewstone pointed me that it could be because of the commit to the bitcoin client that added Chainstate obfuscation to help protect against antivirus detection [2]

However @dagurval have pointed that even without obfuscation blocks are downloaded and stored randomly.

Is there a way in which two different nodes can create the same database, same files so those files could be verified independently then shared one IPFS or similar systems?

  1. Data directory - Bitcoin Wiki
  2. Add chainstate obfuscation to avoid spurious antivirus detection · bitcoin/bitcoin@42cb388 · GitHub
1 Like
#2

So, a feature request: import/export the entire blockchain from/to a raw format with canonical ordering

2 Likes
#3

So, a feature request: import/export the entire blockchain from/to a raw format with canonical ordering

This is probably a better idea than trying to get uniform datadir structure.

Note that you can already export raw blocks via RPC. These adhere to a specified format.

3 Likes
#4

I created the CashStrap page to address this issue. Pre-synced databases for full nodes and indexers can be downloaded over torrent, and those files can be mounted by the provided Docker containers on that page. It only takes a few minutes to get a new installation running after downloading the data.

This is how I’m able to rapidly set up new infrastructure for clients of FullStack.cash.

I’ll be updating that page this week with updated databases in preparation for the May 15th upgrade.

2 Likes
#5

Just found something called https://ipld.io/ it’s like IPFS but for linked data and has some tools for blockchains, could be related.

Would like to know if the export import process and canonical ordering is possible with current tools? if available how could someone do it?