After working with Bitcore Node for some time, we were unable to successfully get it to sync the intranet blocks.
Documentation of the process and configuration used can be found here: Bitcore - Google Drive
We were able to get Bitcore Node to sync the first 244 blocks of the test case (i.e. both mainnet blocks and test-specific blocks) but it consistently failed to process block 245, the first large block (185 MB). Unfortunately, when this happened there were no immediately apparent errors. It seemed to silently stop syncing and then later print some recurring socket-related errors. Restarting Bitcore Node would cause it to identify that there were new blocks to sync, but then never give any indication that they were processed (e.g. they were not added to the database). Ultimately, our conclusion is that there is either 1) a hidden limit that causes Bitcore Node to reject large blocks in a way that we weren’t able to see, or 2) a technical failure, such as being unable to allocate a sufficiently large array or something similar, occurred that caused syncing to fail indefinitely.
Reviewing the configuration and source code led us to conclude that we were likely not encountering a block size limit. The only limit we could find in the code was a 1 MB “block size” limit, which appeared to be used only for checking transaction sizes. As a sanity check, we set up the BCHN/Bitcore Node to sync testnet. Our test didn’t complete for unrelated reasons (the server storage was too small and filled up) but before that failure, it was able to get to block 1,337,979. In doing so, it successfully processed many 32 MB blocks, all apparently fairly quickly (spot-checking suggested that all blocks were process in less than one second). Our conclusion was that there is either a 32+ MB block size limit that we were unable to find, or no limit is enforced and there was some other failure leading to the stalling.
In the future, we recommend further investigation into why Bitcore Node was unable to sync the larger blocks. Ideally this would be done by someone familiar with the Bitcore codebase who can evaluate the installation steps or configuration that we used for Bitcore Node. That may reveal improvements that would either fix the problem or reveal more information about what went wrong.
Additionally, whether as a part of follow-up testing for Bitcore, or generally for the sake of additional scaling testing, it may be beneficial to create an alternate test chain that more slowly ramps up the block size. In the current test chain, blocks go from 217 bytes to 185 megabytes. For applications that are unable to process large blocks, it would be helpful to see which it could process on a chain that had blocks starting at 217 bytes, then jumping to 16 MB, 32 MB, 48 MB, 64 MB, etc. In this case (Bitcore) it may have provided helpful information for where to start with further investigation. For now, we are also leaving this as a future task, to be revisited for the purposes of revisiting/triaging applications that fail on block 245 of the current test chain, should the others be found to have the same behavior.
Next, we will be testing mining pool software. Specifics are still to be determined, following additional evaluation of the available options.