Deep History Squad
This page describes the Deep History Squad, which holds the entire trie data, so it can be used to reconstruct the state of the network at any point in time.
Overview
A variant of the standard observing squad is one that retains a non-pruned history of the blockchain and allows one to query the state of an account at an arbitrary block in the past. Such a setup is called a deep-history observing squad.
A deep-history setup is able to resolve historical account (state) queries, that is, to answer questions such as:
What was Alice's balance on May the 4th?
GET http://squad:8080/address/erd1qyu5wthldzr8wx5c9ucg8kjagg0jfs53s8nr3zpz3hypefsdd8ssycr6th?blockNonce=9250000
Currently, the API client has to perform the conversion from desired timestamp to block nonce. In a future release, the API will directly support timestamp-based queries.
How much UTK were in the
UTK / WEGLDLiquidity Pool on 1st of October?
GET http://squad:8080/address/erd1qqqqqqqqqqqqqpgq0lzzvt2faev4upyf586tg38s84d7zsaj2jpsglugga/key/726573657276650000000a55544b2d326638306539?blockNonce=11410000
In the example above, the key 726573657276650000000a55544b2d326638306539 is decoded as reserve\x00\x00\x00\nUTK-2f80e9.
Public instance
As of October 2022, a public deep-history squad isn't yet available. The instance is being prepared and should be ready in November 2022.
On-premises instance
Deep-history squads can be set up on-premises, just as regular observing squads. However, the storage requirements might increase over time - we'll tackle this in a separate section.
Since each observer of a deep-history squad must have a non-pruned history, their non-regular databases have to be either downloaded or reconstructed, in advance.
Downloading non-pruned database
As of October 2022, a public repository with non-pruned databases for both mainnet and devnet is under construction. This repository would take the shape of a Digital Ocean (S3-compatible) Space. Once the repository is ready, the data can be downloaded via db-archive-scripts - documentation will follow.
Reconstructing non-pruned database
An alternative to downloading a non-pruned history is to reconstruct it locally (on your own infrastructure).
Under the hood, the reconstruction process relies on the import-db feature, which allows us to reprocess previously processed blocks - and, while doing so, for our purposes, we'll also retain the whole, non-pruned history. For our purposes, the import-db procedure requires a target database (placed in the folder node-workdir/db) and a source database (usually placed in the folder node-workdir/import-db/db).
It follows that, in order to reconstruct the history for an observer, we need (to download) two database archives: an old archive and a new archive. For reconstructing the history of a whole squad, 4 x 2 archives are required (to be downloaded).
Downloading the necessary archives and unarchiving them is encapsulated in a step called reconstruction bootstrapping.
Bootstrapping
First, choose an empty folder to serve as the workspace (working directory) of the squad instance - for example, ~/deep-history-workspace.
Afterward, prepare a configuration file called reconstruction.json, following the example of default.reconstruction.json, and save it in the chosen workspace. For the fields oldestArchive and newestArchive, use URLs towards the MultiversX public archive (which are available on request). The URLs in the example below are mere placeholders.
// ~/deep-history-workspace/reconstruction.json
{
"networks": {
"devnet": {
"shards": {
"0": {
"oldestArchive": "https://.../shard-0/2022-October-15.tar",
"newestArchive": "https://.../shard-0/2022-October-25.tar"
},
"1": {
"oldestArchive": "https://.../shard-1/2022-October-15.tar",
"newestArchive": "https://.../shard-1/2022-October-25.tar"
},
"2": {
"oldestArchive": "https://.../shard-2/2022-October-15.tar",
"newestArchive": "https://.../shard-2/2022-October-25.tar"
},
"metachain": {
"oldestArchive": "https://.../shard-metachain/2022-October-15.tar",
"newestArchive": "https://.../shard-metachain/2022-October-25.tar"
}
}
}
}
}
Above, we've chosen (as an example) the archives in such a way as to reconstruct the history between the 15th and 25th of October (~10 days, ~120 devnet epochs).
Now, bootstrap the reconstruction as follows:
# Download the docker-compose configuration
wget https://github.com/multiversx/mx-chain-deep-history/blob/main/docker-compose.yml
# Run the "bootstrap" Docker service
DEEP_HISTORY_WORKSPACE=${HOME}/deep-history-workspace DOCKER_USER=$(id -u):$(id -g) docker compose \
--file ./docker-compose.yml \
--profile bootstrap \
--project-name deep-history-reconstruction up --detach
If you prefer to wait in the current shell until the bootstrap finishes, omit the --detach flag.
Downloading the archives and extracting them might take a while.
Start the reconstruction
Once the bootstrap step is ready, you can proceed with running the reconstruction containers. The example below if for devnet:
# Download the docker-compose configuration (skip this step if performed before)
wget https://github.com/multiversx/mx-chain-deep-history/blob/main/docker-compose.yml
# Possible profiles: reconstruction-devnet, reconstruction-devnet-0, reconstruction-devnet-1, reconstruction-devnet-2, reconstruction-devnet-metachain
DEEP_HISTORY_WORKSPACE=${HOME}/deep-history-workspace DOCKER_USER=$(id -u):$(id -g) docker compose \
--file ./docker-compose.yml \
--profile reconstruction-devnet \
--project-name deep-history-reconstruction up --detach
The reconstruction (which uses import-db under the hood, as previously stated) takes a long time - depending on machine's resources (CPU & memory), and on the distance between the chosen archives.
Once a container finishes reconstruction (for a shard), it will shut down. Once all containers of the compose project deep-history-reconstruction have stopped, the reconstruction is ready, and you can proceed with starting the squad (next section).
Starting the squad
The squad can be started using docker-compose, as follows (the example is for devnet):
# Download the docker-compose configuration (skip this step if performed before)
wget https://github.com/multiversx/mx-chain-deep-history/blob/main/docker-compose.yml
# Possible profiles: squad-devnet, squad-devnet-0, squad-devnet-1, squad-devnet-2, squad-devnet-metachain, squad-devnet-proxy
DEEP_HISTORY_WORKSPACE=${HOME}/deep-history-workspace DOCKER_USER=$(id -u):$(id -g) docker compose \
--file ./docker-compose.yml \
--profile squad-devnet \
--project-name deep-history-squad-devnet up --detach
Congratulations, you've set up a deep-history observing squad! The gateway should be ready to resolve historical account (state) queries.
Handling storage requirements
Documentation in this section is preliminary and subject to change.