This page describes the Deep History Squad, which holds the entire trie data, so it can be used to reconstruct the state of the network at any point in time.

## Overview​

A variant of the standard observing squad is one that retains a non-pruned history of the blockchain and allows one to query the state of an account at an arbitrary block in the past. Such a setup is called a deep-history observing squad.

A deep-history setup is able to resolve historical account (state) queries, that is, to answer questions such as:

What was Alice's balance on May the 4th?

GET http://squad:8080/address/erd1qyu5wthldzr8wx5c9ucg8kjagg0jfs53s8nr3zpz3hypefsdd8ssycr6th?blockNonce=9250000
tip

Currently, the API client has to perform the conversion from desired timestamp to block nonce. In a future release, the API will directly support timestamp-based queries.

How much UTK were in the UTK / WEGLD Liquidity Pool on 1st of October?

GET http://squad:8080/address/erd1qqqqqqqqqqqqqpgq0lzzvt2faev4upyf586tg38s84d7zsaj2jpsglugga/key/726573657276650000000a55544b2d326638306539?blockNonce=11410000

In the example above, the key 726573657276650000000a55544b2d326638306539 is decoded as reserve\x00\x00\x00\nUTK-2f80e9.

## Public instance​

tip

As of October 2022, a public deep-history squad isn't yet available. The instance is being prepared and should be ready in November 2022.

## On-premises instance​

Deep-history squads can be set up on-premises, just as regular observing squads. However, the storage requirements might increase over time - we'll tackle this in a separate section.

Since each observer of a deep-history squad must have a non-pruned history, their non-regular databases have to be either downloaded or reconstructed, in advance.

tip

As of October 2022, a public repository with non-pruned databases for both mainnet and devnet is under construction. This repository would take the shape of a Digital Ocean (S3-compatible) Space. Once the repository is ready, the data can be downloaded via db-archive-scripts - documentation will follow.

### Reconstructing non-pruned database​

Under the hood, the reconstruction process relies on the import-db feature, which allows us to reprocess previously processed blocks - and, while doing so, for our purposes, we'll also retain the whole, non-pruned history. For our purposes, the import-db procedure requires a target database (placed in the folder node-workdir/db) and a source database (usually placed in the folder node-workdir/import-db/db).

It follows that, in order to reconstruct the history for an observer, we need (to download) two database archives: an old archive and a new archive. For reconstructing the history of a whole squad, 4 x 2 archives are required (to be downloaded).

#### Bootstrapping​

First, choose an empty folder to serve as the workspace (working directory) of the squad instance - for example, ~/deep-history-workspace.

Afterward, prepare a configuration file called reconstruction.json, following the example of default.reconstruction.json, and save it in the chosen workspace. For the fields oldestArchive and newestArchive, use URLs towards the MultiversX public archive (which are available on request). The URLs in the example below are mere placeholders.

// ~/deep-history-workspace/reconstruction.json{    "networks": {        "devnet": {            "shards": {                "0": {                    "oldestArchive": "https://.../shard-0/2022-October-15.tar",                    "newestArchive": "https://.../shard-0/2022-October-25.tar"                },                "1": {                    "oldestArchive": "https://.../shard-1/2022-October-15.tar",                    "newestArchive": "https://.../shard-1/2022-October-25.tar"                },                "2": {                    "oldestArchive": "https://.../shard-2/2022-October-15.tar",                    "newestArchive": "https://.../shard-2/2022-October-25.tar"                },                "metachain": {                    "oldestArchive": "https://.../shard-metachain/2022-October-15.tar",                    "newestArchive": "https://.../shard-metachain/2022-October-25.tar"                }            }        }    }}

Above, we've chosen (as an example) the archives in such a way as to reconstruct the history between the 15th and 25th of October (~10 days, ~120 devnet epochs).

Now, bootstrap the reconstruction as follows:

# Download the docker-compose configurationwget https://github.com/multiversx/mx-chain-deep-history/blob/main/docker-compose.yml# Run the "bootstrap" Docker serviceDEEP_HISTORY_WORKSPACE=${HOME}/deep-history-workspace DOCKER_USER=$(id -u):$(id -g) docker compose \ --file ./docker-compose.yml \ --profile bootstrap \ --project-name deep-history-reconstruction up --detach If you prefer to wait in the current shell until the bootstrap finishes, omit the --detach flag. tip Downloading the archives and extracting them might take a while. #### Start the reconstruction​ Once the bootstrap step is ready, you can proceed with running the reconstruction containers. The example below if for devnet: # Download the docker-compose configuration (skip this step if performed before)wget https://github.com/multiversx/mx-chain-deep-history/blob/main/docker-compose.yml# Possible profiles: reconstruction-devnet, reconstruction-devnet-0, reconstruction-devnet-1, reconstruction-devnet-2, reconstruction-devnet-metachainDEEP_HISTORY_WORKSPACE=${HOME}/deep-history-workspace DOCKER_USER=$(id -u):$(id -g) docker compose \    --file ./docker-compose.yml \    --profile reconstruction-devnet \    --project-name deep-history-reconstruction up --detach
tip

The reconstruction (which uses import-db under the hood, as previously stated) takes a long time - depending on machine's resources (CPU & memory), and on the distance between the chosen archives.

Once a container finishes reconstruction (for a shard), it will shut down. Once all containers of the compose project deep-history-reconstruction have stopped, the reconstruction is ready, and you can proceed with starting the squad (next section).

# Download the docker-compose configuration (skip this step if performed before)wget https://github.com/multiversx/mx-chain-deep-history/blob/main/docker-compose.yml# Possible profiles: squad-devnet, squad-devnet-0, squad-devnet-1, squad-devnet-2, squad-devnet-metachain, squad-devnet-proxyDEEP_HISTORY_WORKSPACE=${HOME}/deep-history-workspace DOCKER_USER=$(id -u):\$(id -g) docker compose \    --file ./docker-compose.yml \    --profile squad-devnet \    --project-name deep-history-squad-devnet up --detach