Running a Terrad validator on GCP

For me, I like to automate things.

So I created a set of scripts and a run book (which is what you are looking at right now) on how I set up a validator. This is a first pass, and hopefully will continue to improve when we get feedback.

and firstly, a big thank you to Christian from Block42 who created this guide to help me get started.

I’m going to document how to get it going on Terra’s testnet, which is called ‘Tequila’. so the keys you see if Islip up are really example only.

A note on security

Security is *H*A*R*D* .

I do *NOT* recommend you use this approach. I do not use this approach. you may lose millions, and the have all your delegates get slashed if you follow this advice.

I am sure to get some things wrong. In fact I accidently had a major screwup while I was making this note. (it’s remedied now thankfully)

Feel free to correct me, and add your advice. The aim here to build a secure network, not to show how smart I am (or am not).

I’ll even add a bug bounty to this page.

1 LUNA for the first person to find an active vulnerability in the tooling. (you only get the reward if no one else identified it)

Limit of 50 LUNA in total, (I may increase this once my validator starts generating LUNA). feel free to mail me at petes-fan-club@protonmail.com

TL;DR Location of the scripts

All the scripts I use are located here. Pull Requests are always welcome.

petes-fan-club/terra-scripts

Why a Cloud provider?

Some people like running on their own hardware, as it is ‘faster’ or more secure, or maybe like the blue flashing lights when they walk into their data center.

I wish them well. and hope they can apply some of the things here to their setup. I personally find that cloud providers have a lot of things solved.

I am initially going to start with a VM-based model. Later, I might go into setting up a multi-zone kubernetes cluster which can might offer higher reliability and less care & feeding.

As for why GCP vs Amazon or Azure. I like GCP. If you want to contribute a similar setup which runs on them, feel free to submit a pull request and I’ll add them to the collection.

The Design

We are trying to build a Highly available, Secure, and Fault tolerant design. One of the nuances about terra is that if you send multiple oracle updates you run the risk of being penalized, and if you miss a couple of blocks, you lose your 100% uptime status, and also run the risk of being penalized.

Now while I firmly believe that penalties are ok for the validator, I don’t like the idea of people who have delegated their savings to me should suffer. So we are going to try our best not to let that happen. (even if it means a bit more $$) in real life. but in this initial post, the aim will be to get this running as easily as possible. (It is a test-net after all). In further posts I will aim to make add some more resilience into it.

There are 3 major components needed to run a validator. (nb: I am not discussing TMKMS but this will be covered in further posts when we run multiple terrad nodes)

1. The terrad software itself. This is typically referred to as ‘the blockchain node’.
2. The price feed server. This interacts with various FOREX currency providers to obtain exchange rates, and provides a REST api that the oracle reads from.
3. The oracle that pushes the exchange rates onto terrad. This reads information from the price feed server, signs it with our private key, and then submits that as an action

The RPC (26657) port is not exposed to the network for now. We have a local terracli binary on our ‘local control’ machine and use port forwarding to connect to it.

The aim is to keep private keys on the local machine as much as possible.

Step 1: Create a GCP project

head over to the google’s GUI and create yourself a new project. pick a name you like, and set it as your default.

for this I plan will be using the project name ‘petes-fan-club’.

If you plan on using my scripts, you will need to copy the remote-scripts/settings.default to remote-scripts/settings.private and put in your details. I have tried to put in ‘cheap’ defaults.

If you are planning on using this to run Columbus-4 (the production instance) you should go ahead and submit a quota increase to raise your quota (search for SSD) from 500G SSD to ~3T. (This might take a few days, or a few minutes).

This doesn’t mean you will be charged for it, but you will be able to create disks utilizing that amount (and then be charged for it)

Step 2: Install gcloud

Quickstart: Getting started with Cloud SDK | Cloud SDK Documentation

follow the instructions, log into your google account, and select your project, and default zone. for the scripts I assume asia-south1-c.

Step 3: create / setup your SSH key

There is an assumption here that you have ssh-agent running. This will create a SSH key for you and add it to your agent. This key will be used to connect to the machines. you can probably skip this step and just use the key google generates too.

$ ./setup_key.sh

Step 4: setup the firewall

as we are setting up things on different machines which need to communicate *AND* we need to set up things to talk externally, lets let google worry about it at the router level.

we are opening up port 26656 on the validator for it to communicate with it’s peers.

we are opening up ports 1317 & 8532 on the validator & price feed respectively to the ‘oracle’ machine.

$ ./ setup_firewall.sh

Step 5: create the VMs

I’m using VMs here. The initial thought was to dockerize them or stick them in kubernetes. For now I am choosing simple VMs as I plan on using a vTPM for the oracle. Instead of ‘secrets’.

Remember to check the settings.default for machine & disk sizes and other things.

$ ./create_vms.sh

Step 6: install terrad core onto the validator node

this setups an additional disk for the database, copies over the quicksync file, and configures terrad to run in ‘pruned’ mode. We reboot the machine to start up everything

this will take a LONG time… 30–45m for the test network, way longer for the production one.

$ ./setup_validator.sh

this copies over the files, and then gets to work via ‘validator_run’

once it is rebooted it will start to sync up. This will also take a while.. this will depend on how large a VM your setup, as well as how recent the sync file is.

once rebooted, and fully synced there should have 2 ports

• 26656, which is open to the general public. This could/should be further locked down to a set of persistent peers
• 1317 the ‘rest’ service which is how the oracle communicates to the node. This should only be open to the ‘oracle’ machines

$ gcloud compute instances describe validator-01|grep natIP

natIP: xxx.xxx.xxx.xxx

$ curl xxx.xxx.xxx.xxx:1317

should timeout (due to the firewall working) and

$ gcloud compute ssh user@oracle-01 — ssh-key-file=./setup_key_private
user@oracle-01 ~ $ curl validator-01:1317/info
404 page not found
should be successful.

Step 7: Install the price server

You will need to put your currencylayer API key into settings.private

$ ./setup_price_server.sh

Step 8: Register the validator on the network

It might take some time, but eventually your node will eventually sync up.

$./validator_status.sh | jq .sync_info.catching_up

false

will connect to the validator node and run ‘terracli status’. if it returns ‘true’ it is still syncing, and you can continue to wait

so the next steps are to create a validator & oracle account.

I suggest you do some research on this bit, and for production stuff use some form of hardware enabled storage. If you have some errors about dbus, it means you should be using a X-windows desktop to run this, so it can ask you for the password. for now I am just doing

$ terracli config keyring-backend file

so that it will use the file system. this is not secure. you should use a ledger device or some other hardware wallet here.

you should also open up a 2nd terminal and run ./validator_connect.sh

this actually does a local port-forward, and sets up your terracli config so your local machine can communicate easily to your remote.

$ ./setup_accounts.sh

please take a note of the 24-keys. the first set is the validator. the 2nd set is the oracle.

You will be required to enter it in later in the oracle.

Transfer some luna tokens over to the validator address (either from the tequilla-4 faucet, or your funds).

and transfer some KRW tokens over to the oracle address.

some of the important decisions here is what commission you would like to generate, and your maximum commission rate.

ok.. crunch time… let’s make the validator visible!

$./register_validator.sh

check it out on https://tequila.stake.id/#/

Step 9: Install the oracle

The final step is hooking up the price-feed to the terra node and pushing exchange information onto the blockchain.

If you have set up and funded your oracle account this should be relatively painless.

1. set your ORACLE_PASSWORD in your settings.private. This will be stored in plain text on your ‘feeder’ machine
2. have your 24 words for your oracle key handy.

$ ./setup_feeder.sh

BUG/TODO:

this still doesn’t work.. it has

broadcast error: txhash not found: ….

this will cause you to get slashed.

Step 10: Tell your friends

give it a day or so to make sure it’s all up, and then let your friends know about your validator.

you can also fill out the validator-profile area, and set up your description so others know who you are.

If you feel inclined, feel free to delegate some luna on my validator. It will help the costs of running it, and keeping it up to date.