Version: 1.3.8
Created: 15.07.2022
Last edit: 30.11.2023
The current document describes initial deployment process for the Marionette Stack.
To be able to use this guide below you need a root level access to the operating system of your PC, basic knowledge on Git and SSH as well, an account on the TuneX Gitlab (address: gitlab.tunex.io).
1. Prerequisites
0. Mandatory data to start deployment
Registered domain name
List of endpoints to all blockchain cryptonodes
in case utxo-based cryptonodes credentials to access (repeat for each node you want to add):
rpc-host
rpc-port
user
password
one wallet must be created on the node
in case evm-based cryptonodes credentials to access (repeat for each node you want to add)
HTTPS endpoint from Chainstack
Storage access credentials (use only one of: AWS S3, GCP Cloud Storage, DO Storage):
AWS S3:
Bucket name
Region
Access key
Secret key
GCP:
Storage name
Upload key as google_storage.json
DO:
Storage Name
Digital Ocean Endpoint
Access key
Secret key
Company name
Logo of company in *.png
Color code for the footer and buttons in the mail temlplates (default used: #A32AFF)
Email for support
Email for SSL-certificate
Phone verification credentials (use only one of: Twilio, AWS SNS):
Twilio:
account sid
auth token
paid phone number
AWS SNS:
region
IAM sns user access_key
IAM sns user secret_access_key
PAID Phone number
Email service credentials ( use only Sendgrid or AWS SES)
Sendgrid (Note: add qa@tunex.io to registered emails list- SendGrid Essentials Paid Plan):
User ‘apikey’
password
mail from (trusted to Sendgrid email address)
AWS SES (provide Paid Subscription):
region
IAM sns user access_key
IAM sns user secret_access_key
mail from (trusted to AWS SES email address)
1.1 VM for deploy
For new deployment Marionette Stack with 1(or 2) Payment Interfaces ("PI") based on external cryptonodes ssh access to VM with the following minimal set is required:
Ubuntu 22.04+, min SSD/HDD 40 GB, 6 cores, 10 RAM.
or
Ubuntu 22.04+, with resources based/calculated on Marionette Stack configuration settings
Note: Hardware resources must be estimated during technical interview according to Marionette Stack settings
1.2 TuneX Gitlab group for Marionette Stack
TuneX gitlab Group has been created to deploy Marionette Stack. It contains reuired dependencies. First of all ask TuneX manager path to your group on the TuneX Gitlab and sign into gitlab.tunex.io Your gitlab Group url has the following view: gitlab.tunex.io/[group]
where [group] - your group name (lowercase with "-" instead whitespaces)
For example You got group name "My Project" and the url to your group on the TuneX Gitlab will be:
gitlab.tunex.io/myprojectproject.
So, [group] = myproject
2. Check VM connection and resources
Connect to the VM ( via SSH in terminal or command line prompt, if you need more details on launching it and connecting to remote machine or using Recovery Console please read How to Use SSH to Connect to a Remote Server in Linux or Windows article. Current guide uses linux terminal.
If this is the first time you access this VM via SSH, the user name should be 'root' and you wouldn't be asked for a password. If you're getting Marionette Stacksages like 'wrong user name' etc. you need to find out your user name and/or password.
2.1. Connect VM through ssh
ssh root@<IP_address>
where <IP_address> - the address of the VM for deploy
You should got something like:
2.2. Check Ubuntu disk space
Run:
df -h
make sure there is enough available space according to estimated in the Prerequisites section:
In case of available disk space is less than required please contact system administrator.
2.3. Check available memory and cpu cores
Run:
htop
Note: If you have no 'htop' please install it yourself:
apt install htop
and make sure that the resources of the virtual machine correspond to the Marionette Stack.
Here is an example screen:
3. Required Environment on the VM
3.1 Set up Docker and logging limit
3.1.1. Install Docker engine
Follow actual https://docs.docker.com/engine/install/ubuntu/ and you can skip current step.
Run command:
docker version
If you got the screen like:
follow 3.1.2. Set docker log limit skipping next commands in the current step.
Else if you got the answer similar to:
Update the apt package index and install packages to allow apt to use a repository over HTTPS:
apt-get update
apt-get install \
ca-certificates \
curl \
gnupg \
lsb-release
Add Docker’s official GPG key:
mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
Use the following command to set up the repository:
echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Install Docker Engine:
apt-get update apt-get install docker-ce docker-ce-cli containerd.io
Check the docker engine was successfully installed through viewing the version:
docker version
the result should be like:
3.1.2. Set docker log limit
You should set docker log limit: max-size: 2m, max-file 3 (take for example: https://docs.docker.com/config/containers/logging/json-file/ ):
Below used the 'vim' text editor.
Note: If you have issues using this editor please use 'nano' as alternative or install visual shell for your remote machine
vim /etc/docker/daemon.json
Write next log options data inside daemon.json file and save it
{
"log-driver": "json-file",
"log-opts": {
"max-size": "2m",
"max-file": "3"
}
}
restart docker service to apply log limit:
service docker restart
check docker service to apply log limit:
service docker status
You should get green status if service is OK:
3.2 Set up Docker-compose
Connect to the VM again (in case you exit or loose connection):
ssh root@<IP address>
where - the address of the VM for deploy
The source to install docker-compose https://docs.docker.com/compose/install/ can be used instead current step.
Run this command to download the current stable release of Docker Compose
Note: To install a different version of Compose, substitute 1.29.2 with the version of Compose you want to use
curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
make executable permissions to the binary
chmod +x /usr/local/bin/docker-compose
Note: If the command docker-compose fails after installation, check your path. Here is the nice acticle explaining what is path for Ubuntu and how to make components visible to system and command line using it. You can also create a symbolic link to /usr/bin or any other directory in your path.
For example:
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
Check the docker-compose installation:
docker-compose --version
the result should be like:
If you see a Marionette Stacksage like 'docker-compose version 1.29.2, build 40524192', you're all set. In case you're getting some error Marionette Stacksage please visit this official guide for Docker Compose Versioning.
3.3 Node JS required and settings
Connect to the VM again (in case you exit or loose connection):
ssh root@<IP address>
where - the address of the VM for deploy
3.3.1. Install node LTS version
Source to install for Ubuntu only for one user: https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-20-04
If You are going to use autodeploy under other user (for example gitlab-runner user) each changes of the Marionette project settings (Marionette Stack) You should install Node JS available to all users of the VM operation system.
Here is installing Node JS available to all Users with nvm:
cd ~ apt update
make dir for nvm:
mkdir /usr/local/nvm
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.38.0/install.sh | NVM_DIR=/usr/local/nvm bash
Enter to folder /etc/profile.d add create there file nvm.sh
cd /etc/profile.d && vi nvm.sh
Fill the nvm.sh file next
#!/usr/bin/env bash export NVM_DIR="/usr/local/nvm" source /usr/local/nvm/nvm.sh [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" #loads nvm
Make /etc/profile.d/nvm.sh as source:
source /etc/profile.d/nvm.sh
Install nvm:
nvm install --lts
or
# Look at the list of the node versions nvm list-remote # install any nvm install [version]
where [version] - newest recommended LTS version from the node list versions
Type and run next command for export necessary paths to all users (not necessary):
n=$(which node);n=${n%/bin/node}; chmod -R 755 $n/bin/*; sudo cp -r $n/{bin,lib,share} /usr/local
Verify you’ve installed the required version
node -v
You'll get, for example, similar answer of the version:
3.4 Install git
Run command:
git --version
If you got the screen like:
run command
apt install git
or follow https://git-scm.com/download/linux
4. Requred Gitlab settings and access keys to deploy Marionette Stack
First of all sign in to gitlab.tunex.io and enter to your group on the TuneX Gitlab (mentioned TuneX Gitlab group for Marionette Stack)
4.1. Prepare global/config.yaml in "Marionette-stage" due the “Guidline”
Note: “Guidline” means the “Marionette: STAGING ENVIRONMENT Deployment Guideline” document
Enter to your Gitlab group on the TuneX Gitlab (address: gitlab.tunex.io) and click on "Marionette-stage" project:
Open "Marionette-stage" project, enter to the "global" folder:
Find config.yaml in folder "global" and open it:
Click 'Edit' for config.yaml:
4.1.2. Finish editing config.json and save it
4.2. Generate Access-token for the "Marionette-stage" project
If you are a client and can't perform the actions below yourself, or you are not sure experienced in this topic enough, please contact us and our support will execute this part on your behalf.
Enter to your Gitlab group on the TuneX Gitlab and click on "Marionette-stage" project:
Create new access-token for project Marionette-stage
Call gitlab menu Settings -> Access Tokens
Type 'stage' to the "Token name" field, check Scope only 'read_repository', push button “Create project access token”
On the next step you'll get “project access token” and you must remember it for next chapter:
Save it in your notes for further use cause You can't see it again after generation
4.3. "Access token" in TuneX Gitlab to access docker images repository
If you are a client and can't perform the actions below yourself, or not sure you are experienced in this topic enough, please contact us and our support will execute this part on your behalf.
Enter to your profile on the TuneX Gitlab by quick link https://gitlab.tunex.io/-/profile and push "Access Tokens":
Fill "Token Name" field (e.g: docker-image-token), check box "read_registry" and push button "Create personal access token":
On the next step you'll get “personal access token” and you must remember it for next chapters:
Save "Token Name" and “personal access token” to your notes for further use cause You can't see it again after generation
Ask our technical staff through the issue to give enough rights to your own account in TuneX Gitlab for the docker images repository according to your Marionette Stack.
5. Clone "Marionette-stage" project to VM
Connect to the VM again (in case you exit or loose connection):
ssh root@<IP address>
where - the address of the VM for deploy
5.1. Check the path where you are
pwd
You should start from /root:
5.2 Clone project Marionette-stage from Your Gitlab group to your VM
Copy git path "Clone with HTTPS" from "Marionette-stage" project in your TuneX gitlab Group:
You'll get path like:
where [group] - part of your group URL on the Gitlab TuneX (mentioned "Prepared group on TuneX Gitlab")
Edit git "HTTPS" path and run command with correct data:
git clone https://[Token-name]:[project-token]@gitlab.tunex.io/[group]/marionette-stage.git marionette
where [Token-name] = 'stage' (if you didn't type other data to the field "Token name" generated in previous chapter Generate Access-token for the "Marionette-stage" project),
and [project-token] is "project access token" generated in previous chapter Generate Access-token for the "Marionette-stage" project)
In case of the right "Token Name" and "project access token" you should get similar result:
6. Deploy Marionette Stack
Connect to the VM again (in case you exit or loose connection):
ssh root@<IP address>
where - the address of the VM for deploy
Before starting deployment of Marionette project You'll need the openssl tool to be installed on the VM. To check if this is so please execute command 'openssl version' and check the output.
Here is success result:
If you get 'command not found' or the version is below 1.0.2 please install it and update the tool following instruction.
If you are a client and don't feel confident with the conceptions and processes below please contact our support for assistance.
6.1. Enter to the directory marionette/tool on your VM (the path where you’ve cloned repo):
cd /root/marionette/tool
Run:
npm install
do nothing in case you got next result:
6.2. Check run of the marionette tool:
cd ../
Run command:
./m
You should receive:
6.3. Create keys in the secrets folder:
Check the path where you are
pwd
You should get /root/marionette:
create folder 'secrets' inside root/marionette/config and enter it:
cd config && mkdir secrets && cd secrets
Create couple of rsa keys for using by Marionette project (use the source for information)
openssl genrsa -out key.pem 2048 && openssl rsa -in key.pem -outform PEM -pubout -out public.pem
You may get similar to:
Check generated keys inside current folder:
ls -lah
The result should be 2 generated keys:
6.4. Return to the "marionette" folder
cd ../..
6.5. Run render-config to create compose and config files from templates
Check the path where you are:
pwd
You should get /root/marionette:
make render-config:
./m render-config
The result should be like next screen (all templates successfully 'Done'):
6.6. Download all docker images used for deploy Marionette Stack.
Connect to the docker repository on TuneX Gitlab using command:
docker login gitlab.tunex.io:5050
write your "Token Name" (e.g.: docker-image-token) as login and put "personal access token" generated in previous chapter (chapter Generate your own "Access token" in TuneX Gitlab to access docker images repository)
Check the path where you are:
pwd
You should get /root/marionette:
Run pull for downloading:
docker-compose pull
In success case You should get (all docker images pull will 'done'):
Note: There are cases of troubleshooting You can mostly get on the reason of not enough user rights for docker images or errors in configuration
To resolve similar troubles You should:
check properly settings in the Marionette settings (use guide: Marionette Configuring and Maintenance) in case of errors in configuration
provide the screen evidence to our technical staff for check your rights on the TuneX Gitlab docker images repository in case not enough user rights
Note: Follow next step only if current success
6.7. Before starting all services of Marionette Stack run database and nats services by the command:
Run database, nats:
docker-compose up -d db && docker-compose up -d nats
Check these services are started:
docker-compose ps
6.8. Check the actual count of migrations for your initial deployment of Marionette Stack
6.8.1. View migrations folder
Open in browser your group on TuneX Gitlab and click on "backend" project:
Find "api" folder inside "backend" and enter it:
Find "migrations" folder and click it:
Inside "migrations" folder you find a list of migrations:
6.8.2. Find last actual migration name
Roll down the list and find the migration name with the major number (it depends of revision you deploy):
Remember the migration name with the major number for next steps.
6.9. Run db-bridge service with logs and check it's OK
Connect to the VM again (in case you exit or loose connection):
ssh root@<IP address>
where - the address of the VM for deploy
go to the project folder "marionette":
cd /root/marionette
run db-bridge service with logs:
sleep 30 && docker-compose up -d db-bridge && docker-compose logs -f db-bridge
You should see the logs list and find the rows with "Finished migrating db" and name of the last migrainion:
If the number inside the name of the last migration equals to the magor number of migrations the service "db-bridge" is OK (successfully started)
The last rows of the "db-bridge" service logs looks like:
As you make sure "db-bridge" successfully started skip the next step 8.10 (recommend read it for information).
Follow 8.10. step in trobleshooting case.
Note: Don't loop on 8.9-8.10 steps more then 2 tiMarionette Stack and create the issue to the TuneX team on TuneX Gitlab with screen of "db-bridge" service logs with errors
6.10. Other way to check "db-bridge" service successfully started and solve unsuccess case
8.10.1. Take database parameters to connect it
Enter to your Gitlab group on the TuneX Gitlab (address: gitlab.tunex.io) and click on "Marionette-stage" project:
Open "Marionette-stage" project, enter to the "global" folder:
Find config.yaml in folder "global" and open it:
Find "database name" and "database password":
6.10.2. Connect the database to check success/unsuccess initial start of "db-bridge" service
Connect to the VM again (in case you exit or loose connection):
ssh root@<IP address>
where - the address of the VM for deploy
go to the project folder "marionette":
cd /root/marionette
run connection to the db:
docker-compose exec db mysql -p[database password]
where [database password] - password from global/config.yaml in 8.10.4 step
the successful connection to the db looks like:
in the mysql command line client choose the database of the Marionette project:
use [database name]
where [database name]- name of the database from global/config.yaml in 8.10.4 step
type SQL-request to observe the last successful migration name:
select name from SequelizeMeta order by name desc limit 1;
e.g. of the answer:
Compare the answer of the SQL-request to the migration name with the magor number pointed in 8.8.2 step
You can see similar migration name in success case (example screen):
The similar migration name means success initial start of "db-bridge" service
In other case exit mysql cli to return the terminal cli:
exit;
you'll get to the "marionette" project again:
Follow next step to solve unsuccess initial start of "db-bridge" service
6.10.3. Remove "db-bridge" service process
Check the path where you are
pwd
You should start from the project path /root/marionette:
If you aren't type:
cd /root/marionette
Remove "db-bridge" service process:
docker-compose rm -sf db-bridge
6.10.4. Drop database and return to create "db-bridge" service again
Run connection to the db:
docker-compose exec db mysql -p[database password]
where [database password] - password from global/config.yaml in 8.10.4 step
the successful connection to the db looks like:
Type SQL-request to drop database in the mysql cli:
drop [database name]
where [database name]- name of the database from global/config.yaml in 8.10.4 step
You should get success delete of the db:
exit mysql cli to return the terminal cli:
exit;
You'll get to the "marionette" project again:
Follow "Run db-bridge service with logs and check it's OK" again
6.11. Run all Marionette Stack services
docker-compose up -d
All services will start and you should see all "green" done for each service:
6.12. Check services:
docker-compose ps
example screen:
7. After deploy. Admin Panel URL of Marionette Stack
Admin Panel is a web admin panel of Marionette Stack.
7.1. Enter to your Gitlab group on the TuneX Gitlab (https://gitlab.tunex.io/) and click on "Marionette-stage" project:
7.2. Open "Marionette-stage" project, enter to "global" folder:
7.3. Find config.yaml in folder "global" and open it:
7.4 Click "Edit" button for config.yaml:
7.5. Find row with "base_url" parameter and use it for the next step:
For example:
project_name: marionette base_url: stage.myproject.io marionette_name: MyProject company_name: "My Project LTD"
So, Admin Panel url is "stage.myproject.io/admin"
7.6. Open Admin Panel in browser
Follow "[base_url]/admin", where [base_url] - url from the "base_url" parameter in config.yaml:
8. After deploy. Reset height of blockchains in Admin Panel.
8.1. Login to Admin Panel by default admin
Open in browser URL of your Admin Panel (was pointed in the chapter: Admin Panel URL of Marionette Stack)
Use default login: qa@tunex.io, password: 1234 to login:
8.2. Open menu Configuration -> Blockchains and push the button "..." for each row of the blockchains list:
You should "reset" the height for each blockchain because the last height(right number) has a huge difference with height (left number) and repeat "reset"" again for another blockchain (if you have more than 1 blockchain).
So, return to the current and next step in the loop until you reset the height for all blockchains.
8.3. Reset last height of the blockchain
Copy the number near the button "Reset" and click "Reset":
Paste copied number to the window and click "Reset" and "Reload" button to aply:
9. Compare blockchain height accordingly to etherscan source.
9.1. Login to Admin Panel by default admin
repeat step 8.1. Login to Admin Panel by default admin
9.2. Enter to Configuration -> Blockchains and click the button "..." for each row of the blockchains list:
You should compare blockchain height accordingly to etherscan source again for another blockchain (if you have more than 1 blockchain).
So, return to the current and next step in the loop until you compare blockchain height accordingly to etherscan source for all blockchains.
9.3. Open etherscan source for comparing
Open browser and type the address of the etherscan (as pointed on the previous screen) and find view blocks:
Put browser windows of blockchain Admin Panel tab near the Etherscan viewing the Last block and compare the numbers:
10. After deploy. Create wallets in Marionette Stack.
10.1. Login to Admin Panel by default admin
repeat step 8.1. Login to Admin Panel by default admin
10.2. Open menu Configuration -> Blockchains and push the button "..." for each row of the blockchains list:
You should create wallet again for another blockchain (if you have more than 1 blockchain)
So, return to the current and next step in the loop until you create wallets for all blockchains.
10.3.Create system wallet for selected blockchain
Push button "Create wallet":
Click "Reload" for make sure you got the address of the wallet:
11. After deploy. Check node services
11.1. Login to Admin Panel by default admin
repeat step 10.1. Login to Admin Panel by default admin
11.2. Open "DevOps" in the Admin Panel
Pay attention to the information on system nodes, their up-time and statuses.
Nodes can contain one/several services.
Note: The perfect maximum status of memory usage for each service is less than 60%.
11.3. Overview all nodes in terminal to check it statuses/work
Connect to the VM again (in case you exit or loose connection):
ssh root@<IP address>
where - the address of the VM for deploy
Enter to the directory marionette/tool on your VM (the path where you’ve cloned repo):
cd /root/marionette/
Run next command to enter the marionette command line interface (cli):
./marionette cli
Type inside marionette cli:
nodes
and you'll get the list of the nodes (the same visualized on the Dashboard in the Admin Panel above):
For exit the marionette cli run:
q
12. Conclusion
You have finished the initial deployment of the Marionette Stack.
TODO: insert pat of gitlab-runner configuring from doc. "Gitlab-runner for deployment"