Version: 1.3.8
Created: 15.07.2022
Last update: 15.12.2023

The current document describes initial deployment process for the Marionette Stack.

To be able to use this guide below you need a root level access to the operating system of your PC, basic knowledge on Git and SSH as well, an account on the TuneX Gitlab (address: gitlab.tunex.io).

1. Prerequisites

0. Mandatory data to start deployment

1. Registered domain name

2. List of endpoints to all blockchain cryptonodes

   1. in case utxo-based cryptonodes credentials to access (repeat for each node you want to add):

      • rpc-host

      • rpc-port

      • user

      • password

      • one wallet must be created on the node

   2. in case evm-based cryptonodes credentials to access (repeat for each node you want to add)

      • HTTPS endpoint from Chainstack

3. Storage access credentials (AWS S3, GCP Cloud Storage, DO Storage, Minio, etc.):

   1. AWS S3:

      • Bucket name

      • Region

      • Access key

      • Secret key

   2. GCP:

      • Storage name

      • Upload key as google_storage.json

   3. DO:

      •  Storage Name

      • Digital Ocean Endpoint

      • Access key

      • Secret key

4. Company name

5. Logo of company in *.png

6. Color code for the footer and buttons in the mail temlplates (default used: #A32AFF)

7. Email for support

8. Email for SSL-certificate

9. Phone verification credentials (Twilio, AWS SNS, etc.):

   1. Twilio:

      • account sid

      • auth token

      • paid phone number

   2. AWS SNS:

      • region

      • IAM sns user access_key

      • IAM sns user secret_access_key

      • PAID Phone number

10. Email service credentials (Sendgrid or AWS SES or custom service)

    1. Sendgrid:

       • User ‘apikey’

       • password

       • mail from (trusted to Sendgrid email address)

    2. AWS SES (provide Paid Subscription):

       • region

       • IAM sns user access_key

       • IAM sns user secret_access_key

       • mail from (trusted to AWS SES email address)

1.1 VM for deploy

For new deployment Marionette Stack with 1(or 2) Payment Interfaces ("PI") based on external cryptonodes ssh access to VM with the following minimal set is required:

Ubuntu 22.04+, min SSD/HDD 40 GB, 6 cores, 10 RAM.

or

Ubuntu 22.04+, with resources based/calculated on Marionette Stack configuration settings

Note: Hardware resources must be estimated during technical interview according to Marionette Stack settings

1.2 TuneX Gitlab group for Marionette Stack

TuneX gitlab Group has been created to deploy Marionette Stack. It contains reuired dependencies. First of all ask TuneX manager path to your group on the TuneX Gitlab and sign into gitlab.tunex.io Your gitlab Group url has the following view: gitlab.tunex.io/[group]

where [group] - your group name (lowercase with "-" instead whitespaces)

For example You got group name "My Project" and the url to your group on the TuneX Gitlab will be:

gitlab.tunex.io/myprojectproject.

So, [group] = myproject

2. Check VM connection and resources

Connect to the VM ( via SSH in terminal or command line prompt, if you need more details on launching it and connecting to remote machine or using Recovery Console please read How to Use SSH to Connect to a Remote Server in Linux or Windows article. Current guide uses linux terminal.

If this is the first time you access this VM via SSH, the user name should be 'root' and you wouldn't be asked for a password. If you're getting Marionette Stacksages like 'wrong user name' etc. you need to find out your user name and/or password.

2.1. Connect VM through ssh

where <IP_address> - the address of the VM for deploy

You should got something like:

Open

2.2. Check Ubuntu disk space

Run:

make sure there is enough available space according to estimated in the Prerequisites section:

Open

In case of available disk space is less than required please contact system administrator.

2.3. Check available memory and cpu cores

Run:

Note: If you have no 'htop' please install it yourself:

apt install htop

and make sure that the resources of the virtual machine correspond to the Marionette Stack.
Here is an example screen:

3. Required Environment on the VM

3.1 Set up Docker and logging limit

3.1.1. Install Docker engine

Follow actual https://docs.docker.com/engine/install/ubuntu/ and you can skip current step.

Run command:

If you got the screen like:

follow 3.1.2. Set docker log limit skipping next commands in the current step.

Else if you got the answer similar to:

Update the apt package index and install packages to allow apt to use a repository over HTTPS:

Add Docker’s official GPG key:

Use the following command to set up the repository:

Install Docker Engine:

Check the docker engine was successfully installed through viewing the version:

the result should be like:

3.1.2. Set docker log limit

You should set docker log limit: max-size: 2m, max-file 3 (take for example: https://docs.docker.com/config/containers/logging/json-file/ ):

Below used the 'vim' text editor.

Note: If you have issues using this editor please use 'nano' as alternative or install visual shell for your remote machine

Write next log options data inside daemon.json file and save it

restart docker service to apply log limit:

check docker service to apply log limit:

You should get green status if service is OK:

3.2 Set up Docker-compose

Connect to the VM again (in case you exit or loose connection):

where - the address of the VM for deploy

The source to install docker-compose https://docs.docker.com/compose/install/ can be used instead current step.

Run this command to download the current stable release of Docker Compose

Note: To install a different version of Compose, substitute 1.29.2 with the version of Compose you want to use

make executable permissions to the binary

chmod +x /usr/local/bin/docker-compose

Note: If the command docker-compose fails after installation, check your path. Here is the nice acticle explaining what is path for Ubuntu and how to make components visible to system and command line using it. You can also create a symbolic link to /usr/bin or any other directory in your path.

For example:

Check the docker-compose installation:

the result should be like:

If you see a Marionette Stacksage like 'docker-compose version 1.29.2, build 40524192', you're all set. In case you're getting some error Marionette Stacksage please visit this official guide for Docker Compose Versioning.

3.3 Node JS required and settings

Connect to the VM again (in case you exit or loose connection):

where - the address of the VM for deploy

3.3.1. Install node LTS version

Source to install for Ubuntu only for one user: https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-20-04

If You are going to use autodeploy under other user (for example gitlab-runner user) each changes of the Marionette project settings (Marionette Stack) You should install Node JS available to all users of the VM operation system.

Here is installing Node JS available to all Users with nvm:

make dir for nvm:

Enter to folder /etc/profile.d add create there file nvm.sh

Fill the nvm.sh file next

Make /etc/profile.d/nvm.sh as source:

Install nvm:

or

where [version] - newest recommended LTS version from the node list versions

Type and run next command for export necessary paths to all users (not necessary):

Verify you’ve installed the required version

You'll get, for example, similar answer of the version:

3.4 Install git

Run command:

If you got the screen like:

run command

or follow https://git-scm.com/download/linux

4. Requred Gitlab settings and access keys to deploy Marionette Stack

First of all sign in to gitlab.tunex.io and enter to your group on the TuneX Gitlab (mentioned TuneX Gitlab group for Marionette Stack)

4.1. Prepare global/config.yaml in "Marionette-stage" due the “Guidline”

Note: “Guidline” means the “Marionette: STAGING ENVIRONMENT Deployment Guideline” document

Enter to your Gitlab group on the TuneX Gitlab (address: gitlab.tunex.io) and click on "Marionette-stage" project:

Open "Marionette-stage" project, enter to the "global" folder:

Find config.yaml in folder "global" and open it:

Click 'Edit' for config.yaml:

4.1.2. Finish editing config.json and save it

4.2. Generate Access-token for the "Marionette-stage" project

If you are a client and can't perform the actions below yourself, or you are not sure experienced in this topic enough, please contact us and our support will execute this part on your behalf.

Enter to your Gitlab group on the TuneX Gitlab and click on "Marionette-stage" project:

Create new access-token for project Marionette-stage

Call gitlab menu Settings -> Access Tokens

Type 'stage' to the "Token name" field, check Scope only 'read_repository', push button “Create project access token”

On the next step you'll get “project access token” and you must remember it for next chapter:

Save it in your notes for further use cause You can't see it again after generation

4.3. "Access token" in TuneX Gitlab to access docker images repository

If you are a client and can't perform the actions below yourself, or not sure you are experienced in this topic enough, please contact us and our support will execute this part on your behalf.

Enter to your profile on the TuneX Gitlab by quick link https://gitlab.tunex.io/-/profile and push "Access Tokens":

Fill "Token Name" field (e.g: docker-image-token), check box "read_registry" and push button "Create personal access token":

On the next step you'll get “personal access token” and you must remember it for next chapters:

Save "Token Name" and “personal access token” to your notes for further use cause You can't see it again after generation

Ask our technical staff through the issue to give enough rights to your own account in TuneX Gitlab for the docker images repository according to your Marionette Stack.

5. Clone "Marionette-stage" project to VM

Connect to the VM again (in case you exit or loose connection):

where - the address of the VM for deploy

5.1. Check the path where you are

You should start from /root:

5.2 Clone project Marionette-stage from Your Gitlab group to your VM

Copy git path "Clone with HTTPS" from "Marionette-stage" project in your TuneX gitlab Group:

You'll get path like:

https://gitlab.tunex.io/[group]/marionette-stage.git

where [group] - part of your group URL on the Gitlab TuneX (mentioned "Prepared group on TuneX Gitlab")

Edit git "HTTPS" path and run command with correct data:

where [Token-name] = 'stage' (if you didn't type other data to the field "Token name" generated in previous chapter Generate Access-token for the "Marionette-stage" project),

and [project-token] is "project access token" generated in previous chapter Generate Access-token for the "Marionette-stage" project)

In case of the right "Token Name" and "project access token" you should get similar result:

6. Deploy Marionette Stack

Connect to the VM again (in case you exit or loose connection):

where - the address of the VM for deploy

Before starting deployment of Marionette project You'll need the openssl tool to be installed on the VM. To check if this is so please execute command 'openssl version' and check the output.

Here is success result:

If you get 'command not found' or the version is below 1.0.2 please install it and update the tool following instruction.

If you are a client and don't feel confident with the conceptions and processes below please contact our support for assistance.

6.1. Enter to the directory marionette/tool on your VM (the path where you’ve cloned repo):

Run:

do nothing in case you got next result:

6.2. Check run of the marionette tool:

Run command:

You should receive:

6.3. Create keys in the secrets folder:

Check the path where you are

You should get /root/marionette:

create folder 'secrets' inside root/marionette/config and enter it:

Create couple of rsa keys for using by Marionette project (use the source for information)

You may get similar to:

Check generated keys inside current folder:

The result should be 2 generated keys:

6.4. Return to the "marionette" folder

6.5. Run render-config to create compose and config files from templates

Check the path where you are:

You should get /root/marionette:

make render-config:

The result should be like next screen (all templates successfully 'Done'):

6.6. Download all docker images used for deploy Marionette Stack.

Connect to the docker repository on TuneX Gitlab using command:

write your "Token Name" (e.g.: docker-image-token) as login and put "personal access token" generated in previous chapter (chapter Generate your own "Access token" in TuneX Gitlab to access docker images repository)

Check the path where you are:

You should get /root/marionette:

Run pull for downloading:

In success case You should get (all docker images pull will 'done'):

Note: There are cases of troubleshooting You can mostly get on the reason of not enough user rights for docker images or errors in configuration

To resolve similar troubles You should:

• check properly settings in the Marionette settings (use guide: Marionette Configuring and Maintenance) in case of errors in configuration

• provide the screen evidence to our technical staff for check your rights on the TuneX Gitlab docker images repository in case not enough user rights

Note: Follow next step only if the current was successful

6.7. Before starting all services of Marionette Stack run database and nats services by the command:

Run database, nats:

Check these services are started:

6.8. Check the actual count of migrations for your initial deployment of Marionette Stack

6.8.1. View migrations folder

Open in browser your group on TuneX Gitlab and click on "backend" project:

Find "api" folder inside "backend" and enter it:

Find "migrations" folder and click it:

Inside "migrations" folder you find a list of migrations:

6.8.2. Find last actual migration name

Roll down the list and find the migration name with the major number (it depends of revision you deploy):

Remember the migration name with the major number for next steps.

6.9. Run db-bridge service with logs and check it's OK

Connect to the VM again (in case you exit or loose connection):

where - the address of the VM for deploy

go to the project folder "marionette":

run db-bridge service with logs:

You should see the logs list and find the rows with "Finished migrating db" and name of the last migrainion:

If the number inside the name of the last migration equals to the magor number of migrations the service "db-bridge" is OK (successfully started)

The last rows of the "db-bridge" service logs looks like:

As you make sure "db-bridge" successfully started skip the next step 8.10 (recommend read it for information).

Follow 8.10. step in trobleshooting case.

Note: Don't loop on 8.9-8.10 steps more then 2 tiMarionette Stack and create the issue to the TuneX team on TuneX Gitlab with screen of "db-bridge" service logs with errors

6.10. Other way to check "db-bridge" service successfully started and solve unsuccess case

8.10.1. Take database parameters to connect it

Enter to your Gitlab group on the TuneX Gitlab (address: gitlab.tunex.io) and click on "Marionette-stage" project:

Open "Marionette-stage" project, enter to the "global" folder:

Find config.yaml in folder "global" and open it:

Find "database name" and "database password":

6.10.2. Connect the database to check success/unsuccess initial start of "db-bridge" service

Connect to the VM again (in case you exit or loose connection):

where - the address of the VM for deploy

go to the project folder "marionette":

run connection to the db:

where [database password] - password from global/config.yaml in 8.10.4 step

the successful connection to the db looks like:

in the mysql command line client choose the database of the Marionette project:

where [database name]- name of the database from global/config.yaml in 8.10.4 step

type SQL-request to observe the last successful migration name:

e.g. of the answer:

Compare the answer of the SQL-request to the migration name with the magor number pointed in 8.8.2 step

You can see similar migration name in success case (example screen):

The similar migration name means success initial start of "db-bridge" service

In other case exit mysql cli to return the terminal cli:

you'll get to the "marionette" project again:

Follow next step to solve unsuccess initial start of "db-bridge" service

6.10.3. Remove "db-bridge" service process

Check the path where you are

You should start from the project path /root/marionette:

If you aren't type:

Remove "db-bridge" service process:

6.10.4. Drop database and return to create "db-bridge" service again

Run connection to the db:

where [database password] - password from global/config.yaml in 8.10.4 step

the successful connection to the db looks like:

Type SQL-request to drop database in the mysql cli:

where [database name]- name of the database from global/config.yaml in 8.10.4 step

You should get success delete of the db:

exit mysql cli to return the terminal cli:

You'll get to the "marionette" project again:

Follow "Run db-bridge service with logs and check it's OK" again

6.11. Run all Marionette Stack services

All services will start and you should see all "green" done for each service:

6.12. Check services:

example screen:

7. After deploy. Admin Panel URL of Marionette Stack

Admin Panel is a web admin panel of Marionette Stack.

7.1. Enter to your Gitlab group on the TuneX Gitlab (https://gitlab.tunex.io/) and click on "Marionette-stage" project:

7.2. Open "Marionette-stage" project, enter to "global" folder:

7.3. Find config.yaml in folder "global" and open it:

7.4 Click "Edit" button for config.yaml:

7.5. Find row with "base_url" parameter and use it for the next step:

For example:

So, Admin Panel url is "stage.myproject.io/admin"

7.6. Open Admin Panel in browser

Follow "[base_url]/admin", where [base_url] - url from the "base_url" parameter in config.yaml:

8. After deploy. Reset height of blockchains in Admin Panel.

8.1. Login to Admin Panel by default admin

Open in browser URL of your Admin Panel (was pointed in the chapter: Admin Panel URL of Marionette Stack)

Use default login: qa@tunex.io, password: changeP@ssword123 to login:

8.2. Open menu Configuration -> Blockchains and push the button "..." for each row of the blockchains list:

You should "reset" the height for each blockchain because the last height(right number) has a huge difference with height (left number) and repeat "reset"" again for another blockchain (if you have more than 1 blockchain).

So, return to the current and next step in the loop until you reset the height for all blockchains.

8.3. Reset last height of the blockchain

Copy the number near the button "Reset" and click "Reset":

Paste copied number to the window and click "Reset" and "Reload" button to aply:

9. Compare blockchain height accordingly to etherscan source.

9.1. Login to Admin Panel by default admin

repeat step 8.1. Login to Admin Panel by default admin

9.2. Enter to Configuration -> Blockchains and click the button "..." for each row of the blockchains list:

You should compare blockchain height accordingly to etherscan source again for another blockchain (if you have more than 1 blockchain).

So, return to the current and next step in the loop until you compare blockchain height accordingly to etherscan source for all blockchains.

9.3. Open etherscan source for comparing

Open browser and type the address of the etherscan (as pointed on the previous screen) and find view blocks:

Put browser windows of blockchain Admin Panel tab near the Etherscan viewing the Last block and compare the numbers:

10. After deploy. Create wallets in Marionette Stack.

10.1. Login to Admin Panel by default admin

repeat step 8.1. Login to Admin Panel by default admin

10.2. Open menu Configuration -> Blockchains and push the button "..." for each row of the blockchains list:

You should create wallet again for another blockchain (if you have more than 1 blockchain)

So, return to the current and next step in the loop until you create wallets for all blockchains.

10.3.Create system wallet for selected blockchain

Push button "Create wallet":

Click "Reload" for make sure you got the address of the wallet:

11. After deploy. Check node services

11.1. Login to Admin Panel by default admin

repeat step 10.1. Login to Admin Panel by default admin

11.2. Open "DevOps" in the Admin Panel

Pay attention to the information on system nodes, their up-time and statuses.

Nodes can contain one/several services.

Note: The perfect maximum status of memory usage for each service is less than 60%.

11.3. Overview all nodes in terminal to check it statuses/work

Connect to the VM again (in case you exit or loose connection):

where - the address of the VM for deploy

Enter to the directory marionette/tool on your VM (the path where you’ve cloned repo):

Run next command to enter the marionette command line interface (cli):

Type inside marionette cli:

and you'll get the list of the nodes (the same visualized on the Dashboard in the Admin Panel above):

For exit the marionette cli run:

12. Conclusion

You have finished the initial deployment of the Marionette Stack.

TODO: insert pat of gitlab-runner configuring from doc. "Gitlab-runner for deployment"