/
DevOps - v2 to v3 migration docs

DevOps - v2 to v3 migration docs

Owned by toni
Last updated: Feb 06, 2025
4 min read

 

! PRE-RELEASE DEMO DOCUMENTATION !

GitLab Configurator Updates:

Screenshot from 2025-01-21 15-34-10.png

 

File: global/config.yaml
Paste the code below into the position indicated on the screen:

# at the top under DB clickhouse: image: clickhouse/clickhouse-server:25.1.2 # ... operations: gitlab.tunex.io:5050/marionette-develop/core/operations:latest balances: gitlab.tunex.io:5050/marionette-develop/core/balances:latest accounting: gitlab.tunex.io:5050/marionette-develop/core/accounting:latest trading: enabled: true image: gitlab.tunex.io:5050/marionette-develop/core/trading:latest separate_trading_user_account: false default_market: "ETH-USDT"

Docker Compose Templates:

File: templates/compose/required.yaml.t
Paste the code below into the position indicated on the screen:

image-20250121-162816.png
clickhouse: image: {{clickhouse.image}} volumes: - ../data/clickhouse_data/:/var/lib/clickhouse/ - ../data/clickhouse_logs/:/var/log/clickhouse-server/ ulimits: nproc: 65535 nofile: soft: 262144 hard: 262144

 

File: templates/compose/backend.yaml.t
Paste the code below into the position indicated on the screen:

image-20250121-163158.png

Replace the whole operations.

operations: restart: always image: {{components.operations}} env_file: - ../config/database.env environment: CANCEL_OVERAMOUNT: 1 {{#ifEquals components.trading.enabled true}} SEPARATE_TRADING_USER_ACCOUNT: {{#ifEquals components.trading.separate_trading_user_account true}}1{{else}}0{{/ifEquals}} {{/ifEquals}} depends_on: - clickhouse {{#ifEquals mode "swarm"}} deploy: labels: - "traefik.http.routers.operations.rule=Host(`{{base_url}}`) && PathPrefix(`/components/operations`)" - "traefik.enable=true" - "traefik.http.services.operations.loadbalancer.server.port=3000" {{#if components.traefik.ssl}} - "traefik.http.routers.operations.entrypoints=websecure" - "traefik.http.routers.operations.tls=true" - "traefik.http.routers.operations.tls.certresolver=myresolver" - "traefik.http.routers.operations.middlewares=secureheaders" {{else}} - "traefik.http.routers.operations.entrypoints=web" {{/if}} {{/ifEquals}} {{#ifEquals mode "compose"}} labels: - "traefik.http.routers.operations.rule=Host(`{{base_url}}`) && PathPrefix(`/components/operations`)" - "traefik.enable=true" - "traefik.http.services.operations.loadbalancer.server.port=3000" {{#if components.traefik.ssl}} - "traefik.http.routers.operations.entrypoints=websecure" - "traefik.http.routers.operations.tls=true" - "traefik.http.routers.operations.tls.certresolver=myresolver" - "traefik.http.routers.operations.middlewares=secureheaders" {{else}} - "traefik.http.routers.operations.entrypoints=web" {{/if}} {{/ifEquals}}

 

File: templates/compose/backend.yaml.t
Paste the code below into the position indicated on the screen:

image-20250121-163820.png
image-20250121-163841.png

Replace the balances

balances: restart: always image: {{components.balances}} env_file: - ../config/database.env environment: LOGLEVEL: {{log_level}} NODE_OPTIONS: --max-old-space-size=14000 depends_on: - clickhouse {{#ifEquals mode "swarm"}} deploy: labels: - "traefik.http.routers.balances.rule=Host(`{{base_url}}`) && PathPrefix(`/components/balances`)" - "traefik.enable=true" - "traefik.http.services.balances.loadbalancer.server.port=3000" {{#if components.traefik.ssl}} - "traefik.http.routers.balances.entrypoints=websecure" - "traefik.http.routers.balances.tls=true" - "traefik.http.routers.balances.tls.certresolver=myresolver" - "traefik.http.routers.balances.middlewares=secureheaders" {{else}} - "traefik.http.routers.balances.entrypoints=web" {{/if}} {{/ifEquals}} {{#ifEquals mode "compose"}} labels: - "traefik.http.routers.balances.rule=Host(`{{base_url}}`) && PathPrefix(`/components/balances`)" - "traefik.enable=true" - "traefik.http.services.balances.loadbalancer.server.port=3000" {{#if components.traefik.ssl}} - "traefik.http.routers.balances.entrypoints=websecure" - "traefik.http.routers.balances.tls=true" - "traefik.http.routers.balances.tls.certresolver=myresolver" - "traefik.http.routers.balances.middlewares=secureheaders" {{else}} - "traefik.http.routers.balances.entrypoints=web" {{/if}} {{/ifEquals}}

 

File: templates/compose/backend.yaml.t
Paste the code below into the position indicated on the screen:

image-20250121-163601.png

Replace the trading and add the accounting in the code bellow.

trading: restart: always image: {{components.trading.image}} env_file: - ../config/database.env environment: LOGLEVEL: {{log_level}} SEPARATE_TRADING_USER_ACCOUNT: {{#ifEquals components.trading.separate_trading_user_account true}}1{{else}}0{{/ifEquals}} depends_on: - clickhouse {{#ifEquals mode "swarm"}} deploy: labels: - "traefik.http.routers.trading.rule=Host(`{{base_url}}`) && PathPrefix(`/components/trading`)" - "traefik.enable=true" - "traefik.http.services.trading.loadbalancer.server.port=3000" {{#if components.traefik.ssl}} - "traefik.http.routers.trading.entrypoints=websecure" - "traefik.http.routers.trading.tls=true" - "traefik.http.routers.trading.tls.certresolver=myresolver" - "traefik.http.routers.trading.middlewares=secureheaders" {{else}} - "traefik.http.routers.trading.entrypoints=web" {{/if}} {{/ifEquals}} {{#ifEquals mode "compose"}} labels: - "traefik.http.routers.trading.rule=Host(`{{base_url}}`) && PathPrefix(`/components/trading`)" - "traefik.enable=true" - "traefik.http.services.trading.loadbalancer.server.port=3000" {{#if components.traefik.ssl}} - "traefik.http.routers.trading.entrypoints=websecure" - "traefik.http.routers.trading.tls=true" - "traefik.http.routers.trading.tls.certresolver=myresolver" - "traefik.http.routers.trading.middlewares=secureheaders" {{else}} - "traefik.http.routers.trading.entrypoints=web" {{/if}} {{/ifEquals}} accounting: restart: always image: {{components.accounting}} env_file: - ../config/database.env depends_on: - clickhouse {{#ifEquals mode "swarm"}} deploy: labels: - "traefik.http.routers.accounting.rule=Host(`{{base_url}}`) && PathPrefix(`/components/accounting`)" - "traefik.enable=true" - "traefik.http.services.accounting.loadbalancer.server.port=3000" {{#if components.traefik.ssl}} - "traefik.http.routers.accounting.entrypoints=websecure" - "traefik.http.routers.accounting.tls=true" - "traefik.http.routers.accounting.tls.certresolver=myresolver" - "traefik.http.routers.accounting.middlewares=secureheaders" {{else}} - "traefik.http.routers.accounting.entrypoints=web" {{/if}} {{/ifEquals}} {{#ifEquals mode "compose"}} labels: - "traefik.http.routers.accounting.rule=Host(`{{base_url}}`) && PathPrefix(`/components/accounting`)" - "traefik.enable=true" - "traefik.http.services.accounting.loadbalancer.server.port=3000" {{#if components.traefik.ssl}} - "traefik.http.routers.accounting.entrypoints=websecure" - "traefik.http.routers.accounting.tls=true" - "traefik.http.routers.accounting.tls.certresolver=myresolver" - "traefik.http.routers.accounting.middlewares=secureheaders" {{else}} - "traefik.http.routers.accounting.entrypoints=web" {{/if}} {{/ifEquals}}

 

 

Server Migration:

This release introduces a new database, ClickHouse. To ensure a successful migration, follow the steps below for the initial setup.

Migration Process:

Step 1: Start the migration. Allow the accounting container to migrate with the db-bridge. Wait approximately 30 minutes before proceeding.
Run the following commands:

  1. Bring down the current setup:

    docker-compose down

  2. Start the required services:

    docker-compose up -d nats db clickhouse

  3. Start the db-bridge container and monitor the logs:

    docker-compose up -d db-bridge && docker-compose logs -f db-bridge

  4. Start the accounting container:

    docker-compose up -d accounting

If the migration fails:
Run this command to remove all ClickHouse data and retry:

docker-compose rm -fs accounting clickhouse && rm -rf data/clickhouse_* && docker-compose up -d clickhouse && docker-compose up -d accounting

Step 2: Verify the migration for the balances container. Ensure there are no errors in the logs.
Run the following commands:

docker-compose up -d balances docker-compose logs -f balances

Step 3: If everything is functioning correctly, start all containers:

docker-compose up -d

Re-migration.

If you saw the error this is how you do migration manually.

  1. Chcek puppeteer and remember balance of someone.

  2. start accounting

  3. open clickhouse client, then check if the tables exist

  4.  

    delete from ledgerRecords where true

     

  5. open logs for accounting in other terminal

    docker compose logs -f accounting

  6. In new windows run the marionette cli tool. ./m cli

  7. run the migration

    call 'accounting.migrate'

  8. make sure it is finished, check the logs

  9. if no errors migration is done

  10. restart balances

    docker compose restart balances

  11. check puppeteer for balances and make sure the balances are same and there are not duplicate records

Related content