3.0.0 Backend Changelog - Jan 20, 2025
- Elvira Myro
- Kate Tun
- Denys Tun
Marionette Backend v.3.0.0 Release Highlights
This release introduces significant updates to enhance system performance and reliability, including a shift to integer arithmetic for financial records and an improved architecture.
1. Key Updates:
1.1 Accounting Update: Added Integer Arithmetic for Financial Records
All monetary values are now stored as integers with 18 digits of precision. This means that every unit of currency is represented in the smallest possible denomination. For example, 1 cent (0.01) is now represented as '10000000000000000'.
Why this change?
This change ensures greater accuracy in financial calculations. Storing monetary values as integers eliminates the possibility of rounding errors caused by floating-point arithmetic. This approach ensures precise and consistent calculations for financial records, making the system more reliable and accurate.
1.2 Migration from MySQL to Clickhouse (What is ClickHouse?):
The LedgerRecords table has been moved from MySQL to Clickhouse for improved performance and scalability.
Why this change?
This change benefits high-volume data operations and enhances query efficiency.
1.3 New Component: 'Accounting'
Accounting has been added as a connector to the Clickhouse database. This service is responsible for critical operations such as retrieving user data (e.g., balances) and recording transaction details.
1.4 Separation of Services
Marionette’s modular architecture has been further extended. The following services have been separated into independent images:
- Trading
- Operations
- Balances
Why this change?
This extension is introduced to further simplify maintenance and improve deployment flexibility.
2. Deployment Instructions v3.0.0 from v2.0.0
For step 2.2 Configuration Updates, see section 3. Changes to the Configurator v3.0.0 and apply the covered updates for individual components. Migration to v3 doc.
Follow these steps to ensure a smooth upgrade:
2.1 Stop Services: Completely shut down all running Docker containers.
2.2 Configuration Updates: Update the settings in the Configurator as needed (see Configurator)
2.3 Prepare Databases: Ensure that both Clickhouse and MySQL are running, especially if using local servers.
2.4 Clean Records in Clickhouse: Run the following command to clear existing records in case of failed upgrades:
# bash
docker compose exec clickhouse clickhouse-client
# inside clickhouse-client
use marionette
delete from ledgerRecords where true; 2.5 Initialize Services: Start NATS to enable messaging between services.
2.6 Backup Existing Data in MySQL: Take a snapshot of the database for reference before migration.
2.7 Migrate Data:
2.7.1. Start the db-bridge service and wait for database migrations to complete
2.7.2. Launch the Accounting service to create a new database and finalize migrations
2.7.3. Migrate ledger records from MySQL to Clickhouse by running the following command (in Marionette CLI ./m cli): call "accounting.migrate";
2.8 Start Remaining Services: Once the migration is complete, start all other services to resume operations.
docker compose up -d3. Changes to the Configurator v3.0.0
To deploy new components using the compose template, follow these steps:
3.1 Update/Add the following components:
- update Trading
- update Operations
- update Balances (update)
- (NEW) add Accounting
- (NEW) add ClickHouse (if you are using a local database)
clickhouse:
image: clickhouse/clickhouse-server
volumes:
- ../data/clickhouse_data/:/var/lib/clickhouse/
- ../data/clickhouse_logs/:/var/log/clickhouse-server/
ulimits:
nproc: 65535
nofile:
soft: 262144
hard: 2621443.2 Backend v.3.0.0 Key Updates
3.2.1 Data Storage: Implemented migrations to accommodate the storage of integer (UInt256) data within operations, orders, and trades.
3.2.2 Authentication: Upgraded API authentication mechanisms for improved security. Added authentication mechanism enhancements for the API.
3.3.3 Monitoring: To enhanced monitoring services, added stability and lowered system resources spent on monitoring.
3.3.4 Service Changes: Removal of trading, operations, and balance services.
4. Changes to Accounting v.3.0.0 Updates
Database Communication: Accounting is integrated with ClickHouse for efficient data handling. This service facilitates communication with the Clickhouse database.
4.1 Initial deployment requires the prior instantiation of the following containers: MySQL, Clickhouse, and NATS.
4.2 After containers are running, the service will create a new database and a LedgerRecords table.
4.3 Upon successful container initialization, the service will create a new database and the LedgerRecords table.
4.4 For systems upgrading from earlier (pre v3.0.0) versions, execute the command ./m cli (in Marionette console) after service startup.
4.5 Configuration:
Add an accounting section to the compose file. Use variables defined in global/config.yaml. Example of accounting section in compose file - in template use variables defined in global/config.yaml
Include a new variable in config/database.env for improved customization. Add env variable to the config/database.env:
5. Changes to Trading v.3.0.0 Updates
5.1 Service Independence: The service now operates independently of the backend image
5.2 Data Type Updates: All calculations have been modified to utilize UInt256 data types
5.3 Configuration Updates: Removed deprecated sections from templates/compose/backend.yaml.t
5.4 Use variables from global/config.yaml to configure the trading section in the compose file
5.5 Configuration of the trading section in the Compose file should utilize variables defined within the global configuration file, config.yaml.
6. Changes to Balances v.3.0.0 Update
6.1 Service Decoupled This service has been fully decoupled from the backend image.
6.2 Data Migration: Migrated all calculations to utilize UInt256
6.3 Configuration Update: Removed outdated sections from templates/compose/backend.yaml.t.
6.4 Configuration Update: Provided an example balances section in the compose file. Use variables defined in global/config.yaml.
6.5 Configuration Update: Following section has been removed from templates/compose/backend.yaml.t.
6.6 Example of the balances section within the compose file is provided; in the template, variables defined in global/config.yaml should be used.
7. Changes to Operations v.3.0.0 Update
7.1 Service Decoupled This service has been fully decoupled from the backend image.
7.2 Data Migration: Updated all calculations to utilize the UInt256 data type. All calculations have been migrated to utilize UInt256.
7.3 Configuration Updates: Removed deprecated sections from templates/compose/backend.yaml.t.
7.4 Configuration Updates: Included an example of the operations section in the compose file, configured with global/config.yaml variables.
7.5 Remove the preceding section from the templates/compose/backend.yaml.t file.
7.6 Following is an example of an operations section in a compose file, utilizing variables defined within the global/config.yaml template.
8. Changes to TradingView v.3.0.0 Update
8.1 Bug Fixes: Resolved an issue with the default time zone, which now uses system settings.
8.2 Image Use: To use a public image, refer to: gitlab.tunex.io:5050/marionette-public/tradingview:latest
9. Closing Recommendations
9.1 After deployment, ensure superadmin role permissions are updated:
9.2 Use Puppeteer at the following path: /puppeteer/security/roles.
9.3 Grant all new permissions to maintain access and operational integrity.
10. Marionette 3.0.0 Component Version Table:
# | Component / | Last Version | Updated Release Version | Additional notes / Compatibility |
|---|---|---|---|---|
Main Components | ||||
1 | Backend (core) | 2.4.9 | 3.0.0 |
|
2 | Accounting |
| 3.0.0 |
|
3 | Trading |
| 3.0.0 |
|
4 | Balances |
| 3.0.0 |
|
5 | Operations |
| 3.0.0 |
|
6 | Puppeteer | 2.4.7 | 3.0.0 |
|
7 | CoreConfig | 2.4.9 | 3.0.0 |
|
8 | Workflow | 2.4.6 | 3.0.0 |
|
9 | Roles | 2.4.5 | 3.0.0 |
|
10 | PaymentGateway | 1.0.4 | 3.0.0 |
|
11 | WithdrawRestrictions | 2.4.5 | 3.0.0 |
|
12 | SwapGateway | 2.4.5 | 3.0.0 |
|
13 | GoogleAuth | 2.4.3 | 3.0.0 |
|
14 | ManualRateSource | 2.4.4 | 3.0.0 |
|
15 | InternalSwap | 0.0.1 |
| 3.0.0 compatible |
16 | KYCGateway | 1.0.2 |
| 3.0.0 compatible |
17 | ManualPaymentInterface | 2.2.1 |
| 3.0.0 compatible |
18 | Proxy | 1.4.1 |
| 3.0.0 compatible |
Public Components | ||||
19 | TradingView | 0.1.0 | 3.0.0 |
|
20 | Elboto | 1.0.0-37 |
|
|
Additional Components | ||||
21 | UtxoPaymentInterface | 2.1.3 |
| 3.0.0 compatible |
22 | BankPaymentInterface | 1.4.1 |
| 3.0.0 compatible |
23 | EthGoPI | 1.0.2 |
| 3.0.0 compatible |
24 | TronPaymentInterface | 1.0.6 |
| 3.0.0 compatible |
25 | ReferralProgram | 2.4.4 | 3.0.0 |
|
26 | Staking | 2.4.2 | 3.0.0 |
|
27 | SumSub | 2.2.1 |
| 3.0.0 compatible |
28 | WhiteBit | 2.4.12 | 3.0.0 |
|
29 | KYCGlorep | 1.0.0 |
| 3.0.0 compatible |
30 | BankPaymentInterfaceGloRep | 2.4.2 |
| 3.0.0 compatible |