Mirocana Development Update

In this letter, I would like to address all investors, clients and token-holders who have concerns regarding Mirocana development and our communication practices.

The structure of our core product is very sophisticated and it caused unexpected bugs and difficulties to appear as we develop. I would like to share some tasks that were resolved for the past two weeks (data from our Jira task tracker). The majority of these tasks belongs to our private repository and do not affect our public interfaces. Some tasks have titles and descriptions that could be hard to understand without a context. Each task has it's own time consumption (in hours) that was spent by developer to resolve it:

• Speeding up super strategies - 8 hours

Now full run of super strategy takes about 6 hours (fix it).We have to explore and to understand which steps is necessary to do to make it faster.

• Optimizing MQBSequence - 6 hours

The problem is: data is copied on each step of iteration from self.data to self.show. The solution is to rewrite MQBSequence to not use self.show at all.

• Clickhouse duplicates - 10 hours

There are some duplicates in ClickHouse predictions table. It's necessary to insert all the data exactly once.

• Calculating super strategies only on existing data - 4 hours

Now all the super strategies read data up to the last day. Some strategies may not be calculated for all their period of time and return empty prediction set. The problem appears when predictions for some additional period of time because super strategies should be recalculated.

• Remove using .show in strategies - 1 hour
• Internal: last_with_shift problem - 1 hour

Check last_with_shift on GappingDojiStrategy, USDT_BTC symbol. It says "zero-size array to reduction operation maximum which has no identity"

• Task manager + queue implementation - 18 hours
• Slug queue testing - 1 hour

Test SlugQueue class + apply ideas of some basic testing env.

• Strategies sets essence - 2 hours

Understand and recollect all the strategies in such way to make the strategy manipulation easy.

• Slug queue features + test coverage - 6 hours

There are possible improvements to SlugQueue: 1) calculating mean and deviation to extract it in O(1) + check all strategies for correct using this thing 2) writing tests which check logic for cutting tail and etc. 3) checking speed for last_ methods and choose the best way. 4) return None in last_ methods in case when no data can be extracted, then run all the strategies and fix them. 5) understand carefully about choosing MAX_LAST. 6) rename MAX_LAST. 7) rename MQB and sequence to their true meaning

• A better way for collect_strategy_references - 3 hours

There are some small issues here. 1) building a path to the strategies. 2) collecting strategies inside the file

• MQB clickhouse tests - 4 hours

It's important to check that all data in MQBs are the same as the data in clickhouse. The way of doing it: 1) create a small piece of data in clickhouse and put it into test db. 2) run at the same time MQB iterator and SELECT from db to check any differerence

• Fixing existing tests - 3 hours

Some tests in core_test directory look weird. The task is to check them manually and rewrite if possible.

• Integrate Confluence with JIRA - 1 hour
• SSH authorization + run as user, not as a root - 1 hour
• Clickhouse is down on srv008 - 1 hour
• Backup system for main-node server - 3 hours
• Choose rabbit queue from settings file - 1 hour
• Create context table and save contexts into ClickHouse rarely - 8 hours

Use some period like 10 seconds or 15 minutes - can be set up manually in settings

• Change logic of task assigning in task_manager - 4 hours

Sometimes task_manager assignes a super strategy task for a worker which kills the worker forever until all ordinary strategies got calculated. The solution consists of 2 parts: 1. Worker should throw away the task which cannot be done yet 2. Task manager must pessimize the priority of tasks which didn't executed well

• Restore the state of the mongo DB from snapshot - 1 hour
• RERUN feature - 6 hours

RERUN means we can recalculate the results of applying some strategy or super_strategy or even simulation. For RERUN we need to create a set of things to rerun and not load their context from snapshot.

• Load markets in a proper way - 3 hours

It's a good idea to combine some stuff from delta.py and combinations to build a whole market essences which will be loaded from settings.

• Syncing mongo with settings - 1 hour
• Not to write predictions second time after snapshot recovery - 2 hours

We need to take real iteration time from predictions in predictions table to compare truly when it's necessary to insert new predictions.

• Using iteration_number instead of iteration_time to validate progress - 2 hours
• Task manager: kill if the freshest one is not me - 1 hour
• Workers: check status function instead of set online function - 4 hours
• Task manager: update logic of task assigning - 3 hours
• Reference fast-node.mirocana.com - 1 hour

Please, reference 9192 port to fast-node.mirocana.com similarly to node.mirocana.com

• Create item ids in Zabbix - 2 hours

Create item ids for all data points in node.mirocana.com/stats. Mention the full list of item ids in this task

• Recalculate delta table in production - 7 hours

Super strategies cannot be done until deltas are not calculated with 'market' field.

• Kill TASK_IN_PROGRESS which isn't responding for a long time - 2 hours

Update task manager to check all IN PROGRESS tasks.

• Super predictions: 17 MB context cannot be stored in mongo - 6 hours

See commit 66568c08bccff44d12978aaae137a3f2c898c59e.

• Pymongo.errors.OperationFailure - 4 hours

Mongo cannot store huge number of keys 1. increase RAM for mongo (it's only 32 mb for now) 2. sort everything outside mongo "find" operations 3. garbage collector for mongo collections

• mongo stats query - 1 hours
• GraphEasy simulation - 3 hours
• Remove contexts values from clickhouse and add gridfs id - 4 hours
• Snapshots to super strategies and simulations - 3 hours