events.action: opened
events.author: mocchira
events.masked_author: username_0
events.text: With this PR, the time to finish leofs_test2 has been boosted from 1,900 secs to 1,300 secs on my dev-box (ubuntu 16.04 LTS on virtualbox with vcpu2 and 8GB), which means around 10 minutes has been reduced per one running.
  
  The key points are
  - Get rid of all timer:sleep(s) at src/leofs_test_scenario.erl which added unnecessary delays on each test step
  - Instead of the above timer:sleep(s), I added timer:sleep to the points where timer:sleep actually was needed such as waiting for the queue being filled up after leofs-adm (recover|rebalance|delete-bucket).
events.title: Fix #1057 - Improve the test time
events.type: issue
events.action: created
events.author: yosukehara
events.masked_author: username_1
events.text: @username_0 LGTM
events.type: comment
events.action: created
events.author: yosukehara
events.masked_author: username_1
events.text: I realized that the time spent on the test was shortened:
  
  ```bash
  $ ./leofs_test -b test -d /home/yosuke/dev/projects/leo-project/leofs/package/
   [{bucket,"test"},
    {leofs_dir,"/home/yosuke/dev/projects/leo-project/leofs/package/"}]
  
  ::: START :::
  
   [SCENARIO-1]
     * create_bucket (create a bucket)
     * recover_file (recover data of a file)
          #1
              * 'storage_0@127.0.0.1':[{leo_recovery_node_queue,1}]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #2
              * 'storage_0@127.0.0.1':[{leo_recovery_node_queue,1}]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #3
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_recovery_node_queue,1}]
              * 'storage_3@127.0.0.1':[N/A]
          #2
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,1}]
              * 'storage_3@127.0.0.1':[N/A]
          #3
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[{leo_recovery_node_queue,1}]
          #2
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[{leo_per_object_queue,1}]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #2
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,1}]
              * 'storage_3@127.0.0.1':[N/A]
          #2
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
          #1
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
     * get_objects_not_found (get objects_not_found)....................................................................................................|
     * put_zero_byte_objects (put zero byte objects)....................................................................................................|
     * get_objects (get objects)....................................................................................................|
     * put_objects (put objects)....................................................................................................|
     * get_objects (get objects)....................................................................................................|
     * check_redundancies (check redundancies of replicas)....................................................................................................|
     * del_objects (remove objects)....................................................................................................|
     * check_redundancies (check redundancies of replicas)....................................................................................................|
     * mp_upload_normal (multipart upload)
     * mp_upload_normal_in_parallel (multipart upload in parallel)
     * mp_upload_abort (abort multipart upload)
          #1
              * 'storage_0@127.0.0.1':[{leo_delete_dir_queue_1,2}]
              * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,2}]
              * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,3}]
              * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,2}]
          #2
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,1}]
              * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,3}]
              * 'storage_3@127.0.0.1':[N/A]
          #3
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
     * mp_upload_invalid_complete (invalid complete multipart upload)
   [SCENARIO-2]
     * put_objects (put objects)....................................................................................................|
     * detach_node (detach a node)
     * watch_mq (watch state of mq)
          #1
              * 'storage_0@127.0.0.1':[{leo_sync_by_vnode_id_queue,29}, {leo_rebalance_queue,552}, {leo_per_object_queue,152}]
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,73}, {leo_rebalance_queue,374}]
              * 'storage_2@127.0.0.1':[N/A]
          #2
              * 'storage_0@127.0.0.1':[{leo_sync_by_vnode_id_queue,15}, {leo_rebalance_queue,418}, {leo_per_object_queue,808}]
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,61}, {leo_rebalance_queue,711}, {leo_per_object_queue,206}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,90}]
          #3
              * 'storage_0@127.0.0.1':[{leo_sync_by_vnode_id_queue,5}, {leo_rebalance_queue,313}, {leo_per_object_queue,993}]
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,33}, {leo_rebalance_queue,443}, {leo_per_object_queue,652}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,65}, {leo_rebalance_queue,524}, {leo_per_object_queue,182}]
          #4
              * 'storage_0@127.0.0.1':[{leo_rebalance_queue,67}, {leo_per_object_queue,672}]
              * 'storage_1@127.0.0.1':[{leo_rebalance_queue,3}, {leo_per_object_queue,1277}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,29}, {leo_rebalance_queue,929}, {leo_per_object_queue,584}]
          #5
              * 'storage_0@127.0.0.1':[{leo_rebalance_queue,26}, {leo_per_object_queue,49}]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,11}, {leo_rebalance_queue,32}, {leo_per_object_queue,1354}]
          #6
              * 'storage_0@127.0.0.1':[{leo_per_object_queue,49}]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,279}]
          #7
              * 'storage_0@127.0.0.1':[{leo_per_object_queue,41}]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
          #8
              * 'storage_0@127.0.0.1':[{leo_per_object_queue,5}]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
          #9
              * 'storage_0@127.0.0.1':[N/A]
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
     * get_objects (get objects)....................................................................................................|
     * check_redundancies (check redundancies of replicas)....................................................................................................|
   [SCENARIO-3]
     * put_objects (put objects)....................................................................................................|
     * takeover (takeover)
     * watch_mq (watch state of mq)
          #1
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,175}, {leo_rebalance_queue,71}, {leo_per_object_queue,1359}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,153}, {leo_per_object_queue,2629}]
              * 'storage_4@127.0.0.1':[N/A]
          #2
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,146}, {leo_rebalance_queue,438}, {leo_per_object_queue,1280}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,132}, {leo_rebalance_queue,294}, {leo_per_object_queue,2192}]
              * 'storage_4@127.0.0.1':[N/A]
          #3
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,103}, {leo_rebalance_queue,786}, {leo_per_object_queue,1345}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,83}, {leo_rebalance_queue,722}, {leo_per_object_queue,2174}]
              * 'storage_4@127.0.0.1':[N/A]
          #4
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,68}, {leo_rebalance_queue,1154}, {leo_per_object_queue,1351}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,77}, {leo_rebalance_queue,1060}, {leo_per_object_queue,2140}]
              * 'storage_4@127.0.0.1':[N/A]
          #5
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,59}, {leo_rebalance_queue,1217}, {leo_per_object_queue,1249}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,29}, {leo_rebalance_queue,1217}, {leo_per_object_queue,1930}]
              * 'storage_4@127.0.0.1':[N/A]
          #6
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,17}, {leo_rebalance_queue,1315}, {leo_per_object_queue,1238}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,8}, {leo_rebalance_queue,985}, {leo_per_object_queue,1906}]
              * 'storage_4@127.0.0.1':[N/A]
          #7
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,6}, {leo_rebalance_queue,163}, {leo_per_object_queue,1426}]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,383}]
              * 'storage_4@127.0.0.1':[N/A]
          #8
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
     * get_objects (get objects)....................................................................................................|
     * check_redundancies (check redundancies of replicas)....................................................................................................|
   [SCENARIO-4]
     * put_objects (put objects)....................................................................................................|
     * suspend_node (suspend a node)
     * stop_node (stop a node)
     * get_objects (get objects)....................................................................................................|
     * start_node (start a node)
     * resume_node (resume a node)
     * put_objects (put objects)....................................................................................................|
     * check_redundancies (check redundancies of replicas)....................................................................................................|
     * get_objects (get objects)....................................................................................................|
   [SCENARIO-5]
     * put_objects (put objects)....................................................................................................|
     * attach_node (attach a node)
     * watch_mq (watch state of mq)
          #1
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,77}, {leo_rebalance_queue,500}, {leo_per_object_queue,219}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,83}, {leo_rebalance_queue,262}, {leo_per_object_queue,68}]
          #2
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,118}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,43}, {leo_rebalance_queue,337}, {leo_per_object_queue,786}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,38}, {leo_rebalance_queue,214}, {leo_per_object_queue,527}]
          #3
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,111}, {leo_rebalance_queue,166}, {leo_per_object_queue,19}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,38}, {leo_rebalance_queue,129}, {leo_per_object_queue,814}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,25}, {leo_rebalance_queue,285}, {leo_per_object_queue,487}]
          #4
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,111}, {leo_rebalance_queue,352}, {leo_per_object_queue,21}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,29}, {leo_rebalance_queue,133}, {leo_per_object_queue,482}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,16}, {leo_rebalance_queue,347}, {leo_per_object_queue,434}]
          #5
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,92}, {leo_rebalance_queue,396}, {leo_per_object_queue,96}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,23}, {leo_rebalance_queue,126}, {leo_per_object_queue,215}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,12}, {leo_rebalance_queue,129}, {leo_per_object_queue,468}]
          #6
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,92}, {leo_rebalance_queue,507}, {leo_per_object_queue,184}]
              * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,11}, {leo_rebalance_queue,61}, {leo_per_object_queue,122}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_rebalance_queue,28}, {leo_per_object_queue,81}]
          #7
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,58}, {leo_rebalance_queue,726}, {leo_per_object_queue,368}]
              * 'storage_2@127.0.0.1':[{leo_rebalance_queue,61}, {leo_per_object_queue,77}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_rebalance_queue,15}]
          #8
              * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,19}, {leo_rebalance_queue,706}, {leo_per_object_queue,561}]
              * 'storage_2@127.0.0.1':[{leo_rebalance_queue,61}, {leo_per_object_queue,74}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_rebalance_queue,4}]
          #9
              * 'storage_1@127.0.0.1':[{leo_per_object_queue,616}]
              * 'storage_2@127.0.0.1':[{leo_rebalance_queue,49}, {leo_per_object_queue,40}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_per_object_queue,18}]
          #10
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,27}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #11
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,27}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #12
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
     * get_objects (get objects)....................................................................................................|
     * check_redundancies (check redundancies of replicas)....................................................................................................|
     * compaction (execute data-compaction).......................................................|
     * check_redundancies (check redundancies of replicas)....................................................................................................|
     * get_objects (get objects)....................................................................................................|
   [SCENARIO-6]
     * put_objects (put objects)....................................................................................................|
     * remove_avs (remove avs of a node)
     * recover_node (recover data of a node)
     * watch_mq (watch state of mq)
          #1
              * 'storage_1@127.0.0.1':[{leo_recovery_node_queue,1}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[{leo_recovery_node_queue,1}]
              * 'storage_4@127.0.0.1':[{leo_per_object_queue,730}]
          #2
              * 'storage_1@127.0.0.1':[{leo_recovery_node_queue,1}, {leo_per_object_queue,129}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[{leo_recovery_node_queue,1}, {leo_per_object_queue,1424}]
              * 'storage_4@127.0.0.1':[{leo_per_object_queue,609}]
          #3
              * 'storage_1@127.0.0.1':[{leo_recovery_node_queue,1}, {leo_per_object_queue,954}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[{leo_per_object_queue,572}]
              * 'storage_4@127.0.0.1':[N/A]
          #4
              * 'storage_1@127.0.0.1':[{leo_per_object_queue,550}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[{leo_per_object_queue,424}]
              * 'storage_4@127.0.0.1':[N/A]
          #5
              * 'storage_1@127.0.0.1':[{leo_per_object_queue,232}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #6
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
     * get_objects (get objects)....................................................................................................|
     * check_redundancies (check redundancies of replicas)....................................................................................................|
   [SCENARIO-7]
     * delete_bucket (delete a bucket)
     * watch_mq (watch state of mq)
          #1
              * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,4633}]
              * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,5020}]
              * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,4743}]
              * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,4617}]
          #2
              * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,3917}]
              * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,4354}]
              * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,4220}]
              * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,3816}]
          #3
              * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,2892}]
              * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,3215}]
              * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,3144}]
              * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,2916}]
          #4
              * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,2424}]
              * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,2466}]
              * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,2407}]
              * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,2151}]
          #5
              * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,1395}]
              * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,1545}]
              * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,1514}]
              * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,1247}]
          #6
              * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,682}]
              * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,873}]
              * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,677}]
              * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,324}]
          #7
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
     * create_bucket (create a bucket)
     * get_objects_not_found (get objects_not_found)....................................................................................................|
     * put_inconsistent_objects (put inconsistent objects)....................................................................................................|
     * scrub_cluster (scrub the whole cluster)
          #1
              * 'storage_1@127.0.0.1':[{leo_recovery_node_queue,1}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #2
              * 'storage_1@127.0.0.1':[{leo_per_object_queue,4527}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #3
              * 'storage_1@127.0.0.1':[{leo_per_object_queue,2921}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #4
              * 'storage_1@127.0.0.1':[{leo_per_object_queue,1649}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #5
              * 'storage_1@127.0.0.1':[{leo_per_object_queue,174}]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #6
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #1
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,4879}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #2
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,4443}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #3
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,2370}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #4
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[{leo_per_object_queue,57}]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #5
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #1
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[{leo_per_object_queue,3948}]
              * 'storage_4@127.0.0.1':[N/A]
          #2
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[{leo_per_object_queue,3128}]
              * 'storage_4@127.0.0.1':[N/A]
          #3
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[{leo_per_object_queue,503}]
              * 'storage_4@127.0.0.1':[N/A]
          #4
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
          #1
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_recovery_node_queue,1}]
          #2
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_recovery_node_queue,1}]
          #3
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_per_object_queue,3745}]
          #4
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_per_object_queue,3318}]
          #5
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_per_object_queue,1416}]
          #6
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[{leo_per_object_queue,211}]
          #7
              * 'storage_1@127.0.0.1':[N/A]
              * 'storage_2@127.0.0.1':[N/A]
              * 'storage_3@127.0.0.1':[N/A]
              * 'storage_4@127.0.0.1':[N/A]
     * check_redundancies (check redundancies of replicas)....................................................................................................|
  ::: Finished (1548sec) :::
  ```
events.type: comment