HPCC-34683 Add tests to compare round trip times for Dali save

[streeterd]

1. Added a new unit test that can be used to compare the round trip time for loading and saving a running dali store for ipt_none vs ipt_lowmem from running Bare Metal.

Type of change:

• This change is a bug fix (non-breaking change which fixes an issue).
• This change is a new feature (non-breaking change which adds functionality).
• This change improves the code (refactor or other change that does not change the functionality)
• This change fixes warnings (the fix does not alter the functionality or the generated code)
• This change is a breaking change (fix or feature that will cause existing behavior to change).
• This change alters the query API (existing queries will have to be recompiled)

Checklist:

• My code follows the code style of this project.
  • My code does not create any new warnings from compiler, build system, or lint.
• The commit message is properly formatted and free of typos.
  • The commit message title makes sense in a changelog, by itself.
  • The commit is signed.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly, or...
  • I have created a JIRA ticket to update the documentation.
  • Any new interfaces or exported functions are appropriately commented.
• I have read the CONTRIBUTORS document.
• The change has been fully tested:
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • I have checked that this change does not introduce memory leaks.
  • I have used Valgrind or similar tools to check for potential issues.
• I have given due consideration to all of the following potential concerns:
  • Scalability
  • Performance
  • Security
  • Thread-safety
  • Cloud-compatibility
  • Premature optimization
  • Existing deployed queries will not be broken
  • This change fixes the problem, not just the symptom
  • The target branch of this pull request is appropriate for such a change.
• There are no similar instances of the same problem that should be addressed
  • I have addressed them here
  • I have raised JIRA issues to address them separately
• This is a user interface / front-end modification
  • I have tested my changes in multiple modern browsers
  • The component(s) render as expected

Smoketest:

• Send notifications about my Pull Request position in Smoketest queue.
• Test my draft Pull Request.

Testing:

Steps:

1. Run up bare metal
2. Run commands:

• ~/hpccDebug/etc/init.d/hpcc-init stop -c mydali
• ~/hpccDebug/etc/init.d/hpcc-init start -c mydali
• ~/build/Debug-install/Debug/bin/unittests -v -e DaliStoreLoadAndSaveTimingStressTest

Output:

16:04:02.109974 14390 UNK UNK Including test DaliStoreLoadAndSaveTimingStressTest
.16:04:02.134749 14390 UNK UNK MP TLS: off acceptThreadPoolSize: 0
16:04:02.140532 14390 UNK UNK Auto-selected dali store file '//192.168.1.115/home/dave/hpccDebug/var/lib/HPCCSystems/hpcc-data/dali/dalisds1.xml' (edition=1)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/normal/i0
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/normal/i0 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/normal/i0/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/normal/i1
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/normal/i1 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/normal/i1/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/normal/i2
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/normal/i2 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/normal/i2/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/normal/i3
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/normal/i3 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/normal/i3/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/normal/i4
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/normal/i4 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/normal/i4/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/lowmem/i0
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i0 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i0/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/lowmem/i1
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i1 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i1/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/lowmem/i2
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i2 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i2/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/lowmem/i3
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i3 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i3/Store (attrs=0 children=15)
sds-ls -la /DaliStoreLoadAndSaveTimingStressTest/lowmem/i4
drwxr-xr-x 1 sds sds     1 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i4 (attrs=0)
drw-r--r-- 1 sds sds 67686 /DaliStoreLoadAndSaveTimingStressTest/lowmem/i4/Store (attrs=0 children=15)

=== XML TIMING COMPARISON TEST ===
Store file: //192.168.1.115/home/dave/hpccDebug/var/lib/HPCCSystems/hpcc-data/dali/dalisds1.xml
XML data size: 67682 bytes
Iterations: 5
┌──────────────────────┬─────────────────┬─────────────────┬─────────────────┐
│ Mode                 │ Avg SDS Load    │ Avg SDS Save    │ Avg Diag Save   │
│                      │ (ms)            │ (ms)            │ (ms)            │
├──────────────────────┼─────────────────┼─────────────────┼─────────────────┤
│ XML Normal           │        0.000286 │       11.184660 │       14.168543 │
│ XML Low Memory       │        0.000395 │       11.656103 │       26.447048 │
│ XML LowMem Diff      │         +37.97% │          +4.22% │         +86.66% │
└──────────────────────┴─────────────────┴─────────────────┴─────────────────┘



OK (1 tests)