Performance and usability enhancements, bug fixes, and CI/CD automation for Shuffle#22
Open
imranakram wants to merge 10 commits into
Open
Performance and usability enhancements, bug fixes, and CI/CD automation for Shuffle#22imranakram wants to merge 10 commits into
imranakram wants to merge 10 commits into
Conversation
- Add ExecuteMultipleRequest batching for import (configurable BatchSize, UI field, schema support) - Support Multi-Select OptionSet (OptionSetValueCollection) in export/import - Ensure deterministic XML export ordering (alphabetical attributes) - Optimize attribute filtering and deduplication for performance - Hoist metadata lookups out of inner loops - Fix CSV/text export off-by-one error and improve update error logging - Add GitHub Actions for CI build and release automation - Update solution, README, and nuspecs for new features and copyright
…ogs. Updated ShuffleDataImport.cs and ShuffleSolutionImport.cs to use formatted values where available, improving clarity during data matching and solution import operations. Also adjusted logging in Shuffler.cs to avoid double-formatting messages.
…s available natively
Expanded README to include detailed documentation for Shuffle Builder, Runner, and Deployer tools, including features, usage, and XrmToolBox availability. Added a comprehensive schema reference for Shuffle Definition XML with tables for all elements and attributes. Clarified terminology and reorganized recent changes and bug fix sections for improved clarity and usability.
imranakram
added
documentation
Owner
|
Before I read the details: I recommend that we use |
High-performance bulk import now uses CreateMultiple/UpdateMultiple on Dataverse (online), with automatic detection and fallback to ExecuteMultipleRequest for on-premises or unsupported entities. Batch size default lowered to 100 per Microsoft's guidance. Batch logic refactored for robust fallback and detailed logging. Updated docs and solution items accordingly. No breaking changes to public API or config.
Introduce DeferStateAndOwner option to enable two-pass import: statecode, statuscode, and ownerid are stripped for bulk import, then applied in a second pass using bulk operations. This significantly improves performance (3-5×) for datasets with state/owner attributes. Includes README documentation, new struct definitions, and robust error handling. Feature is opt-in and backwards compatible.
Automatically uses UpsertMultiple for imports with Save="CreateUpdate" and CreateWithId="true" on Dataverse, eliminating pre-retrieval queries and significantly improving performance. Adds batching and fallback logic for all CRM versions. Updates documentation to explain UpsertMultiple optimization, usage, and compatibility. No breaking changes; full backward compatibility maintained.
Added a detailed "Import Path Selection" section to the README, explaining how Shuffle chooses between Upsert and Match-based import strategies based on configuration. Clarified when PreRetrieveAll is used or bypassed, and included a summary table for quick reference. Improved documentation for PreRetrieveAll to specify its relevance to each path, helping users optimize import performance.
Collaborator
Author
|
I have added support for CreateMultiple etc, please have a look and test as well if you can @rappen |
Previously, Upsert was enabled for CreateUpdate with IDs, match attributes, and no delete. Now, it also requires UpdateIdentical=true, ensuring Upsert is only used when identical records can be updated, since Upsert cannot skip identical records.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Classification
Implements major performance and usability enhancements, bug fixes, and CI/CD automation for Shuffle tools.
PR Summary
This PR introduces batch processing for import/export, improved Multi-Select OptionSet support, deterministic export ordering, and automated build/release workflows.
ExecuteMultipleRequestbatching for Create/Update/Delete, improves error handling, and optimizes attribute processing.BatchSizeattribute to schema and logic for configurable batching.