add sql script generation for deleting obsolete cocs by eperedo · Pull Request #94 · EyeSeeTea/d2-tools

eperedo · 2026-01-28T19:02:39Z

📌 References

Issue: https://app.clickup.com/t/869btxm35

📝 Implementation

Generating a sql script for deleting categoryOptionCombos.

yarn start categoryOptionCombos regenerate \
    --url=https://play.im.dhis2.org/stable-2-40-10 \
    --auth="admin:district" \
     --generate-sql-delete-script

Before deleting a categoryOptionCombo the script checks for existing data in the datavalue and datavalueaudit tables. If you have thousands or millions of records, this process can be very slow.

Adding an index to these tables improves performance significantly:

CREATE INDEX CONCURRENTLY idx_datavalue_catoptcombo ON datavalue(categoryoptioncomboid);
CREATE INDEX CONCURRENTLY idx_datavalue_attoptcombo ON datavalue(attributeoptioncomboid);
CREATE INDEX CONCURRENTLY idx_datavalueaudit_catoptcombo ON datavalueaudit(categoryoptioncomboid);
CREATE INDEX CONCURRENTLY idx_datavalueaudit_attoptcombo ON datavalueaudit(attributeoptioncomboid);

📹 Screenshots/Screen capture

🔥 Notes to the tester

#869btxm35

tokland · 2026-01-30T10:40:45Z

        await this.saveCocs(categoryComboWithGeneratedCocs, options);
-        await this.deleteCocs(categoryComboWithGeneratedCocs, options);
+        if (options.deleteCocs) {
+            await this.deleteCocs(categoryComboWithGeneratedCocs, options);


method deleteCocs also takes option deleteCocs, which is confusing, 1) it will only be called when true (as we now have this if). 2) you'd expect to delete, given the method name.

tokland

In-line comments:

tokland · 2026-02-26T12:56:05Z

+                    categoryIndex: categoryIndexByCocOptionPosition[inputPosition],
+                    inputPosition,
+                }))
+                .sortBy(item => [item.categoryIndex, item.inputPosition])


This pattern has been on my mind for some time, so I took the time to explore it. According to the docs, it should work: https://lodash.com/docs/4.17.23#sortBy (_.sortBy(users, ['user', 'age']); it the equivalent example). However, I was worried if this was sorting things as one would expect (lexycographical order) or there was some string conversion that messes things up.

> var users = [{ 'user': 'fred', 'age': 48 }, { 'user': 'barney', 'age': 31 }, { 'user': 'barney', 'age': 4 }] > JSON.stringify(_.sortBy(users, ["user", "age"])) '[{"user":"barney","age":4},{"user":"barney","age":31},{"user":"fred","age":48}]' > JSON.stringify(_.sortBy(users, [u => u.user, u => u.age])) '[{"user":"barney","age":4},{"user":"barney","age":31},{"user":"fred","age":48}]' > JSON.stringify(_.sortBy(users, u => [u.user, u.age])) '[{"user":"barney","age":31},{"user":"barney","age":4},{"user":"fred","age":48}]'

So the code that works is passing mappers as an array (with string -we do not use that, not type-safe, or an arrow function), but not returning an array, as JS compares them as strings.

Also, in addition of fixing, the code if you can tweak the spec so we have a RED before going to GREEN, great!

tokland · 2026-02-26T12:57:11Z

+    private createDeleteUnusedCategoryOptionCombosSQL(ids: Id[], batchSize: number = 500): string {
+        const batches = Array.from({ length: Math.ceil(ids.length / batchSize) }, (_, i) =>
+            ids.slice(i * batchSize, (i + 1) * batchSize)
+        );


Is the same than _.chunk ?

tokland · 2026-02-26T12:57:59Z

+DROP TABLE temp_uids_batch;
+DROP TABLE temp_ids_batch;
+
+`;


[subjective] To avoid a long multi-line variable in the middle of a function/method, we can move it to a constant/instance value and use lodash template to interpolate when used, what do you think?

tokland · 2026-02-26T12:59:26Z

+`;
+        });
+
+        return `BEGIN;${batchStatements.join("\n")}COMMIT;`.trim();


Very minor, but id' write an a multiline array with a final join., something like this

return [
"BEGIN;", //
...batchStatements,
"COMMIT;",
].join("\n");

(in the hope that it's more readable)

tokland · 2026-02-26T13:01:37Z

                return {
                    regeneratedCoc: RegeneratedCoc.create({
-                        id: getUid(combinationKey, categoryCombo.id),
+                        id: categoryComboId,


So categoryOptionComboId?

tokland · 2026-02-26T13:09:23Z

+import { Id } from "domain/entities/Base";
+
+export interface CategoryOptionComboDeleteExporter {
+    exportDeleteScript(cocIds: Id[]): void;


Proposal: instead of a side-effect (saving the file), return here a string. And make the use case return also that string somewhere in the response object. And it's the script that writes the final file from that string. What do you think?

…k in presentation layer

tokland

Some very minor things:

tokland · 2026-03-03T13:05:45Z

-    async getAll(): Promise<CategoryCombo[]> {
-        return this.getAllByPages({ page: 1, pageSize: 100, categoryCombos: [] });
+    async getAll(options: { ids?: Id[] }): Promise<CategoryCombo[]> {
+        return this.getAllByPages({ page: 1, pageSize: 100, categoryCombos: [], ids: options.ids });


[Minor, subjective] Having getAll with an optional ids filter feels a bit confusing. Maybe it would be clearer to have two separate methods: getAll() and getByIds(ids)?

tokland · 2026-03-03T13:09:42Z

+function writeSqlScriptToDisk(sqlScript: string, fileName: string): void {
+    writeFileSync(fileName, sqlScript);
+    logger.info(`SQL generated: ${fileName}`);
+    logger.info(`You can execute the sql with d2-docker: d2-docker run-sql ${fileName}`);


we may have N instances running, I'd add the explicit image option

tokland · 2026-03-03T13:10:37Z

+    --auth="admin:district" \
+    --persist \
+    --delete-cocs \
+    --cat-combos-ids=id1,id2,id3


Proposal: --category-combo-ids (no abbreviation, plural only the last term)

tokland · 2026-03-03T13:11:04Z

    --auth="admin:district" \
    --persist \
-    --generate-sql-delete-script
+    --generate-sql-delete-script \


do we need it?

tokland · 2026-03-03T13:13:50Z

                page: options.page,
                pageSize: options.pageSize,
+                // TODO: change to request in chunks when ids are provided
+                filter: options.ids ? { id: { in: options.ids } } : undefined,


Is this TODO to do now, as we have the ids?

eperedo added 2 commits January 28, 2026 13:59

add sql script generation for deleting obsolete cocs

3af96bb

update readme

a2ae0e8

eperedo requested review from MiquelAdell and tokland January 28, 2026 19:02

tokland reviewed Jan 30, 2026

View reviewed changes

eperedo added 2 commits February 25, 2026 17:29

fix order of categoryOptions inside categoryOptionCombos

61d270e

move sql script from presentation to domain layer + infra

bbc1c99

eperedo requested a review from tokland February 26, 2026 00:40

tokland reviewed Feb 26, 2026

View reviewed changes

eperedo added 4 commits March 2, 2026 15:55

fix order by categoryIndex

fa64588

improvements on coc delete sql generation

9f696ee

remove side effect by returning the sql script and saving file to dis…

f77897f

…k in presentation layer

add support to regenerate specific cat. combos by providing ids

8e4e885

eperedo requested a review from tokland March 2, 2026 22:32

tokland reviewed Mar 3, 2026

View reviewed changes

Conversation

eperedo commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📌 References

📝 Implementation

📹 Screenshots/Screen capture

🔥 Notes to the tester

Uh oh!

tokland Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tokland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tokland Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tokland Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

tokland Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tokland Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tokland Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tokland Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tokland left a comment

Choose a reason for hiding this comment

Uh oh!

tokland Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

tokland Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

tokland Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

tokland Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

tokland Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

eperedo commented Jan 28, 2026 •

edited

Loading

tokland Jan 30, 2026 •

edited

Loading

tokland Feb 26, 2026 •

edited

Loading

tokland Feb 26, 2026 •

edited

Loading

tokland Feb 26, 2026 •

edited

Loading

tokland Feb 26, 2026 •

edited

Loading

tokland Feb 26, 2026 •

edited

Loading

tokland Mar 3, 2026 •

edited

Loading