feat: added tracking for temporary fix!

amindadgar · amindadgar · commit aaf252c1c3c2 · 2025-06-29T10:28:27.000+03:30
diff --git a/hivemind_etl/mediawiki/etl.py b/hivemind_etl/mediawiki/etl.py
@@ -103,6 +103,8 @@ def load(self, documents: list[Document]) -> None:
         )
         
         # Process batches in parallel using ThreadPoolExecutor
+        # TODO: Revert to larger batch size once llama-index loading issue is resolved
+        # See: https://github.com/TogetherCrew/temporal-worker-python/issues/60
         batch_size = 1
         batches = [documents[i:i + batch_size] for i in range(0, len(documents), batch_size)]
         

Original file line number	Diff line number	Diff line change
`@@ -103,6 +103,8 @@ def load(self, documents: list[Document]) -> None:`
`103`	`103`	`)`
`104`	`104`
`105`	`105`	`# Process batches in parallel using ThreadPoolExecutor`
	`106`	`+ # TODO: Revert to larger batch size once llama-index loading issue is resolved`
	`107`	`+ # See: https://github.com/TogetherCrew/temporal-worker-python/issues/60`
`106`	`108`	`batch_size = 1`
`107`	`109`	`batches = [documents[i:i + batch_size] for i in range(0, len(documents), batch_size)]`
`108`	`110`