Filesystem: Migrate local filesystem access to Apache Arrow

[amotl]

About

Performance improvements for LocalFileSystem using Arrow.

References

• Filesystem: Migrate from fsspec.LocalFileSystem to pyarrow.fs.LocalFileSystem #232

[coderabbitai]

Walkthrough

Local filesystem initialization now wraps Apache Arrow’s LocalFileSystem for fsspec compatibility. The project adds the required fsspec dependency and records the migration in the changelog.

Changes

Local filesystem migration

Layer / File(s)	Summary
Arrow local filesystem integration src/dlt_filesystem/source/impl/local.py, pyproject.toml, docs/changelog.md	LocalFilesystemSource uses ArrowFSWrapper(LocalFileSystem()); fsspec>2021.9 is added as a project dependency, and the migration is recorded in the changelog.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

A rabbit hops through Arrow’s bright stream,
Wrapping local files in a filesystem dream.
fsspec joins the burrow, neat and spry,
While changelog carrots wait nearby.
Hop, hop—local paths now fly!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: migrating local filesystem access to Apache Arrow.
Description check	✅ Passed	The description is directly related and matches the Arrow-based LocalFileSystem performance change.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch localfs-arrow

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[read-the-docs-community]

Documentation build overview

📚 omniload | 🛠️ Build #33650131 | 📁 Comparing ca2b90d against latest (bf9bf05)

  🔍 Preview build  

1 file changed

± changelog.html

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.37%. Comparing base (bf9bf05) to head (ca2b90d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #238   +/-   ##
=======================================
  Coverage   55.37%   55.37%           
=======================================
  Files         210      210           
  Lines        9830     9831    +1     
=======================================
+ Hits         5443     5444    +1     
  Misses       4387     4387           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.