[Bug]: Memory Exception

[souzadevinicius]

What happened?

python3(61861,0x16ba5b000) malloc: *** error for object 0x49b400000: pointer being freed was not allocated
python3(61861,0x16ba5b000) malloc: *** set a breakpoint in malloc_error_break to debug

What did you expect to see?

No response

What version of fastdup were you runnning on?

2.45

What version of Python were you running on?

3.11.11

Operating System

macOS Tahoe Apple M4 Pro

Reproduction steps

import fastdup


def main():
    fd = fastdup.create(
        work_dir="fastdup_work_dir/",
        input_dir="./Pictures",
    )
    fd.run(run_mode=1)


if __name__ == "__main__":
    main()

Relevant log output

python3(61861,0x16ba5b000) malloc: *** error for object 0x49b400000: pointer being freed was not allocated
python3(61861,0x16ba5b000) malloc: *** set a breakpoint in malloc_error_break to debug
[61921:573795:20251109,153132.294307:WARNING process_memory_mac.cc:93] mach_vm_read(0x14ce14000, 0x8000): (os/kern) invalid address (1)
[61921:573795:20251109,153132.294855:WARNING process_memory_mac.cc:93] mach_vm_read(0x14cf40000, 0x8000): (os/kern) invalid address (1)
[61921:573795:20251109,153132.295271:WARNING process_memory_mac.cc:93] mach_vm_read(0x14d09c000, 0x8000): (os/kern) invalid address (1)
[61921:573795:20251109,153132.299924:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
[61921:573795:20251109,153132.299951:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libobjc.A.dylib
[61921:573795:20251109,153132.299965:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
[61921:573795:20251109,153132.299979:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/liboah.dylib
[61921:573795:20251109,153132.300009:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libc++abi.dylib
[61921:573795:20251109,153132.300103:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libRosetta.dylib
[61921:573795:20251109,153132.300143:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libc++.1.dylib
[61921:573795:20251109,153132.300165:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
[61921:573795:20251109,153132.300194:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
[61921:573795:20251109,153132.300222:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/Network.framework/Versions/A/Network
[61921:573795:20251109,153132.300238:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/Security.framework/Versions/A/Security
[61921:573795:20251109,153132.300252:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/LoggingSupport.framework/Versions/A/LoggingSupport
[61921:573795:20251109,153132.300269:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libquic.dylib
[61921:573795:20251109,153132.300284:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libsqlite3.dylib
[61921:573795:20251109,153132.300299:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libdispatch.dylib
[61921:573795:20251109,153132.300311:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libdyld.dylib
[61921:573795:20251109,153132.300323:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_asl.dylib
[61921:573795:20251109,153132.300337:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_c.dylib
[61921:573795:20251109,153132.300354:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_malloc.dylib
[61921:573795:20251109,153132.300366:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_notify.dylib
[61921:573795:20251109,153132.300377:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_secinit.dylib
[61921:573795:20251109,153132.300387:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_kernel.dylib
[61921:573795:20251109,153132.300398:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_platform.dylib
[61921:573795:20251109,153132.300409:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_pthread.dylib
[61921:573795:20251109,153132.300421:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libsystem_trace.dylib
[61921:573795:20251109,153132.300434:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/system/libxpc.dylib
[61921:573795:20251109,153132.300451:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
[61921:573795:20251109,153132.300470:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
[61921:573795:20251109,153132.300496:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
[61921:573795:20251109,153132.300532:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/SkyLight
[61921:573795:20251109,153132.300551:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/BaseBoard.framework/Versions/A/BaseBoard
[61921:573795:20251109,153132.300570:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/CoreDisplay.framework/Versions/A/CoreDisplay
[61921:573795:20251109,153132.300583:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/VideoToolbox.framework/Versions/A/VideoToolbox
[61921:573795:20251109,153132.300627:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/ExtensionFoundation.framework/Versions/A/ExtensionFoundation
[61921:573795:20251109,153132.300644:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/AppServerSupport.framework/Versions/A/AppServerSupport
[61921:573795:20251109,153132.300681:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/AudioToolboxCore.framework/Versions/A/AudioToolboxCore
[61921:573795:20251109,153132.300702:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/OSLog.framework/Versions/A/OSLog
[61921:573795:20251109,153132.300715:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libspindump.dylib
[61921:573795:20251109,153132.300729:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
[61921:573795:20251109,153132.300742:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/GPUWrangler.framework/Versions/A/GPUWrangler
[61921:573795:20251109,153132.300752:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/IOPresentment.framework/Versions/A/IOPresentment
[61921:573795:20251109,153132.300860:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
[61921:573795:20251109,153132.300874:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
[61921:573795:20251109,153132.300892:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/IconFoundation.framework/Versions/A/IconFoundation
[61921:573795:20251109,153132.300906:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/RenderBox.framework/Versions/A/RenderBox
[61921:573795:20251109,153132.300935:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libTelephonyUtilDynamic.dylib
[61921:573795:20251109,153132.300956:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/libsandbox.1.dylib
[61921:573795:20251109,153132.300969:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/AVFCore.framework/Versions/A/AVFCore
[61921:573795:20251109,153132.300996:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/PrivateFrameworks/AttributeGraph.framework/Versions/A/AttributeGraph
[61921:573795:20251109,153132.301020:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
[61921:573795:20251109,153132.301068:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL
[61921:573795:20251109,153132.301183:WARNING mach_o_image_annotations_reader.cc:85] unexpected crash info version 7 in /usr/lib/dyld

Attach a screenshot [Optional]

No response

Contact Details [Optional]

No response

[dbickson]

Hi @souzadevinicius
This is what chatgpt says:

Thanks for the super-clear repro. That malloc “pointer being freed was not allocated” + the presence of libRosetta.dylib in your crash logs screams architecture mix-up on Apple Silicon (some x86_64 piece under Rosetta crossing ABI boundaries with arm64 code). That’s the #1 cause I’ve seen for this exact error pattern in Python+C++ image libs (OpenCV/libjpeg/etc.).

Here’s a tight, do-now checklist to isolate and fix:

0. Quick sanity checks

Run these to see what’s actually running:

python3 - <<'PY'
import platform, struct, sys
print("Python:", sys.version)
print("machine:", platform.machine()) # expect: arm64
print("pointer_bits:", struct.calcsize("P")*8) # expect: 64
PY

file "$(which python3)" # should say arm64 or universal
pip show fastdup opencv-python opencv-python-headless numpy

If you see x86_64 anywhere (or “i386”), you’re under Rosetta.

1. Use a clean, arm64-native Python env

Avoid mixing Homebrew prefixes (/usr/local = Intel, /opt/homebrew = ARM):

Make sure your terminal itself is NOT running under Rosetta.

Then:

which brew
brew --prefix # expect /opt/homebrew

Fresh venv using the system or /opt/homebrew Python

/opt/homebrew/bin/python3 -m venv ~/venvs/fd-arm
source ~/venvs/fd-arm/bin/activate
python -m pip install --upgrade pip wheel setuptools
python -m pip install fastdup==2.45 "opencv-python>=4.8" --only-binary=:all:

If you use Conda, create an arm64 env (conda config --show | grep subdir → osx-arm64) and ensure all packages are osx-arm64.

2. Force single-thread to rule out race/double-free

Some image libs and BLAS mixes can double-free under thread pressure. Try:

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export VECLIB_MAXIMUM_THREADS=1
export MKL_NUM_THREADS=1

Then run your minimal script again.

3. Reduce the repro to “known-safe” files

Make sure the crash isn’t triggered by a specific format/decoder:

mkdir /tmp/fd_test && cp /System/Library/Desktop\ Pictures/Solid\ Colors/Blue.png /tmp/fd_test/
python - <<'PY'
import fastdup
fd = fastdup.create(work_dir="/tmp/fd_work", input_dir="/tmp/fd_test")
fd.run(run_mode=1)
print("OK")
PY

If that works, the issue could be specific codecs (e.g., HEIC) or corrupt files in ~/Pictures. Then try running on a subset, or convert HEIC → PNG/JPEG first to test.

4. Enable malloc + Python alloc debuggers (to get a real backtrace)

This will point at the offending binary:

export PYTHONMALLOC=debug
export MallocScribble=1
export MallocStackLogging=1
lldb -- python -X dev your_script.py

when it crashes:

(lldb) bt all

Share the backtrace frame that first leaves Python and enters native code (it will show the exact dylib that frees the pointer).

5. Verify no “Intel” wheels slipped in

List platform tags:

python - <<'PY'
import pkg_resources, sys
for d in pkg_resources.working_set:
if any(tag in d.location for tag in ("macosx_10", "x86_64")):
print("Suspicious:", d.project_name, d.location)
PY

If you see an x86_64 wheel (common culprits: opencv-python, numpy, Pillow, pyarrow), reinstall it as arm64:

pip install --force-reinstall --no-binary=:all: # build arm64 from source

or prefer official arm/universal wheels if available:

pip install --force-reinstall --only-binary=:all:

6. Try the alternate OpenCV build

If you installed opencv-python, test swapping to headless (or vice-versa):

pip uninstall -y opencv-python opencv-python-headless
pip install opencv-python-headless==4.10.0.84

(Headless avoids linking against macOS GUI bits that sometimes interact poorly under Rosetta.)

7. Small code tweaks (defensive)

Make sure your minimal script is exactly this (to avoid accidental import side effects):

import fastdup

def main():
fd = fastdup.create(work_dir="fastdup_work_dir/", input_dir="./Pictures")
fd.run(run_mode=1)

if name == "main":
main()

8. If it still crashes

Please capture and share these (I can pinpoint the offending binary/function):

Output of:

platform.machine, pointer bits, and file $(which python3)

pip freeze | egrep 'fastdup|opencv|numpy|pillow|pyarrow'

otool -L $(python -c "import cv2,inspect,os;print(os.path.realpath(inspect.getfile(cv2)))")

The bt all from lldb with PYTHONMALLOC=debug, MallocStackLogging=1.

Why this likely fixes it

Your log includes /usr/lib/libRosetta.dylib, which indicates some part of the process is translated (x86_64) while other parts are native (arm64). When C/C++ extensions from different architectures/wheels mix, you can get mismatched allocators (e.g., new[] in one binary, free() in another), leading exactly to “pointer being freed was not allocated.”

Locking everything to arm64 (or everything to Rosetta, but arm64 is strongly preferred) and keeping OpenCV/NumPy/Pillow consistent almost always resolves this class of crash.

If you want, send me the lldb backtrace + the package/arch outputs and I’ll map the failing frame to the exact library call and suggest a precise package swap.

[souzadevinicius]

>>> import platform, struct, sys
>>> print("Python:", sys.version)
Python: 3.11.11 (main, Feb 19 2025, 06:06:38) [Clang 16.0.0 (clang-1600.0.26.6)]
>>> print("machine:", platform.machine()) # expect: arm64
machine: arm64
>>> print("pointer_bits:", struct.calcsize("P")*8) # expect: 64
pointer_bits: 64
>>> 
## Reduce the repro to “known-safe” files

It worked with safe files, but when I got back to my files, new crash described below

mkdir /tmp/fd_test && cp /System/Library/Desktop\ Pictures/Solid\ Colors/Blue.png /tmp/fd_test/

python - <<'PY'
import fastdup
fd = fastdup.create(work_dir="/tmp/fd_work", input_dir="./Pictures")
fd.run(run_mode=1)
print("OK")
PY

.asdf/installs/python/3.11.11/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown        

List platform tags:

python - <<'PY'
import pkg_resources, sys
for d in pkg_resources.working_set:
if any(tag in d.location for tag in ("macosx_10", "x86_64")):
print("Suspicious:", d.project_name, d.location)
PY

DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources, sys

[dbickson]

HI @souzadevinicius please try version 2.46 for mac m4, we have added some additional printouts to try and nail the issue. The best would be to take the minimal dataset size where the error happens and run with verbose=1 to get the full debug output.