Skip to content

Commit 2ac10fa

Browse files
ashaoashao
committed
Initial commit for ML Library Builder
The ML Library Builder is a Makefile-based tool to help buildt ML backends for Apple arm64 architectures. This is primarily meant to support SmartSim users doing development and prototyping, however we hope that it is also useful more broadly
0 parents  commit 2ac10fa

7 files changed

Lines changed: 168 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
build
2+
install
3+
*.tgz
4+
*.zip
5+
*.tar.gz

‎.gitmodules‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "pytorch"]
2+
path = pytorch
3+
url = https://github.com/pytorch/pytorch.git

‎LICENSE.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
ML Library Builder is licensed under BSD 2-Clause
2+
---
3+
BSD 2-Clause License
4+
5+
Copyright (c) 2024, Hewlett Packard Enterprise
6+
All rights reserved.
7+
8+
Redistribution and use in source and binary forms, with or without
9+
modification, are permitted provided that the following conditions are met:
10+
11+
1. Redistributions of source code must retain the above copyright notice, this
12+
list of conditions and the following disclaimer.
13+
14+
2. Redistributions in binary form must reproduce the above copyright notice,
15+
this list of conditions and the following disclaimer in the documentation
16+
and/or other materials provided with the distribution.
17+
18+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
19+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
20+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
21+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
22+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
23+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
24+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
25+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
26+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
27+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎Makefile‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# BSD 2-Clause License
2+
#
3+
# Copyright (c) 2024, Hewlett Packard Enterprise
4+
# All rights reserved.
5+
#
6+
# Redistribution and use in source and binary forms, with or without
7+
# modification, are permitted provided that the following conditions are met:
8+
#
9+
# 1. Redistributions of source code must retain the above copyright notice, this
10+
# list of conditions and the following disclaimer.
11+
#
12+
# 2. Redistributions in binary form must reproduce the above copyright notice,
13+
# this list of conditions and the following disclaimer in the documentation
14+
# and/or other materials provided with the distribution.
15+
#
16+
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
17+
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18+
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
19+
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
20+
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
21+
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
22+
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
23+
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
24+
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
25+
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
26+
27+
PYTORCH_VERSION=2.0.1
28+
OSX_ARCHITECTURE=arm64
29+
30+
TORCH_TARGET = libtorch-macos-$(OSX_ARCHITECTURE)-$(PYTORCH_VERSION).zip
31+
TORCH_BUILD = $(PWD)/build/libtorch
32+
TORCH_INSTALL = $(PWD)/install/libtorch
33+
34+
TORCH_CMAKE_OPTIONS =
35+
TORCH_CMAKE_OPTIONS += -DCMAKE_OSX_ARCHITECTURES=$(OSX_ARCHITECTURE)
36+
TORCH_CMAKE_OPTIONS += -DUSE_MKL=OFF -DUSE_MKLDNN=OFF -DUSE_ITT=OFF
37+
TORCH_CMAKE_OPTIONS += -DUSE_QNNPACK=OFF -DUSE_KINETO=OFF
38+
39+
.PHONY: help
40+
help:
41+
@grep "^# help\:" Makefile | grep -v grep | sed 's/\# help\: //' | sed 's/\# help\://'
42+
43+
ifneq ($(shell uname), Darwin)
44+
$(error This tool requires Mac OSX)
45+
endif
46+
47+
# help:
48+
# help: ----Overview----
49+
# help: This makefile can be used to builds ML backends for use on arm64. Generally
50+
# help: all that needs to be done to accomplish this is
51+
# help:
52+
# help: pip install -r pytorch/requirements.txt
53+
# help: make torch
54+
# help:
55+
# help: ----Meta targets----
56+
# help: clean -- Cleans all build and install directories
57+
.PHONY: clean
58+
clean: clean_torch
59+
60+
# help:
61+
# help: ----Build Targets----
62+
# help: torch -- Builds libtorch
63+
# help:
64+
.PHONY: torch
65+
torch: $(TORCH_TARGET)
66+
67+
# Checkout a specific version of Torch and update all of the torch submodules
68+
.PHONY: checkout_torch
69+
checkout_torch:
70+
cd pytorch && git checkout v$(PYTORCH_VERSION) && \
71+
git submodule foreach --recursive git reset --hard && \
72+
git submodule update --init --recursive
73+
74+
$(TORCH_BUILD) $(TORCH_INSTALL):
75+
mkdir -p $@
76+
77+
.PHONY: build_torch
78+
build_torch: $(TORCH_BUILD) $(TORCH_INSTALL) checkout_torch
79+
cd $< && \
80+
cmake -DCMAKE_INSTALL_PREFIX=$(TORCH_INSTALL) $(TORCH_CMAKE_OPTIONS) ../../pytorch && \
81+
make install -j 6
82+
83+
$(TORCH_TARGET): build_torch
84+
cd install && zip -r ../$@ libtorch
85+
86+
.PHONY: clean_torch
87+
clean_torch:
88+
rm -rf $(TORCH_BUILD) $(TORCH_TARGET) $(TORCH_INSTALL)
89+

‎NOTICE.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
====================
2+
Third-Party Licenses
3+
====================
4+
pyTorch and Torchlib are licensed under a modified BSD clause and can be found here:
5+
https://github.com/pytorch/pytorch/blob/main/LICENSE

‎README.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# ML Library Builder
2+
3+
## Description
4+
Many popular machine-learning packages have C++ libraries associated with them.
5+
Like with any compiled library, many options exist to modify their
6+
functionality. This repository was originally intended to build libtorch for use
7+
on Mac OSX platforms using ARM64 architectures (e.g. Apple Silicon M1). While
8+
targeted primarily for users and developers of SmartSim, we hope that it will
9+
be useful more broadly. Note that because of this, we are likely only going
10+
to upgrade the backend versions as we need to and will likely not do a new
11+
release for every version.
12+
13+
Future work includes building other variants of the libraries as needed or
14+
including support for other backends like TensorFlow and ONNX.
15+
16+
## Requirements
17+
To use this library, please ensure that the following requirements are fulfilled:
18+
19+
- Mac OSX (Intel or Apple Silicon)
20+
- Xcode version >=12.2
21+
- Make >=4.3
22+
- Cmake >=3.24
23+
- Python 3.10
24+
25+
Note: lower versions than those might also work
26+
27+
## Instructions
28+
Briefly, the following should be sufficient to clone and build from this repository.
29+
30+
```
31+
git clone --recursive https://github.com/CrayLabs/ml_lib_builder.git
32+
cd ml_lib_builder
33+
pip install -r pytorch/requirements.txt
34+
make torch
35+
```
36+
37+
This will result in a zipped tarfile containing all the shared libraries that
38+
comprise libtorch.

‎pytorch‎

Submodule pytorch added at e9ebda2

0 commit comments

Comments
 (0)