Skip to content

Commit 7c2ded8

Browse files
benelogclaude
andcommitted
docs: add ai-standard note and reorganize note hierarchy
Add new note on HuggingFace open model compatibility standards (safetensors, config conventions). Reorganize Children/Related sections across multiple notes for better hierarchy. Fix heading levels and formatting in architecture, network, hard-disk, and cqrs notes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 0673e2f commit 7c2ded8

11 files changed

Lines changed: 117 additions & 34 deletions

content/ai-standard.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# 허깅페이스 오픈 모델의 호환성을 가능하게 하는 표준들
2+
3+
크게 **모델 가중치 포맷**, **모델 아키텍처 규약**, **메타데이터/설정 표준** 세 층위가 있다.
4+
5+
## 1. 모델 가중치 저장 포맷
6+
7+
| 포맷 | 특징 |
8+
|------|------|
9+
| **Safetensors** | 현재 사실상 표준. 메모리 매핑 가능, 보안 취약점 없음 (pickle 미사용) |
10+
| **GGUF** | llama.cpp 생태계 표준. 양자화 정보를 파일 자체에 포함. CPU/엣지 추론에 최적화 |
11+
| **PyTorch (.bin/.pt)** | pickle 기반의 레거시 포맷. 여전히 많이 쓰이지만 보안 우려 |
12+
13+
**Safetensors**가 핵심이다. 텐서의 이름(key)과 shape, dtype, 바이너리 데이터를 단순한 헤더+바이너리 구조로 저장하므로, 어떤 프레임워크든 쉽게 읽을 수 있다.
14+
15+
## 2. 모델 아키텍처 규약 — Transformers 라이브러리의 역할
16+
17+
허깅페이스 `transformers` 라이브러리가 사실상의 표준 역할을 한다.
18+
19+
- **`config.json`** — 모델 구조 정의 (`model_type`, `hidden_size`, `num_attention_heads` 등)
20+
- **`tokenizer.json` / `tokenizer_config.json`** — 토크나이저 정의
21+
- **`generation_config.json`** — 생성 파라미터
22+
23+
```json
24+
// config.json 예시
25+
{
26+
"model_type": "llama",
27+
"hidden_size": 4096,
28+
"num_attention_heads": 32,
29+
"num_hidden_layers": 32,
30+
"vocab_size": 32000
31+
}
32+
```
33+
34+
`model_type``"llama"`이면, 라이브러리가 `LlamaForCausalLM` 클래스를 자동으로 선택하고 가중치를 로드한다. 그래서 Llama 아키텍처 기반 모델은 누가 학습했든 동일한 코드로 실행된다.
35+
36+
## 3. 추론 엔진 간 호환성
37+
38+
```
39+
HF 모델 (safetensors + config.json)
40+
41+
├─→ transformers (Python)
42+
├─→ vLLM (고성능 서빙)
43+
├─→ TGI (HF 자체 서빙)
44+
├─→ Ollama / llama.cpp (GGUF 변환 후)
45+
└─→ ONNX Runtime (ONNX 변환 후)
46+
```
47+
48+
이것이 가능한 이유는 **아키텍처가 표준화**되어 있기 때문이다. 대부분의 오픈 모델이 소수의 아키텍처(Llama, Mistral, Qwen, Gemma 등)를 따르고, 각 추론 엔진이 이 아키텍처들을 구현해 둔다.
49+
50+
## 핵심 정리
51+
52+
공식적인 "표준 기구"가 정한 스펙이라기보다는:
53+
54+
1. **Safetensors** — 텐서를 저장하는 단순하고 안전한 바이너리 포맷
55+
2. **HF transformers의 config 규약**`config.json``model_type`으로 아키텍처를 식별
56+
3. **소수 아키텍처로의 수렴** — 대부분의 모델이 Llama/Mistral 계열 아키텍처를 채택
57+
58+
이 세 가지가 맞물려서 "누가 학습했든 다운받으면 바로 돌릴 수 있는" 생태계가 만들어진 것이다. 사실상의 표준(de facto standard)이지, ISO 같은 공식 표준은 아니다.

content/architecture.md

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
- <https://www.educative.io/prepare-for-interview/system-design>
44
- [\[마틴 파울러](https://www.youtube.com/watch?v=4E1BHTvhB7Y) 소프트웨어 아키텍쳐의 중요성 (한글 자막)\]
55

6-
# 용어
6+
## 용어
77

88
UML에서 정의한 컴포넌트의 정의 (참고자료 : UML 1.3 Specification, OMG)
99

@@ -17,7 +17,7 @@ which represent services implemented by the elements resident in the
1717
Component. These services define behavior offered by instances of the
1818
Component as a whole to other client Component instances.
1919

20-
# MSA
20+
## MSA
2121

2222
- <https://medium.com/@madhukaudantha/microservice-architecture-and-design-patterns-for-microservices-e0e5013fd58a>
2323
- <https://particular.net/blog/goodbye-microservices-hello-right-sized-services>
@@ -34,22 +34,22 @@ Component as a whole to other client Component instances.
3434
- <https://allegro.tech/2016/03/Managing-Frontend-in-the-microservices-architecture.html>
3535
- <https://blog.bernd-ruecker.com/3-common-pitfalls-in-microservice-integration-and-how-to-avoid-them-3f27a442cd07>
3636

37-
# Transactional out box ( Store and forward)
37+
## Transactional out box ( Store and forward)
3838

3939
- [MSA에서 메시징 트랜잭션 처리하기](https://www.popit.kr/msa%ec%97%90%ec%84%9c-%eb%a9%94%ec%8b%9c%ec%a7%95-%ed%8a%b8%eb%9e%9c%ec%9e%ad%ec%85%98-%ec%b2%98%eb%a6%ac%ed%95%98%ea%b8%b0/)
4040

41-
# Events
41+
## Events
4242

4343
- <https://www.infoq.com/articles/events-workflow-automation>
4444

45-
# TCC
45+
## TCC
4646

4747
- <https://www.codota.com/code/java/methods/org.bytesoft.compensable.Compensable/%3Cinit%3E>
4848
- <https://github.com/liuyangming/ByteTCC>
4949
- <https://github.com/apache/servicecomb-pack>
5050
- <https://github.com/apache/servicecomb-pack/blob/133a2fa7247b2ba00a380762a38369849390560e/integration-tests/pack-tests/src/test/java/org/apache/servicecomb/pack/integration/tests/resttemplate/GreetingService.java#L38>
5151

52-
# 아키텍트의 역할
52+
## 아키텍트의 역할
5353

5454
- Meet the achitect <http://www.youtube.com/watch?v=Df4RnVl03Es&eurl=http%3A%2F%2Fbobbyryu%2Eblogspot%2Ecom%2F2007%2F05%2Fit%2Ehtml>
5555
- [아키텍처 건전성](http://www.ibm.com/developerworks/kr/library/j-ap07107/index.html)
@@ -71,27 +71,27 @@ Component as a whole to other client Component instances.
7171

7272
- Simplex architecture :
7373

74-
# Architecture pattern
74+
## Architecture pattern
7575

7676
- <https://docs.microsoft.com/ko-kr/azure/architecture/patterns/anti-corruption-layer>
7777
- <http://en.wikipedia.org/wiki/Architectural_pattern_%28computer_science%29>
7878

7979
- 4+1 View : <http://www3.software.ibm.com/ibmdl/pub/software/rational/web/whitepapers/2003/Pbk4p1.pdf>
8080

81-
# Module
81+
## Module
8282

8383
- [멀티모듈 설계 이야기 with Spring, Gradle](http://woowabros.github.io/study/2019/07/01/multi-module.html)
8484

85-
# 오픈소스 아키텍처 : <http://www.aosabook.org/en/>
85+
## 오픈소스 아키텍처 : <http://www.aosabook.org/en/>
8686

8787
## Children
8888
- [[architecture-cases]]
89-
89+
- [[design-pattern]]
90+
- [[scalability]]
91+
- [[cqrs]]
9092
## Related
9193
- [[platform-choice]]
9294
- [[design]]
93-
- [[design-pattern]]
94-
- [[domain-driven-design]]
9595
- [[msa-solutions]]
96-
- [[scalability]]
96+
9797
- [[spring-cloud]]

content/cqrs.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,4 @@
2222
- <https://discuss.elastic.co/t/elasticsearch-as-eventstore-for-cqrs/13252/2>
2323
- <http://blog.2mas.xyz/cqrs-the-simple-way-with-eventstore-and-elasticsearch-integrating-elasticsearch/>
2424

25-
[Online Event Processing - Achieving consistency where distributed
26-
transactions have failed](https://queue.acm.org/detail.cfm?id=3321612)
25+
[Online Event Processing - Achieving consistency where distributed transactions have failed](https://queue.acm.org/detail.cfm?id=3321612)

content/documentation-tools.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,12 @@ surge.sh로 배포
119119
- Pattern
120120
121121
## Children
122+
- [[ascii-tools]]
122123
- [[asciidoc]]
124+
- [[diagram]]
123125
- [[docbook]]
124126
- [[documentation-cases]]
127+
- [[excel]]
125128
- [[markdown]]
129+
- [[pdf]]
130+
- [[wiki]]

content/hard-disk.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Failure Trends in a large disk drive population :
22

3-
# SMART
3+
## SMART
44

55
- drive’s self monitoring facility, <http://en.wikipedia.org/wiki/S.M.A.R.T>
66
- scan errors, reallocation counts, offline reallocation counts, and probational counts

content/java.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
## Children
2+
- [[android]]
3+
- [[eclipse]]
24
- [[java-build]]
35
- [[effective-java]]
46
- [[java-annotation]]
@@ -29,10 +31,15 @@
2931
- [[java-thread-dump]]
3032
- [[java-tips]]
3133
- [[java-web-framework]]
34+
- [[jca]]
3235
- [[jcip-annotation]]
3336
- [[jdbc]]
3437
- [[jpa]]
38+
- [[jpms]]
39+
- [[jshell]]
3540
- [[jvm]]
41+
- [[memory]]
3642
- [[openjdk]]
3743
- [[osgi]]
3844
- [[quartz]]
45+
- [[spring]]

content/linux.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
## Children
22
- [[docker]]
3+
- [[hard-disk]]
34
- [[linux-desktop]]
45
- [[linux-dist]]
56
- [[linux-kernel]]
@@ -9,4 +10,8 @@
910
- [[linux-shell]]
1011
- [[linux-tips]]
1112
- [[ssh]]
13+
- [[vi]]
14+
- [[virtualization]]
15+
- [[web-shell]]
1216
- [[wget]]
17+
- [[windows-command]]

content/network.md

Lines changed: 15 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,27 @@
1-
# Class
1+
## Class
22

3-
Class A 0~127
3+
* Class A : 0~127
4+
* Class B : 128~191
5+
* Class C : 192~233
6+
* Class D : 234 이상
47

5-
Class B 128~191
8+
### 사설네트워크 영역 (RFC 1597)
9+
* 10.0.0.0 ~ 10.255.255.255
10+
* 172.16.0.0 ~ 172.31.255.255
11+
* 192.168.0.0 ~ 192.168.255.255
612

7-
Class C 192~233
8-
9-
Class D 234 이상
10-
11-
사설네트워크 영역 (RFC 1597)
12-
13-
10.0.0.0 ~ 10.255.255.255
14-
15-
172.16.0.0 ~ 172.31.255.255
16-
17-
192.168.0.0 ~ 192.168.255.255
18-
19-
# CIDR
13+
## CIDR
2014

2115
<http://en.wikipedia.org/wiki/CIDR>
2216

23-
# Switch
17+
## Switch
2418

2519
[\[네트워크 이야기](http://mkhouse.info/tt/58) L4 스위치\]
2620

27-
## Related
21+
## Children
2822
- [[linux-network]]
2923
- [[tcp-ip]]
3024
- [[tshark]]
25+
- [[ftp]]
26+
- [[http]]
27+
- [[smtp]]

content/project-management.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,7 @@ IT조직들](http://www.zdnet.co.kr/itbiz/column/anchor/yscho/0,39044217,3917247
3232
# 추정
3333

3434
- <https://www.brightball.com/articles/story-points-are-pointless-measure-queues>
35+
36+
## Children
37+
- [[issue-tracker]]
38+
- [[planning]]

content/sw-engineering.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,3 +144,6 @@ Maintenance Programmers. Englewood Cliffs, NH: Prentice-Hall
144144
필름기법: FilmStrip
145145

146146
StateChart Diagram
147+
148+
## Children
149+
- [[project-management]]

0 commit comments

Comments
 (0)