From ff3c4cce43d30dfa827a3028bc7510837298b73c Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Wed, 17 Jun 2026 23:58:31 +0200 Subject: [PATCH 01/11] Add Core AI provider package --- README.md | 52 +- bun.lock | 18 + packages/core-ai/CoreAI.podspec | 21 + packages/core-ai/README.md | 70 ++ packages/core-ai/ios/CoreAI.mm | 231 +++++ packages/core-ai/ios/CoreAIError.swift | 49 + packages/core-ai/ios/CoreAIImpl.swift | 995 ++++++++++++++++++++ packages/core-ai/package.json | 91 ++ packages/core-ai/src/NativeCoreAI.ts | 136 +++ packages/core-ai/src/ai-sdk.ts | 394 ++++++++ packages/core-ai/src/catalog.ts | 216 +++++ packages/core-ai/src/core.ts | 371 ++++++++ packages/core-ai/src/index.ts | 37 + packages/core-ai/src/types.ts | 202 ++++ packages/core-ai/tsconfig.build.json | 9 + packages/core-ai/tsconfig.json | 7 + website/src/docs/_meta.json | 7 + website/src/docs/core-ai/_meta.json | 5 + website/src/docs/core-ai/api.md | 79 ++ website/src/docs/core-ai/getting-started.md | 87 ++ website/src/docs/core-ai/models.md | 35 + website/src/docs/index.md | 11 + 22 files changed, 3118 insertions(+), 5 deletions(-) create mode 100644 packages/core-ai/CoreAI.podspec create mode 100644 packages/core-ai/README.md create mode 100644 packages/core-ai/ios/CoreAI.mm create mode 100644 packages/core-ai/ios/CoreAIError.swift create mode 100644 packages/core-ai/ios/CoreAIImpl.swift create mode 100644 packages/core-ai/package.json create mode 100644 packages/core-ai/src/NativeCoreAI.ts create mode 100644 packages/core-ai/src/ai-sdk.ts create mode 100644 packages/core-ai/src/catalog.ts create mode 100644 packages/core-ai/src/core.ts create mode 100644 packages/core-ai/src/index.ts create mode 100644 packages/core-ai/src/types.ts create mode 100644 packages/core-ai/tsconfig.build.json create mode 100644 packages/core-ai/tsconfig.json create mode 100644 website/src/docs/core-ai/_meta.json create mode 100644 website/src/docs/core-ai/api.md create mode 100644 website/src/docs/core-ai/getting-started.md create mode 100644 website/src/docs/core-ai/models.md diff --git a/README.md b/README.md index a6236474..c4bf91ad 100644 --- a/README.md +++ b/README.md @@ -45,11 +45,12 @@ receiving the current app's telemetry stream. ## Available Providers -| Provider | Built-in | Platforms | Runtime | Description | -| --------------- | -------- | ------------ | ------------------------------------------------------------------- | ---------------------------------------------------------- | -| [Apple](#apple) | ✅ Yes | iOS | [Apple](https://developer.apple.com/documentation/FoundationModels) | Apple Foundation Models, embeddings, transcription, speech | -| [Llama](#llama) | ❌ No | iOS, Android | [llama.rn](https://github.com/mybigday/llama.rn) | Run GGUF models via llama.rn | -| [MLC](#mlc) | ❌ No | iOS, Android | [MLC LLM](https://github.com/mlc-ai/mlc-llm) | Run open-source LLMs via MLC runtime | +| Provider | Built-in | Platforms | Runtime | Description | +| ------------------- | -------- | ------------ | ------------------------------------------------------------------- | ---------------------------------------------------------- | +| [Apple](#apple) | ✅ Yes | iOS | [Apple](https://developer.apple.com/documentation/FoundationModels) | Apple Foundation Models, embeddings, transcription, speech | +| [Core AI](#core-ai) | ❌ No | iOS | [Core AI](https://developer.apple.com/core-ai/) | Run app-provided Core AI model bundles | +| [Llama](#llama) | ❌ No | iOS, Android | [llama.rn](https://github.com/mybigday/llama.rn) | Run GGUF models via llama.rn | +| [MLC](#mlc) | ❌ No | iOS, Android | [MLC LLM](https://github.com/mlc-ai/mlc-llm) | Run open-source LLMs via MLC runtime | --- @@ -119,6 +120,47 @@ See the [Apple documentation](https://react-native-ai.dev/docs/apple/getting-sta --- +### Core AI + +Run app-provided Apple Core AI model bundles exported from [`apple/coreai-models`](https://github.com/apple/coreai-models). This provider is for custom `.aimodel` assets, not Apple system models. + +- **Language Sessions** - Core AI language models with persistent native sessions +- **AI SDK Adapters** - Language, embeddings, image generation, and transcription where AI SDK has matching interfaces +- **Native-only APIs** - Segmentation, object detection, depth, super-resolution, model inspection, specialization, and raw `.aimodel` functions +- **Catalog Metadata** - Starter model metadata from Apple’s registry, including iOS/macOS support + +#### Installation + +```bash +npm install @react-native-ai/core-ai +``` + +Core AI also requires adding `https://github.com/apple/coreai-models` as a Swift Package dependency in the host iOS app and linking the products you use, such as `CoreAILM`, `CoreAIDiffusion`, `CoreAISegmentation`, and `CoreAIObjectDetection`. + +React Native installs the package through CocoaPods, but the Apple helpers are Swift Package products. Expo apps should use a config plugin to patch the generated Xcode project; bare React Native apps should add the package in Xcode or through a predictable project patch. + +#### Usage + +```typescript +import { coreAI } from '@react-native-ai/core-ai' +import { generateText } from 'ai' + +const model = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +const { text } = await generateText({ + model, + prompt: 'Explain Core AI', +}) +``` + +See the [Core AI documentation](https://react-native-ai.dev/docs/core-ai/getting-started) for setup, starter models, and direct native APIs. + +--- + ### Llama Run any GGUF model on-device using [llama.rn](https://github.com/mybigday/llama.rn). **Requires download** - models are downloaded from HuggingFace. diff --git a/bun.lock b/bun.lock index 97e35c71..57480cfe 100644 --- a/bun.lock +++ b/bun.lock @@ -100,6 +100,22 @@ "react-native": ">=0.76.0", }, }, + "packages/core-ai": { + "name": "@react-native-ai/core-ai", + "version": "0.12.0", + "dependencies": { + "@ai-sdk/provider": "^3.0.5", + "@ai-sdk/provider-utils": "^4.0.1", + "zod": "^4.2.1", + }, + "peerDependencies": { + "expo": "*", + "react-native": ">=0.76.0", + }, + "optionalPeers": [ + "expo", + ], + }, "packages/dev-tools": { "name": "@react-native-ai/dev-tools", "version": "0.12.0", @@ -756,6 +772,8 @@ "@react-native-ai/apple": ["@react-native-ai/apple@workspace:packages/apple-llm"], + "@react-native-ai/core-ai": ["@react-native-ai/core-ai@workspace:packages/core-ai"], + "@react-native-ai/dev-tools": ["@react-native-ai/dev-tools@workspace:packages/dev-tools"], "@react-native-ai/example": ["@react-native-ai/example@workspace:apps/expo-example"], diff --git a/packages/core-ai/CoreAI.podspec b/packages/core-ai/CoreAI.podspec new file mode 100644 index 00000000..20b78f74 --- /dev/null +++ b/packages/core-ai/CoreAI.podspec @@ -0,0 +1,21 @@ +require "json" + +package = JSON.parse(File.read(File.join(__dir__, "package.json"))) + +Pod::Spec.new do |s| + s.name = "ReactNativeAICoreAI" + + s.version = package["version"] + s.summary = package["description"] + s.homepage = package["homepage"] + s.license = package["license"] + s.authors = package["author"] + + s.platforms = { :ios => min_ios_version_supported } + s.source = { :git => "https://github.com/callstackincubator/ai.git", :tag => "#{s.version}" } + + s.source_files = "ios/**/*.{h,m,mm,swift}" + s.exclude_files = "ios/Tests/**/*" + + install_modules_dependencies(s) +end diff --git a/packages/core-ai/README.md b/packages/core-ai/README.md new file mode 100644 index 00000000..6de5120b --- /dev/null +++ b/packages/core-ai/README.md @@ -0,0 +1,70 @@ +# @react-native-ai/core-ai + +Apple Core AI provider for React Native AI. + +This package is for app-provided Core AI model bundles exported from +[`apple/coreai-models`](https://github.com/apple/coreai-models). It is separate +from `@react-native-ai/apple`, which wraps Apple system capabilities such as +Foundation Models, NLContextualEmbedding, SpeechAnalyzer, and AVSpeechSynthesizer. + +## API Layers + +Core AI exposes two layers: + +- AI SDK adapters where AI SDK has a matching model interface: language models, + image generation, text embeddings, and transcription. +- Direct native APIs for Core AI-specific capabilities: model loading, + persistent language sessions, model inspection, specialization, segmentation, + object detection, depth, super-resolution, and raw `.aimodel` functions. + +The first native wrappers cover language sessions, diffusion image generation, +segmentation, and object detection. Embeddings, transcription, depth, +super-resolution, classification, reranking, speech generation, video +generation, and arbitrary raw tensor calls stay explicit: the JS APIs exist +where useful, but they reject with a setup/unsupported error until a real +task-specific Core AI runner is wired. + +## Native Setup + +Add `https://github.com/apple/coreai-models` as a Swift Package dependency in +the host iOS app and link the products used by your wrappers: + +- `CoreAILM` for language models and sessions +- `CoreAIDiffusion` for image generation +- `CoreAISegmentation` for segmentation +- `CoreAIObjectDetection` for object detection + +React Native installs this package through CocoaPods, but Apple's helpers are +Swift Package products. The app target still needs those SPM products linked. +For Expo, use a config plugin to patch the generated Xcode project. For bare +React Native, add the Swift Package in Xcode or with a project patching script. + +## Usage + +```ts +import { coreAI } from '@react-native-ai/core-ai' +import { streamText } from 'ai' + +const model = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +const result = streamText({ + model, + prompt: 'Explain Core AI in one sentence.', +}) +``` + +For persistent native sessions: + +```ts +await model.prepare() + +const session = await model.createSession({ + instructions: 'Answer tersely.', +}) + +const response = await session.respond('What is Qwen3?') +``` diff --git a/packages/core-ai/ios/CoreAI.mm b/packages/core-ai/ios/CoreAI.mm new file mode 100644 index 00000000..1a228c2e --- /dev/null +++ b/packages/core-ai/ios/CoreAI.mm @@ -0,0 +1,231 @@ +// +// CoreAI.mm +// ReactNativeAICoreAI +// + +#if __has_include("ReactNativeAICoreAI/ReactNativeAICoreAI-Swift.h") +#import "ReactNativeAICoreAI/ReactNativeAICoreAI-Swift.h" +#else +#import "ReactNativeAICoreAI-Swift.h" +#endif + +#import +#import +#import + +#import + +@interface ReactNativeAICoreAI : NativeCoreAISpecBase +@property (strong, nonatomic) CoreAIImpl *coreAI; +@end + +using namespace facebook; +using namespace JS::NativeCoreAI; + +static NSDictionary *CoreAIConfigToDictionary(NativeCoreAIModelConfig &config) { + NSMutableDictionary *dict = [NSMutableDictionary new]; + dict[@"id"] = config.id_(); + dict[@"sourceType"] = config.sourceType(); + if (config.sourceUri()) { + dict[@"sourceUri"] = config.sourceUri(); + } + if (config.bundleName()) { + dict[@"bundleName"] = config.bundleName(); + } + if (config.bundleExtension()) { + dict[@"bundleExtension"] = config.bundleExtension(); + } + if (config.bundleSubdirectory()) { + dict[@"bundleSubdirectory"] = config.bundleSubdirectory(); + } + if (config.task()) { + dict[@"task"] = config.task(); + } + if (config.family()) { + dict[@"family"] = config.family(); + } + if (config.variant()) { + dict[@"variant"] = config.variant(); + } + return dict; +} + +@implementation ReactNativeAICoreAI + +@synthesize callInvoker; + +- (instancetype)init { + self = [super init]; + if (self) { + _coreAI = [CoreAIImpl new]; + } + return self; +} + ++ (NSString *)moduleName { + return @"NativeCoreAI"; +} + +- (std::shared_ptr)getTurboModule:(const react::ObjCTurboModule::InitParams &)params { + return std::make_shared(params); +} + +- (void)getCapabilities:(nonnull RCTPromiseResolveBlock)resolve reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI getCapabilities:resolve reject:reject]; +} + +- (void)inspectModel:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI inspectModel:CoreAIConfigToDictionary(config) resolve:resolve reject:reject]; +} + +- (void)loadModel:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI loadModel:CoreAIConfigToDictionary(config) options:options resolve:resolve reject:reject]; +} + +- (void)unloadModel:(nonnull NSString *)modelHandle + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI unloadModel:modelHandle resolve:resolve reject:reject]; +} + +- (void)removeModel:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI removeModel:CoreAIConfigToDictionary(config) resolve:resolve reject:reject]; +} + +- (void)specializeModel:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI specializeModel:CoreAIConfigToDictionary(config) options:options resolve:resolve reject:reject]; +} + +- (void)createLanguageSession:(nonnull NSString *)modelHandle + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI createLanguageSession:modelHandle options:options resolve:resolve reject:reject]; +} + +- (void)releaseLanguageSession:(nonnull NSString *)sessionHandle + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI releaseLanguageSession:sessionHandle resolve:resolve reject:reject]; +} + +- (void)respondToLanguageSession:(nonnull NSString *)sessionHandle + prompt:(nonnull NSString *)prompt + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI respondToLanguageSession:sessionHandle prompt:prompt options:options resolve:resolve reject:reject]; +} + +- (void)streamLanguageSession:(nonnull NSString *)streamId + sessionHandle:(nonnull NSString *)sessionHandle + prompt:(nonnull NSString *)prompt + options:(nonnull NSDictionary *)options { + [_coreAI streamLanguageSession:streamId + sessionHandle:sessionHandle + prompt:prompt + options:options + onUpdate:^(NSString *streamId, NSString *content) { + [self emitOnStreamUpdate:@{@"streamId": streamId, @"content": content}]; + } + onComplete:^(NSString *streamId) { + [self emitOnStreamComplete:@{@"streamId": streamId}]; + } + onError:^(NSString *streamId, NSString *code, NSString *error) { + NSMutableDictionary *payload = [@{@"streamId": streamId, @"error": error} mutableCopy]; + if (code.length > 0) { + payload[@"code"] = code; + } + [self emitOnStreamError:payload]; + }]; +} + +- (void)generateText:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + messages:(nonnull NSArray *)messages + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI generateText:CoreAIConfigToDictionary(config) messages:messages options:options resolve:resolve reject:reject]; +} + +- (void)streamText:(nonnull NSString *)streamId + config:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + messages:(nonnull NSArray *)messages + options:(nonnull NSDictionary *)options { + [_coreAI streamText:streamId + config:CoreAIConfigToDictionary(config) + messages:messages + options:options + onUpdate:^(NSString *streamId, NSString *content) { + [self emitOnStreamUpdate:@{@"streamId": streamId, @"content": content}]; + } + onComplete:^(NSString *streamId) { + [self emitOnStreamComplete:@{@"streamId": streamId}]; + } + onError:^(NSString *streamId, NSString *code, NSString *error) { + NSMutableDictionary *payload = [@{@"streamId": streamId, @"error": error} mutableCopy]; + if (code.length > 0) { + payload[@"code"] = code; + } + [self emitOnStreamError:payload]; + }]; +} + +- (void)cancelStream:(nonnull NSString *)streamId { + [_coreAI cancelStream:streamId]; +} + +- (void)embed:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + values:(nonnull NSArray *)values + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI embed:CoreAIConfigToDictionary(config) values:values options:options resolve:resolve reject:reject]; +} + +- (void)transcribe:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + audioBase64:(nonnull NSString *)audioBase64 + mediaType:(nonnull NSString *)mediaType + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI transcribe:CoreAIConfigToDictionary(config) audioBase64:audioBase64 mediaType:mediaType options:options resolve:resolve reject:reject]; +} + +- (void)generateImage:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + prompt:(nonnull NSString *)prompt + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI generateImage:CoreAIConfigToDictionary(config) prompt:prompt options:options resolve:resolve reject:reject]; +} + +- (void)runTask:(nonnull NSString *)task + config:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + input:(nonnull NSDictionary *)input + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI runTask:task config:CoreAIConfigToDictionary(config) input:input options:options resolve:resolve reject:reject]; +} + +- (void)runRawFunction:(nonnull NSString *)modelHandle + functionName:(nonnull NSString *)functionName + inputs:(nonnull NSDictionary *)inputs + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI runRawFunction:modelHandle functionName:functionName inputs:inputs options:options resolve:resolve reject:reject]; +} + +@end diff --git a/packages/core-ai/ios/CoreAIError.swift b/packages/core-ai/ios/CoreAIError.swift new file mode 100644 index 00000000..9dd70fd5 --- /dev/null +++ b/packages/core-ai/ios/CoreAIError.swift @@ -0,0 +1,49 @@ +import Foundation + +enum CoreAIError: LocalizedError { + case unsupportedOS + case missingSource + case missingSwiftPackage(String) + case modelNotLoaded(String) + case sessionNotFound(String) + case unsupportedTask(String) + case invalidInput(String) + + var errorDescription: String? { + switch self { + case .unsupportedOS: + return "Core AI requires iOS 27 or macOS 27 or newer." + case .missingSource: + return "Core AI model source is missing. Provide a file URI or bundle resource." + case .missingSwiftPackage(let product): + return "Missing Apple Core AI Swift Package product: \(product). Add https://github.com/apple/coreai-models to the app target and link \(product)." + case .modelNotLoaded(let handle): + return "Core AI model is not loaded: \(handle)." + case .sessionNotFound(let handle): + return "Core AI language session was not found: \(handle)." + case .unsupportedTask(let task): + return "Core AI task is not implemented for this wrapper yet: \(task)." + case .invalidInput(let message): + return message + } + } + + var code: String { + switch self { + case .unsupportedOS: + return "CORE_AI_UNSUPPORTED_OS" + case .missingSource: + return "CORE_AI_MISSING_SOURCE" + case .missingSwiftPackage: + return "CORE_AI_MISSING_SPM_PRODUCT" + case .modelNotLoaded: + return "CORE_AI_MODEL_NOT_LOADED" + case .sessionNotFound: + return "CORE_AI_SESSION_NOT_FOUND" + case .unsupportedTask: + return "CORE_AI_UNSUPPORTED_TASK" + case .invalidInput: + return "CORE_AI_INVALID_INPUT" + } + } +} diff --git a/packages/core-ai/ios/CoreAIImpl.swift b/packages/core-ai/ios/CoreAIImpl.swift new file mode 100644 index 00000000..0fdbbbc0 --- /dev/null +++ b/packages/core-ai/ios/CoreAIImpl.swift @@ -0,0 +1,995 @@ +import CoreGraphics +import Foundation +import ImageIO + +#if canImport(FoundationModels) +import FoundationModels +#endif + +#if canImport(CoreAI) +import CoreAI +#endif + +#if canImport(CoreAIDiffusionPipeline) +import CoreAIDiffusionPipeline +#endif + +#if canImport(CoreAIImageSegmenter) +import CoreAIImageSegmenter +#endif + +#if canImport(CoreAILanguageModels) +import CoreAILanguageModels +#endif + +#if canImport(CoreAIObjectDetector) +import CoreAIObjectDetector +#endif + +@objc +public class CoreAIImpl: NSObject { + private var loadedModels: [String: Any] = [:] + private var languageSessions: [String: Any] = [:] + private var streamTasks: [String: Task] = [:] + + @objc + public func getCapabilities( + _ resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + var missingProducts: [String] = [] + +#if !canImport(CoreAILanguageModels) + missingProducts.append("CoreAILM") +#endif +#if !canImport(CoreAIDiffusionPipeline) + missingProducts.append("CoreAIDiffusion") +#endif +#if !canImport(CoreAIImageSegmenter) + missingProducts.append("CoreAISegmentation") +#endif +#if !canImport(CoreAIObjectDetector) + missingProducts.append("CoreAIObjectDetection") +#endif + + resolve([ + "isCoreAIRuntimeAvailable": Self.isCoreAIRuntimeAvailable(), + "isCoreAILMAvailable": Self.isCoreAILMAvailable(), + "isCoreAIDiffusionAvailable": Self.isCoreAIDiffusionAvailable(), + "isCoreAISegmentationAvailable": Self.isCoreAISegmentationAvailable(), + "isCoreAIObjectDetectionAvailable": Self.isCoreAIObjectDetectionAvailable(), + "supportedPlatform": Self.isSupportedPlatform(), + "missingProducts": missingProducts + ]) + } + + @objc + public func inspectModel( + _ config: [String: Any], + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + do { + let sourceURL = try resolveSourceURL(config) + var isDirectory: ObjCBool = false + let exists = FileManager.default.fileExists(atPath: sourceURL.path, isDirectory: &isDirectory) + let attributes = try? FileManager.default.attributesOfItem(atPath: sourceURL.path) + let size = attributes?[.size] as? NSNumber + + resolve(modelInfo(config: config, metadata: [ + "exists": exists, + "isDirectory": isDirectory.boolValue, + "path": sourceURL.path + ], size: size)) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + + @objc + public func loadModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + let task = (config["task"] as? String) ?? "unknown" + + switch task { + case "language": + loadLanguageModel(config, resolve: resolve, reject: reject) + return + case "diffusion": + loadDiffusionModel(config, options: options, resolve: resolve, reject: reject) + return + case "segmentation": + loadSegmentationModel(config, options: options, resolve: resolve, reject: reject) + return + case "object-detection": + loadObjectDetectionModel(config, options: options, resolve: resolve, reject: reject) + return + default: + break + } + + let handle = makeHandle("model") + loadedModels[handle] = [ + "config": config, + "options": options ?? [:] + ] + resolve([ + "modelHandle": handle, + "info": modelInfo(config: config) + ]) + } + + @objc + public func unloadModel( + _ modelHandle: String, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + loadedModels.removeValue(forKey: modelHandle) + resolve(nil) + } + + @objc + public func removeModel( + _ config: [String: Any], + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + do { + let sourceURL = try resolveSourceURL(config) + try FileManager.default.removeItem(at: sourceURL) + resolve(nil) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + + @objc + public func specializeModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + resolve(modelInfo(config: config, metadata: [ + "specializationRequested": true, + "persistence": options?["persistence"] as? String ?? "default" + ])) + } + + @objc + public func createLanguageSession( + _ modelHandle: String, + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(FoundationModels) && canImport(CoreAILanguageModels) + if #available(iOS 27, macOS 27, *) { + guard let model = loadedModels[modelHandle] as? CoreAILanguageModel else { + let error = CoreAIError.modelNotLoaded(modelHandle) + reject(error.code, error.localizedDescription, error) + return + } + + let session = LanguageModelSession(model: model) + let sessionHandle = makeHandle("session") + languageSessions[sessionHandle] = session + resolve(sessionHandle) + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAILM"), reject: reject) +#endif + } + + @objc + public func releaseLanguageSession( + _ sessionHandle: String, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + languageSessions.removeValue(forKey: sessionHandle) + resolve(nil) + } + + @objc + public func respondToLanguageSession( + _ sessionHandle: String, + prompt: String, + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(FoundationModels) && canImport(CoreAILanguageModels) + if #available(iOS 27, macOS 27, *) { + guard let session = languageSessions[sessionHandle] as? LanguageModelSession else { + let error = CoreAIError.sessionNotFound(sessionHandle) + reject(error.code, error.localizedDescription, error) + return + } + + Task { + do { + let response = try await session.respond(to: prompt) + resolve([["type": "text", "text": String(describing: response)]]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAILM"), reject: reject) +#endif + } + + @objc + public func streamLanguageSession( + _ streamId: String, + sessionHandle: String, + prompt: String, + options: [String: Any]?, + onUpdate: @escaping (String, String) -> Void, + onComplete: @escaping (String) -> Void, + onError: @escaping (String, String, String) -> Void + ) { +#if canImport(FoundationModels) && canImport(CoreAILanguageModels) + if #available(iOS 27, macOS 27, *) { + guard let session = languageSessions[sessionHandle] as? LanguageModelSession else { + emitError(.sessionNotFound(sessionHandle), streamId: streamId, onError: onError) + return + } + + let task = Task { + do { + let responseStream = session.streamResponse(to: prompt) + for try await chunk in responseStream { + if Task.isCancelled { + return + } + onUpdate(streamId, String(describing: chunk.content)) + } + onComplete(streamId) + } catch { + if !Task.isCancelled { + onError(streamId, errorCode(error), error.localizedDescription) + } + } + self.streamTasks.removeValue(forKey: streamId) + } + streamTasks[streamId] = task + } else { + emitError(.unsupportedOS, streamId: streamId, onError: onError) + } +#else + emitError(.missingSwiftPackage("CoreAILM"), streamId: streamId, onError: onError) +#endif + } + + @objc + public func generateText( + _ config: [String: Any], + messages: [[String: Any]], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + loadLanguageModel(config, resolve: { loaded in + guard let loadedDict = loaded as? [String: Any], + let modelHandle = loadedDict["modelHandle"] as? String else { + let error = CoreAIError.invalidInput("Core AI language model did not return a handle.") + reject(error.code, error.localizedDescription, error) + return + } + + self.createLanguageSession(modelHandle, options: options, resolve: { session in + guard let sessionHandle = session as? String else { + let error = CoreAIError.invalidInput("Core AI language session did not return a handle.") + reject(error.code, error.localizedDescription, error) + return + } + + let prompt = self.lastUserPrompt(messages) + self.respondToLanguageSession(sessionHandle, prompt: prompt, options: options, resolve: resolve, reject: reject) + }, reject: reject) + }, reject: reject) + } + + @objc + public func streamText( + _ streamId: String, + config: [String: Any], + messages: [[String: Any]], + options: [String: Any]?, + onUpdate: @escaping (String, String) -> Void, + onComplete: @escaping (String) -> Void, + onError: @escaping (String, String, String) -> Void + ) { + loadLanguageModel(config, resolve: { loaded in + guard let loadedDict = loaded as? [String: Any], + let modelHandle = loadedDict["modelHandle"] as? String else { + self.emitError(.invalidInput("Core AI language model did not return a handle."), streamId: streamId, onError: onError) + return + } + + self.createLanguageSession(modelHandle, options: options, resolve: { session in + guard let sessionHandle = session as? String else { + self.emitError(.invalidInput("Core AI language session did not return a handle."), streamId: streamId, onError: onError) + return + } + + self.streamLanguageSession( + streamId, + sessionHandle: sessionHandle, + prompt: self.lastUserPrompt(messages), + options: options, + onUpdate: onUpdate, + onComplete: onComplete, + onError: onError + ) + }, reject: { code, message, _ in + onError(streamId, code, message) + }) + }, reject: { code, message, _ in + onError(streamId, code, message) + }) + } + + @objc + public func cancelStream(_ streamId: String) { + streamTasks[streamId]?.cancel() + streamTasks.removeValue(forKey: streamId) + } + + @objc + public func embed( + _ config: [String: Any], + values: [String], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + rejectCoreAI(.unsupportedTask("embedding"), reject: reject) + } + + @objc + public func transcribe( + _ config: [String: Any], + audioBase64: String, + mediaType: String, + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + rejectCoreAI(.unsupportedTask("asr"), reject: reject) + } + + @objc + public func generateImage( + _ config: [String: Any], + prompt: String, + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIDiffusionPipeline) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + Task { + do { + let pipeline = try await self.makeDiffusionPipeline(at: sourceURL, options: options) + let configuration = try self.makePipelineConfiguration(prompt: prompt, options: options) + let result = try await pipeline.generateImages(configuration: configuration) { _ in true } + let images = try result.images.map(Self.pngBase64) + resolve([ + "images": images, + "metadata": [ + "count": images.count, + "imageFormat": "image/png" + ] + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAIDiffusion"), reject: reject) +#endif + } + + @objc + public func runTask( + _ task: String, + config: [String: Any], + input: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + switch task { + case "segmentation": + runSegmentationTask(config: config, input: input, options: options, resolve: resolve, reject: reject) + case "object-detection": + runObjectDetectionTask(config: config, input: input, options: options, resolve: resolve, reject: reject) + default: + rejectCoreAI(.unsupportedTask(task), reject: reject) + } + } + + @objc + public func runRawFunction( + _ modelHandle: String, + functionName: String, + inputs: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + rejectCoreAI(.unsupportedTask("raw"), reject: reject) + } + + private func loadLanguageModel( + _ config: [String: Any], + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(FoundationModels) && canImport(CoreAILanguageModels) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + Task { + do { + let model = try await CoreAILanguageModel(resourcesAt: sourceURL) + let handle = self.makeHandle("model") + self.loadedModels[handle] = model + resolve([ + "modelHandle": handle, + "info": self.modelInfo(config: config) + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAILM"), reject: reject) +#endif + } + + private func loadDiffusionModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIDiffusionPipeline) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + Task { + do { + let pipeline = try await self.makeDiffusionPipeline(at: sourceURL, options: options) + let handle = self.makeHandle("model") + self.loadedModels[handle] = pipeline + resolve([ + "modelHandle": handle, + "info": self.modelInfo(config: config) + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAIDiffusion"), reject: reject) +#endif + } + + private func loadSegmentationModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIImageSegmenter) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + let parameters = makeSegmentationParameters(options: options) + Task { + do { + let segmenter = try await ImageSegmenter(resourcesAt: sourceURL.path, parameters: parameters) + let handle = self.makeHandle("model") + self.loadedModels[handle] = segmenter + resolve([ + "modelHandle": handle, + "info": self.modelInfo(config: config) + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAISegmentation"), reject: reject) +#endif + } + + private func loadObjectDetectionModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIObjectDetector) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + Task { + do { + let detector = try await ObjectDetector(resourcesAt: sourceURL.path) + let handle = self.makeHandle("model") + self.loadedModels[handle] = detector + resolve([ + "modelHandle": handle, + "info": self.modelInfo(config: config) + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAIObjectDetection"), reject: reject) +#endif + } + + private func runSegmentationTask( + config: [String: Any], + input: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIImageSegmenter) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + let image = try Self.loadCGImage(from: input) + let parameters = makeSegmentationParameters(options: options) + Task { + do { + let segmenter = try await ImageSegmenter(resourcesAt: sourceURL.path, parameters: parameters) + let response: SegmentationResponse + if let prompt = input["text"] as? String, !prompt.isEmpty { + response = try await segmenter.segment(image: image, prompt: prompt, parameters: parameters) + } else { + response = try await segmenter.segment( + image: image, + pointQuery: Self.makePointQuery(input: input), + parameters: parameters + ) + } + resolve([ + "task": "segmentation", + "output": Self.serializeSegmentationResponse(response), + "metadata": [ + "imageWidth": image.width, + "imageHeight": image.height + ] + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAISegmentation"), reject: reject) +#endif + } + + private func runObjectDetectionTask( + config: [String: Any], + input: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIObjectDetector) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + let image = try Self.loadCGImage(from: input) + let parameters = makeDetectionParameters(options: options) + Task { + do { + let detector = try await ObjectDetector(resourcesAt: sourceURL.path) + let detections = try await detector.detect(image: image, parameters: parameters) + resolve([ + "task": "object-detection", + "output": [ + "detections": detections.map(Self.serializeDetection) + ], + "metadata": [ + "imageWidth": image.width, + "imageHeight": image.height + ] + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAIObjectDetection"), reject: reject) +#endif + } + + private func resolveSourceURL(_ config: [String: Any]) throws -> URL { + if let sourceUri = config["sourceUri"] as? String { + if sourceUri.hasPrefix("file://") { + return URL(string: sourceUri) ?? URL(fileURLWithPath: sourceUri.replacingOccurrences(of: "file://", with: "")) + } + return URL(fileURLWithPath: sourceUri) + } + + if let bundleName = config["bundleName"] as? String { + let ext = config["bundleExtension"] as? String + let subdirectory = config["bundleSubdirectory"] as? String + if let url = Bundle.main.url(forResource: bundleName, withExtension: ext, subdirectory: subdirectory) { + return url + } + } + + throw CoreAIError.missingSource + } + + private func modelInfo( + config: [String: Any], + metadata: [String: Any] = [:], + size: NSNumber? = nil + ) -> [String: Any] { + var info: [String: Any] = [ + "id": config["id"] as? String ?? "unknown", + "family": config["family"] as? String ?? "", + "task": config["task"] as? String ?? "unknown", + "platforms": ["iOS", "macOS"], + "functions": [], + "metadata": metadata + ] + if let size { + info["modelSizeBytes"] = size + } + return info + } + + private func lastUserPrompt(_ messages: [[String: Any]]) -> String { + return messages.last(where: { ($0["role"] as? String) == "user" })?["content"] as? String ?? "" + } + + private func makeHandle(_ prefix: String) -> String { + return "\(prefix)-\(UUID().uuidString)" + } + + private func rejectCoreAI( + _ error: CoreAIError, + reject: @escaping (String, String, Error?) -> Void + ) { + reject(error.code, error.localizedDescription, error) + } + + private func emitError( + _ error: CoreAIError, + streamId: String, + onError: @escaping (String, String, String) -> Void + ) { + onError(streamId, error.code, error.localizedDescription) + } + + private static func isSupportedPlatform() -> Bool { + if #available(iOS 27, macOS 27, *) { + return true + } + return false + } + + private static func isCoreAIRuntimeAvailable() -> Bool { +#if canImport(CoreAI) + return isSupportedPlatform() +#else + return false +#endif + } + + private static func isCoreAILMAvailable() -> Bool { +#if canImport(CoreAILanguageModels) + return isSupportedPlatform() +#else + return false +#endif + } + + private static func isCoreAIDiffusionAvailable() -> Bool { +#if canImport(CoreAIDiffusionPipeline) + return isSupportedPlatform() +#else + return false +#endif + } + + private static func isCoreAISegmentationAvailable() -> Bool { +#if canImport(CoreAIImageSegmenter) + return isSupportedPlatform() +#else + return false +#endif + } + + private static func isCoreAIObjectDetectionAvailable() -> Bool { +#if canImport(CoreAIObjectDetector) + return isSupportedPlatform() +#else + return false +#endif + } +} + +#if canImport(CoreAIDiffusionPipeline) +extension CoreAIImpl { + private func makeDiffusionPipeline( + at sourceURL: URL, + options: [String: Any]? + ) async throws -> any DiffusionPipeline { + let descriptor = try PipelineDescriptor.resolve(at: sourceURL) + + switch descriptor.type { + case .stableDiffusion, .stableDiffusionXL, .none: + return try await StableDiffusionPipeline.load(from: sourceURL) + case .stableDiffusion3: + return try await SD3Pipeline(from: sourceURL) + case .flux2: + let mode = DecodeResolution(rawValue: options?["decodeResolution"] as? String ?? "auto") ?? .auto + return try await Flux2Pipeline(from: sourceURL, mode: mode) + } + } + + private func makePipelineConfiguration( + prompt: String, + options: [String: Any]? + ) throws -> PipelineConfiguration { + let scheduler = SchedulerType(rawValue: options?["schedulerType"] as? String ?? "") ?? .dpmSolverMultistep + let decodeResolution = DecodeResolution(rawValue: options?["decodeResolution"] as? String ?? "full") ?? .full + let seed = UInt32(options?["seed"] as? Double ?? 0) + let stepCount = Int(options?["stepCount"] as? Double ?? 50) + let guidanceScale = Float(options?["guidanceScale"] as? Double ?? 7.5) + + return PipelineConfiguration( + prompt: prompt, + negativePrompt: options?["negativePrompt"] as? String ?? "", + seed: seed, + stepCount: stepCount, + guidanceScale: guidanceScale, + schedulerType: scheduler, + strength: Float(options?["strength"] as? Double ?? 1.0), + decodeResolution: decodeResolution, + lazyModelLoading: options?["lazyModelLoading"] as? Bool ?? true + ) + } +} +#endif + +#if canImport(CoreAIImageSegmenter) +extension CoreAIImpl { + private func makeSegmentationParameters(options: [String: Any]?) -> SegmentationParameters { + var parameters = SegmentationParameters.default + if let maskThreshold = options?["maskThreshold"] as? Double { + parameters.maskThreshold = Float(maskThreshold) + } + if let maxSegments = options?["maxSegments"] as? Double { + parameters.maxSegments = Int(maxSegments) + } + if let tokenizerContextLength = options?["tokenizerContextLength"] as? Double { + parameters.tokenizerContextLength = Int(tokenizerContextLength) + } + return parameters + } + + private static func makePointQuery(input: [String: Any]) -> PointQuery { + guard let data = input["data"] as? [String: Any] else { + return PointQuery() + } + + if let queries = data["queries"] as? [[Any]] { + return PointQuery(queries: queries.map { query in + query.compactMap { item in + guard let point = item as? [String: Any] else { + return nil + } + return makePoint(point) + } + }) + } + + if let points = data["points"] as? [[String: Any]] { + return PointQuery(points: points.compactMap(makePoint)) + } + + if let box = data["box"] as? [String: Any], + let x = box["x"] as? Double, + let y = box["y"] as? Double, + let width = box["width"] as? Double, + let height = box["height"] as? Double { + return PointQuery(points: [ + PointQuery.Point(x: Float(x), y: Float(y), label: .boxTopLeft), + PointQuery.Point(x: Float(x + width), y: Float(y + height), label: .boxBottomRight) + ]) + } + + return PointQuery() + } + + private static func makePoint(_ point: [String: Any]) -> PointQuery.Point? { + guard let x = point["x"] as? Double, + let y = point["y"] as? Double else { + return nil + } + + let labelValue = Int32(point["label"] as? Double ?? 1) + let label = PointQuery.Label(rawValue: labelValue) ?? .foreground + return PointQuery.Point(x: Float(x), y: Float(y), label: label) + } + + private static func serializeSegmentationResponse(_ response: SegmentationResponse) -> [String: Any] { + var output: [String: Any] = [ + "segments": response.segments.map { segment in + [ + "score": segment.score, + "box": serializeRect(segment.box), + "maskWidth": segment.maskWidth, + "maskHeight": segment.maskHeight, + "mask": segment.mask + ] + } + ] + + if let probabilityMap = response.probabilityMap { + output["probabilityMap"] = [ + "width": probabilityMap.width, + "height": probabilityMap.height, + "probabilities": probabilityMap.probabilities + ] + } + + return output + } +} +#endif + +#if canImport(CoreAIObjectDetector) +extension CoreAIImpl { + private func makeDetectionParameters(options: [String: Any]?) -> DetectionParameters { + var parameters = DetectionParameters.default + if let threshold = options?["threshold"] as? Double { + parameters.threshold = Float(threshold) + } + if let maxDetections = options?["maxDetections"] as? Double { + parameters.maxDetections = Int(maxDetections) + } + if let inputHeight = options?["inputHeight"] as? Double { + parameters.inputHeight = Int(inputHeight) + } + if let inputWidth = options?["inputWidth"] as? Double { + parameters.inputWidth = Int(inputWidth) + } + return parameters + } + + private static func serializeDetection(_ detection: DetectedObject) -> [String: Any] { + return [ + "label": detection.label, + "labelIndex": detection.labelIndex, + "confidence": detection.confidence, + "box": serializeRect(detection.boundingBox) + ] + } +} +#endif + +extension CoreAIImpl { + private static func loadCGImage(from input: [String: Any]) throws -> CGImage { + guard let imageUri = input["imageUri"] as? String else { + throw CoreAIError.invalidInput("Expected input.imageUri for this Core AI task.") + } + + let url: URL + if imageUri.hasPrefix("file://") { + url = URL(string: imageUri) ?? URL(fileURLWithPath: imageUri.replacingOccurrences(of: "file://", with: "")) + } else { + url = URL(fileURLWithPath: imageUri) + } + + guard let source = CGImageSourceCreateWithURL(url as CFURL, nil), + let image = CGImageSourceCreateImageAtIndex(source, 0, nil) else { + throw CoreAIError.invalidInput("Could not load image at \(url.path).") + } + return image + } + + private static func pngBase64(_ image: CGImage) throws -> String { + let data = NSMutableData() + guard let destination = CGImageDestinationCreateWithData( + data, + "public.png" as CFString, + 1, + nil + ) else { + throw CoreAIError.invalidInput("Could not create PNG destination.") + } + + CGImageDestinationAddImage(destination, image, nil) + guard CGImageDestinationFinalize(destination) else { + throw CoreAIError.invalidInput("Could not encode generated image as PNG.") + } + return (data as Data).base64EncodedString() + } + + private static func serializeRect(_ rect: CGRect) -> [String: Double] { + return [ + "x": rect.origin.x, + "y": rect.origin.y, + "width": rect.width, + "height": rect.height + ].mapValues(Double.init) + } +} + +private func errorCode(_ error: Error) -> String { + if let coreAIError = error as? CoreAIError { + return coreAIError.code + } + return "CORE_AI_ERROR" +} diff --git a/packages/core-ai/package.json b/packages/core-ai/package.json new file mode 100644 index 00000000..65ab5468 --- /dev/null +++ b/packages/core-ai/package.json @@ -0,0 +1,91 @@ +{ + "name": "@react-native-ai/core-ai", + "version": "0.12.0", + "description": "Apple Core AI provider for React Native AI", + "main": "lib/commonjs/index", + "module": "lib/module/index", + "types": "lib/typescript/index.d.ts", + "react-native": "src/index", + "source": "src/index", + "files": [ + "src", + "lib", + "ios", + "*.podspec", + "!ios/build", + "!**/__tests__", + "!**/__fixtures__", + "!**/__mocks__", + "!**/.*" + ], + "scripts": { + "clean": "del-cli lib", + "typecheck": "tsc --noEmit", + "prepare": "bob build" + }, + "keywords": [ + "react-native", + "apple", + "core-ai", + "llm", + "ai", + "sdk", + "vercel", + "expo", + "spm" + ], + "repository": { + "type": "git", + "url": "git+https://github.com/callstackincubator/ai.git" + }, + "author": "Mike Grabowski ", + "contributors": [ + "Callstack " + ], + "license": "MIT", + "bugs": { + "url": "https://github.com/callstackincubator/ai/issues" + }, + "homepage": "https://github.com/callstackincubator/ai#readme", + "publishConfig": { + "registry": "https://registry.npmjs.org/" + }, + "dependencies": { + "@ai-sdk/provider": "^3.0.5", + "@ai-sdk/provider-utils": "^4.0.1", + "zod": "^4.2.1" + }, + "peerDependencies": { + "react-native": ">=0.76.0", + "expo": "*" + }, + "peerDependenciesMeta": { + "expo": { + "optional": true + } + }, + "react-native-builder-bob": { + "source": "src", + "output": "lib", + "targets": [ + "commonjs", + "module", + [ + "typescript", + { + "project": "tsconfig.build.json" + } + ] + ] + }, + "codegenConfig": { + "name": "NativeCoreAI", + "type": "modules", + "jsSrcsDir": "src", + "ios": { + "modulesProvider": { + "NativeCoreAI": "ReactNativeAICoreAI" + } + } + } +} diff --git a/packages/core-ai/src/NativeCoreAI.ts b/packages/core-ai/src/NativeCoreAI.ts new file mode 100644 index 00000000..3be50468 --- /dev/null +++ b/packages/core-ai/src/NativeCoreAI.ts @@ -0,0 +1,136 @@ +import type { TurboModule } from 'react-native' +import { TurboModuleRegistry } from 'react-native' +import type { + EventEmitter, + UnsafeObject, +} from 'react-native/Libraries/Types/CodegenTypes' + +export interface NativeCoreAIModelConfig { + id: string + sourceType: string + sourceUri?: string + bundleName?: string + bundleExtension?: string + bundleSubdirectory?: string + task?: string + family?: string + variant?: string +} + +export interface CoreAIModelInfo { + id?: string + family?: string + task: string + platforms: string[] + functions: UnsafeObject[] + maxContextLength?: number + modelSizeBytes?: number + metadata?: UnsafeObject +} + +export interface CoreAILoadedModel { + modelHandle: string + info: CoreAIModelInfo +} + +export interface CoreAIMessage { + role: 'assistant' | 'system' | 'tool' | 'user' + content: string +} + +export type CoreAIStreamUpdateEvent = { + streamId: string + content: string +} + +export type CoreAIStreamCompleteEvent = { + streamId: string +} + +export type CoreAIStreamErrorEvent = { + streamId: string + code?: string + error: string +} + +export interface Spec extends TurboModule { + getCapabilities(): Promise + + inspectModel(config: NativeCoreAIModelConfig): Promise + loadModel( + config: NativeCoreAIModelConfig, + options?: UnsafeObject + ): Promise + unloadModel(modelHandle: string): Promise + removeModel(config: NativeCoreAIModelConfig): Promise + specializeModel( + config: NativeCoreAIModelConfig, + options?: UnsafeObject + ): Promise + + createLanguageSession( + modelHandle: string, + options?: UnsafeObject + ): Promise + releaseLanguageSession(sessionHandle: string): Promise + respondToLanguageSession( + sessionHandle: string, + prompt: string, + options?: UnsafeObject + ): Promise + streamLanguageSession( + streamId: string, + sessionHandle: string, + prompt: string, + options?: UnsafeObject + ): void + + generateText( + config: NativeCoreAIModelConfig, + messages: CoreAIMessage[], + options?: UnsafeObject + ): Promise + streamText( + streamId: string, + config: NativeCoreAIModelConfig, + messages: CoreAIMessage[], + options?: UnsafeObject + ): void + cancelStream(streamId: string): void + + embed( + config: NativeCoreAIModelConfig, + values: string[], + options?: UnsafeObject + ): Promise + transcribe( + config: NativeCoreAIModelConfig, + audioBase64: string, + mediaType: string, + options?: UnsafeObject + ): Promise + generateImage( + config: NativeCoreAIModelConfig, + prompt: string, + options?: UnsafeObject + ): Promise + + runTask( + task: string, + config: NativeCoreAIModelConfig, + input: UnsafeObject, + options?: UnsafeObject + ): Promise + runRawFunction( + modelHandle: string, + functionName: string, + inputs: UnsafeObject, + options?: UnsafeObject + ): Promise + + onStreamUpdate: EventEmitter + onStreamComplete: EventEmitter + onStreamError: EventEmitter +} + +export default TurboModuleRegistry.getEnforcing('NativeCoreAI') diff --git a/packages/core-ai/src/ai-sdk.ts b/packages/core-ai/src/ai-sdk.ts new file mode 100644 index 00000000..1932c640 --- /dev/null +++ b/packages/core-ai/src/ai-sdk.ts @@ -0,0 +1,394 @@ +import type { + EmbeddingModelV3, + EmbeddingModelV3CallOptions, + EmbeddingModelV3Result, + ImageModelV3, + ImageModelV3CallOptions, + LanguageModelV3, + LanguageModelV3CallOptions, + LanguageModelV3FinishReason, + LanguageModelV3Prompt, + LanguageModelV3StreamPart, + RerankingModelV3, + RerankingModelV3CallOptions, + SpeechModelV3, + SpeechModelV3CallOptions, + TranscriptionModelV3, + TranscriptionModelV3CallOptions, +} from '@ai-sdk/provider' +import { generateId } from '@ai-sdk/provider-utils' + +import { + coreAI, + CoreAIEmbeddingModel, + CoreAIImageModel, + CoreAILanguageModel, + CoreAITranscriptionModel, + prepareMessages, +} from './core' +import NativeCoreAI from './NativeCoreAI' +import type { + CoreAIGenerationOptions, + CoreAIGenerationPart, + CoreAIModelConfig, + CoreAIStreamCompleteEvent, + CoreAIStreamErrorEvent, + CoreAIStreamUpdateEvent, +} from './types' +import { toNativeModelConfig } from './types' + +export function createCoreAIProvider() { + const provider = function (config: CoreAIModelConfig) { + return provider.languageModel(config) + } + + provider.getCapabilities = () => coreAI.getCapabilities() + provider.languageModel = (config: CoreAIModelConfig) => { + return new CoreAILanguageModelAdapter(config) + } + provider.textEmbeddingModel = (config: CoreAIModelConfig) => { + return new CoreAITextEmbeddingModel(config) + } + provider.embeddingModel = provider.textEmbeddingModel + provider.imageModel = (config: CoreAIModelConfig) => { + return new CoreAIImageGenerationModel(config) + } + provider.transcriptionModel = (config: CoreAIModelConfig) => { + return new CoreAITranscriptionAdapter(config) + } + provider.rerankingModel = (config: CoreAIModelConfig) => { + return new UnsupportedCoreAIRerankingModel(config) + } + provider.speechModel = (config: CoreAIModelConfig) => { + return new UnsupportedCoreAISpeechModel(config) + } + + return provider +} + +export const coreAIProvider = createCoreAIProvider() + +export class CoreAILanguageModelAdapter + extends CoreAILanguageModel + implements LanguageModelV3 +{ + readonly specificationVersion = 'v3' + readonly supportedUrls = {} + readonly provider = 'core-ai' + readonly modelId: string + + constructor(config: CoreAIModelConfig) { + super({ ...config, task: 'language' }) + this.modelId = config.id + } + + async doGenerate(options: LanguageModelV3CallOptions) { + const response = (await NativeCoreAI.generateText( + toNativeModelConfig(this.config), + toCoreAIMessages(options.prompt), + toCoreAIGenerationOptions(options) + )) as CoreAIGenerationPart[] + + return { + content: response.map(toLanguageModelContent), + finishReason: toFinishReason('stop'), + usage: emptyUsage(), + warnings: [], + } + } + + async doStream(options: LanguageModelV3CallOptions) { + if (typeof ReadableStream === 'undefined') { + throw new Error( + 'ReadableStream is not available. Load a web stream polyfill before streaming Core AI responses.' + ) + } + + const streamId = generateId() + let listeners: { remove(): void }[] = [] + + const cleanup = () => { + listeners.forEach((listener) => listener.remove()) + listeners = [] + } + + const stream = new ReadableStream({ + start: (controller) => { + controller.enqueue({ type: 'text-start', id: streamId }) + + const updateListener = NativeCoreAI.onStreamUpdate( + (event: CoreAIStreamUpdateEvent) => { + if (event.streamId === streamId) { + controller.enqueue({ + type: 'text-delta', + id: streamId, + delta: event.content, + }) + } + } + ) + const completeListener = NativeCoreAI.onStreamComplete( + (event: CoreAIStreamCompleteEvent) => { + if (event.streamId === streamId) { + controller.enqueue({ type: 'text-end', id: streamId }) + controller.enqueue({ + type: 'finish', + finishReason: toFinishReason('stop'), + usage: emptyUsage(), + }) + cleanup() + controller.close() + } + } + ) + const errorListener = NativeCoreAI.onStreamError( + (event: CoreAIStreamErrorEvent) => { + if (event.streamId === streamId) { + controller.enqueue({ + type: 'error', + error: new Error(event.error), + }) + cleanup() + controller.close() + } + } + ) + + listeners = [updateListener, completeListener, errorListener] + NativeCoreAI.streamText( + streamId, + toNativeModelConfig(this.config), + toCoreAIMessages(options.prompt), + toCoreAIGenerationOptions(options) + ) + }, + cancel: () => { + cleanup() + NativeCoreAI.cancelStream(streamId) + }, + }) + + return { + stream, + rawCall: { + rawPrompt: options.prompt, + rawSettings: {}, + }, + } + } +} + +export class CoreAITextEmbeddingModel + extends CoreAIEmbeddingModel + implements EmbeddingModelV3 +{ + readonly specificationVersion = 'v3' + readonly provider = 'core-ai' + readonly modelId: string + readonly maxEmbeddingsPerCall = Infinity + readonly supportsParallelCalls = false + + constructor(config: CoreAIModelConfig) { + super({ ...config, task: 'embedding' }) + this.modelId = config.id + } + + async doEmbed( + options: EmbeddingModelV3CallOptions + ): Promise { + const result = await this.embed(options.values) + return { + embeddings: result.embeddings, + warnings: [], + } + } +} + +export class CoreAIImageGenerationModel + extends CoreAIImageModel + implements ImageModelV3 +{ + readonly specificationVersion = 'v3' + readonly provider = 'core-ai' + readonly modelId: string + readonly maxImagesPerCall = 1 + + constructor(config: CoreAIModelConfig) { + super({ ...config, task: 'diffusion' }) + this.modelId = config.id + } + + async doGenerate(options: ImageModelV3CallOptions) { + const result = await this.generate(options.prompt ?? '', { + n: options.n, + size: options.size, + aspectRatio: options.aspectRatio, + seed: options.seed, + ...(options.providerOptions?.['core-ai'] ?? {}), + }) + + return { + images: result.images, + warnings: [], + response: { + timestamp: new Date(), + modelId: this.modelId, + headers: undefined, + }, + providerMetadata: result.metadata + ? ({ + 'core-ai': { + images: [], + ...result.metadata, + }, + } as any) + : undefined, + } + } +} + +export class CoreAITranscriptionAdapter + extends CoreAITranscriptionModel + implements TranscriptionModelV3 +{ + readonly specificationVersion = 'v3' + readonly provider = 'core-ai' + readonly modelId: string + + constructor(config: CoreAIModelConfig) { + super({ ...config, task: 'asr' }) + this.modelId = config.id + } + + async doGenerate(options: TranscriptionModelV3CallOptions) { + const result = await this.transcribe(options.audio, options.mediaType, { + ...(options.providerOptions?.['core-ai'] ?? {}), + }) + + return { + text: result.text, + segments: result.segments, + language: result.language, + durationInSeconds: result.durationInSeconds, + warnings: [], + response: { + timestamp: new Date(), + modelId: this.modelId, + }, + providerMetadata: result.metadata + ? ({ + 'core-ai': result.metadata, + } as any) + : undefined, + } + } +} + +class UnsupportedCoreAIRerankingModel implements RerankingModelV3 { + readonly specificationVersion = 'v3' + readonly provider = 'core-ai' + readonly modelId: string + + constructor(config: CoreAIModelConfig) { + this.modelId = config.id + } + + doRerank(_options: RerankingModelV3CallOptions) { + return Promise.reject( + new Error( + 'Core AI reranking is not available yet. Apple coreai-models does not currently list a reranking starter model, so @react-native-ai/core-ai exposes no real reranking provider.' + ) + ) as ReturnType + } +} + +class UnsupportedCoreAISpeechModel implements SpeechModelV3 { + readonly specificationVersion = 'v3' + readonly provider = 'core-ai' + readonly modelId: string + + constructor(config: CoreAIModelConfig) { + this.modelId = config.id + } + + doGenerate(_options: SpeechModelV3CallOptions) { + return Promise.reject( + new Error( + 'Core AI speech generation is not available yet. Use @react-native-ai/apple for AVSpeechSynthesizer-backed speech unless a real Core AI TTS model is added.' + ) + ) as ReturnType + } +} + +function toCoreAIMessages(prompt: LanguageModelV3Prompt) { + return prepareMessages(prompt) +} + +function toCoreAIGenerationOptions( + options: LanguageModelV3CallOptions +): CoreAIGenerationOptions { + return { + maxTokens: options.maxOutputTokens, + temperature: options.temperature, + topP: options.topP, + topK: options.topK, + schema: + options.responseFormat?.type === 'json' + ? options.responseFormat.schema + : undefined, + tools: options.tools, + } +} + +function toLanguageModelContent(part: CoreAIGenerationPart) { + switch (part.type) { + case 'text': + return part + case 'reasoning': + return { + type: 'reasoning' as const, + text: part.text, + } + case 'tool-call': + return { + type: 'tool-call' as const, + toolCallId: generateId(), + providerExecuted: true, + toolName: part.toolName, + input: part.input, + } + case 'tool-result': + return { + type: 'tool-result' as const, + toolCallId: generateId(), + providerExecuted: true, + toolName: part.toolName, + result: part.output, + } + } +} + +function toFinishReason( + raw: 'stop' | 'length' | 'tool-calls' | 'error' +): LanguageModelV3FinishReason { + return { + unified: raw === 'tool-calls' ? 'tool-calls' : raw, + raw, + } +} + +function emptyUsage() { + return { + inputTokens: { + total: 0, + noCache: undefined, + cacheRead: undefined, + cacheWrite: undefined, + }, + outputTokens: { + total: 0, + text: undefined, + reasoning: undefined, + }, + } +} diff --git a/packages/core-ai/src/catalog.ts b/packages/core-ai/src/catalog.ts new file mode 100644 index 00000000..c2decb22 --- /dev/null +++ b/packages/core-ai/src/catalog.ts @@ -0,0 +1,216 @@ +import type { CoreAIModelTask, CoreAIPlatform } from './types' + +export interface CoreAIModelCatalogEntry { + id: string + hfId: string + family: string + task: CoreAIModelTask + platforms: CoreAIPlatform[] + exportScript?: string + exportCommand: string + notes?: string +} + +const ENTRIES: CoreAIModelCatalogEntry[] = [ + { + id: 'qwen3-0.6b', + hfId: 'Qwen/Qwen3-0.6B', + family: 'qwen3', + task: 'language', + platforms: ['iOS', 'macOS'], + exportCommand: 'uv run coreai.llm.export Qwen/Qwen3-0.6B --platform iOS', + notes: 'Recommended first iOS language-session example.', + }, + { + id: 'qwen2.5-1.5b-instruct', + hfId: 'Qwen/Qwen2.5-1.5B-Instruct', + family: 'qwen2.5', + task: 'language', + platforms: ['iOS', 'macOS'], + exportCommand: + 'uv run coreai.llm.export Qwen/Qwen2.5-1.5B-Instruct --platform iOS', + }, + { + id: 'qwen3-4b', + hfId: 'Qwen/Qwen3-4B', + family: 'qwen3', + task: 'language', + platforms: ['iOS', 'macOS'], + exportCommand: 'uv run coreai.llm.export Qwen/Qwen3-4B --platform iOS', + }, + { + id: 'gpt-oss-20b', + hfId: 'openai/gpt-oss-20b', + family: 'gpt-oss', + task: 'language', + platforms: ['macOS'], + exportCommand: 'uv run coreai.llm.export openai/gpt-oss-20b', + notes: 'macOS-only in the current Apple registry.', + }, + { + id: 'gemma3-4b-it', + hfId: 'google/gemma-3-4b-it', + family: 'gemma3', + task: 'language', + platforms: ['macOS'], + exportCommand: 'uv run coreai.llm.export google/gemma-3-4b-it', + notes: 'macOS-only in the current Apple registry.', + }, + { + id: 'clip-vit-b32', + hfId: 'openai/clip-vit-base-patch32', + family: 'clip', + task: 'embedding', + platforms: ['iOS', 'macOS'], + exportScript: 'models/clip/export.py', + exportCommand: + 'uv run models/clip/export.py --model openai/clip-vit-base-patch32', + }, + { + id: 'clap-htsat', + hfId: 'laion/clap-htsat-unfused', + family: 'clap', + task: 'embedding', + platforms: ['iOS', 'macOS'], + exportScript: 'models/clap/export.py', + exportCommand: + 'uv run models/clap/export.py --model laion/clap-htsat-unfused', + }, + { + id: 'sd-1.5', + hfId: 'runwayml/stable-diffusion-v1-5', + family: 'stable-diffusion', + task: 'diffusion', + platforms: ['iOS', 'macOS'], + exportCommand: + 'uv run coreai.diffusion.export runwayml/stable-diffusion-v1-5', + }, + { + id: 'flux2-klein-4b', + hfId: 'black-forest-labs/FLUX.2-klein-4B', + family: 'flux2', + task: 'diffusion', + platforms: ['iOS', 'macOS'], + exportCommand: + 'uv run coreai.diffusion.export flux2-klein-4b --platform iOS', + }, + { + id: 'wav2vec2-base', + hfId: 'wav2vec2_asr_base_960h', + family: 'wav2vec2', + task: 'asr', + platforms: ['iOS', 'macOS'], + exportScript: 'models/wav2vec2/export.py', + exportCommand: + 'uv run models/wav2vec2/export.py --model wav2vec2_asr_base_960h', + }, + { + id: 'whisper-large-v3-turbo', + hfId: 'openai/whisper-large-v3-turbo', + family: 'whisper', + task: 'asr', + platforms: ['iOS', 'macOS'], + exportScript: 'models/whisper/export.py', + exportCommand: + 'uv run models/whisper/export.py --model openai/whisper-large-v3-turbo', + }, + { + id: 'efficient-sam-vitt', + hfId: 'efficient_sam_vitt', + family: 'efficient-sam', + task: 'segmentation', + platforms: ['iOS', 'macOS'], + exportScript: 'models/efficient-sam/export.py', + exportCommand: + 'uv run models/efficient-sam/export.py --model efficient_sam_vitt', + }, + { + id: 'sam3', + hfId: 'facebook/sam3', + family: 'sam3', + task: 'segmentation', + platforms: ['iOS', 'macOS'], + exportScript: 'models/sam3/export.py', + exportCommand: 'uv run models/sam3/export.py --model facebook/sam3', + notes: 'Gated on Hugging Face.', + }, + { + id: 'yolos-tiny', + hfId: 'hustvl/yolos-tiny', + family: 'yolo', + task: 'object-detection', + platforms: ['iOS', 'macOS'], + exportScript: 'models/yolo/export.py', + exportCommand: 'uv run models/yolo/export.py --model hustvl/yolos-tiny', + }, + { + id: 'yolos-base', + hfId: 'hustvl/yolos-base', + family: 'yolo', + task: 'object-detection', + platforms: ['iOS', 'macOS'], + exportScript: 'models/yolo/export.py', + exportCommand: 'uv run models/yolo/export.py --model hustvl/yolos-base', + }, + { + id: 'depth-anything-3-small', + hfId: 'depth-anything/da3-small', + family: 'depth-anything', + task: 'depth', + platforms: ['macOS'], + exportScript: 'models/depth-anything/export.py', + exportCommand: + 'uv run models/depth-anything/export.py --model depth-anything/da3-small', + notes: 'macOS-only in the current Apple registry.', + }, + { + id: 'edsr-x2', + hfId: 'edsr_r16f64_x2', + family: 'edsr', + task: 'super-resolution', + platforms: ['iOS', 'macOS'], + exportScript: 'models/edsr/export.py', + exportCommand: 'uv run models/edsr/export.py --model edsr_r16f64_x2', + }, + { + id: 'pvt-v2-b0', + hfId: 'pvt_v2_b0', + family: 'pvt', + task: 'classification', + platforms: ['iOS', 'macOS'], + exportScript: 'models/pvt/export.py', + exportCommand: 'uv run models/pvt/export.py --model pvt_v2_b0', + }, +] + +export const catalog = { + list({ + task, + platform, + }: { task?: CoreAIModelTask; platform?: CoreAIPlatform } = {}) { + return ENTRIES.filter((entry) => { + if (task && entry.task !== task) { + return false + } + if (platform && !entry.platforms.includes(platform)) { + return false + } + return true + }) + }, + + get(id: string) { + return ENTRIES.find((entry) => entry.id === id) + }, + + getExportCommand(id: string, options: { platform?: CoreAIPlatform } = {}) { + const entry = catalog.get(id) + if (!entry) { + return undefined + } + if (options.platform === 'iOS' && !entry.platforms.includes('iOS')) { + return undefined + } + return entry.exportCommand + }, +} diff --git a/packages/core-ai/src/core.ts b/packages/core-ai/src/core.ts new file mode 100644 index 00000000..7b0d90ef --- /dev/null +++ b/packages/core-ai/src/core.ts @@ -0,0 +1,371 @@ +import { generateId } from '@ai-sdk/provider-utils' +import type { UnsafeObject } from 'react-native/Libraries/Types/CodegenTypes' + +import { catalog } from './catalog' +import NativeCoreAI from './NativeCoreAI' +import type { + CoreAICapabilities, + CoreAIEmbeddingResult, + CoreAIGenerationOptions, + CoreAIGenerationPart, + CoreAIImageGenerationOptions, + CoreAIImageGenerationResult, + CoreAILoadedModel, + CoreAIMessage, + CoreAIModelConfig, + CoreAIModelInfo, + CoreAIModelTask, + CoreAIStreamCompleteEvent, + CoreAIStreamErrorEvent, + CoreAIStreamUpdateEvent, + CoreAITaskInput, + CoreAITaskResult, + CoreAITranscriptionResult, +} from './types' +import { toNativeModelConfig } from './types' + +export class CoreAIModel { + protected loadedModel?: CoreAILoadedModel + + constructor(readonly config: CoreAIModelConfig) {} + + get modelHandle() { + return this.loadedModel?.modelHandle + } + + async inspect(): Promise { + return NativeCoreAI.inspectModel( + toNativeModelConfig(this.config) + ) as Promise + } + + async prepare(options: UnsafeObject = {}): Promise { + const loadedModel = (await NativeCoreAI.loadModel( + toNativeModelConfig(this.config), + options + )) as CoreAILoadedModel + this.loadedModel = loadedModel + return loadedModel.info + } + + async specialize(options: UnsafeObject = {}): Promise { + return NativeCoreAI.specializeModel( + toNativeModelConfig(this.config), + options + ) as Promise + } + + async unload(): Promise { + if (!this.loadedModel) { + return + } + await NativeCoreAI.unloadModel(this.loadedModel.modelHandle) + this.loadedModel = undefined + } + + async remove(): Promise { + await NativeCoreAI.removeModel(toNativeModelConfig(this.config)) + this.loadedModel = undefined + } + + protected async ensureLoaded(): Promise { + if (!this.loadedModel) { + await this.prepare() + } + return this.loadedModel! + } +} + +export class CoreAILanguageSession { + constructor(readonly sessionHandle: string) {} + + async respond( + prompt: string, + options: CoreAIGenerationOptions = {} + ): Promise { + return NativeCoreAI.respondToLanguageSession( + this.sessionHandle, + prompt, + options + ) as Promise + } + + async stream( + prompt: string, + options: CoreAIGenerationOptions = {} + ): Promise> { + if (typeof ReadableStream === 'undefined') { + throw new Error( + 'ReadableStream is not available. Load a web stream polyfill before streaming Core AI responses.' + ) + } + + const streamId = generateId() + const sessionHandle = this.sessionHandle + let listeners: { remove(): void }[] = [] + + const cleanup = () => { + listeners.forEach((listener) => listener.remove()) + listeners = [] + } + + const stream = new ReadableStream({ + start(controller) { + const updateListener = NativeCoreAI.onStreamUpdate( + (event: CoreAIStreamUpdateEvent) => { + if (event.streamId === streamId) { + controller.enqueue(event) + } + } + ) + const completeListener = NativeCoreAI.onStreamComplete( + (event: CoreAIStreamCompleteEvent) => { + if (event.streamId === streamId) { + cleanup() + controller.close() + } + } + ) + const errorListener = NativeCoreAI.onStreamError( + (event: CoreAIStreamErrorEvent) => { + if (event.streamId === streamId) { + cleanup() + controller.error(new Error(event.error)) + } + } + ) + + listeners = [updateListener, completeListener, errorListener] + NativeCoreAI.streamLanguageSession( + streamId, + sessionHandle, + prompt, + options + ) + }, + cancel() { + cleanup() + NativeCoreAI.cancelStream(streamId) + }, + }) + + return stream + } + + async close(): Promise { + await NativeCoreAI.releaseLanguageSession(this.sessionHandle) + } +} + +export class CoreAILanguageModel extends CoreAIModel { + async createSession( + options: UnsafeObject = {} + ): Promise { + const model = await this.ensureLoaded() + const sessionHandle = await NativeCoreAI.createLanguageSession( + model.modelHandle, + options + ) + return new CoreAILanguageSession(sessionHandle) + } +} + +export class CoreAIEmbeddingModel extends CoreAIModel { + async embed( + values: string[], + options: UnsafeObject = {} + ): Promise { + return NativeCoreAI.embed( + toNativeModelConfig(this.config), + values, + options + ) as Promise + } +} + +export class CoreAITranscriptionModel extends CoreAIModel { + async transcribe( + audio: Uint8Array | string, + mediaType: string, + options: UnsafeObject = {} + ): Promise { + return NativeCoreAI.transcribe( + toNativeModelConfig(this.config), + toBase64(audio), + mediaType, + options + ) as Promise + } +} + +export class CoreAIImageModel extends CoreAIModel { + async generate( + prompt: string, + options: CoreAIImageGenerationOptions = {} + ): Promise { + return NativeCoreAI.generateImage( + toNativeModelConfig(this.config), + prompt, + options + ) as Promise + } +} + +class CoreAITaskModel extends CoreAIModel { + constructor( + config: CoreAIModelConfig, + private readonly task: CoreAIModelTask + ) { + super({ ...config, task: config.task ?? task }) + } + + async run( + input: CoreAITaskInput, + options: UnsafeObject = {} + ): Promise { + return NativeCoreAI.runTask( + this.task, + toNativeModelConfig(this.config), + input as UnsafeObject, + options + ) as Promise + } +} + +export class CoreAIRawModel extends CoreAIModel { + async load(options: UnsafeObject = {}) { + return this.prepare(options) + } + + async runFunction( + functionName: string, + inputs: UnsafeObject, + options: UnsafeObject = {} + ): Promise { + const model = await this.ensureLoaded() + return NativeCoreAI.runRawFunction( + model.modelHandle, + functionName, + inputs, + options + ) + } +} + +export const coreAI = { + catalog, + + getCapabilities(): Promise { + return NativeCoreAI.getCapabilities() as Promise + }, + + languageModel(config: CoreAIModelConfig) { + return new CoreAILanguageModel({ ...config, task: 'language' }) + }, + + embeddingModel(config: CoreAIModelConfig) { + return new CoreAIEmbeddingModel({ ...config, task: 'embedding' }) + }, + + transcriptionModel(config: CoreAIModelConfig) { + return new CoreAITranscriptionModel({ ...config, task: 'asr' }) + }, + + imageModel(config: CoreAIModelConfig) { + return new CoreAIImageModel({ ...config, task: 'diffusion' }) + }, + + segmenter(config: CoreAIModelConfig) { + return new CoreAITaskModel(config, 'segmentation') + }, + + objectDetector(config: CoreAIModelConfig) { + return new CoreAITaskModel(config, 'object-detection') + }, + + depthEstimator(config: CoreAIModelConfig) { + return new CoreAITaskModel(config, 'depth') + }, + + superResolution(config: CoreAIModelConfig) { + return new CoreAITaskModel(config, 'super-resolution') + }, + + classifier(config: CoreAIModelConfig) { + return new CoreAITaskModel(config, 'classification') + }, + + models: { + inspect(config: CoreAIModelConfig) { + return NativeCoreAI.inspectModel( + toNativeModelConfig(config) + ) as Promise + }, + specialize(config: CoreAIModelConfig, options: UnsafeObject = {}) { + return NativeCoreAI.specializeModel( + toNativeModelConfig(config), + options + ) as Promise + }, + remove(config: CoreAIModelConfig) { + return NativeCoreAI.removeModel(toNativeModelConfig(config)) + }, + }, + + embeddings: { + embed( + config: CoreAIModelConfig, + values: string[], + options: UnsafeObject = {} + ) { + return NativeCoreAI.embed( + toNativeModelConfig({ ...config, task: 'embedding' }), + values, + options + ) as Promise + }, + }, + + unstable: { + loadModel(config: CoreAIModelConfig) { + return new CoreAIRawModel({ ...config, task: 'raw' }) + }, + }, +} + +function toBase64(data: Uint8Array | string): string { + if (typeof data === 'string') { + return data + } + + let binary = '' + const chunkSize = 0x8000 + for (let i = 0; i < data.length; i += chunkSize) { + const chunk = data.subarray(i, i + chunkSize) + binary += String.fromCharCode(...chunk) + } + return btoa(binary) +} + +export function prepareMessages(messages: unknown): CoreAIMessage[] { + if (!Array.isArray(messages)) { + return [] + } + + return messages.map((message: any): CoreAIMessage => { + const content = Array.isArray(message.content) + ? message.content.reduce((acc: string, part: any) => { + if (part.type === 'text') { + return acc + part.text + } + console.warn('Unsupported Core AI message content type:', part) + return acc + }, '') + : String(message.content ?? '') + + return { + role: message.role, + content, + } + }) +} diff --git a/packages/core-ai/src/index.ts b/packages/core-ai/src/index.ts new file mode 100644 index 00000000..117110bd --- /dev/null +++ b/packages/core-ai/src/index.ts @@ -0,0 +1,37 @@ +export { + CoreAIImageGenerationModel, + CoreAILanguageModelAdapter, + coreAIProvider, + CoreAITextEmbeddingModel, + CoreAITranscriptionAdapter, + createCoreAIProvider, +} from './ai-sdk' +export { catalog } from './catalog' +export { + coreAI, + CoreAIEmbeddingModel, + CoreAIImageModel, + CoreAILanguageModel, + CoreAILanguageSession, + CoreAIModel, + CoreAIRawModel, + CoreAITranscriptionModel, +} from './core' +export type { + CoreAICapabilities, + CoreAIEmbeddingResult, + CoreAIGenerationOptions, + CoreAIGenerationPart, + CoreAIImageGenerationOptions, + CoreAIImageGenerationResult, + CoreAILoadedModel, + CoreAIMessage, + CoreAIModelConfig, + CoreAIModelInfo, + CoreAIModelSource, + CoreAIModelTask, + CoreAIPlatform, + CoreAITaskInput, + CoreAITaskResult, + CoreAITranscriptionResult, +} from './types' diff --git a/packages/core-ai/src/types.ts b/packages/core-ai/src/types.ts new file mode 100644 index 00000000..77cf91ce --- /dev/null +++ b/packages/core-ai/src/types.ts @@ -0,0 +1,202 @@ +import type { UnsafeObject } from 'react-native/Libraries/Types/CodegenTypes' + +export type CoreAIPlatform = 'iOS' | 'macOS' + +export type CoreAIModelTask = + | 'language' + | 'embedding' + | 'asr' + | 'diffusion' + | 'segmentation' + | 'object-detection' + | 'depth' + | 'super-resolution' + | 'classification' + | 'encoding' + | 'raw' + | 'unknown' + +export type CoreAIModelSource = + | { + type: 'file' + uri: string + } + | { + type: 'bundle' + name: string + extension?: string + subdirectory?: string + } + +export interface CoreAIModelConfig { + id: string + source: CoreAIModelSource + task?: CoreAIModelTask + family?: string + variant?: CoreAIPlatform +} + +export interface NativeCoreAIModelConfig { + id: string + sourceType: string + sourceUri?: string + bundleName?: string + bundleExtension?: string + bundleSubdirectory?: string + task?: string + family?: string + variant?: string +} + +export interface CoreAITensorDescriptor { + name?: string + dataType?: string + shape?: number[] +} + +export interface CoreAIModelInfo { + id?: string + family?: string + task: CoreAIModelTask + platforms: CoreAIPlatform[] + functions: { + name: string + inputs: CoreAITensorDescriptor[] + outputs: CoreAITensorDescriptor[] + }[] + maxContextLength?: number + modelSizeBytes?: number + metadata?: UnsafeObject +} + +export interface CoreAILoadedModel { + modelHandle: string + info: CoreAIModelInfo +} + +export interface CoreAILanguageSession { + sessionHandle: string +} + +export interface CoreAIMessage { + role: 'assistant' | 'system' | 'tool' | 'user' + content: string +} + +export interface CoreAIGenerationOptions { + temperature?: number + maxTokens?: number + topP?: number + topK?: number + schema?: UnsafeObject + tools?: UnsafeObject +} + +export type CoreAIGenerationPart = + | { type: 'text'; text: string } + | { type: 'reasoning'; text: string } + | { type: 'tool-call'; toolName: string; input: string } + | { type: 'tool-result'; toolName: string; output: string } + +export interface CoreAIStreamUpdateEvent { + streamId: string + content: string +} + +export interface CoreAIStreamCompleteEvent { + streamId: string +} + +export interface CoreAIStreamErrorEvent { + streamId: string + code?: string + error: string +} + +export interface CoreAIEmbeddingResult { + embeddings: number[][] + metadata?: UnsafeObject +} + +export interface CoreAITranscriptionSegment { + text: string + startSecond: number + endSecond: number +} + +export interface CoreAITranscriptionResult { + text: string + segments: CoreAITranscriptionSegment[] + language?: string + durationInSeconds?: number + metadata?: UnsafeObject +} + +export interface CoreAIImageGenerationOptions { + n?: number + size?: `${number}x${number}` + aspectRatio?: `${number}:${number}` + negativePrompt?: string + seed?: number + stepCount?: number + guidanceScale?: number + schedulerType?: string + decodeResolution?: 'auto' | 'full' | 'half' | 'tiled' + strength?: number + lazyModelLoading?: boolean +} + +export interface CoreAIImageGenerationResult { + images: string[] + metadata?: UnsafeObject +} + +export interface CoreAITaskInput { + imageUri?: string + audioUri?: string + text?: string + values?: string[] + data?: UnsafeObject +} + +export interface CoreAITaskResult { + task: CoreAIModelTask + output: UnsafeObject + metadata?: UnsafeObject +} + +export interface CoreAICapabilities { + isCoreAIRuntimeAvailable: boolean + isCoreAILMAvailable: boolean + isCoreAIDiffusionAvailable: boolean + isCoreAISegmentationAvailable: boolean + isCoreAIObjectDetectionAvailable: boolean + supportedPlatform: boolean + missingProducts: string[] +} + +export function toNativeModelConfig( + config: CoreAIModelConfig +): NativeCoreAIModelConfig { + if (config.source.type === 'file') { + return { + id: config.id, + sourceType: 'file', + sourceUri: config.source.uri, + task: config.task, + family: config.family, + variant: config.variant, + } + } + + return { + id: config.id, + sourceType: 'bundle', + bundleName: config.source.name, + bundleExtension: config.source.extension, + bundleSubdirectory: config.source.subdirectory, + task: config.task, + family: config.family, + variant: config.variant, + } +} diff --git a/packages/core-ai/tsconfig.build.json b/packages/core-ai/tsconfig.build.json new file mode 100644 index 00000000..50db0b50 --- /dev/null +++ b/packages/core-ai/tsconfig.build.json @@ -0,0 +1,9 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib/typescript", + "noEmit": false, + "emitDeclarationOnly": true + } +} diff --git a/packages/core-ai/tsconfig.json b/packages/core-ai/tsconfig.json new file mode 100644 index 00000000..d2fecb28 --- /dev/null +++ b/packages/core-ai/tsconfig.json @@ -0,0 +1,7 @@ +{ + "extends": "../../tsconfig.base.json", + "include": ["src"], + "compilerOptions": { + "noEmit": true + } +} diff --git a/website/src/docs/_meta.json b/website/src/docs/_meta.json index e30bf51a..9ec5cb02 100644 --- a/website/src/docs/_meta.json +++ b/website/src/docs/_meta.json @@ -21,6 +21,13 @@ "collapsible": true, "collapsed": false }, + { + "type": "dir", + "name": "core-ai", + "label": "Core AI", + "collapsible": true, + "collapsed": false + }, { "type": "dir", "name": "llama", diff --git a/website/src/docs/core-ai/_meta.json b/website/src/docs/core-ai/_meta.json new file mode 100644 index 00000000..75f9cbd0 --- /dev/null +++ b/website/src/docs/core-ai/_meta.json @@ -0,0 +1,5 @@ +[ + { "type": "file", "name": "getting-started", "label": "Getting Started" }, + { "type": "file", "name": "api", "label": "API Surface" }, + { "type": "file", "name": "models", "label": "Starter Models" } +] diff --git a/website/src/docs/core-ai/api.md b/website/src/docs/core-ai/api.md new file mode 100644 index 00000000..6f97f96d --- /dev/null +++ b/website/src/docs/core-ai/api.md @@ -0,0 +1,79 @@ +# API Surface + +Core AI has two API layers. + +## AI SDK Adapters + +Use these when you want portable AI SDK app code: + +```typescript +import { coreAI } from '@react-native-ai/core-ai' +import { embed, experimental_transcribe, generateImage, generateText } from 'ai' + +const language = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +await generateText({ model: language, prompt: 'Hello' }) +``` + +AI SDK adapter status: + +| Capability | AI SDK API | Core AI API | Status | +| ----------------- | --------------------------------------------- | -------------------------------- | -------------------------------- | +| Language text | `generateText`, `streamText` | `coreAI.languageModel(...)` | Native wrapper implemented | +| Structured output | `generateText` / `streamText` response format | `coreAI.languageModel(...)` | Routed through language sessions | +| Image generation | `generateImage` | `coreAI.imageModel(...)` | Native diffusion wrapper | +| Text embeddings | `embed`, `embedMany` | `coreAI.embeddingModel(...)` | Explicit unsupported error | +| Transcription | `experimental_transcribe` | `coreAI.transcriptionModel(...)` | Explicit unsupported error | + +The embedding and transcription adapters are present because AI SDK has matching interfaces and Apple’s registry has starter export recipes. They reject until this package gets task-specific native runners for CLIP/CLAP and wav2vec2/Whisper outputs. + +Reranking, speech generation, and video generation are intentionally not exposed as working Core AI adapters yet. AI SDK has model interfaces for reranking and speech, but Apple’s current `coreai-models` registry does not provide a Core AI reranker, TTS model, or video-generation starter. Use `@react-native-ai/apple` for system speech synthesis. + +## Direct Native APIs + +Use direct APIs when Core AI has no AI SDK equivalent or when you need Core AI-specific lifecycle control: + +```typescript +const model = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +await model.prepare({ specialize: true }) + +const session = await model.createSession({ + instructions: 'Keep answers short.', +}) + +await session.respond('Create a vocabulary card for "flower".') +``` + +Native-only API status: + +| Capability | API | Status | Why native-only | +| -------------------- | -------------------------------- | -------------------------- | -------------------------------------------------------------------------- | +| Model inspection | `coreAI.models.inspect(...)` | Implemented | AI SDK does not manage Core AI asset metadata. | +| Specialization | `coreAI.models.specialize(...)` | Metadata stub | Core AI-specific preparation step. | +| Persistent sessions | `model.createSession(...)` | Implemented for LLMs | AI SDK owns message history; Core AI sessions own native transcript state. | +| Segmentation | `coreAI.segmenter(...)` | Implemented | AI SDK image APIs do not model segmentation masks. | +| Object detection | `coreAI.objectDetector(...)` | Implemented | AI SDK has no detection primitive. | +| Depth estimation | `coreAI.depthEstimator(...)` | Explicit unsupported error | AI SDK has no depth primitive. | +| Super-resolution | `coreAI.superResolution(...)` | Explicit unsupported error | AI SDK has no image-to-image upscaling primitive. | +| Raw `.aimodel` calls | `coreAI.unstable.loadModel(...)` | Explicit unsupported error | Raw tensor APIs need Core AI-specific ownership and performance rules. | + +## Catalog + +The package ships lightweight metadata generated from Apple’s model registry: + +```typescript +coreAI.catalog.list({ task: 'language', platform: 'iOS' }) +coreAI.catalog.get('qwen3-0.6b') +coreAI.catalog.getExportCommand('qwen3-0.6b', { platform: 'iOS' }) +``` + +This metadata helps avoid downloading or exporting a model that cannot run on the target platform. diff --git a/website/src/docs/core-ai/getting-started.md b/website/src/docs/core-ai/getting-started.md new file mode 100644 index 00000000..de21ddc7 --- /dev/null +++ b/website/src/docs/core-ai/getting-started.md @@ -0,0 +1,87 @@ +# Getting Started + +`@react-native-ai/core-ai` lets React Native apps run app-provided Apple Core AI model bundles. Use it when you export models from [`apple/coreai-models`](https://github.com/apple/coreai-models) and ship or download those bundles in your app. + +Use `@react-native-ai/apple` for Apple system models. Use `@react-native-ai/core-ai` for custom Core AI model assets. + +## Installation + +```bash +npm install @react-native-ai/core-ai +``` + +If you use the AI SDK, install `ai` v6 and the required polyfills: + +```bash +npm install ai +``` + +## Requirements + +- React Native New Architecture +- iOS 27 or newer for Core AI runtime APIs +- Xcode 27 or newer for building apps with Core AI +- Exported `.aimodel` bundles from `apple/coreai-models` +- The Apple Core AI Models Swift Package linked to the host app target + +## Swift Package Setup + +Add `https://github.com/apple/coreai-models` to the iOS app as a Swift Package dependency, then link the products you use: + +- `CoreAILM` for `coreAI.languageModel(...)` and sessions +- `CoreAIDiffusion` for `coreAI.imageModel(...)` +- `CoreAISegmentation` for `coreAI.segmenter(...)` +- `CoreAIObjectDetection` for `coreAI.objectDetector(...)` + +React Native installs this package through CocoaPods, but CocoaPods will not automatically link those Swift Package products into your app target. Treat SPM as a host-app integration step. + +For Expo, use a config plugin that patches the generated Xcode project with `XCRemoteSwiftPackageReference` and links the selected products. For bare React Native, add the Swift Package in Xcode or use a project patching script that modifies `project.pbxproj` predictably. + +The native module checks for missing products and returns clear setup errors instead of failing silently. + +The current native implementation wires language sessions, diffusion image generation, image segmentation, and object detection. Catalog entries for embeddings, transcription, depth, super-resolution, and classification are included so apps can discover Apple starter models, but those task runners still need follow-up native wrappers. + +## First iOS Model + +Start with Qwen3 0.6B because Apple’s registry includes an iOS preset: + +```bash +uv run coreai.llm.export Qwen/Qwen3-0.6B --platform iOS --output-dir ./models +``` + +Then load it from React Native: + +```typescript +import { coreAI } from '@react-native-ai/core-ai' +import { streamText } from 'ai' + +const model = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +const result = streamText({ + model, + prompt: 'Explain Core AI in one sentence.', +}) + +for await (const delta of result.textStream) { + console.log(delta) +} +``` + +For a persistent native session: + +```typescript +await model.prepare() + +const session = await model.createSession({ + instructions: 'Answer in one short paragraph.', +}) + +const response = await session.respond('What is on-device AI?') + +await session.close() +await model.unload() +``` diff --git a/website/src/docs/core-ai/models.md b/website/src/docs/core-ai/models.md new file mode 100644 index 00000000..ff41a078 --- /dev/null +++ b/website/src/docs/core-ai/models.md @@ -0,0 +1,35 @@ +# Starter Models + +Source this list from `apple/coreai-models/python/src/coreai_models/model_registry.py` when updating the package catalog. Prefer iOS examples. Use macOS only when the registry has no iOS entry. + +| Feature | Starter model | Platform | Package status | Notes | +| ------------------------------------------------------------ | ----------------------------- | ----------- | ------------------------------ | -------------------------------------------------------------- | +| Language sessions, text generation, structured output, tools | `qwen3-0.6b` | iOS + macOS | Implemented | Best first Core AI session demo. | +| Better iOS language quality | `qwen2.5-1.5b-instruct` | iOS + macOS | Implemented | Good second example after the session API works. | +| Larger iOS language validation | `qwen3-4b` | iOS + macOS | Implemented | Useful for memory and specialization testing. | +| macOS-only LLM validation | `gpt-oss-20b`, `gemma3-4b-it` | macOS | Implemented when running macOS | Not iOS demo candidates in the current registry. | +| Text/image embeddings | `clip-vit-b32` | iOS + macOS | Export starter only | Needs a CLIP embedding runner before the AI SDK adapter works. | +| Audio/text embeddings | `clap-htsat` | iOS + macOS | Export starter only | Needs a CLAP embedding runner. | +| Reranking | None | None | Not exposed | Do not ship a fake provider until there is a real reranker. | +| Image generation | `sd-1.5` | iOS + macOS | Implemented | Lowest-friction diffusion starter. | +| Modern image generation | `flux2-klein-4b` | iOS + macOS | Implemented | Later demo; heavier setup and memory profile. | +| Transcription smoke test | `wav2vec2-base` | iOS + macOS | Export starter only | Needs an ASR output decoder before the AI SDK adapter works. | +| Transcription API parity | `whisper-large-v3-turbo` | iOS + macOS | Export starter only | Better match for AI SDK transcription expectations. | +| Speech generation | None | None | Not exposed | Keep using `@react-native-ai/apple` for `AVSpeechSynthesizer`. | +| Video generation | None | None | Not exposed | No Core AI video-generation starter in the current registry. | +| Image segmentation | `efficient-sam-vitt` | iOS + macOS | Implemented | Least-gated segmentation starter. | +| Promptable segmentation validation | `sam3` | iOS + macOS | Implemented | Gated on Hugging Face; not first-run docs. | +| Object detection | `yolos-tiny` | iOS + macOS | Implemented | Smaller detection starter. | +| Object detection quality check | `yolos-base` | iOS + macOS | Implemented | Larger detector after wrapper is stable. | +| Depth estimation | `depth-anything-3-small` | macOS | Export starter only | macOS-only in the current Apple registry. | +| Super-resolution | `edsr-x2` | iOS + macOS | Export starter only | Needs an image-to-image result wrapper. | +| Raw model example | `pvt-v2-b0` | iOS + macOS | Load/inspect only | Good for validating raw `.aimodel` loading. | + +Example exports: + +```bash +uv run coreai.llm.export Qwen/Qwen3-0.6B --platform iOS --output-dir ./models +uv run coreai.diffusion.export runwayml/stable-diffusion-v1-5 +uv run models/efficient-sam/export.py --model efficient_sam_vitt +uv run models/yolo/export.py --model hustvl/yolos-tiny +``` diff --git a/website/src/docs/index.md b/website/src/docs/index.md index 07dd6091..420a2f19 100644 --- a/website/src/docs/index.md +++ b/website/src/docs/index.md @@ -30,6 +30,17 @@ Native integration with Apple's on-device AI capabilities through `@react-native Production-ready with instant availability on supported iOS devices. +### Apple Core AI + +Run app-provided Core AI model bundles through `@react-native-ai/core-ai`: + +- **Language Sessions** - Custom Core AI language models with persistent native sessions +- **AI SDK Adapters** - Language, embeddings, image generation, and transcription where AI SDK has matching APIs +- **Native-only Tasks** - Segmentation, object detection, depth, super-resolution, model inspection, specialization, and raw `.aimodel` calls +- **Model Catalog** - Lightweight metadata from Apple’s `coreai-models` registry so apps can pick iOS-safe starters + +Requires exported Core AI assets and Apple’s `coreai-models` Swift Package products linked into the host app target. + ### Llama Engine Run any GGUF model from HuggingFace locally using `llama.rn` through `@react-native-ai/llama`: From e89a8176fca54defa29e1cf01c139bb60061f3c3 Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:25:39 +0200 Subject: [PATCH 02/11] Drop unsupported Core AI adapter helpers --- packages/core-ai/README.md | 7 +++--- packages/core-ai/src/ai-sdk.ts | 46 ---------------------------------- 2 files changed, 4 insertions(+), 49 deletions(-) diff --git a/packages/core-ai/README.md b/packages/core-ai/README.md index 6de5120b..4da031a9 100644 --- a/packages/core-ai/README.md +++ b/packages/core-ai/README.md @@ -20,9 +20,10 @@ Core AI exposes two layers: The first native wrappers cover language sessions, diffusion image generation, segmentation, and object detection. Embeddings, transcription, depth, super-resolution, classification, reranking, speech generation, video -generation, and arbitrary raw tensor calls stay explicit: the JS APIs exist -where useful, but they reject with a setup/unsupported error until a real -task-specific Core AI runner is wired. +generation, and arbitrary raw tensor calls stay explicit: embeddings and +transcription keep AI SDK-shaped adapters because Apple has starter export +recipes, while reranking, speech, and video have no provider helpers until a +real Core AI model/runtime exists. ## Native Setup diff --git a/packages/core-ai/src/ai-sdk.ts b/packages/core-ai/src/ai-sdk.ts index 1932c640..ea31f21b 100644 --- a/packages/core-ai/src/ai-sdk.ts +++ b/packages/core-ai/src/ai-sdk.ts @@ -9,10 +9,6 @@ import type { LanguageModelV3FinishReason, LanguageModelV3Prompt, LanguageModelV3StreamPart, - RerankingModelV3, - RerankingModelV3CallOptions, - SpeechModelV3, - SpeechModelV3CallOptions, TranscriptionModelV3, TranscriptionModelV3CallOptions, } from '@ai-sdk/provider' @@ -56,12 +52,6 @@ export function createCoreAIProvider() { provider.transcriptionModel = (config: CoreAIModelConfig) => { return new CoreAITranscriptionAdapter(config) } - provider.rerankingModel = (config: CoreAIModelConfig) => { - return new UnsupportedCoreAIRerankingModel(config) - } - provider.speechModel = (config: CoreAIModelConfig) => { - return new UnsupportedCoreAISpeechModel(config) - } return provider } @@ -284,42 +274,6 @@ export class CoreAITranscriptionAdapter } } -class UnsupportedCoreAIRerankingModel implements RerankingModelV3 { - readonly specificationVersion = 'v3' - readonly provider = 'core-ai' - readonly modelId: string - - constructor(config: CoreAIModelConfig) { - this.modelId = config.id - } - - doRerank(_options: RerankingModelV3CallOptions) { - return Promise.reject( - new Error( - 'Core AI reranking is not available yet. Apple coreai-models does not currently list a reranking starter model, so @react-native-ai/core-ai exposes no real reranking provider.' - ) - ) as ReturnType - } -} - -class UnsupportedCoreAISpeechModel implements SpeechModelV3 { - readonly specificationVersion = 'v3' - readonly provider = 'core-ai' - readonly modelId: string - - constructor(config: CoreAIModelConfig) { - this.modelId = config.id - } - - doGenerate(_options: SpeechModelV3CallOptions) { - return Promise.reject( - new Error( - 'Core AI speech generation is not available yet. Use @react-native-ai/apple for AVSpeechSynthesizer-backed speech unless a real Core AI TTS model is added.' - ) - ) as ReturnType - } -} - function toCoreAIMessages(prompt: LanguageModelV3Prompt) { return prepareMessages(prompt) } From 2f966f9c722ac2e33a321f7e42850a7149f8b39d Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:27:52 +0200 Subject: [PATCH 03/11] Remove static Core AI catalog API --- packages/core-ai/src/catalog.ts | 216 ----------------------------- packages/core-ai/src/core.ts | 3 - packages/core-ai/src/index.ts | 1 - website/src/docs/core-ai/api.md | 12 +- website/src/docs/core-ai/models.md | 4 +- 5 files changed, 5 insertions(+), 231 deletions(-) delete mode 100644 packages/core-ai/src/catalog.ts diff --git a/packages/core-ai/src/catalog.ts b/packages/core-ai/src/catalog.ts deleted file mode 100644 index c2decb22..00000000 --- a/packages/core-ai/src/catalog.ts +++ /dev/null @@ -1,216 +0,0 @@ -import type { CoreAIModelTask, CoreAIPlatform } from './types' - -export interface CoreAIModelCatalogEntry { - id: string - hfId: string - family: string - task: CoreAIModelTask - platforms: CoreAIPlatform[] - exportScript?: string - exportCommand: string - notes?: string -} - -const ENTRIES: CoreAIModelCatalogEntry[] = [ - { - id: 'qwen3-0.6b', - hfId: 'Qwen/Qwen3-0.6B', - family: 'qwen3', - task: 'language', - platforms: ['iOS', 'macOS'], - exportCommand: 'uv run coreai.llm.export Qwen/Qwen3-0.6B --platform iOS', - notes: 'Recommended first iOS language-session example.', - }, - { - id: 'qwen2.5-1.5b-instruct', - hfId: 'Qwen/Qwen2.5-1.5B-Instruct', - family: 'qwen2.5', - task: 'language', - platforms: ['iOS', 'macOS'], - exportCommand: - 'uv run coreai.llm.export Qwen/Qwen2.5-1.5B-Instruct --platform iOS', - }, - { - id: 'qwen3-4b', - hfId: 'Qwen/Qwen3-4B', - family: 'qwen3', - task: 'language', - platforms: ['iOS', 'macOS'], - exportCommand: 'uv run coreai.llm.export Qwen/Qwen3-4B --platform iOS', - }, - { - id: 'gpt-oss-20b', - hfId: 'openai/gpt-oss-20b', - family: 'gpt-oss', - task: 'language', - platforms: ['macOS'], - exportCommand: 'uv run coreai.llm.export openai/gpt-oss-20b', - notes: 'macOS-only in the current Apple registry.', - }, - { - id: 'gemma3-4b-it', - hfId: 'google/gemma-3-4b-it', - family: 'gemma3', - task: 'language', - platforms: ['macOS'], - exportCommand: 'uv run coreai.llm.export google/gemma-3-4b-it', - notes: 'macOS-only in the current Apple registry.', - }, - { - id: 'clip-vit-b32', - hfId: 'openai/clip-vit-base-patch32', - family: 'clip', - task: 'embedding', - platforms: ['iOS', 'macOS'], - exportScript: 'models/clip/export.py', - exportCommand: - 'uv run models/clip/export.py --model openai/clip-vit-base-patch32', - }, - { - id: 'clap-htsat', - hfId: 'laion/clap-htsat-unfused', - family: 'clap', - task: 'embedding', - platforms: ['iOS', 'macOS'], - exportScript: 'models/clap/export.py', - exportCommand: - 'uv run models/clap/export.py --model laion/clap-htsat-unfused', - }, - { - id: 'sd-1.5', - hfId: 'runwayml/stable-diffusion-v1-5', - family: 'stable-diffusion', - task: 'diffusion', - platforms: ['iOS', 'macOS'], - exportCommand: - 'uv run coreai.diffusion.export runwayml/stable-diffusion-v1-5', - }, - { - id: 'flux2-klein-4b', - hfId: 'black-forest-labs/FLUX.2-klein-4B', - family: 'flux2', - task: 'diffusion', - platforms: ['iOS', 'macOS'], - exportCommand: - 'uv run coreai.diffusion.export flux2-klein-4b --platform iOS', - }, - { - id: 'wav2vec2-base', - hfId: 'wav2vec2_asr_base_960h', - family: 'wav2vec2', - task: 'asr', - platforms: ['iOS', 'macOS'], - exportScript: 'models/wav2vec2/export.py', - exportCommand: - 'uv run models/wav2vec2/export.py --model wav2vec2_asr_base_960h', - }, - { - id: 'whisper-large-v3-turbo', - hfId: 'openai/whisper-large-v3-turbo', - family: 'whisper', - task: 'asr', - platforms: ['iOS', 'macOS'], - exportScript: 'models/whisper/export.py', - exportCommand: - 'uv run models/whisper/export.py --model openai/whisper-large-v3-turbo', - }, - { - id: 'efficient-sam-vitt', - hfId: 'efficient_sam_vitt', - family: 'efficient-sam', - task: 'segmentation', - platforms: ['iOS', 'macOS'], - exportScript: 'models/efficient-sam/export.py', - exportCommand: - 'uv run models/efficient-sam/export.py --model efficient_sam_vitt', - }, - { - id: 'sam3', - hfId: 'facebook/sam3', - family: 'sam3', - task: 'segmentation', - platforms: ['iOS', 'macOS'], - exportScript: 'models/sam3/export.py', - exportCommand: 'uv run models/sam3/export.py --model facebook/sam3', - notes: 'Gated on Hugging Face.', - }, - { - id: 'yolos-tiny', - hfId: 'hustvl/yolos-tiny', - family: 'yolo', - task: 'object-detection', - platforms: ['iOS', 'macOS'], - exportScript: 'models/yolo/export.py', - exportCommand: 'uv run models/yolo/export.py --model hustvl/yolos-tiny', - }, - { - id: 'yolos-base', - hfId: 'hustvl/yolos-base', - family: 'yolo', - task: 'object-detection', - platforms: ['iOS', 'macOS'], - exportScript: 'models/yolo/export.py', - exportCommand: 'uv run models/yolo/export.py --model hustvl/yolos-base', - }, - { - id: 'depth-anything-3-small', - hfId: 'depth-anything/da3-small', - family: 'depth-anything', - task: 'depth', - platforms: ['macOS'], - exportScript: 'models/depth-anything/export.py', - exportCommand: - 'uv run models/depth-anything/export.py --model depth-anything/da3-small', - notes: 'macOS-only in the current Apple registry.', - }, - { - id: 'edsr-x2', - hfId: 'edsr_r16f64_x2', - family: 'edsr', - task: 'super-resolution', - platforms: ['iOS', 'macOS'], - exportScript: 'models/edsr/export.py', - exportCommand: 'uv run models/edsr/export.py --model edsr_r16f64_x2', - }, - { - id: 'pvt-v2-b0', - hfId: 'pvt_v2_b0', - family: 'pvt', - task: 'classification', - platforms: ['iOS', 'macOS'], - exportScript: 'models/pvt/export.py', - exportCommand: 'uv run models/pvt/export.py --model pvt_v2_b0', - }, -] - -export const catalog = { - list({ - task, - platform, - }: { task?: CoreAIModelTask; platform?: CoreAIPlatform } = {}) { - return ENTRIES.filter((entry) => { - if (task && entry.task !== task) { - return false - } - if (platform && !entry.platforms.includes(platform)) { - return false - } - return true - }) - }, - - get(id: string) { - return ENTRIES.find((entry) => entry.id === id) - }, - - getExportCommand(id: string, options: { platform?: CoreAIPlatform } = {}) { - const entry = catalog.get(id) - if (!entry) { - return undefined - } - if (options.platform === 'iOS' && !entry.platforms.includes('iOS')) { - return undefined - } - return entry.exportCommand - }, -} diff --git a/packages/core-ai/src/core.ts b/packages/core-ai/src/core.ts index 7b0d90ef..c7bb4acf 100644 --- a/packages/core-ai/src/core.ts +++ b/packages/core-ai/src/core.ts @@ -1,7 +1,6 @@ import { generateId } from '@ai-sdk/provider-utils' import type { UnsafeObject } from 'react-native/Libraries/Types/CodegenTypes' -import { catalog } from './catalog' import NativeCoreAI from './NativeCoreAI' import type { CoreAICapabilities, @@ -253,8 +252,6 @@ export class CoreAIRawModel extends CoreAIModel { } export const coreAI = { - catalog, - getCapabilities(): Promise { return NativeCoreAI.getCapabilities() as Promise }, diff --git a/packages/core-ai/src/index.ts b/packages/core-ai/src/index.ts index 117110bd..3c124688 100644 --- a/packages/core-ai/src/index.ts +++ b/packages/core-ai/src/index.ts @@ -6,7 +6,6 @@ export { CoreAITranscriptionAdapter, createCoreAIProvider, } from './ai-sdk' -export { catalog } from './catalog' export { coreAI, CoreAIEmbeddingModel, diff --git a/website/src/docs/core-ai/api.md b/website/src/docs/core-ai/api.md index 6f97f96d..4abb2195 100644 --- a/website/src/docs/core-ai/api.md +++ b/website/src/docs/core-ai/api.md @@ -66,14 +66,6 @@ Native-only API status: | Super-resolution | `coreAI.superResolution(...)` | Explicit unsupported error | AI SDK has no image-to-image upscaling primitive. | | Raw `.aimodel` calls | `coreAI.unstable.loadModel(...)` | Explicit unsupported error | Raw tensor APIs need Core AI-specific ownership and performance rules. | -## Catalog +## Starter Models -The package ships lightweight metadata generated from Apple’s model registry: - -```typescript -coreAI.catalog.list({ task: 'language', platform: 'iOS' }) -coreAI.catalog.get('qwen3-0.6b') -coreAI.catalog.getExportCommand('qwen3-0.6b', { platform: 'iOS' }) -``` - -This metadata helps avoid downloading or exporting a model that cannot run on the target platform. +The starter model list in these docs is informational only. The package does not ship a static model catalog; apps should load available models dynamically from their own bundle, download state, or registry integration. diff --git a/website/src/docs/core-ai/models.md b/website/src/docs/core-ai/models.md index ff41a078..325e76d1 100644 --- a/website/src/docs/core-ai/models.md +++ b/website/src/docs/core-ai/models.md @@ -1,6 +1,8 @@ # Starter Models -Source this list from `apple/coreai-models/python/src/coreai_models/model_registry.py` when updating the package catalog. Prefer iOS examples. Use macOS only when the registry has no iOS entry. +Source this list from `apple/coreai-models/python/src/coreai_models/model_registry.py` when updating the docs. Prefer iOS examples. Use macOS only when the registry has no iOS entry. + +This is not a package API. Apps should discover installed/downloaded Core AI bundles dynamically. | Feature | Starter model | Platform | Package status | Notes | | ------------------------------------------------------------ | ----------------------------- | ----------- | ------------------------------ | -------------------------------------------------------------- | From 7000133e958a0758be04091edbbdaabb7b0ebbfa Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:38:35 +0200 Subject: [PATCH 04/11] Simplify Core AI provider surface --- packages/core-ai/README.md | 8 + packages/core-ai/src/ai-sdk.ts | 286 +++++++++++++-- packages/core-ai/src/core.ts | 368 -------------------- packages/core-ai/src/index.ts | 19 +- website/src/docs/core-ai/api.md | 32 +- website/src/docs/core-ai/getting-started.md | 6 +- 6 files changed, 286 insertions(+), 433 deletions(-) delete mode 100644 packages/core-ai/src/core.ts diff --git a/packages/core-ai/README.md b/packages/core-ai/README.md index 4da031a9..096969aa 100644 --- a/packages/core-ai/README.md +++ b/packages/core-ai/README.md @@ -69,3 +69,11 @@ const session = await model.createSession({ const response = await session.respond('What is Qwen3?') ``` + +For lower-level calls, import `CoreAI` and call the native TurboModule directly: + +```ts +import { CoreAI, toNativeModelConfig } from '@react-native-ai/core-ai' + +await CoreAI.inspectModel(toNativeModelConfig(modelConfig)) +``` diff --git a/packages/core-ai/src/ai-sdk.ts b/packages/core-ai/src/ai-sdk.ts index ea31f21b..efaea3c3 100644 --- a/packages/core-ai/src/ai-sdk.ts +++ b/packages/core-ai/src/ai-sdk.ts @@ -7,29 +7,28 @@ import type { LanguageModelV3, LanguageModelV3CallOptions, LanguageModelV3FinishReason, - LanguageModelV3Prompt, LanguageModelV3StreamPart, TranscriptionModelV3, TranscriptionModelV3CallOptions, } from '@ai-sdk/provider' import { generateId } from '@ai-sdk/provider-utils' +import type { UnsafeObject } from 'react-native/Libraries/Types/CodegenTypes' -import { - coreAI, - CoreAIEmbeddingModel, - CoreAIImageModel, - CoreAILanguageModel, - CoreAITranscriptionModel, - prepareMessages, -} from './core' import NativeCoreAI from './NativeCoreAI' import type { + CoreAIEmbeddingResult, CoreAIGenerationOptions, CoreAIGenerationPart, + CoreAIImageGenerationOptions, + CoreAIImageGenerationResult, + CoreAILoadedModel, + CoreAIMessage, CoreAIModelConfig, + CoreAIModelInfo, CoreAIStreamCompleteEvent, CoreAIStreamErrorEvent, CoreAIStreamUpdateEvent, + CoreAITranscriptionResult, } from './types' import { toNativeModelConfig } from './types' @@ -38,9 +37,9 @@ export function createCoreAIProvider() { return provider.languageModel(config) } - provider.getCapabilities = () => coreAI.getCapabilities() + provider.getCapabilities = () => NativeCoreAI.getCapabilities() provider.languageModel = (config: CoreAIModelConfig) => { - return new CoreAILanguageModelAdapter(config) + return new CoreAILanguageModel(config) } provider.textEmbeddingModel = (config: CoreAIModelConfig) => { return new CoreAITextEmbeddingModel(config) @@ -50,32 +49,95 @@ export function createCoreAIProvider() { return new CoreAIImageGenerationModel(config) } provider.transcriptionModel = (config: CoreAIModelConfig) => { - return new CoreAITranscriptionAdapter(config) + return new CoreAITranscriptionModel(config) } return provider } -export const coreAIProvider = createCoreAIProvider() +export const coreAI = createCoreAIProvider() + +export interface CoreAILanguageSession { + sessionHandle: string + respond( + prompt: string, + options?: CoreAIGenerationOptions + ): Promise + stream( + prompt: string, + options?: CoreAIGenerationOptions + ): Promise> + close(): Promise +} -export class CoreAILanguageModelAdapter - extends CoreAILanguageModel - implements LanguageModelV3 -{ +export class CoreAILanguageModel implements LanguageModelV3 { readonly specificationVersion = 'v3' readonly supportedUrls = {} readonly provider = 'core-ai' readonly modelId: string + readonly config: CoreAIModelConfig + + private loadedModel?: CoreAILoadedModel constructor(config: CoreAIModelConfig) { - super({ ...config, task: 'language' }) + this.config = { ...config, task: 'language' } this.modelId = config.id } + get modelHandle() { + return this.loadedModel?.modelHandle + } + + async inspect(): Promise { + return NativeCoreAI.inspectModel( + toNativeModelConfig(this.config) + ) as Promise + } + + async prepare(options: UnsafeObject = {}): Promise { + const loadedModel = (await NativeCoreAI.loadModel( + toNativeModelConfig(this.config), + options + )) as CoreAILoadedModel + this.loadedModel = loadedModel + return loadedModel.info + } + + async specialize(options: UnsafeObject = {}): Promise { + return NativeCoreAI.specializeModel( + toNativeModelConfig(this.config), + options + ) as Promise + } + + async unload(): Promise { + if (!this.loadedModel) { + return + } + await NativeCoreAI.unloadModel(this.loadedModel.modelHandle) + this.loadedModel = undefined + } + + async remove(): Promise { + await NativeCoreAI.removeModel(toNativeModelConfig(this.config)) + this.loadedModel = undefined + } + + async createSession( + options: UnsafeObject = {} + ): Promise { + const model = await this.ensureLoaded() + const sessionHandle = await NativeCoreAI.createLanguageSession( + model.modelHandle, + options + ) + return createLanguageSession(sessionHandle) + } + async doGenerate(options: LanguageModelV3CallOptions) { const response = (await NativeCoreAI.generateText( toNativeModelConfig(this.config), - toCoreAIMessages(options.prompt), + prepareMessages(options.prompt), toCoreAIGenerationOptions(options) )) as CoreAIGenerationPart[] @@ -148,7 +210,7 @@ export class CoreAILanguageModelAdapter NativeCoreAI.streamText( streamId, toNativeModelConfig(this.config), - toCoreAIMessages(options.prompt), + prepareMessages(options.prompt), toCoreAIGenerationOptions(options) ) }, @@ -166,23 +228,39 @@ export class CoreAILanguageModelAdapter }, } } + + private async ensureLoaded(): Promise { + if (!this.loadedModel) { + await this.prepare() + } + return this.loadedModel! + } } -export class CoreAITextEmbeddingModel - extends CoreAIEmbeddingModel - implements EmbeddingModelV3 -{ +export class CoreAITextEmbeddingModel implements EmbeddingModelV3 { readonly specificationVersion = 'v3' readonly provider = 'core-ai' readonly modelId: string readonly maxEmbeddingsPerCall = Infinity readonly supportsParallelCalls = false + readonly config: CoreAIModelConfig constructor(config: CoreAIModelConfig) { - super({ ...config, task: 'embedding' }) + this.config = { ...config, task: 'embedding' } this.modelId = config.id } + async embed( + values: string[], + options: UnsafeObject = {} + ): Promise { + return NativeCoreAI.embed( + toNativeModelConfig(this.config), + values, + options + ) as Promise + } + async doEmbed( options: EmbeddingModelV3CallOptions ): Promise { @@ -194,20 +272,29 @@ export class CoreAITextEmbeddingModel } } -export class CoreAIImageGenerationModel - extends CoreAIImageModel - implements ImageModelV3 -{ +export class CoreAIImageGenerationModel implements ImageModelV3 { readonly specificationVersion = 'v3' readonly provider = 'core-ai' readonly modelId: string readonly maxImagesPerCall = 1 + readonly config: CoreAIModelConfig constructor(config: CoreAIModelConfig) { - super({ ...config, task: 'diffusion' }) + this.config = { ...config, task: 'diffusion' } this.modelId = config.id } + async generate( + prompt: string, + options: CoreAIImageGenerationOptions = {} + ): Promise { + return NativeCoreAI.generateImage( + toNativeModelConfig(this.config), + prompt, + options + ) as Promise + } + async doGenerate(options: ImageModelV3CallOptions) { const result = await this.generate(options.prompt ?? '', { n: options.n, @@ -237,19 +324,30 @@ export class CoreAIImageGenerationModel } } -export class CoreAITranscriptionAdapter - extends CoreAITranscriptionModel - implements TranscriptionModelV3 -{ +export class CoreAITranscriptionModel implements TranscriptionModelV3 { readonly specificationVersion = 'v3' readonly provider = 'core-ai' readonly modelId: string + readonly config: CoreAIModelConfig constructor(config: CoreAIModelConfig) { - super({ ...config, task: 'asr' }) + this.config = { ...config, task: 'asr' } this.modelId = config.id } + async transcribe( + audio: Uint8Array | string, + mediaType: string, + options: UnsafeObject = {} + ): Promise { + return NativeCoreAI.transcribe( + toNativeModelConfig(this.config), + toBase64(audio), + mediaType, + options + ) as Promise + } + async doGenerate(options: TranscriptionModelV3CallOptions) { const result = await this.transcribe(options.audio, options.mediaType, { ...(options.providerOptions?.['core-ai'] ?? {}), @@ -274,8 +372,108 @@ export class CoreAITranscriptionAdapter } } -function toCoreAIMessages(prompt: LanguageModelV3Prompt) { - return prepareMessages(prompt) +function createLanguageSession(sessionHandle: string): CoreAILanguageSession { + return { + sessionHandle, + respond(prompt, options = {}) { + return NativeCoreAI.respondToLanguageSession( + sessionHandle, + prompt, + options + ) as Promise + }, + stream(prompt, options = {}) { + return streamLanguageSession(sessionHandle, prompt, options) + }, + close() { + return NativeCoreAI.releaseLanguageSession(sessionHandle) + }, + } +} + +function streamLanguageSession( + sessionHandle: string, + prompt: string, + options: CoreAIGenerationOptions +): Promise> { + if (typeof ReadableStream === 'undefined') { + throw new Error( + 'ReadableStream is not available. Load a web stream polyfill before streaming Core AI responses.' + ) + } + + const streamId = generateId() + let listeners: { remove(): void }[] = [] + + const cleanup = () => { + listeners.forEach((listener) => listener.remove()) + listeners = [] + } + + const stream = new ReadableStream({ + start(controller) { + const updateListener = NativeCoreAI.onStreamUpdate( + (event: CoreAIStreamUpdateEvent) => { + if (event.streamId === streamId) { + controller.enqueue(event) + } + } + ) + const completeListener = NativeCoreAI.onStreamComplete( + (event: CoreAIStreamCompleteEvent) => { + if (event.streamId === streamId) { + cleanup() + controller.close() + } + } + ) + const errorListener = NativeCoreAI.onStreamError( + (event: CoreAIStreamErrorEvent) => { + if (event.streamId === streamId) { + cleanup() + controller.error(new Error(event.error)) + } + } + ) + + listeners = [updateListener, completeListener, errorListener] + NativeCoreAI.streamLanguageSession( + streamId, + sessionHandle, + prompt, + options + ) + }, + cancel() { + cleanup() + NativeCoreAI.cancelStream(streamId) + }, + }) + + return Promise.resolve(stream) +} + +export function prepareMessages(messages: unknown): CoreAIMessage[] { + if (!Array.isArray(messages)) { + return [] + } + + return messages.map((message: any): CoreAIMessage => { + const content = Array.isArray(message.content) + ? message.content.reduce((acc: string, part: any) => { + if (part.type === 'text') { + return acc + part.text + } + console.warn('Unsupported Core AI message content type:', part) + return acc + }, '') + : String(message.content ?? '') + + return { + role: message.role, + content, + } + }) } function toCoreAIGenerationOptions( @@ -346,3 +544,17 @@ function emptyUsage() { }, } } + +function toBase64(data: Uint8Array | string): string { + if (typeof data === 'string') { + return data + } + + let binary = '' + const chunkSize = 0x8000 + for (let i = 0; i < data.length; i += chunkSize) { + const chunk = data.subarray(i, i + chunkSize) + binary += String.fromCharCode(...chunk) + } + return btoa(binary) +} diff --git a/packages/core-ai/src/core.ts b/packages/core-ai/src/core.ts deleted file mode 100644 index c7bb4acf..00000000 --- a/packages/core-ai/src/core.ts +++ /dev/null @@ -1,368 +0,0 @@ -import { generateId } from '@ai-sdk/provider-utils' -import type { UnsafeObject } from 'react-native/Libraries/Types/CodegenTypes' - -import NativeCoreAI from './NativeCoreAI' -import type { - CoreAICapabilities, - CoreAIEmbeddingResult, - CoreAIGenerationOptions, - CoreAIGenerationPart, - CoreAIImageGenerationOptions, - CoreAIImageGenerationResult, - CoreAILoadedModel, - CoreAIMessage, - CoreAIModelConfig, - CoreAIModelInfo, - CoreAIModelTask, - CoreAIStreamCompleteEvent, - CoreAIStreamErrorEvent, - CoreAIStreamUpdateEvent, - CoreAITaskInput, - CoreAITaskResult, - CoreAITranscriptionResult, -} from './types' -import { toNativeModelConfig } from './types' - -export class CoreAIModel { - protected loadedModel?: CoreAILoadedModel - - constructor(readonly config: CoreAIModelConfig) {} - - get modelHandle() { - return this.loadedModel?.modelHandle - } - - async inspect(): Promise { - return NativeCoreAI.inspectModel( - toNativeModelConfig(this.config) - ) as Promise - } - - async prepare(options: UnsafeObject = {}): Promise { - const loadedModel = (await NativeCoreAI.loadModel( - toNativeModelConfig(this.config), - options - )) as CoreAILoadedModel - this.loadedModel = loadedModel - return loadedModel.info - } - - async specialize(options: UnsafeObject = {}): Promise { - return NativeCoreAI.specializeModel( - toNativeModelConfig(this.config), - options - ) as Promise - } - - async unload(): Promise { - if (!this.loadedModel) { - return - } - await NativeCoreAI.unloadModel(this.loadedModel.modelHandle) - this.loadedModel = undefined - } - - async remove(): Promise { - await NativeCoreAI.removeModel(toNativeModelConfig(this.config)) - this.loadedModel = undefined - } - - protected async ensureLoaded(): Promise { - if (!this.loadedModel) { - await this.prepare() - } - return this.loadedModel! - } -} - -export class CoreAILanguageSession { - constructor(readonly sessionHandle: string) {} - - async respond( - prompt: string, - options: CoreAIGenerationOptions = {} - ): Promise { - return NativeCoreAI.respondToLanguageSession( - this.sessionHandle, - prompt, - options - ) as Promise - } - - async stream( - prompt: string, - options: CoreAIGenerationOptions = {} - ): Promise> { - if (typeof ReadableStream === 'undefined') { - throw new Error( - 'ReadableStream is not available. Load a web stream polyfill before streaming Core AI responses.' - ) - } - - const streamId = generateId() - const sessionHandle = this.sessionHandle - let listeners: { remove(): void }[] = [] - - const cleanup = () => { - listeners.forEach((listener) => listener.remove()) - listeners = [] - } - - const stream = new ReadableStream({ - start(controller) { - const updateListener = NativeCoreAI.onStreamUpdate( - (event: CoreAIStreamUpdateEvent) => { - if (event.streamId === streamId) { - controller.enqueue(event) - } - } - ) - const completeListener = NativeCoreAI.onStreamComplete( - (event: CoreAIStreamCompleteEvent) => { - if (event.streamId === streamId) { - cleanup() - controller.close() - } - } - ) - const errorListener = NativeCoreAI.onStreamError( - (event: CoreAIStreamErrorEvent) => { - if (event.streamId === streamId) { - cleanup() - controller.error(new Error(event.error)) - } - } - ) - - listeners = [updateListener, completeListener, errorListener] - NativeCoreAI.streamLanguageSession( - streamId, - sessionHandle, - prompt, - options - ) - }, - cancel() { - cleanup() - NativeCoreAI.cancelStream(streamId) - }, - }) - - return stream - } - - async close(): Promise { - await NativeCoreAI.releaseLanguageSession(this.sessionHandle) - } -} - -export class CoreAILanguageModel extends CoreAIModel { - async createSession( - options: UnsafeObject = {} - ): Promise { - const model = await this.ensureLoaded() - const sessionHandle = await NativeCoreAI.createLanguageSession( - model.modelHandle, - options - ) - return new CoreAILanguageSession(sessionHandle) - } -} - -export class CoreAIEmbeddingModel extends CoreAIModel { - async embed( - values: string[], - options: UnsafeObject = {} - ): Promise { - return NativeCoreAI.embed( - toNativeModelConfig(this.config), - values, - options - ) as Promise - } -} - -export class CoreAITranscriptionModel extends CoreAIModel { - async transcribe( - audio: Uint8Array | string, - mediaType: string, - options: UnsafeObject = {} - ): Promise { - return NativeCoreAI.transcribe( - toNativeModelConfig(this.config), - toBase64(audio), - mediaType, - options - ) as Promise - } -} - -export class CoreAIImageModel extends CoreAIModel { - async generate( - prompt: string, - options: CoreAIImageGenerationOptions = {} - ): Promise { - return NativeCoreAI.generateImage( - toNativeModelConfig(this.config), - prompt, - options - ) as Promise - } -} - -class CoreAITaskModel extends CoreAIModel { - constructor( - config: CoreAIModelConfig, - private readonly task: CoreAIModelTask - ) { - super({ ...config, task: config.task ?? task }) - } - - async run( - input: CoreAITaskInput, - options: UnsafeObject = {} - ): Promise { - return NativeCoreAI.runTask( - this.task, - toNativeModelConfig(this.config), - input as UnsafeObject, - options - ) as Promise - } -} - -export class CoreAIRawModel extends CoreAIModel { - async load(options: UnsafeObject = {}) { - return this.prepare(options) - } - - async runFunction( - functionName: string, - inputs: UnsafeObject, - options: UnsafeObject = {} - ): Promise { - const model = await this.ensureLoaded() - return NativeCoreAI.runRawFunction( - model.modelHandle, - functionName, - inputs, - options - ) - } -} - -export const coreAI = { - getCapabilities(): Promise { - return NativeCoreAI.getCapabilities() as Promise - }, - - languageModel(config: CoreAIModelConfig) { - return new CoreAILanguageModel({ ...config, task: 'language' }) - }, - - embeddingModel(config: CoreAIModelConfig) { - return new CoreAIEmbeddingModel({ ...config, task: 'embedding' }) - }, - - transcriptionModel(config: CoreAIModelConfig) { - return new CoreAITranscriptionModel({ ...config, task: 'asr' }) - }, - - imageModel(config: CoreAIModelConfig) { - return new CoreAIImageModel({ ...config, task: 'diffusion' }) - }, - - segmenter(config: CoreAIModelConfig) { - return new CoreAITaskModel(config, 'segmentation') - }, - - objectDetector(config: CoreAIModelConfig) { - return new CoreAITaskModel(config, 'object-detection') - }, - - depthEstimator(config: CoreAIModelConfig) { - return new CoreAITaskModel(config, 'depth') - }, - - superResolution(config: CoreAIModelConfig) { - return new CoreAITaskModel(config, 'super-resolution') - }, - - classifier(config: CoreAIModelConfig) { - return new CoreAITaskModel(config, 'classification') - }, - - models: { - inspect(config: CoreAIModelConfig) { - return NativeCoreAI.inspectModel( - toNativeModelConfig(config) - ) as Promise - }, - specialize(config: CoreAIModelConfig, options: UnsafeObject = {}) { - return NativeCoreAI.specializeModel( - toNativeModelConfig(config), - options - ) as Promise - }, - remove(config: CoreAIModelConfig) { - return NativeCoreAI.removeModel(toNativeModelConfig(config)) - }, - }, - - embeddings: { - embed( - config: CoreAIModelConfig, - values: string[], - options: UnsafeObject = {} - ) { - return NativeCoreAI.embed( - toNativeModelConfig({ ...config, task: 'embedding' }), - values, - options - ) as Promise - }, - }, - - unstable: { - loadModel(config: CoreAIModelConfig) { - return new CoreAIRawModel({ ...config, task: 'raw' }) - }, - }, -} - -function toBase64(data: Uint8Array | string): string { - if (typeof data === 'string') { - return data - } - - let binary = '' - const chunkSize = 0x8000 - for (let i = 0; i < data.length; i += chunkSize) { - const chunk = data.subarray(i, i + chunkSize) - binary += String.fromCharCode(...chunk) - } - return btoa(binary) -} - -export function prepareMessages(messages: unknown): CoreAIMessage[] { - if (!Array.isArray(messages)) { - return [] - } - - return messages.map((message: any): CoreAIMessage => { - const content = Array.isArray(message.content) - ? message.content.reduce((acc: string, part: any) => { - if (part.type === 'text') { - return acc + part.text - } - console.warn('Unsupported Core AI message content type:', part) - return acc - }, '') - : String(message.content ?? '') - - return { - role: message.role, - content, - } - }) -} diff --git a/packages/core-ai/src/index.ts b/packages/core-ai/src/index.ts index 3c124688..20aee593 100644 --- a/packages/core-ai/src/index.ts +++ b/packages/core-ai/src/index.ts @@ -1,21 +1,13 @@ +export type { CoreAILanguageSession } from './ai-sdk' export { + coreAI, CoreAIImageGenerationModel, - CoreAILanguageModelAdapter, - coreAIProvider, + CoreAILanguageModel, CoreAITextEmbeddingModel, - CoreAITranscriptionAdapter, + CoreAITranscriptionModel, createCoreAIProvider, } from './ai-sdk' -export { - coreAI, - CoreAIEmbeddingModel, - CoreAIImageModel, - CoreAILanguageModel, - CoreAILanguageSession, - CoreAIModel, - CoreAIRawModel, - CoreAITranscriptionModel, -} from './core' +export { default as CoreAI } from './NativeCoreAI' export type { CoreAICapabilities, CoreAIEmbeddingResult, @@ -34,3 +26,4 @@ export type { CoreAITaskResult, CoreAITranscriptionResult, } from './types' +export { toNativeModelConfig } from './types' diff --git a/website/src/docs/core-ai/api.md b/website/src/docs/core-ai/api.md index 4abb2195..f72befdd 100644 --- a/website/src/docs/core-ai/api.md +++ b/website/src/docs/core-ai/api.md @@ -38,13 +38,21 @@ Reranking, speech generation, and video generation are intentionally not exposed Use direct APIs when Core AI has no AI SDK equivalent or when you need Core AI-specific lifecycle control: ```typescript -const model = coreAI.languageModel({ +import { CoreAI, coreAI, toNativeModelConfig } from '@react-native-ai/core-ai' + +const config = { id: 'qwen3-0.6b', source: { type: 'file', uri: qwen3ModelDirectory }, variant: 'iOS', +} as const + +await CoreAI.inspectModel(toNativeModelConfig({ ...config, task: 'language' })) + +const model = coreAI.languageModel({ + ...config, }) -await model.prepare({ specialize: true }) +await model.prepare() const session = await model.createSession({ instructions: 'Keep answers short.', @@ -55,16 +63,16 @@ await session.respond('Create a vocabulary card for "flower".') Native-only API status: -| Capability | API | Status | Why native-only | -| -------------------- | -------------------------------- | -------------------------- | -------------------------------------------------------------------------- | -| Model inspection | `coreAI.models.inspect(...)` | Implemented | AI SDK does not manage Core AI asset metadata. | -| Specialization | `coreAI.models.specialize(...)` | Metadata stub | Core AI-specific preparation step. | -| Persistent sessions | `model.createSession(...)` | Implemented for LLMs | AI SDK owns message history; Core AI sessions own native transcript state. | -| Segmentation | `coreAI.segmenter(...)` | Implemented | AI SDK image APIs do not model segmentation masks. | -| Object detection | `coreAI.objectDetector(...)` | Implemented | AI SDK has no detection primitive. | -| Depth estimation | `coreAI.depthEstimator(...)` | Explicit unsupported error | AI SDK has no depth primitive. | -| Super-resolution | `coreAI.superResolution(...)` | Explicit unsupported error | AI SDK has no image-to-image upscaling primitive. | -| Raw `.aimodel` calls | `coreAI.unstable.loadModel(...)` | Explicit unsupported error | Raw tensor APIs need Core AI-specific ownership and performance rules. | +| Capability | API | Status | Why native-only | +| -------------------- | ----------------------------------------- | -------------------------- | -------------------------------------------------------------------------- | +| Model inspection | `CoreAI.inspectModel(...)` | Implemented | AI SDK does not manage Core AI asset metadata. | +| Specialization | `CoreAI.specializeModel(...)` | Metadata stub | Core AI-specific preparation step. | +| Persistent sessions | `model.createSession(...)` | Implemented for LLMs | AI SDK owns message history; Core AI sessions own native transcript state. | +| Segmentation | `CoreAI.runTask('segmentation', ...)` | Implemented | AI SDK image APIs do not model segmentation masks. | +| Object detection | `CoreAI.runTask('object-detection', ...)` | Implemented | AI SDK has no detection primitive. | +| Depth estimation | `CoreAI.runTask('depth', ...)` | Explicit unsupported error | AI SDK has no depth primitive. | +| Super-resolution | `CoreAI.runTask('super-resolution', ...)` | Explicit unsupported error | AI SDK has no image-to-image upscaling primitive. | +| Raw `.aimodel` calls | `CoreAI.runRawFunction(...)` | Explicit unsupported error | Raw tensor APIs need Core AI-specific ownership and performance rules. | ## Starter Models diff --git a/website/src/docs/core-ai/getting-started.md b/website/src/docs/core-ai/getting-started.md index de21ddc7..dc00554b 100644 --- a/website/src/docs/core-ai/getting-started.md +++ b/website/src/docs/core-ai/getting-started.md @@ -30,8 +30,8 @@ Add `https://github.com/apple/coreai-models` to the iOS app as a Swift Package d - `CoreAILM` for `coreAI.languageModel(...)` and sessions - `CoreAIDiffusion` for `coreAI.imageModel(...)` -- `CoreAISegmentation` for `coreAI.segmenter(...)` -- `CoreAIObjectDetection` for `coreAI.objectDetector(...)` +- `CoreAISegmentation` for `CoreAI.runTask('segmentation', ...)` +- `CoreAIObjectDetection` for `CoreAI.runTask('object-detection', ...)` React Native installs this package through CocoaPods, but CocoaPods will not automatically link those Swift Package products into your app target. Treat SPM as a host-app integration step. @@ -39,7 +39,7 @@ For Expo, use a config plugin that patches the generated Xcode project with `XCR The native module checks for missing products and returns clear setup errors instead of failing silently. -The current native implementation wires language sessions, diffusion image generation, image segmentation, and object detection. Catalog entries for embeddings, transcription, depth, super-resolution, and classification are included so apps can discover Apple starter models, but those task runners still need follow-up native wrappers. +The current native implementation wires language sessions, diffusion image generation, image segmentation, and object detection. Embeddings, transcription, depth, super-resolution, and classification are listed as starter-model docs only; those task runners still need follow-up native wrappers. ## First iOS Model From b4bd4022c118b789eac77b3626a95d9f20d54726 Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:41:00 +0200 Subject: [PATCH 05/11] Document Core AI SPM verification status --- packages/core-ai/README.md | 4 ++++ website/src/docs/core-ai/getting-started.md | 2 ++ 2 files changed, 6 insertions(+) diff --git a/packages/core-ai/README.md b/packages/core-ai/README.md index 096969aa..32565742 100644 --- a/packages/core-ai/README.md +++ b/packages/core-ai/README.md @@ -30,6 +30,10 @@ real Core AI model/runtime exists. Add `https://github.com/apple/coreai-models` as a Swift Package dependency in the host iOS app and link the products used by your wrappers: +This is the intended integration path, but it still needs full verification in +an Xcode 27 / iOS 27 host app with Apple Core AI products linked into the app +target. + - `CoreAILM` for language models and sessions - `CoreAIDiffusion` for image generation - `CoreAISegmentation` for segmentation diff --git a/website/src/docs/core-ai/getting-started.md b/website/src/docs/core-ai/getting-started.md index dc00554b..289e3b1e 100644 --- a/website/src/docs/core-ai/getting-started.md +++ b/website/src/docs/core-ai/getting-started.md @@ -28,6 +28,8 @@ npm install ai Add `https://github.com/apple/coreai-models` to the iOS app as a Swift Package dependency, then link the products you use: +This is the intended integration path, but it still needs full verification in an Xcode 27 / iOS 27 host app with Apple Core AI products linked into the app target. + - `CoreAILM` for `coreAI.languageModel(...)` and sessions - `CoreAIDiffusion` for `coreAI.imageModel(...)` - `CoreAISegmentation` for `CoreAI.runTask('segmentation', ...)` From 33e685330e5fdb9d4528168f00a568f32c1f1301 Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:42:49 +0200 Subject: [PATCH 06/11] Add Core AI Expo plugin product options --- packages/core-ai/README.md | 21 +- packages/core-ai/app.plugin.js | 1 + packages/core-ai/package.json | 2 + packages/core-ai/src/expo-plugin.ts | 257 ++++++++++++++++++++ website/src/docs/core-ai/getting-started.md | 19 +- 5 files changed, 297 insertions(+), 3 deletions(-) create mode 100644 packages/core-ai/app.plugin.js create mode 100644 packages/core-ai/src/expo-plugin.ts diff --git a/packages/core-ai/README.md b/packages/core-ai/README.md index 32565742..6bfb8367 100644 --- a/packages/core-ai/README.md +++ b/packages/core-ai/README.md @@ -41,8 +41,25 @@ target. React Native installs this package through CocoaPods, but Apple's helpers are Swift Package products. The app target still needs those SPM products linked. -For Expo, use a config plugin to patch the generated Xcode project. For bare -React Native, add the Swift Package in Xcode or with a project patching script. +For Expo, add the config plugin and choose the products your app uses: + +```json +{ + "expo": { + "plugins": [ + [ + "@react-native-ai/core-ai", + { + "products": ["CoreAILM"] + } + ] + ] + } +} +``` + +For bare React Native, add the Swift Package in Xcode or with a project patching +script. ## Usage diff --git a/packages/core-ai/app.plugin.js b/packages/core-ai/app.plugin.js new file mode 100644 index 00000000..aef8b9d8 --- /dev/null +++ b/packages/core-ai/app.plugin.js @@ -0,0 +1 @@ +module.exports = require('./lib/commonjs/expo-plugin') diff --git a/packages/core-ai/package.json b/packages/core-ai/package.json index 65ab5468..043e5d11 100644 --- a/packages/core-ai/package.json +++ b/packages/core-ai/package.json @@ -12,6 +12,7 @@ "lib", "ios", "*.podspec", + "app.plugin.js", "!ios/build", "!**/__tests__", "!**/__fixtures__", @@ -32,6 +33,7 @@ "sdk", "vercel", "expo", + "config-plugin", "spm" ], "repository": { diff --git a/packages/core-ai/src/expo-plugin.ts b/packages/core-ai/src/expo-plugin.ts new file mode 100644 index 00000000..71891d56 --- /dev/null +++ b/packages/core-ai/src/expo-plugin.ts @@ -0,0 +1,257 @@ +import { + ConfigPlugin, + createRunOncePlugin, + IOSConfig, + WarningAggregator, + withDangerousMod, +} from 'expo/config-plugins' +import { existsSync, readFileSync, writeFileSync } from 'fs' + +const pkg = require('../../package.json') + +export type CoreAISwiftPackageProduct = + | 'CoreAILM' + | 'CoreAIDiffusion' + | 'CoreAISegmentation' + | 'CoreAIObjectDetection' + +export type CoreAIExpoPluginOptions = { + products?: CoreAISwiftPackageProduct[] + targetName?: string +} + +const CORE_AI_PACKAGE_URL = 'https://github.com/apple/coreai-models' +const CORE_AI_PACKAGE_NAME = 'coreai-models' +const DEFAULT_PRODUCTS: CoreAISwiftPackageProduct[] = ['CoreAILM'] + +const withCoreAISwiftPackage: ConfigPlugin = ( + config, + options = {} +) => { + return withDangerousMod(config, [ + 'ios', + (config) => { + const projectPath = IOSConfig.Paths.getPBXProjectPath( + config.modRequest.projectRoot + ) + + if (!existsSync(projectPath)) { + WarningAggregator.addWarningIOS( + 'plugins', + `Could not find project.pbxproj to link ${CORE_AI_PACKAGE_URL}. Add the Swift Package manually.` + ) + return config + } + + const contents = readFileSync(projectPath, 'utf8') + const nextContents = addSwiftPackageProducts(contents, { + products: options?.products ?? DEFAULT_PRODUCTS, + targetName: options?.targetName ?? config.modRequest.projectName, + }) + + if (contents !== nextContents) { + writeFileSync(projectPath, nextContents) + } + + return config + }, + ]) +} + +type AddSwiftPackageOptions = { + products: CoreAISwiftPackageProduct[] + targetName?: string +} + +function addSwiftPackageProducts( + contents: string, + options: AddSwiftPackageOptions +) { + const products = uniqueProducts(options.products) + const projectId = findProjectId(contents) + const target = findApplicationTarget(contents, options.targetName) + const frameworksPhaseId = findFrameworksBuildPhaseId(target.block) + + if (!projectId || !target.id || !frameworksPhaseId) { + WarningAggregator.addWarningIOS( + 'plugins', + `Could not safely link ${CORE_AI_PACKAGE_URL}. Add the Swift Package products manually in Xcode.` + ) + return contents + } + + let nextContents = contents + const packageId = stablePbxId('CoreAI Swift package') + + nextContents = ensureSectionEntry( + nextContents, + 'XCRemoteSwiftPackageReference', + `${packageId} /* XCRemoteSwiftPackageReference "${CORE_AI_PACKAGE_NAME}" */ = { + isa = XCRemoteSwiftPackageReference; + repositoryURL = "${CORE_AI_PACKAGE_URL}"; + requirement = { + branch = main; + kind = branch; + }; + };` + ) + + nextContents = ensureListItem( + nextContents, + projectId, + 'packageReferences', + `${packageId} /* XCRemoteSwiftPackageReference "${CORE_AI_PACKAGE_NAME}" */` + ) + + for (const product of products) { + const productId = stablePbxId(`CoreAI product ${product}`) + const buildFileId = stablePbxId(`CoreAI build file ${product}`) + + nextContents = ensureSectionEntry( + nextContents, + 'XCSwiftPackageProductDependency', + `${productId} /* ${product} */ = { + isa = XCSwiftPackageProductDependency; + package = ${packageId} /* XCRemoteSwiftPackageReference "${CORE_AI_PACKAGE_NAME}" */; + productName = ${product}; + };` + ) + + nextContents = ensureSectionEntry( + nextContents, + 'PBXBuildFile', + `${buildFileId} /* ${product} in Frameworks */ = {isa = PBXBuildFile; productRef = ${productId} /* ${product} */; };` + ) + + nextContents = ensureListItem( + nextContents, + target.id, + 'packageProductDependencies', + `${productId} /* ${product} */` + ) + + nextContents = ensureListItem( + nextContents, + frameworksPhaseId, + 'files', + `${buildFileId} /* ${product} in Frameworks */` + ) + } + + return nextContents +} + +function uniqueProducts(products: CoreAISwiftPackageProduct[]) { + return [...new Set(products)] +} + +function findProjectId(contents: string) { + const projectObject = contents.match( + /([A-F0-9]{24}) \/\* Project object \*\/ = {\n\s+isa = PBXProject;/ + ) + if (projectObject) { + return projectObject[1] + } + + return contents.match( + /([A-F0-9]{24}) \/\* .* \*\/ = {\n\s+isa = PBXProject;/ + )?.[1] +} + +function findApplicationTarget(contents: string, targetName?: string) { + const targetRegex = + /([A-F0-9]{24}) \/\* ([^*]+) \*\/ = {\n\s+isa = PBXNativeTarget;[\s\S]*?\n\t\t};/g + const targets = Array.from(contents.matchAll(targetRegex)).map((match) => ({ + id: match[1], + name: match[2].trim(), + block: match[0], + })) + + const namedTarget = targetName + ? targets.find((target) => target.name === targetName) + : undefined + + return ( + namedTarget ?? + targets.find((target) => + target.block.includes( + 'productType = "com.apple.product-type.application"' + ) + ) ?? + targets[0] ?? { id: undefined, block: '' } + ) +} + +function findFrameworksBuildPhaseId(targetBlock: string) { + return targetBlock.match(/([A-F0-9]{24}) \/\* Frameworks \*\//)?.[1] +} + +function ensureSectionEntry( + contents: string, + sectionName: string, + entry: string +) { + const id = entry.slice(0, 24) + + if (contents.includes(id)) { + return contents + } + + const sectionStart = `/* Begin ${sectionName} section */` + const sectionEnd = `/* End ${sectionName} section */` + + if (contents.includes(sectionStart)) { + return contents.replace(sectionEnd, `\t\t${entry}\n${sectionEnd}`) + } + + return contents.replace( + '/* Begin XCBuildConfiguration section */', + `${sectionStart}\n\t\t${entry}\n${sectionEnd}\n\n/* Begin XCBuildConfiguration section */` + ) +} + +function ensureListItem( + contents: string, + objectId: string, + propertyName: string, + item: string +) { + const objectRegex = new RegExp( + `(${objectId} /\\* [^*]+ \\*/ = \\{[\\s\\S]*?\\n\\t\\t\\};)` + ) + const objectMatch = contents.match(objectRegex) + + if (!objectMatch || objectMatch[1].includes(item)) { + return contents + } + + const objectBlock = objectMatch[1] + const listRegex = new RegExp( + `(\\n\\t\\t\\t${propertyName} = \\(\\n)([\\s\\S]*?)(\\t\\t\\t\\);)` + ) + const listMatch = objectBlock.match(listRegex) + const nextBlock = listMatch + ? objectBlock.replace(listRegex, `$1$2\t\t\t\t${item},\n$3`) + : objectBlock.replace( + /\n\t\t};$/, + `\n\t\t\t${propertyName} = (\n\t\t\t\t${item},\n\t\t\t);\n\t\t};` + ) + + return contents.replace(objectBlock, nextBlock) +} + +function stablePbxId(value: string) { + let hash = 0 + for (let index = 0; index < value.length; index += 1) { + hash = (hash * 31 + value.charCodeAt(index)) >>> 0 + } + return hash.toString(16).toUpperCase().padStart(24, '0').slice(0, 24) +} + +export { addSwiftPackageProducts } + +export default createRunOncePlugin( + withCoreAISwiftPackage, + pkg.name, + pkg.version +) diff --git a/website/src/docs/core-ai/getting-started.md b/website/src/docs/core-ai/getting-started.md index 289e3b1e..f3d5cd6d 100644 --- a/website/src/docs/core-ai/getting-started.md +++ b/website/src/docs/core-ai/getting-started.md @@ -37,7 +37,24 @@ This is the intended integration path, but it still needs full verification in a React Native installs this package through CocoaPods, but CocoaPods will not automatically link those Swift Package products into your app target. Treat SPM as a host-app integration step. -For Expo, use a config plugin that patches the generated Xcode project with `XCRemoteSwiftPackageReference` and links the selected products. For bare React Native, add the Swift Package in Xcode or use a project patching script that modifies `project.pbxproj` predictably. +For Expo, add the config plugin and choose the products your app uses: + +```json +{ + "expo": { + "plugins": [ + [ + "@react-native-ai/core-ai", + { + "products": ["CoreAILM"] + } + ] + ] + } +} +``` + +The plugin patches the generated Xcode project with `XCRemoteSwiftPackageReference` and links the selected products. For bare React Native, add the Swift Package in Xcode or use a project patching script that modifies `project.pbxproj` predictably. The native module checks for missing products and returns clear setup errors instead of failing silently. From 73ed5a072096860e52b312a43a6f2607142119fd Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:43:17 +0200 Subject: [PATCH 07/11] Default Core AI Expo plugin to all products --- packages/core-ai/README.md | 20 +++++++++++++------- packages/core-ai/src/expo-plugin.ts | 7 ++++++- website/src/docs/core-ai/getting-started.md | 21 ++++++++++++--------- 3 files changed, 31 insertions(+), 17 deletions(-) diff --git a/packages/core-ai/README.md b/packages/core-ai/README.md index 6bfb8367..212ddcce 100644 --- a/packages/core-ai/README.md +++ b/packages/core-ai/README.md @@ -41,18 +41,24 @@ target. React Native installs this package through CocoaPods, but Apple's helpers are Swift Package products. The app target still needs those SPM products linked. -For Expo, add the config plugin and choose the products your app uses: +For Expo, add the config plugin. It links all known Apple Core AI Swift Package +products by default: + +```json +{ + "expo": { + "plugins": ["@react-native-ai/core-ai"] + } +} +``` + +Pass `products` only when you want to link a smaller set: ```json { "expo": { "plugins": [ - [ - "@react-native-ai/core-ai", - { - "products": ["CoreAILM"] - } - ] + ["@react-native-ai/core-ai", { "products": ["CoreAILM"] }] ] } } diff --git a/packages/core-ai/src/expo-plugin.ts b/packages/core-ai/src/expo-plugin.ts index 71891d56..f01fa6c2 100644 --- a/packages/core-ai/src/expo-plugin.ts +++ b/packages/core-ai/src/expo-plugin.ts @@ -22,7 +22,12 @@ export type CoreAIExpoPluginOptions = { const CORE_AI_PACKAGE_URL = 'https://github.com/apple/coreai-models' const CORE_AI_PACKAGE_NAME = 'coreai-models' -const DEFAULT_PRODUCTS: CoreAISwiftPackageProduct[] = ['CoreAILM'] +const DEFAULT_PRODUCTS: CoreAISwiftPackageProduct[] = [ + 'CoreAILM', + 'CoreAIDiffusion', + 'CoreAISegmentation', + 'CoreAIObjectDetection', +] const withCoreAISwiftPackage: ConfigPlugin = ( config, diff --git a/website/src/docs/core-ai/getting-started.md b/website/src/docs/core-ai/getting-started.md index f3d5cd6d..35ed2fe6 100644 --- a/website/src/docs/core-ai/getting-started.md +++ b/website/src/docs/core-ai/getting-started.md @@ -37,19 +37,22 @@ This is the intended integration path, but it still needs full verification in a React Native installs this package through CocoaPods, but CocoaPods will not automatically link those Swift Package products into your app target. Treat SPM as a host-app integration step. -For Expo, add the config plugin and choose the products your app uses: +For Expo, add the config plugin. It links all known Apple Core AI Swift Package products by default: ```json { "expo": { - "plugins": [ - [ - "@react-native-ai/core-ai", - { - "products": ["CoreAILM"] - } - ] - ] + "plugins": ["@react-native-ai/core-ai"] + } +} +``` + +Pass `products` only when you want to link a smaller set: + +```json +{ + "expo": { + "plugins": [["@react-native-ai/core-ai", { "products": ["CoreAILM"] }]] } } ``` From 942e8db64ec92bbc1ef8754a5fa7caf554aec775 Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:43:29 +0200 Subject: [PATCH 08/11] Remove starter models preface --- website/src/docs/core-ai/models.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/website/src/docs/core-ai/models.md b/website/src/docs/core-ai/models.md index 325e76d1..0b8a4d5d 100644 --- a/website/src/docs/core-ai/models.md +++ b/website/src/docs/core-ai/models.md @@ -1,9 +1,5 @@ # Starter Models -Source this list from `apple/coreai-models/python/src/coreai_models/model_registry.py` when updating the docs. Prefer iOS examples. Use macOS only when the registry has no iOS entry. - -This is not a package API. Apps should discover installed/downloaded Core AI bundles dynamically. - | Feature | Starter model | Platform | Package status | Notes | | ------------------------------------------------------------ | ----------------------------- | ----------- | ------------------------------ | -------------------------------------------------------------- | | Language sessions, text generation, structured output, tools | `qwen3-0.6b` | iOS + macOS | Implemented | Best first Core AI session demo. | From 3047107ff0230809cf43e1272ea5565785d8a612 Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:44:47 +0200 Subject: [PATCH 09/11] Group Core AI starter models by type --- website/src/docs/core-ai/getting-started.md | 2 +- website/src/docs/core-ai/models.md | 32 ++++++--------------- website/src/docs/index.md | 6 ++-- 3 files changed, 13 insertions(+), 27 deletions(-) diff --git a/website/src/docs/core-ai/getting-started.md b/website/src/docs/core-ai/getting-started.md index 35ed2fe6..498d90a5 100644 --- a/website/src/docs/core-ai/getting-started.md +++ b/website/src/docs/core-ai/getting-started.md @@ -61,7 +61,7 @@ The plugin patches the generated Xcode project with `XCRemoteSwiftPackageReferen The native module checks for missing products and returns clear setup errors instead of failing silently. -The current native implementation wires language sessions, diffusion image generation, image segmentation, and object detection. Embeddings, transcription, depth, super-resolution, and classification are listed as starter-model docs only; those task runners still need follow-up native wrappers. +The current native implementation wires language sessions, diffusion image generation, image segmentation, and object detection. Starter-model docs only list those exposed feature families; embeddings, transcription, depth, super-resolution, and classification still need follow-up native wrappers. ## First iOS Model diff --git a/website/src/docs/core-ai/models.md b/website/src/docs/core-ai/models.md index 0b8a4d5d..6edda34c 100644 --- a/website/src/docs/core-ai/models.md +++ b/website/src/docs/core-ai/models.md @@ -1,29 +1,15 @@ # Starter Models -| Feature | Starter model | Platform | Package status | Notes | -| ------------------------------------------------------------ | ----------------------------- | ----------- | ------------------------------ | -------------------------------------------------------------- | -| Language sessions, text generation, structured output, tools | `qwen3-0.6b` | iOS + macOS | Implemented | Best first Core AI session demo. | -| Better iOS language quality | `qwen2.5-1.5b-instruct` | iOS + macOS | Implemented | Good second example after the session API works. | -| Larger iOS language validation | `qwen3-4b` | iOS + macOS | Implemented | Useful for memory and specialization testing. | -| macOS-only LLM validation | `gpt-oss-20b`, `gemma3-4b-it` | macOS | Implemented when running macOS | Not iOS demo candidates in the current registry. | -| Text/image embeddings | `clip-vit-b32` | iOS + macOS | Export starter only | Needs a CLIP embedding runner before the AI SDK adapter works. | -| Audio/text embeddings | `clap-htsat` | iOS + macOS | Export starter only | Needs a CLAP embedding runner. | -| Reranking | None | None | Not exposed | Do not ship a fake provider until there is a real reranker. | -| Image generation | `sd-1.5` | iOS + macOS | Implemented | Lowest-friction diffusion starter. | -| Modern image generation | `flux2-klein-4b` | iOS + macOS | Implemented | Later demo; heavier setup and memory profile. | -| Transcription smoke test | `wav2vec2-base` | iOS + macOS | Export starter only | Needs an ASR output decoder before the AI SDK adapter works. | -| Transcription API parity | `whisper-large-v3-turbo` | iOS + macOS | Export starter only | Better match for AI SDK transcription expectations. | -| Speech generation | None | None | Not exposed | Keep using `@react-native-ai/apple` for `AVSpeechSynthesizer`. | -| Video generation | None | None | Not exposed | No Core AI video-generation starter in the current registry. | -| Image segmentation | `efficient-sam-vitt` | iOS + macOS | Implemented | Least-gated segmentation starter. | -| Promptable segmentation validation | `sam3` | iOS + macOS | Implemented | Gated on Hugging Face; not first-run docs. | -| Object detection | `yolos-tiny` | iOS + macOS | Implemented | Smaller detection starter. | -| Object detection quality check | `yolos-base` | iOS + macOS | Implemented | Larger detector after wrapper is stable. | -| Depth estimation | `depth-anything-3-small` | macOS | Export starter only | macOS-only in the current Apple registry. | -| Super-resolution | `edsr-x2` | iOS + macOS | Export starter only | Needs an image-to-image result wrapper. | -| Raw model example | `pvt-v2-b0` | iOS + macOS | Load/inspect only | Good for validating raw `.aimodel` loading. | +| Type | Models | Platform | Model file | +| ------------------ | ------------------------------------------- | ----------- | ---------- | +| Language | `qwen3-0.6b` | iOS + macOS | TBD | +| Language | `qwen2.5-1.5b-instruct`, `qwen3-4b` | iOS + macOS | TBD | +| Language | `gpt-oss-20b`, `gemma3-4b-it` | macOS | TBD | +| Image generation | `sd-1.5`, `flux2-klein-4b` | iOS + macOS | TBD | +| Image segmentation | `efficient-sam-vitt`, `sam3` | iOS + macOS | TBD | +| Object detection | `yolos-tiny`, `yolos-base` | iOS + macOS | TBD | -Example exports: +Example exports for listed models: ```bash uv run coreai.llm.export Qwen/Qwen3-0.6B --platform iOS --output-dir ./models diff --git a/website/src/docs/index.md b/website/src/docs/index.md index 420a2f19..9f37fdb7 100644 --- a/website/src/docs/index.md +++ b/website/src/docs/index.md @@ -35,9 +35,9 @@ Production-ready with instant availability on supported iOS devices. Run app-provided Core AI model bundles through `@react-native-ai/core-ai`: - **Language Sessions** - Custom Core AI language models with persistent native sessions -- **AI SDK Adapters** - Language, embeddings, image generation, and transcription where AI SDK has matching APIs -- **Native-only Tasks** - Segmentation, object detection, depth, super-resolution, model inspection, specialization, and raw `.aimodel` calls -- **Model Catalog** - Lightweight metadata from Apple’s `coreai-models` registry so apps can pick iOS-safe starters +- **AI SDK Adapters** - Language and image generation today, with embedding and transcription surfaces kept explicit until native runners land +- **Native-only Tasks** - Segmentation, object detection, model inspection, specialization, and model lifecycle calls +- **Starter Models** - iOS-first examples sourced from Apple’s `coreai-models` registry, kept as docs/example data rather than a package catalog Requires exported Core AI assets and Apple’s `coreai-models` Swift Package products linked into the host app target. From 66033ce30d1cef3a17615b1e57aee4421c947ffc Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 00:52:50 +0200 Subject: [PATCH 10/11] Add Core AI starter model artifact links --- website/src/docs/core-ai/models.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/website/src/docs/core-ai/models.md b/website/src/docs/core-ai/models.md index 6edda34c..851de553 100644 --- a/website/src/docs/core-ai/models.md +++ b/website/src/docs/core-ai/models.md @@ -1,13 +1,20 @@ # Starter Models -| Type | Models | Platform | Model file | -| ------------------ | ------------------------------------------- | ----------- | ---------- | -| Language | `qwen3-0.6b` | iOS + macOS | TBD | -| Language | `qwen2.5-1.5b-instruct`, `qwen3-4b` | iOS + macOS | TBD | -| Language | `gpt-oss-20b`, `gemma3-4b-it` | macOS | TBD | -| Image generation | `sd-1.5`, `flux2-klein-4b` | iOS + macOS | TBD | -| Image segmentation | `efficient-sam-vitt`, `sam3` | iOS + macOS | TBD | -| Object detection | `yolos-tiny`, `yolos-base` | iOS + macOS | TBD | +Apple's repo publishes export recipes, not official ready-made `.aimodel` downloads. This table links the prebuilt Core AI bundles we found online; remaining starter models should use the Apple export recipe until we publish our own hosted bundles, for example from a Callstack Hugging Face org. + +| Type | Model | Platform | Prebuilt Core AI bundle | Apple export recipe | +| ------------------ | -------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| Language | [`qwen3-0.6b`](https://huggingface.co/Qwen/Qwen3-0.6B) | iOS + macOS | [iOS `.aimodel`](https://huggingface.co/mlboydaisuke/qwen3-0.6b-CoreAI-official/tree/main/ios/qwen3_0_6b_mixed_4bit_8bit_static.aimodel), [macOS `.aimodel`](https://huggingface.co/mlboydaisuke/qwen3-0.6b-CoreAI-official/tree/main/macos/qwen3_0_6b_4bit_dynamic.aimodel) | [Qwen3 recipe](https://github.com/apple/coreai-models/tree/main/models/qwen3) | +| Language | [`qwen2.5-1.5b-instruct`](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) | iOS + macOS | Planned Callstack-hosted bundle | [Qwen2.5 recipe](https://github.com/apple/coreai-models/tree/main/models/qwen2) | +| Language | [`qwen3-4b`](https://huggingface.co/Qwen/Qwen3-4B) | iOS + macOS | [iOS `.aimodel`](https://huggingface.co/mlboydaisuke/qwen3-4b-CoreAI-official/tree/main/ios/qwen3_4b_mixed_4bit_8bit_static.aimodel), [macOS `.aimodel`](https://huggingface.co/mlboydaisuke/qwen3-4b-CoreAI-official/tree/main/macos/qwen3_4b_4bit_dynamic.aimodel) | [Qwen3 recipe](https://github.com/apple/coreai-models/tree/main/models/qwen3) | +| Language | [`gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b) | macOS | [macOS `.aimodel`](https://huggingface.co/mlboydaisuke/gpt-oss-20b-CoreAI-official/tree/main/macos/gpt_oss_20b_dynamic.aimodel) | [GPT-OSS recipe](https://github.com/apple/coreai-models/tree/main/models/gpt_oss) | +| Language | [`gemma3-4b-it`](https://huggingface.co/google/gemma-3-4b-it) | macOS | [macOS `.aimodel`](https://huggingface.co/mlboydaisuke/gemma-3-4b-it-CoreAI-official/tree/main/macos/gemma_3_4b_it_4bit_dynamic.aimodel) | [Gemma 3 recipe](https://github.com/apple/coreai-models/tree/main/models/gemma3) | +| Image generation | [`sd-1.5`](https://huggingface.co/runwayml/stable-diffusion-v1-5) | iOS + macOS | Planned Callstack-hosted bundle | [Stable Diffusion recipe](https://github.com/apple/coreai-models/tree/main/models/stable-diffusion) | +| Image generation | [`flux2-klein-4b`](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B) | iOS + macOS | [pipeline bundle](https://huggingface.co/mlboydaisuke/FLUX.2-klein-4B-CoreAI/tree/main) | [FLUX.2 recipe](https://github.com/apple/coreai-models/tree/main/models/flux2) | +| Image segmentation | [`efficient-sam-vitt`](https://huggingface.co/merve/EfficientSAM/blob/main/efficient_sam_vitt.pt) | iOS + macOS | Planned Callstack-hosted bundle | [EfficientSAM recipe](https://github.com/apple/coreai-models/tree/main/models/efficient-sam) | +| Image segmentation | [`sam3`](https://huggingface.co/facebook/sam3) | iOS + macOS | [float16 `.aimodel`](https://huggingface.co/lenitas/coreai-artifacts/tree/main/artifacts/sam3/float16/sam3_float16.aimodel), [w6 palettized `.aimodel`](https://huggingface.co/lenitas/coreai-artifacts/tree/main/artifacts/sam3/w6palett-fp16/sam3_w6palett_fp16.aimodel) | [SAM 3 recipe](https://github.com/apple/coreai-models/tree/main/models/sam3) | +| Object detection | [`yolos-tiny`](https://huggingface.co/hustvl/yolos-tiny) | iOS + macOS | Planned Callstack-hosted bundle | [YOLOS recipe](https://github.com/apple/coreai-models/tree/main/models/yolo) | +| Object detection | [`yolos-base`](https://huggingface.co/hustvl/yolos-base) | iOS + macOS | Planned Callstack-hosted bundle | [YOLOS recipe](https://github.com/apple/coreai-models/tree/main/models/yolo) | Example exports for listed models: From d58a320c72ac1ea5652ecfec32754d27841b8bf6 Mon Sep 17 00:00:00 2001 From: Mike Grabowski Date: Thu, 18 Jun 2026 01:04:37 +0200 Subject: [PATCH 11/11] Document Core AI model management TBD --- packages/core-ai/README.md | 10 ++++++++++ website/src/docs/core-ai/api.md | 6 ++++++ 2 files changed, 16 insertions(+) diff --git a/packages/core-ai/README.md b/packages/core-ai/README.md index 212ddcce..4f078085 100644 --- a/packages/core-ai/README.md +++ b/packages/core-ai/README.md @@ -104,3 +104,13 @@ import { CoreAI, toNativeModelConfig } from '@react-native-ai/core-ai' await CoreAI.inspectModel(toNativeModelConfig(modelConfig)) ``` + +## Model Management TBD + +Remote Core AI assets still need a first-class download path. Apple’s native +API loads and specializes local `.aimodel` or `.aimodelc` URLs, so this package +should add helpers that download model bundles into app storage, return a local +file source, optionally specialize the model before first use, and persist +bookmark data for cached specialized assets. Where possible, the downloader and +storage pieces should be shared with the other React Native AI runtimes instead +of duplicating provider-specific plumbing. diff --git a/website/src/docs/core-ai/api.md b/website/src/docs/core-ai/api.md index f72befdd..5c1570a2 100644 --- a/website/src/docs/core-ai/api.md +++ b/website/src/docs/core-ai/api.md @@ -74,6 +74,12 @@ Native-only API status: | Super-resolution | `CoreAI.runTask('super-resolution', ...)` | Explicit unsupported error | AI SDK has no image-to-image upscaling primitive. | | Raw `.aimodel` calls | `CoreAI.runRawFunction(...)` | Explicit unsupported error | Raw tensor APIs need Core AI-specific ownership and performance rules. | +## Model Management TBD + +Core AI still needs a model-management layer for remote assets. Apple’s Core AI APIs load and specialize local `.aimodel` or `.aimodelc` URLs, so remote models need to be downloaded into app storage before they can be used. + +The intended follow-up is to add APIs that can download a model bundle, return a local `source: { type: 'file', uri }`, optionally call Core AI specialization ahead of first use, and persist bookmark data for specialized cached assets. This should reuse shared download and storage helpers with other runtimes where practical, instead of creating a separate downloader for every provider. + ## Starter Models The starter model list in these docs is informational only. The package does not ship a static model catalog; apps should load available models dynamically from their own bundle, download state, or registry integration.