diff --git a/README.md b/README.md index a6236474..c4bf91ad 100644 --- a/README.md +++ b/README.md @@ -45,11 +45,12 @@ receiving the current app's telemetry stream. ## Available Providers -| Provider | Built-in | Platforms | Runtime | Description | -| --------------- | -------- | ------------ | ------------------------------------------------------------------- | ---------------------------------------------------------- | -| [Apple](#apple) | ✅ Yes | iOS | [Apple](https://developer.apple.com/documentation/FoundationModels) | Apple Foundation Models, embeddings, transcription, speech | -| [Llama](#llama) | ❌ No | iOS, Android | [llama.rn](https://github.com/mybigday/llama.rn) | Run GGUF models via llama.rn | -| [MLC](#mlc) | ❌ No | iOS, Android | [MLC LLM](https://github.com/mlc-ai/mlc-llm) | Run open-source LLMs via MLC runtime | +| Provider | Built-in | Platforms | Runtime | Description | +| ------------------- | -------- | ------------ | ------------------------------------------------------------------- | ---------------------------------------------------------- | +| [Apple](#apple) | ✅ Yes | iOS | [Apple](https://developer.apple.com/documentation/FoundationModels) | Apple Foundation Models, embeddings, transcription, speech | +| [Core AI](#core-ai) | ❌ No | iOS | [Core AI](https://developer.apple.com/core-ai/) | Run app-provided Core AI model bundles | +| [Llama](#llama) | ❌ No | iOS, Android | [llama.rn](https://github.com/mybigday/llama.rn) | Run GGUF models via llama.rn | +| [MLC](#mlc) | ❌ No | iOS, Android | [MLC LLM](https://github.com/mlc-ai/mlc-llm) | Run open-source LLMs via MLC runtime | --- @@ -119,6 +120,47 @@ See the [Apple documentation](https://react-native-ai.dev/docs/apple/getting-sta --- +### Core AI + +Run app-provided Apple Core AI model bundles exported from [`apple/coreai-models`](https://github.com/apple/coreai-models). This provider is for custom `.aimodel` assets, not Apple system models. + +- **Language Sessions** - Core AI language models with persistent native sessions +- **AI SDK Adapters** - Language, embeddings, image generation, and transcription where AI SDK has matching interfaces +- **Native-only APIs** - Segmentation, object detection, depth, super-resolution, model inspection, specialization, and raw `.aimodel` functions +- **Catalog Metadata** - Starter model metadata from Apple’s registry, including iOS/macOS support + +#### Installation + +```bash +npm install @react-native-ai/core-ai +``` + +Core AI also requires adding `https://github.com/apple/coreai-models` as a Swift Package dependency in the host iOS app and linking the products you use, such as `CoreAILM`, `CoreAIDiffusion`, `CoreAISegmentation`, and `CoreAIObjectDetection`. + +React Native installs the package through CocoaPods, but the Apple helpers are Swift Package products. Expo apps should use a config plugin to patch the generated Xcode project; bare React Native apps should add the package in Xcode or through a predictable project patch. + +#### Usage + +```typescript +import { coreAI } from '@react-native-ai/core-ai' +import { generateText } from 'ai' + +const model = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +const { text } = await generateText({ + model, + prompt: 'Explain Core AI', +}) +``` + +See the [Core AI documentation](https://react-native-ai.dev/docs/core-ai/getting-started) for setup, starter models, and direct native APIs. + +--- + ### Llama Run any GGUF model on-device using [llama.rn](https://github.com/mybigday/llama.rn). **Requires download** - models are downloaded from HuggingFace. diff --git a/bun.lock b/bun.lock index 97e35c71..57480cfe 100644 --- a/bun.lock +++ b/bun.lock @@ -100,6 +100,22 @@ "react-native": ">=0.76.0", }, }, + "packages/core-ai": { + "name": "@react-native-ai/core-ai", + "version": "0.12.0", + "dependencies": { + "@ai-sdk/provider": "^3.0.5", + "@ai-sdk/provider-utils": "^4.0.1", + "zod": "^4.2.1", + }, + "peerDependencies": { + "expo": "*", + "react-native": ">=0.76.0", + }, + "optionalPeers": [ + "expo", + ], + }, "packages/dev-tools": { "name": "@react-native-ai/dev-tools", "version": "0.12.0", @@ -756,6 +772,8 @@ "@react-native-ai/apple": ["@react-native-ai/apple@workspace:packages/apple-llm"], + "@react-native-ai/core-ai": ["@react-native-ai/core-ai@workspace:packages/core-ai"], + "@react-native-ai/dev-tools": ["@react-native-ai/dev-tools@workspace:packages/dev-tools"], "@react-native-ai/example": ["@react-native-ai/example@workspace:apps/expo-example"], diff --git a/packages/core-ai/CoreAI.podspec b/packages/core-ai/CoreAI.podspec new file mode 100644 index 00000000..20b78f74 --- /dev/null +++ b/packages/core-ai/CoreAI.podspec @@ -0,0 +1,21 @@ +require "json" + +package = JSON.parse(File.read(File.join(__dir__, "package.json"))) + +Pod::Spec.new do |s| + s.name = "ReactNativeAICoreAI" + + s.version = package["version"] + s.summary = package["description"] + s.homepage = package["homepage"] + s.license = package["license"] + s.authors = package["author"] + + s.platforms = { :ios => min_ios_version_supported } + s.source = { :git => "https://github.com/callstackincubator/ai.git", :tag => "#{s.version}" } + + s.source_files = "ios/**/*.{h,m,mm,swift}" + s.exclude_files = "ios/Tests/**/*" + + install_modules_dependencies(s) +end diff --git a/packages/core-ai/README.md b/packages/core-ai/README.md new file mode 100644 index 00000000..4f078085 --- /dev/null +++ b/packages/core-ai/README.md @@ -0,0 +1,116 @@ +# @react-native-ai/core-ai + +Apple Core AI provider for React Native AI. + +This package is for app-provided Core AI model bundles exported from +[`apple/coreai-models`](https://github.com/apple/coreai-models). It is separate +from `@react-native-ai/apple`, which wraps Apple system capabilities such as +Foundation Models, NLContextualEmbedding, SpeechAnalyzer, and AVSpeechSynthesizer. + +## API Layers + +Core AI exposes two layers: + +- AI SDK adapters where AI SDK has a matching model interface: language models, + image generation, text embeddings, and transcription. +- Direct native APIs for Core AI-specific capabilities: model loading, + persistent language sessions, model inspection, specialization, segmentation, + object detection, depth, super-resolution, and raw `.aimodel` functions. + +The first native wrappers cover language sessions, diffusion image generation, +segmentation, and object detection. Embeddings, transcription, depth, +super-resolution, classification, reranking, speech generation, video +generation, and arbitrary raw tensor calls stay explicit: embeddings and +transcription keep AI SDK-shaped adapters because Apple has starter export +recipes, while reranking, speech, and video have no provider helpers until a +real Core AI model/runtime exists. + +## Native Setup + +Add `https://github.com/apple/coreai-models` as a Swift Package dependency in +the host iOS app and link the products used by your wrappers: + +This is the intended integration path, but it still needs full verification in +an Xcode 27 / iOS 27 host app with Apple Core AI products linked into the app +target. + +- `CoreAILM` for language models and sessions +- `CoreAIDiffusion` for image generation +- `CoreAISegmentation` for segmentation +- `CoreAIObjectDetection` for object detection + +React Native installs this package through CocoaPods, but Apple's helpers are +Swift Package products. The app target still needs those SPM products linked. +For Expo, add the config plugin. It links all known Apple Core AI Swift Package +products by default: + +```json +{ + "expo": { + "plugins": ["@react-native-ai/core-ai"] + } +} +``` + +Pass `products` only when you want to link a smaller set: + +```json +{ + "expo": { + "plugins": [ + ["@react-native-ai/core-ai", { "products": ["CoreAILM"] }] + ] + } +} +``` + +For bare React Native, add the Swift Package in Xcode or with a project patching +script. + +## Usage + +```ts +import { coreAI } from '@react-native-ai/core-ai' +import { streamText } from 'ai' + +const model = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +const result = streamText({ + model, + prompt: 'Explain Core AI in one sentence.', +}) +``` + +For persistent native sessions: + +```ts +await model.prepare() + +const session = await model.createSession({ + instructions: 'Answer tersely.', +}) + +const response = await session.respond('What is Qwen3?') +``` + +For lower-level calls, import `CoreAI` and call the native TurboModule directly: + +```ts +import { CoreAI, toNativeModelConfig } from '@react-native-ai/core-ai' + +await CoreAI.inspectModel(toNativeModelConfig(modelConfig)) +``` + +## Model Management TBD + +Remote Core AI assets still need a first-class download path. Apple’s native +API loads and specializes local `.aimodel` or `.aimodelc` URLs, so this package +should add helpers that download model bundles into app storage, return a local +file source, optionally specialize the model before first use, and persist +bookmark data for cached specialized assets. Where possible, the downloader and +storage pieces should be shared with the other React Native AI runtimes instead +of duplicating provider-specific plumbing. diff --git a/packages/core-ai/app.plugin.js b/packages/core-ai/app.plugin.js new file mode 100644 index 00000000..aef8b9d8 --- /dev/null +++ b/packages/core-ai/app.plugin.js @@ -0,0 +1 @@ +module.exports = require('./lib/commonjs/expo-plugin') diff --git a/packages/core-ai/ios/CoreAI.mm b/packages/core-ai/ios/CoreAI.mm new file mode 100644 index 00000000..1a228c2e --- /dev/null +++ b/packages/core-ai/ios/CoreAI.mm @@ -0,0 +1,231 @@ +// +// CoreAI.mm +// ReactNativeAICoreAI +// + +#if __has_include("ReactNativeAICoreAI/ReactNativeAICoreAI-Swift.h") +#import "ReactNativeAICoreAI/ReactNativeAICoreAI-Swift.h" +#else +#import "ReactNativeAICoreAI-Swift.h" +#endif + +#import +#import +#import + +#import + +@interface ReactNativeAICoreAI : NativeCoreAISpecBase +@property (strong, nonatomic) CoreAIImpl *coreAI; +@end + +using namespace facebook; +using namespace JS::NativeCoreAI; + +static NSDictionary *CoreAIConfigToDictionary(NativeCoreAIModelConfig &config) { + NSMutableDictionary *dict = [NSMutableDictionary new]; + dict[@"id"] = config.id_(); + dict[@"sourceType"] = config.sourceType(); + if (config.sourceUri()) { + dict[@"sourceUri"] = config.sourceUri(); + } + if (config.bundleName()) { + dict[@"bundleName"] = config.bundleName(); + } + if (config.bundleExtension()) { + dict[@"bundleExtension"] = config.bundleExtension(); + } + if (config.bundleSubdirectory()) { + dict[@"bundleSubdirectory"] = config.bundleSubdirectory(); + } + if (config.task()) { + dict[@"task"] = config.task(); + } + if (config.family()) { + dict[@"family"] = config.family(); + } + if (config.variant()) { + dict[@"variant"] = config.variant(); + } + return dict; +} + +@implementation ReactNativeAICoreAI + +@synthesize callInvoker; + +- (instancetype)init { + self = [super init]; + if (self) { + _coreAI = [CoreAIImpl new]; + } + return self; +} + ++ (NSString *)moduleName { + return @"NativeCoreAI"; +} + +- (std::shared_ptr)getTurboModule:(const react::ObjCTurboModule::InitParams &)params { + return std::make_shared(params); +} + +- (void)getCapabilities:(nonnull RCTPromiseResolveBlock)resolve reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI getCapabilities:resolve reject:reject]; +} + +- (void)inspectModel:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI inspectModel:CoreAIConfigToDictionary(config) resolve:resolve reject:reject]; +} + +- (void)loadModel:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI loadModel:CoreAIConfigToDictionary(config) options:options resolve:resolve reject:reject]; +} + +- (void)unloadModel:(nonnull NSString *)modelHandle + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI unloadModel:modelHandle resolve:resolve reject:reject]; +} + +- (void)removeModel:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI removeModel:CoreAIConfigToDictionary(config) resolve:resolve reject:reject]; +} + +- (void)specializeModel:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI specializeModel:CoreAIConfigToDictionary(config) options:options resolve:resolve reject:reject]; +} + +- (void)createLanguageSession:(nonnull NSString *)modelHandle + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI createLanguageSession:modelHandle options:options resolve:resolve reject:reject]; +} + +- (void)releaseLanguageSession:(nonnull NSString *)sessionHandle + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI releaseLanguageSession:sessionHandle resolve:resolve reject:reject]; +} + +- (void)respondToLanguageSession:(nonnull NSString *)sessionHandle + prompt:(nonnull NSString *)prompt + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI respondToLanguageSession:sessionHandle prompt:prompt options:options resolve:resolve reject:reject]; +} + +- (void)streamLanguageSession:(nonnull NSString *)streamId + sessionHandle:(nonnull NSString *)sessionHandle + prompt:(nonnull NSString *)prompt + options:(nonnull NSDictionary *)options { + [_coreAI streamLanguageSession:streamId + sessionHandle:sessionHandle + prompt:prompt + options:options + onUpdate:^(NSString *streamId, NSString *content) { + [self emitOnStreamUpdate:@{@"streamId": streamId, @"content": content}]; + } + onComplete:^(NSString *streamId) { + [self emitOnStreamComplete:@{@"streamId": streamId}]; + } + onError:^(NSString *streamId, NSString *code, NSString *error) { + NSMutableDictionary *payload = [@{@"streamId": streamId, @"error": error} mutableCopy]; + if (code.length > 0) { + payload[@"code"] = code; + } + [self emitOnStreamError:payload]; + }]; +} + +- (void)generateText:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + messages:(nonnull NSArray *)messages + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI generateText:CoreAIConfigToDictionary(config) messages:messages options:options resolve:resolve reject:reject]; +} + +- (void)streamText:(nonnull NSString *)streamId + config:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + messages:(nonnull NSArray *)messages + options:(nonnull NSDictionary *)options { + [_coreAI streamText:streamId + config:CoreAIConfigToDictionary(config) + messages:messages + options:options + onUpdate:^(NSString *streamId, NSString *content) { + [self emitOnStreamUpdate:@{@"streamId": streamId, @"content": content}]; + } + onComplete:^(NSString *streamId) { + [self emitOnStreamComplete:@{@"streamId": streamId}]; + } + onError:^(NSString *streamId, NSString *code, NSString *error) { + NSMutableDictionary *payload = [@{@"streamId": streamId, @"error": error} mutableCopy]; + if (code.length > 0) { + payload[@"code"] = code; + } + [self emitOnStreamError:payload]; + }]; +} + +- (void)cancelStream:(nonnull NSString *)streamId { + [_coreAI cancelStream:streamId]; +} + +- (void)embed:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + values:(nonnull NSArray *)values + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI embed:CoreAIConfigToDictionary(config) values:values options:options resolve:resolve reject:reject]; +} + +- (void)transcribe:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + audioBase64:(nonnull NSString *)audioBase64 + mediaType:(nonnull NSString *)mediaType + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI transcribe:CoreAIConfigToDictionary(config) audioBase64:audioBase64 mediaType:mediaType options:options resolve:resolve reject:reject]; +} + +- (void)generateImage:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + prompt:(nonnull NSString *)prompt + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI generateImage:CoreAIConfigToDictionary(config) prompt:prompt options:options resolve:resolve reject:reject]; +} + +- (void)runTask:(nonnull NSString *)task + config:(JS::NativeCoreAI::NativeCoreAIModelConfig &)config + input:(nonnull NSDictionary *)input + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI runTask:task config:CoreAIConfigToDictionary(config) input:input options:options resolve:resolve reject:reject]; +} + +- (void)runRawFunction:(nonnull NSString *)modelHandle + functionName:(nonnull NSString *)functionName + inputs:(nonnull NSDictionary *)inputs + options:(nonnull NSDictionary *)options + resolve:(nonnull RCTPromiseResolveBlock)resolve + reject:(nonnull RCTPromiseRejectBlock)reject { + [_coreAI runRawFunction:modelHandle functionName:functionName inputs:inputs options:options resolve:resolve reject:reject]; +} + +@end diff --git a/packages/core-ai/ios/CoreAIError.swift b/packages/core-ai/ios/CoreAIError.swift new file mode 100644 index 00000000..9dd70fd5 --- /dev/null +++ b/packages/core-ai/ios/CoreAIError.swift @@ -0,0 +1,49 @@ +import Foundation + +enum CoreAIError: LocalizedError { + case unsupportedOS + case missingSource + case missingSwiftPackage(String) + case modelNotLoaded(String) + case sessionNotFound(String) + case unsupportedTask(String) + case invalidInput(String) + + var errorDescription: String? { + switch self { + case .unsupportedOS: + return "Core AI requires iOS 27 or macOS 27 or newer." + case .missingSource: + return "Core AI model source is missing. Provide a file URI or bundle resource." + case .missingSwiftPackage(let product): + return "Missing Apple Core AI Swift Package product: \(product). Add https://github.com/apple/coreai-models to the app target and link \(product)." + case .modelNotLoaded(let handle): + return "Core AI model is not loaded: \(handle)." + case .sessionNotFound(let handle): + return "Core AI language session was not found: \(handle)." + case .unsupportedTask(let task): + return "Core AI task is not implemented for this wrapper yet: \(task)." + case .invalidInput(let message): + return message + } + } + + var code: String { + switch self { + case .unsupportedOS: + return "CORE_AI_UNSUPPORTED_OS" + case .missingSource: + return "CORE_AI_MISSING_SOURCE" + case .missingSwiftPackage: + return "CORE_AI_MISSING_SPM_PRODUCT" + case .modelNotLoaded: + return "CORE_AI_MODEL_NOT_LOADED" + case .sessionNotFound: + return "CORE_AI_SESSION_NOT_FOUND" + case .unsupportedTask: + return "CORE_AI_UNSUPPORTED_TASK" + case .invalidInput: + return "CORE_AI_INVALID_INPUT" + } + } +} diff --git a/packages/core-ai/ios/CoreAIImpl.swift b/packages/core-ai/ios/CoreAIImpl.swift new file mode 100644 index 00000000..0fdbbbc0 --- /dev/null +++ b/packages/core-ai/ios/CoreAIImpl.swift @@ -0,0 +1,995 @@ +import CoreGraphics +import Foundation +import ImageIO + +#if canImport(FoundationModels) +import FoundationModels +#endif + +#if canImport(CoreAI) +import CoreAI +#endif + +#if canImport(CoreAIDiffusionPipeline) +import CoreAIDiffusionPipeline +#endif + +#if canImport(CoreAIImageSegmenter) +import CoreAIImageSegmenter +#endif + +#if canImport(CoreAILanguageModels) +import CoreAILanguageModels +#endif + +#if canImport(CoreAIObjectDetector) +import CoreAIObjectDetector +#endif + +@objc +public class CoreAIImpl: NSObject { + private var loadedModels: [String: Any] = [:] + private var languageSessions: [String: Any] = [:] + private var streamTasks: [String: Task] = [:] + + @objc + public func getCapabilities( + _ resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + var missingProducts: [String] = [] + +#if !canImport(CoreAILanguageModels) + missingProducts.append("CoreAILM") +#endif +#if !canImport(CoreAIDiffusionPipeline) + missingProducts.append("CoreAIDiffusion") +#endif +#if !canImport(CoreAIImageSegmenter) + missingProducts.append("CoreAISegmentation") +#endif +#if !canImport(CoreAIObjectDetector) + missingProducts.append("CoreAIObjectDetection") +#endif + + resolve([ + "isCoreAIRuntimeAvailable": Self.isCoreAIRuntimeAvailable(), + "isCoreAILMAvailable": Self.isCoreAILMAvailable(), + "isCoreAIDiffusionAvailable": Self.isCoreAIDiffusionAvailable(), + "isCoreAISegmentationAvailable": Self.isCoreAISegmentationAvailable(), + "isCoreAIObjectDetectionAvailable": Self.isCoreAIObjectDetectionAvailable(), + "supportedPlatform": Self.isSupportedPlatform(), + "missingProducts": missingProducts + ]) + } + + @objc + public func inspectModel( + _ config: [String: Any], + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + do { + let sourceURL = try resolveSourceURL(config) + var isDirectory: ObjCBool = false + let exists = FileManager.default.fileExists(atPath: sourceURL.path, isDirectory: &isDirectory) + let attributes = try? FileManager.default.attributesOfItem(atPath: sourceURL.path) + let size = attributes?[.size] as? NSNumber + + resolve(modelInfo(config: config, metadata: [ + "exists": exists, + "isDirectory": isDirectory.boolValue, + "path": sourceURL.path + ], size: size)) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + + @objc + public func loadModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + let task = (config["task"] as? String) ?? "unknown" + + switch task { + case "language": + loadLanguageModel(config, resolve: resolve, reject: reject) + return + case "diffusion": + loadDiffusionModel(config, options: options, resolve: resolve, reject: reject) + return + case "segmentation": + loadSegmentationModel(config, options: options, resolve: resolve, reject: reject) + return + case "object-detection": + loadObjectDetectionModel(config, options: options, resolve: resolve, reject: reject) + return + default: + break + } + + let handle = makeHandle("model") + loadedModels[handle] = [ + "config": config, + "options": options ?? [:] + ] + resolve([ + "modelHandle": handle, + "info": modelInfo(config: config) + ]) + } + + @objc + public func unloadModel( + _ modelHandle: String, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + loadedModels.removeValue(forKey: modelHandle) + resolve(nil) + } + + @objc + public func removeModel( + _ config: [String: Any], + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + do { + let sourceURL = try resolveSourceURL(config) + try FileManager.default.removeItem(at: sourceURL) + resolve(nil) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + + @objc + public func specializeModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + resolve(modelInfo(config: config, metadata: [ + "specializationRequested": true, + "persistence": options?["persistence"] as? String ?? "default" + ])) + } + + @objc + public func createLanguageSession( + _ modelHandle: String, + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(FoundationModels) && canImport(CoreAILanguageModels) + if #available(iOS 27, macOS 27, *) { + guard let model = loadedModels[modelHandle] as? CoreAILanguageModel else { + let error = CoreAIError.modelNotLoaded(modelHandle) + reject(error.code, error.localizedDescription, error) + return + } + + let session = LanguageModelSession(model: model) + let sessionHandle = makeHandle("session") + languageSessions[sessionHandle] = session + resolve(sessionHandle) + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAILM"), reject: reject) +#endif + } + + @objc + public func releaseLanguageSession( + _ sessionHandle: String, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + languageSessions.removeValue(forKey: sessionHandle) + resolve(nil) + } + + @objc + public func respondToLanguageSession( + _ sessionHandle: String, + prompt: String, + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(FoundationModels) && canImport(CoreAILanguageModels) + if #available(iOS 27, macOS 27, *) { + guard let session = languageSessions[sessionHandle] as? LanguageModelSession else { + let error = CoreAIError.sessionNotFound(sessionHandle) + reject(error.code, error.localizedDescription, error) + return + } + + Task { + do { + let response = try await session.respond(to: prompt) + resolve([["type": "text", "text": String(describing: response)]]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAILM"), reject: reject) +#endif + } + + @objc + public func streamLanguageSession( + _ streamId: String, + sessionHandle: String, + prompt: String, + options: [String: Any]?, + onUpdate: @escaping (String, String) -> Void, + onComplete: @escaping (String) -> Void, + onError: @escaping (String, String, String) -> Void + ) { +#if canImport(FoundationModels) && canImport(CoreAILanguageModels) + if #available(iOS 27, macOS 27, *) { + guard let session = languageSessions[sessionHandle] as? LanguageModelSession else { + emitError(.sessionNotFound(sessionHandle), streamId: streamId, onError: onError) + return + } + + let task = Task { + do { + let responseStream = session.streamResponse(to: prompt) + for try await chunk in responseStream { + if Task.isCancelled { + return + } + onUpdate(streamId, String(describing: chunk.content)) + } + onComplete(streamId) + } catch { + if !Task.isCancelled { + onError(streamId, errorCode(error), error.localizedDescription) + } + } + self.streamTasks.removeValue(forKey: streamId) + } + streamTasks[streamId] = task + } else { + emitError(.unsupportedOS, streamId: streamId, onError: onError) + } +#else + emitError(.missingSwiftPackage("CoreAILM"), streamId: streamId, onError: onError) +#endif + } + + @objc + public func generateText( + _ config: [String: Any], + messages: [[String: Any]], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + loadLanguageModel(config, resolve: { loaded in + guard let loadedDict = loaded as? [String: Any], + let modelHandle = loadedDict["modelHandle"] as? String else { + let error = CoreAIError.invalidInput("Core AI language model did not return a handle.") + reject(error.code, error.localizedDescription, error) + return + } + + self.createLanguageSession(modelHandle, options: options, resolve: { session in + guard let sessionHandle = session as? String else { + let error = CoreAIError.invalidInput("Core AI language session did not return a handle.") + reject(error.code, error.localizedDescription, error) + return + } + + let prompt = self.lastUserPrompt(messages) + self.respondToLanguageSession(sessionHandle, prompt: prompt, options: options, resolve: resolve, reject: reject) + }, reject: reject) + }, reject: reject) + } + + @objc + public func streamText( + _ streamId: String, + config: [String: Any], + messages: [[String: Any]], + options: [String: Any]?, + onUpdate: @escaping (String, String) -> Void, + onComplete: @escaping (String) -> Void, + onError: @escaping (String, String, String) -> Void + ) { + loadLanguageModel(config, resolve: { loaded in + guard let loadedDict = loaded as? [String: Any], + let modelHandle = loadedDict["modelHandle"] as? String else { + self.emitError(.invalidInput("Core AI language model did not return a handle."), streamId: streamId, onError: onError) + return + } + + self.createLanguageSession(modelHandle, options: options, resolve: { session in + guard let sessionHandle = session as? String else { + self.emitError(.invalidInput("Core AI language session did not return a handle."), streamId: streamId, onError: onError) + return + } + + self.streamLanguageSession( + streamId, + sessionHandle: sessionHandle, + prompt: self.lastUserPrompt(messages), + options: options, + onUpdate: onUpdate, + onComplete: onComplete, + onError: onError + ) + }, reject: { code, message, _ in + onError(streamId, code, message) + }) + }, reject: { code, message, _ in + onError(streamId, code, message) + }) + } + + @objc + public func cancelStream(_ streamId: String) { + streamTasks[streamId]?.cancel() + streamTasks.removeValue(forKey: streamId) + } + + @objc + public func embed( + _ config: [String: Any], + values: [String], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + rejectCoreAI(.unsupportedTask("embedding"), reject: reject) + } + + @objc + public func transcribe( + _ config: [String: Any], + audioBase64: String, + mediaType: String, + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + rejectCoreAI(.unsupportedTask("asr"), reject: reject) + } + + @objc + public func generateImage( + _ config: [String: Any], + prompt: String, + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIDiffusionPipeline) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + Task { + do { + let pipeline = try await self.makeDiffusionPipeline(at: sourceURL, options: options) + let configuration = try self.makePipelineConfiguration(prompt: prompt, options: options) + let result = try await pipeline.generateImages(configuration: configuration) { _ in true } + let images = try result.images.map(Self.pngBase64) + resolve([ + "images": images, + "metadata": [ + "count": images.count, + "imageFormat": "image/png" + ] + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAIDiffusion"), reject: reject) +#endif + } + + @objc + public func runTask( + _ task: String, + config: [String: Any], + input: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + switch task { + case "segmentation": + runSegmentationTask(config: config, input: input, options: options, resolve: resolve, reject: reject) + case "object-detection": + runObjectDetectionTask(config: config, input: input, options: options, resolve: resolve, reject: reject) + default: + rejectCoreAI(.unsupportedTask(task), reject: reject) + } + } + + @objc + public func runRawFunction( + _ modelHandle: String, + functionName: String, + inputs: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { + rejectCoreAI(.unsupportedTask("raw"), reject: reject) + } + + private func loadLanguageModel( + _ config: [String: Any], + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(FoundationModels) && canImport(CoreAILanguageModels) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + Task { + do { + let model = try await CoreAILanguageModel(resourcesAt: sourceURL) + let handle = self.makeHandle("model") + self.loadedModels[handle] = model + resolve([ + "modelHandle": handle, + "info": self.modelInfo(config: config) + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAILM"), reject: reject) +#endif + } + + private func loadDiffusionModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIDiffusionPipeline) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + Task { + do { + let pipeline = try await self.makeDiffusionPipeline(at: sourceURL, options: options) + let handle = self.makeHandle("model") + self.loadedModels[handle] = pipeline + resolve([ + "modelHandle": handle, + "info": self.modelInfo(config: config) + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAIDiffusion"), reject: reject) +#endif + } + + private func loadSegmentationModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIImageSegmenter) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + let parameters = makeSegmentationParameters(options: options) + Task { + do { + let segmenter = try await ImageSegmenter(resourcesAt: sourceURL.path, parameters: parameters) + let handle = self.makeHandle("model") + self.loadedModels[handle] = segmenter + resolve([ + "modelHandle": handle, + "info": self.modelInfo(config: config) + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAISegmentation"), reject: reject) +#endif + } + + private func loadObjectDetectionModel( + _ config: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIObjectDetector) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + Task { + do { + let detector = try await ObjectDetector(resourcesAt: sourceURL.path) + let handle = self.makeHandle("model") + self.loadedModels[handle] = detector + resolve([ + "modelHandle": handle, + "info": self.modelInfo(config: config) + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAIObjectDetection"), reject: reject) +#endif + } + + private func runSegmentationTask( + config: [String: Any], + input: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIImageSegmenter) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + let image = try Self.loadCGImage(from: input) + let parameters = makeSegmentationParameters(options: options) + Task { + do { + let segmenter = try await ImageSegmenter(resourcesAt: sourceURL.path, parameters: parameters) + let response: SegmentationResponse + if let prompt = input["text"] as? String, !prompt.isEmpty { + response = try await segmenter.segment(image: image, prompt: prompt, parameters: parameters) + } else { + response = try await segmenter.segment( + image: image, + pointQuery: Self.makePointQuery(input: input), + parameters: parameters + ) + } + resolve([ + "task": "segmentation", + "output": Self.serializeSegmentationResponse(response), + "metadata": [ + "imageWidth": image.width, + "imageHeight": image.height + ] + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAISegmentation"), reject: reject) +#endif + } + + private func runObjectDetectionTask( + config: [String: Any], + input: [String: Any], + options: [String: Any]?, + resolve: @escaping (Any?) -> Void, + reject: @escaping (String, String, Error?) -> Void + ) { +#if canImport(CoreAIObjectDetector) + if #available(iOS 27, macOS 27, *) { + do { + let sourceURL = try resolveSourceURL(config) + let image = try Self.loadCGImage(from: input) + let parameters = makeDetectionParameters(options: options) + Task { + do { + let detector = try await ObjectDetector(resourcesAt: sourceURL.path) + let detections = try await detector.detect(image: image, parameters: parameters) + resolve([ + "task": "object-detection", + "output": [ + "detections": detections.map(Self.serializeDetection) + ], + "metadata": [ + "imageWidth": image.width, + "imageHeight": image.height + ] + ]) + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } + } catch { + reject(errorCode(error), error.localizedDescription, error) + } + } else { + rejectCoreAI(.unsupportedOS, reject: reject) + } +#else + rejectCoreAI(.missingSwiftPackage("CoreAIObjectDetection"), reject: reject) +#endif + } + + private func resolveSourceURL(_ config: [String: Any]) throws -> URL { + if let sourceUri = config["sourceUri"] as? String { + if sourceUri.hasPrefix("file://") { + return URL(string: sourceUri) ?? URL(fileURLWithPath: sourceUri.replacingOccurrences(of: "file://", with: "")) + } + return URL(fileURLWithPath: sourceUri) + } + + if let bundleName = config["bundleName"] as? String { + let ext = config["bundleExtension"] as? String + let subdirectory = config["bundleSubdirectory"] as? String + if let url = Bundle.main.url(forResource: bundleName, withExtension: ext, subdirectory: subdirectory) { + return url + } + } + + throw CoreAIError.missingSource + } + + private func modelInfo( + config: [String: Any], + metadata: [String: Any] = [:], + size: NSNumber? = nil + ) -> [String: Any] { + var info: [String: Any] = [ + "id": config["id"] as? String ?? "unknown", + "family": config["family"] as? String ?? "", + "task": config["task"] as? String ?? "unknown", + "platforms": ["iOS", "macOS"], + "functions": [], + "metadata": metadata + ] + if let size { + info["modelSizeBytes"] = size + } + return info + } + + private func lastUserPrompt(_ messages: [[String: Any]]) -> String { + return messages.last(where: { ($0["role"] as? String) == "user" })?["content"] as? String ?? "" + } + + private func makeHandle(_ prefix: String) -> String { + return "\(prefix)-\(UUID().uuidString)" + } + + private func rejectCoreAI( + _ error: CoreAIError, + reject: @escaping (String, String, Error?) -> Void + ) { + reject(error.code, error.localizedDescription, error) + } + + private func emitError( + _ error: CoreAIError, + streamId: String, + onError: @escaping (String, String, String) -> Void + ) { + onError(streamId, error.code, error.localizedDescription) + } + + private static func isSupportedPlatform() -> Bool { + if #available(iOS 27, macOS 27, *) { + return true + } + return false + } + + private static func isCoreAIRuntimeAvailable() -> Bool { +#if canImport(CoreAI) + return isSupportedPlatform() +#else + return false +#endif + } + + private static func isCoreAILMAvailable() -> Bool { +#if canImport(CoreAILanguageModels) + return isSupportedPlatform() +#else + return false +#endif + } + + private static func isCoreAIDiffusionAvailable() -> Bool { +#if canImport(CoreAIDiffusionPipeline) + return isSupportedPlatform() +#else + return false +#endif + } + + private static func isCoreAISegmentationAvailable() -> Bool { +#if canImport(CoreAIImageSegmenter) + return isSupportedPlatform() +#else + return false +#endif + } + + private static func isCoreAIObjectDetectionAvailable() -> Bool { +#if canImport(CoreAIObjectDetector) + return isSupportedPlatform() +#else + return false +#endif + } +} + +#if canImport(CoreAIDiffusionPipeline) +extension CoreAIImpl { + private func makeDiffusionPipeline( + at sourceURL: URL, + options: [String: Any]? + ) async throws -> any DiffusionPipeline { + let descriptor = try PipelineDescriptor.resolve(at: sourceURL) + + switch descriptor.type { + case .stableDiffusion, .stableDiffusionXL, .none: + return try await StableDiffusionPipeline.load(from: sourceURL) + case .stableDiffusion3: + return try await SD3Pipeline(from: sourceURL) + case .flux2: + let mode = DecodeResolution(rawValue: options?["decodeResolution"] as? String ?? "auto") ?? .auto + return try await Flux2Pipeline(from: sourceURL, mode: mode) + } + } + + private func makePipelineConfiguration( + prompt: String, + options: [String: Any]? + ) throws -> PipelineConfiguration { + let scheduler = SchedulerType(rawValue: options?["schedulerType"] as? String ?? "") ?? .dpmSolverMultistep + let decodeResolution = DecodeResolution(rawValue: options?["decodeResolution"] as? String ?? "full") ?? .full + let seed = UInt32(options?["seed"] as? Double ?? 0) + let stepCount = Int(options?["stepCount"] as? Double ?? 50) + let guidanceScale = Float(options?["guidanceScale"] as? Double ?? 7.5) + + return PipelineConfiguration( + prompt: prompt, + negativePrompt: options?["negativePrompt"] as? String ?? "", + seed: seed, + stepCount: stepCount, + guidanceScale: guidanceScale, + schedulerType: scheduler, + strength: Float(options?["strength"] as? Double ?? 1.0), + decodeResolution: decodeResolution, + lazyModelLoading: options?["lazyModelLoading"] as? Bool ?? true + ) + } +} +#endif + +#if canImport(CoreAIImageSegmenter) +extension CoreAIImpl { + private func makeSegmentationParameters(options: [String: Any]?) -> SegmentationParameters { + var parameters = SegmentationParameters.default + if let maskThreshold = options?["maskThreshold"] as? Double { + parameters.maskThreshold = Float(maskThreshold) + } + if let maxSegments = options?["maxSegments"] as? Double { + parameters.maxSegments = Int(maxSegments) + } + if let tokenizerContextLength = options?["tokenizerContextLength"] as? Double { + parameters.tokenizerContextLength = Int(tokenizerContextLength) + } + return parameters + } + + private static func makePointQuery(input: [String: Any]) -> PointQuery { + guard let data = input["data"] as? [String: Any] else { + return PointQuery() + } + + if let queries = data["queries"] as? [[Any]] { + return PointQuery(queries: queries.map { query in + query.compactMap { item in + guard let point = item as? [String: Any] else { + return nil + } + return makePoint(point) + } + }) + } + + if let points = data["points"] as? [[String: Any]] { + return PointQuery(points: points.compactMap(makePoint)) + } + + if let box = data["box"] as? [String: Any], + let x = box["x"] as? Double, + let y = box["y"] as? Double, + let width = box["width"] as? Double, + let height = box["height"] as? Double { + return PointQuery(points: [ + PointQuery.Point(x: Float(x), y: Float(y), label: .boxTopLeft), + PointQuery.Point(x: Float(x + width), y: Float(y + height), label: .boxBottomRight) + ]) + } + + return PointQuery() + } + + private static func makePoint(_ point: [String: Any]) -> PointQuery.Point? { + guard let x = point["x"] as? Double, + let y = point["y"] as? Double else { + return nil + } + + let labelValue = Int32(point["label"] as? Double ?? 1) + let label = PointQuery.Label(rawValue: labelValue) ?? .foreground + return PointQuery.Point(x: Float(x), y: Float(y), label: label) + } + + private static func serializeSegmentationResponse(_ response: SegmentationResponse) -> [String: Any] { + var output: [String: Any] = [ + "segments": response.segments.map { segment in + [ + "score": segment.score, + "box": serializeRect(segment.box), + "maskWidth": segment.maskWidth, + "maskHeight": segment.maskHeight, + "mask": segment.mask + ] + } + ] + + if let probabilityMap = response.probabilityMap { + output["probabilityMap"] = [ + "width": probabilityMap.width, + "height": probabilityMap.height, + "probabilities": probabilityMap.probabilities + ] + } + + return output + } +} +#endif + +#if canImport(CoreAIObjectDetector) +extension CoreAIImpl { + private func makeDetectionParameters(options: [String: Any]?) -> DetectionParameters { + var parameters = DetectionParameters.default + if let threshold = options?["threshold"] as? Double { + parameters.threshold = Float(threshold) + } + if let maxDetections = options?["maxDetections"] as? Double { + parameters.maxDetections = Int(maxDetections) + } + if let inputHeight = options?["inputHeight"] as? Double { + parameters.inputHeight = Int(inputHeight) + } + if let inputWidth = options?["inputWidth"] as? Double { + parameters.inputWidth = Int(inputWidth) + } + return parameters + } + + private static func serializeDetection(_ detection: DetectedObject) -> [String: Any] { + return [ + "label": detection.label, + "labelIndex": detection.labelIndex, + "confidence": detection.confidence, + "box": serializeRect(detection.boundingBox) + ] + } +} +#endif + +extension CoreAIImpl { + private static func loadCGImage(from input: [String: Any]) throws -> CGImage { + guard let imageUri = input["imageUri"] as? String else { + throw CoreAIError.invalidInput("Expected input.imageUri for this Core AI task.") + } + + let url: URL + if imageUri.hasPrefix("file://") { + url = URL(string: imageUri) ?? URL(fileURLWithPath: imageUri.replacingOccurrences(of: "file://", with: "")) + } else { + url = URL(fileURLWithPath: imageUri) + } + + guard let source = CGImageSourceCreateWithURL(url as CFURL, nil), + let image = CGImageSourceCreateImageAtIndex(source, 0, nil) else { + throw CoreAIError.invalidInput("Could not load image at \(url.path).") + } + return image + } + + private static func pngBase64(_ image: CGImage) throws -> String { + let data = NSMutableData() + guard let destination = CGImageDestinationCreateWithData( + data, + "public.png" as CFString, + 1, + nil + ) else { + throw CoreAIError.invalidInput("Could not create PNG destination.") + } + + CGImageDestinationAddImage(destination, image, nil) + guard CGImageDestinationFinalize(destination) else { + throw CoreAIError.invalidInput("Could not encode generated image as PNG.") + } + return (data as Data).base64EncodedString() + } + + private static func serializeRect(_ rect: CGRect) -> [String: Double] { + return [ + "x": rect.origin.x, + "y": rect.origin.y, + "width": rect.width, + "height": rect.height + ].mapValues(Double.init) + } +} + +private func errorCode(_ error: Error) -> String { + if let coreAIError = error as? CoreAIError { + return coreAIError.code + } + return "CORE_AI_ERROR" +} diff --git a/packages/core-ai/package.json b/packages/core-ai/package.json new file mode 100644 index 00000000..043e5d11 --- /dev/null +++ b/packages/core-ai/package.json @@ -0,0 +1,93 @@ +{ + "name": "@react-native-ai/core-ai", + "version": "0.12.0", + "description": "Apple Core AI provider for React Native AI", + "main": "lib/commonjs/index", + "module": "lib/module/index", + "types": "lib/typescript/index.d.ts", + "react-native": "src/index", + "source": "src/index", + "files": [ + "src", + "lib", + "ios", + "*.podspec", + "app.plugin.js", + "!ios/build", + "!**/__tests__", + "!**/__fixtures__", + "!**/__mocks__", + "!**/.*" + ], + "scripts": { + "clean": "del-cli lib", + "typecheck": "tsc --noEmit", + "prepare": "bob build" + }, + "keywords": [ + "react-native", + "apple", + "core-ai", + "llm", + "ai", + "sdk", + "vercel", + "expo", + "config-plugin", + "spm" + ], + "repository": { + "type": "git", + "url": "git+https://github.com/callstackincubator/ai.git" + }, + "author": "Mike Grabowski ", + "contributors": [ + "Callstack " + ], + "license": "MIT", + "bugs": { + "url": "https://github.com/callstackincubator/ai/issues" + }, + "homepage": "https://github.com/callstackincubator/ai#readme", + "publishConfig": { + "registry": "https://registry.npmjs.org/" + }, + "dependencies": { + "@ai-sdk/provider": "^3.0.5", + "@ai-sdk/provider-utils": "^4.0.1", + "zod": "^4.2.1" + }, + "peerDependencies": { + "react-native": ">=0.76.0", + "expo": "*" + }, + "peerDependenciesMeta": { + "expo": { + "optional": true + } + }, + "react-native-builder-bob": { + "source": "src", + "output": "lib", + "targets": [ + "commonjs", + "module", + [ + "typescript", + { + "project": "tsconfig.build.json" + } + ] + ] + }, + "codegenConfig": { + "name": "NativeCoreAI", + "type": "modules", + "jsSrcsDir": "src", + "ios": { + "modulesProvider": { + "NativeCoreAI": "ReactNativeAICoreAI" + } + } + } +} diff --git a/packages/core-ai/src/NativeCoreAI.ts b/packages/core-ai/src/NativeCoreAI.ts new file mode 100644 index 00000000..3be50468 --- /dev/null +++ b/packages/core-ai/src/NativeCoreAI.ts @@ -0,0 +1,136 @@ +import type { TurboModule } from 'react-native' +import { TurboModuleRegistry } from 'react-native' +import type { + EventEmitter, + UnsafeObject, +} from 'react-native/Libraries/Types/CodegenTypes' + +export interface NativeCoreAIModelConfig { + id: string + sourceType: string + sourceUri?: string + bundleName?: string + bundleExtension?: string + bundleSubdirectory?: string + task?: string + family?: string + variant?: string +} + +export interface CoreAIModelInfo { + id?: string + family?: string + task: string + platforms: string[] + functions: UnsafeObject[] + maxContextLength?: number + modelSizeBytes?: number + metadata?: UnsafeObject +} + +export interface CoreAILoadedModel { + modelHandle: string + info: CoreAIModelInfo +} + +export interface CoreAIMessage { + role: 'assistant' | 'system' | 'tool' | 'user' + content: string +} + +export type CoreAIStreamUpdateEvent = { + streamId: string + content: string +} + +export type CoreAIStreamCompleteEvent = { + streamId: string +} + +export type CoreAIStreamErrorEvent = { + streamId: string + code?: string + error: string +} + +export interface Spec extends TurboModule { + getCapabilities(): Promise + + inspectModel(config: NativeCoreAIModelConfig): Promise + loadModel( + config: NativeCoreAIModelConfig, + options?: UnsafeObject + ): Promise + unloadModel(modelHandle: string): Promise + removeModel(config: NativeCoreAIModelConfig): Promise + specializeModel( + config: NativeCoreAIModelConfig, + options?: UnsafeObject + ): Promise + + createLanguageSession( + modelHandle: string, + options?: UnsafeObject + ): Promise + releaseLanguageSession(sessionHandle: string): Promise + respondToLanguageSession( + sessionHandle: string, + prompt: string, + options?: UnsafeObject + ): Promise + streamLanguageSession( + streamId: string, + sessionHandle: string, + prompt: string, + options?: UnsafeObject + ): void + + generateText( + config: NativeCoreAIModelConfig, + messages: CoreAIMessage[], + options?: UnsafeObject + ): Promise + streamText( + streamId: string, + config: NativeCoreAIModelConfig, + messages: CoreAIMessage[], + options?: UnsafeObject + ): void + cancelStream(streamId: string): void + + embed( + config: NativeCoreAIModelConfig, + values: string[], + options?: UnsafeObject + ): Promise + transcribe( + config: NativeCoreAIModelConfig, + audioBase64: string, + mediaType: string, + options?: UnsafeObject + ): Promise + generateImage( + config: NativeCoreAIModelConfig, + prompt: string, + options?: UnsafeObject + ): Promise + + runTask( + task: string, + config: NativeCoreAIModelConfig, + input: UnsafeObject, + options?: UnsafeObject + ): Promise + runRawFunction( + modelHandle: string, + functionName: string, + inputs: UnsafeObject, + options?: UnsafeObject + ): Promise + + onStreamUpdate: EventEmitter + onStreamComplete: EventEmitter + onStreamError: EventEmitter +} + +export default TurboModuleRegistry.getEnforcing('NativeCoreAI') diff --git a/packages/core-ai/src/ai-sdk.ts b/packages/core-ai/src/ai-sdk.ts new file mode 100644 index 00000000..efaea3c3 --- /dev/null +++ b/packages/core-ai/src/ai-sdk.ts @@ -0,0 +1,560 @@ +import type { + EmbeddingModelV3, + EmbeddingModelV3CallOptions, + EmbeddingModelV3Result, + ImageModelV3, + ImageModelV3CallOptions, + LanguageModelV3, + LanguageModelV3CallOptions, + LanguageModelV3FinishReason, + LanguageModelV3StreamPart, + TranscriptionModelV3, + TranscriptionModelV3CallOptions, +} from '@ai-sdk/provider' +import { generateId } from '@ai-sdk/provider-utils' +import type { UnsafeObject } from 'react-native/Libraries/Types/CodegenTypes' + +import NativeCoreAI from './NativeCoreAI' +import type { + CoreAIEmbeddingResult, + CoreAIGenerationOptions, + CoreAIGenerationPart, + CoreAIImageGenerationOptions, + CoreAIImageGenerationResult, + CoreAILoadedModel, + CoreAIMessage, + CoreAIModelConfig, + CoreAIModelInfo, + CoreAIStreamCompleteEvent, + CoreAIStreamErrorEvent, + CoreAIStreamUpdateEvent, + CoreAITranscriptionResult, +} from './types' +import { toNativeModelConfig } from './types' + +export function createCoreAIProvider() { + const provider = function (config: CoreAIModelConfig) { + return provider.languageModel(config) + } + + provider.getCapabilities = () => NativeCoreAI.getCapabilities() + provider.languageModel = (config: CoreAIModelConfig) => { + return new CoreAILanguageModel(config) + } + provider.textEmbeddingModel = (config: CoreAIModelConfig) => { + return new CoreAITextEmbeddingModel(config) + } + provider.embeddingModel = provider.textEmbeddingModel + provider.imageModel = (config: CoreAIModelConfig) => { + return new CoreAIImageGenerationModel(config) + } + provider.transcriptionModel = (config: CoreAIModelConfig) => { + return new CoreAITranscriptionModel(config) + } + + return provider +} + +export const coreAI = createCoreAIProvider() + +export interface CoreAILanguageSession { + sessionHandle: string + respond( + prompt: string, + options?: CoreAIGenerationOptions + ): Promise + stream( + prompt: string, + options?: CoreAIGenerationOptions + ): Promise> + close(): Promise +} + +export class CoreAILanguageModel implements LanguageModelV3 { + readonly specificationVersion = 'v3' + readonly supportedUrls = {} + readonly provider = 'core-ai' + readonly modelId: string + readonly config: CoreAIModelConfig + + private loadedModel?: CoreAILoadedModel + + constructor(config: CoreAIModelConfig) { + this.config = { ...config, task: 'language' } + this.modelId = config.id + } + + get modelHandle() { + return this.loadedModel?.modelHandle + } + + async inspect(): Promise { + return NativeCoreAI.inspectModel( + toNativeModelConfig(this.config) + ) as Promise + } + + async prepare(options: UnsafeObject = {}): Promise { + const loadedModel = (await NativeCoreAI.loadModel( + toNativeModelConfig(this.config), + options + )) as CoreAILoadedModel + this.loadedModel = loadedModel + return loadedModel.info + } + + async specialize(options: UnsafeObject = {}): Promise { + return NativeCoreAI.specializeModel( + toNativeModelConfig(this.config), + options + ) as Promise + } + + async unload(): Promise { + if (!this.loadedModel) { + return + } + await NativeCoreAI.unloadModel(this.loadedModel.modelHandle) + this.loadedModel = undefined + } + + async remove(): Promise { + await NativeCoreAI.removeModel(toNativeModelConfig(this.config)) + this.loadedModel = undefined + } + + async createSession( + options: UnsafeObject = {} + ): Promise { + const model = await this.ensureLoaded() + const sessionHandle = await NativeCoreAI.createLanguageSession( + model.modelHandle, + options + ) + return createLanguageSession(sessionHandle) + } + + async doGenerate(options: LanguageModelV3CallOptions) { + const response = (await NativeCoreAI.generateText( + toNativeModelConfig(this.config), + prepareMessages(options.prompt), + toCoreAIGenerationOptions(options) + )) as CoreAIGenerationPart[] + + return { + content: response.map(toLanguageModelContent), + finishReason: toFinishReason('stop'), + usage: emptyUsage(), + warnings: [], + } + } + + async doStream(options: LanguageModelV3CallOptions) { + if (typeof ReadableStream === 'undefined') { + throw new Error( + 'ReadableStream is not available. Load a web stream polyfill before streaming Core AI responses.' + ) + } + + const streamId = generateId() + let listeners: { remove(): void }[] = [] + + const cleanup = () => { + listeners.forEach((listener) => listener.remove()) + listeners = [] + } + + const stream = new ReadableStream({ + start: (controller) => { + controller.enqueue({ type: 'text-start', id: streamId }) + + const updateListener = NativeCoreAI.onStreamUpdate( + (event: CoreAIStreamUpdateEvent) => { + if (event.streamId === streamId) { + controller.enqueue({ + type: 'text-delta', + id: streamId, + delta: event.content, + }) + } + } + ) + const completeListener = NativeCoreAI.onStreamComplete( + (event: CoreAIStreamCompleteEvent) => { + if (event.streamId === streamId) { + controller.enqueue({ type: 'text-end', id: streamId }) + controller.enqueue({ + type: 'finish', + finishReason: toFinishReason('stop'), + usage: emptyUsage(), + }) + cleanup() + controller.close() + } + } + ) + const errorListener = NativeCoreAI.onStreamError( + (event: CoreAIStreamErrorEvent) => { + if (event.streamId === streamId) { + controller.enqueue({ + type: 'error', + error: new Error(event.error), + }) + cleanup() + controller.close() + } + } + ) + + listeners = [updateListener, completeListener, errorListener] + NativeCoreAI.streamText( + streamId, + toNativeModelConfig(this.config), + prepareMessages(options.prompt), + toCoreAIGenerationOptions(options) + ) + }, + cancel: () => { + cleanup() + NativeCoreAI.cancelStream(streamId) + }, + }) + + return { + stream, + rawCall: { + rawPrompt: options.prompt, + rawSettings: {}, + }, + } + } + + private async ensureLoaded(): Promise { + if (!this.loadedModel) { + await this.prepare() + } + return this.loadedModel! + } +} + +export class CoreAITextEmbeddingModel implements EmbeddingModelV3 { + readonly specificationVersion = 'v3' + readonly provider = 'core-ai' + readonly modelId: string + readonly maxEmbeddingsPerCall = Infinity + readonly supportsParallelCalls = false + readonly config: CoreAIModelConfig + + constructor(config: CoreAIModelConfig) { + this.config = { ...config, task: 'embedding' } + this.modelId = config.id + } + + async embed( + values: string[], + options: UnsafeObject = {} + ): Promise { + return NativeCoreAI.embed( + toNativeModelConfig(this.config), + values, + options + ) as Promise + } + + async doEmbed( + options: EmbeddingModelV3CallOptions + ): Promise { + const result = await this.embed(options.values) + return { + embeddings: result.embeddings, + warnings: [], + } + } +} + +export class CoreAIImageGenerationModel implements ImageModelV3 { + readonly specificationVersion = 'v3' + readonly provider = 'core-ai' + readonly modelId: string + readonly maxImagesPerCall = 1 + readonly config: CoreAIModelConfig + + constructor(config: CoreAIModelConfig) { + this.config = { ...config, task: 'diffusion' } + this.modelId = config.id + } + + async generate( + prompt: string, + options: CoreAIImageGenerationOptions = {} + ): Promise { + return NativeCoreAI.generateImage( + toNativeModelConfig(this.config), + prompt, + options + ) as Promise + } + + async doGenerate(options: ImageModelV3CallOptions) { + const result = await this.generate(options.prompt ?? '', { + n: options.n, + size: options.size, + aspectRatio: options.aspectRatio, + seed: options.seed, + ...(options.providerOptions?.['core-ai'] ?? {}), + }) + + return { + images: result.images, + warnings: [], + response: { + timestamp: new Date(), + modelId: this.modelId, + headers: undefined, + }, + providerMetadata: result.metadata + ? ({ + 'core-ai': { + images: [], + ...result.metadata, + }, + } as any) + : undefined, + } + } +} + +export class CoreAITranscriptionModel implements TranscriptionModelV3 { + readonly specificationVersion = 'v3' + readonly provider = 'core-ai' + readonly modelId: string + readonly config: CoreAIModelConfig + + constructor(config: CoreAIModelConfig) { + this.config = { ...config, task: 'asr' } + this.modelId = config.id + } + + async transcribe( + audio: Uint8Array | string, + mediaType: string, + options: UnsafeObject = {} + ): Promise { + return NativeCoreAI.transcribe( + toNativeModelConfig(this.config), + toBase64(audio), + mediaType, + options + ) as Promise + } + + async doGenerate(options: TranscriptionModelV3CallOptions) { + const result = await this.transcribe(options.audio, options.mediaType, { + ...(options.providerOptions?.['core-ai'] ?? {}), + }) + + return { + text: result.text, + segments: result.segments, + language: result.language, + durationInSeconds: result.durationInSeconds, + warnings: [], + response: { + timestamp: new Date(), + modelId: this.modelId, + }, + providerMetadata: result.metadata + ? ({ + 'core-ai': result.metadata, + } as any) + : undefined, + } + } +} + +function createLanguageSession(sessionHandle: string): CoreAILanguageSession { + return { + sessionHandle, + respond(prompt, options = {}) { + return NativeCoreAI.respondToLanguageSession( + sessionHandle, + prompt, + options + ) as Promise + }, + stream(prompt, options = {}) { + return streamLanguageSession(sessionHandle, prompt, options) + }, + close() { + return NativeCoreAI.releaseLanguageSession(sessionHandle) + }, + } +} + +function streamLanguageSession( + sessionHandle: string, + prompt: string, + options: CoreAIGenerationOptions +): Promise> { + if (typeof ReadableStream === 'undefined') { + throw new Error( + 'ReadableStream is not available. Load a web stream polyfill before streaming Core AI responses.' + ) + } + + const streamId = generateId() + let listeners: { remove(): void }[] = [] + + const cleanup = () => { + listeners.forEach((listener) => listener.remove()) + listeners = [] + } + + const stream = new ReadableStream({ + start(controller) { + const updateListener = NativeCoreAI.onStreamUpdate( + (event: CoreAIStreamUpdateEvent) => { + if (event.streamId === streamId) { + controller.enqueue(event) + } + } + ) + const completeListener = NativeCoreAI.onStreamComplete( + (event: CoreAIStreamCompleteEvent) => { + if (event.streamId === streamId) { + cleanup() + controller.close() + } + } + ) + const errorListener = NativeCoreAI.onStreamError( + (event: CoreAIStreamErrorEvent) => { + if (event.streamId === streamId) { + cleanup() + controller.error(new Error(event.error)) + } + } + ) + + listeners = [updateListener, completeListener, errorListener] + NativeCoreAI.streamLanguageSession( + streamId, + sessionHandle, + prompt, + options + ) + }, + cancel() { + cleanup() + NativeCoreAI.cancelStream(streamId) + }, + }) + + return Promise.resolve(stream) +} + +export function prepareMessages(messages: unknown): CoreAIMessage[] { + if (!Array.isArray(messages)) { + return [] + } + + return messages.map((message: any): CoreAIMessage => { + const content = Array.isArray(message.content) + ? message.content.reduce((acc: string, part: any) => { + if (part.type === 'text') { + return acc + part.text + } + console.warn('Unsupported Core AI message content type:', part) + return acc + }, '') + : String(message.content ?? '') + + return { + role: message.role, + content, + } + }) +} + +function toCoreAIGenerationOptions( + options: LanguageModelV3CallOptions +): CoreAIGenerationOptions { + return { + maxTokens: options.maxOutputTokens, + temperature: options.temperature, + topP: options.topP, + topK: options.topK, + schema: + options.responseFormat?.type === 'json' + ? options.responseFormat.schema + : undefined, + tools: options.tools, + } +} + +function toLanguageModelContent(part: CoreAIGenerationPart) { + switch (part.type) { + case 'text': + return part + case 'reasoning': + return { + type: 'reasoning' as const, + text: part.text, + } + case 'tool-call': + return { + type: 'tool-call' as const, + toolCallId: generateId(), + providerExecuted: true, + toolName: part.toolName, + input: part.input, + } + case 'tool-result': + return { + type: 'tool-result' as const, + toolCallId: generateId(), + providerExecuted: true, + toolName: part.toolName, + result: part.output, + } + } +} + +function toFinishReason( + raw: 'stop' | 'length' | 'tool-calls' | 'error' +): LanguageModelV3FinishReason { + return { + unified: raw === 'tool-calls' ? 'tool-calls' : raw, + raw, + } +} + +function emptyUsage() { + return { + inputTokens: { + total: 0, + noCache: undefined, + cacheRead: undefined, + cacheWrite: undefined, + }, + outputTokens: { + total: 0, + text: undefined, + reasoning: undefined, + }, + } +} + +function toBase64(data: Uint8Array | string): string { + if (typeof data === 'string') { + return data + } + + let binary = '' + const chunkSize = 0x8000 + for (let i = 0; i < data.length; i += chunkSize) { + const chunk = data.subarray(i, i + chunkSize) + binary += String.fromCharCode(...chunk) + } + return btoa(binary) +} diff --git a/packages/core-ai/src/expo-plugin.ts b/packages/core-ai/src/expo-plugin.ts new file mode 100644 index 00000000..f01fa6c2 --- /dev/null +++ b/packages/core-ai/src/expo-plugin.ts @@ -0,0 +1,262 @@ +import { + ConfigPlugin, + createRunOncePlugin, + IOSConfig, + WarningAggregator, + withDangerousMod, +} from 'expo/config-plugins' +import { existsSync, readFileSync, writeFileSync } from 'fs' + +const pkg = require('../../package.json') + +export type CoreAISwiftPackageProduct = + | 'CoreAILM' + | 'CoreAIDiffusion' + | 'CoreAISegmentation' + | 'CoreAIObjectDetection' + +export type CoreAIExpoPluginOptions = { + products?: CoreAISwiftPackageProduct[] + targetName?: string +} + +const CORE_AI_PACKAGE_URL = 'https://github.com/apple/coreai-models' +const CORE_AI_PACKAGE_NAME = 'coreai-models' +const DEFAULT_PRODUCTS: CoreAISwiftPackageProduct[] = [ + 'CoreAILM', + 'CoreAIDiffusion', + 'CoreAISegmentation', + 'CoreAIObjectDetection', +] + +const withCoreAISwiftPackage: ConfigPlugin = ( + config, + options = {} +) => { + return withDangerousMod(config, [ + 'ios', + (config) => { + const projectPath = IOSConfig.Paths.getPBXProjectPath( + config.modRequest.projectRoot + ) + + if (!existsSync(projectPath)) { + WarningAggregator.addWarningIOS( + 'plugins', + `Could not find project.pbxproj to link ${CORE_AI_PACKAGE_URL}. Add the Swift Package manually.` + ) + return config + } + + const contents = readFileSync(projectPath, 'utf8') + const nextContents = addSwiftPackageProducts(contents, { + products: options?.products ?? DEFAULT_PRODUCTS, + targetName: options?.targetName ?? config.modRequest.projectName, + }) + + if (contents !== nextContents) { + writeFileSync(projectPath, nextContents) + } + + return config + }, + ]) +} + +type AddSwiftPackageOptions = { + products: CoreAISwiftPackageProduct[] + targetName?: string +} + +function addSwiftPackageProducts( + contents: string, + options: AddSwiftPackageOptions +) { + const products = uniqueProducts(options.products) + const projectId = findProjectId(contents) + const target = findApplicationTarget(contents, options.targetName) + const frameworksPhaseId = findFrameworksBuildPhaseId(target.block) + + if (!projectId || !target.id || !frameworksPhaseId) { + WarningAggregator.addWarningIOS( + 'plugins', + `Could not safely link ${CORE_AI_PACKAGE_URL}. Add the Swift Package products manually in Xcode.` + ) + return contents + } + + let nextContents = contents + const packageId = stablePbxId('CoreAI Swift package') + + nextContents = ensureSectionEntry( + nextContents, + 'XCRemoteSwiftPackageReference', + `${packageId} /* XCRemoteSwiftPackageReference "${CORE_AI_PACKAGE_NAME}" */ = { + isa = XCRemoteSwiftPackageReference; + repositoryURL = "${CORE_AI_PACKAGE_URL}"; + requirement = { + branch = main; + kind = branch; + }; + };` + ) + + nextContents = ensureListItem( + nextContents, + projectId, + 'packageReferences', + `${packageId} /* XCRemoteSwiftPackageReference "${CORE_AI_PACKAGE_NAME}" */` + ) + + for (const product of products) { + const productId = stablePbxId(`CoreAI product ${product}`) + const buildFileId = stablePbxId(`CoreAI build file ${product}`) + + nextContents = ensureSectionEntry( + nextContents, + 'XCSwiftPackageProductDependency', + `${productId} /* ${product} */ = { + isa = XCSwiftPackageProductDependency; + package = ${packageId} /* XCRemoteSwiftPackageReference "${CORE_AI_PACKAGE_NAME}" */; + productName = ${product}; + };` + ) + + nextContents = ensureSectionEntry( + nextContents, + 'PBXBuildFile', + `${buildFileId} /* ${product} in Frameworks */ = {isa = PBXBuildFile; productRef = ${productId} /* ${product} */; };` + ) + + nextContents = ensureListItem( + nextContents, + target.id, + 'packageProductDependencies', + `${productId} /* ${product} */` + ) + + nextContents = ensureListItem( + nextContents, + frameworksPhaseId, + 'files', + `${buildFileId} /* ${product} in Frameworks */` + ) + } + + return nextContents +} + +function uniqueProducts(products: CoreAISwiftPackageProduct[]) { + return [...new Set(products)] +} + +function findProjectId(contents: string) { + const projectObject = contents.match( + /([A-F0-9]{24}) \/\* Project object \*\/ = {\n\s+isa = PBXProject;/ + ) + if (projectObject) { + return projectObject[1] + } + + return contents.match( + /([A-F0-9]{24}) \/\* .* \*\/ = {\n\s+isa = PBXProject;/ + )?.[1] +} + +function findApplicationTarget(contents: string, targetName?: string) { + const targetRegex = + /([A-F0-9]{24}) \/\* ([^*]+) \*\/ = {\n\s+isa = PBXNativeTarget;[\s\S]*?\n\t\t};/g + const targets = Array.from(contents.matchAll(targetRegex)).map((match) => ({ + id: match[1], + name: match[2].trim(), + block: match[0], + })) + + const namedTarget = targetName + ? targets.find((target) => target.name === targetName) + : undefined + + return ( + namedTarget ?? + targets.find((target) => + target.block.includes( + 'productType = "com.apple.product-type.application"' + ) + ) ?? + targets[0] ?? { id: undefined, block: '' } + ) +} + +function findFrameworksBuildPhaseId(targetBlock: string) { + return targetBlock.match(/([A-F0-9]{24}) \/\* Frameworks \*\//)?.[1] +} + +function ensureSectionEntry( + contents: string, + sectionName: string, + entry: string +) { + const id = entry.slice(0, 24) + + if (contents.includes(id)) { + return contents + } + + const sectionStart = `/* Begin ${sectionName} section */` + const sectionEnd = `/* End ${sectionName} section */` + + if (contents.includes(sectionStart)) { + return contents.replace(sectionEnd, `\t\t${entry}\n${sectionEnd}`) + } + + return contents.replace( + '/* Begin XCBuildConfiguration section */', + `${sectionStart}\n\t\t${entry}\n${sectionEnd}\n\n/* Begin XCBuildConfiguration section */` + ) +} + +function ensureListItem( + contents: string, + objectId: string, + propertyName: string, + item: string +) { + const objectRegex = new RegExp( + `(${objectId} /\\* [^*]+ \\*/ = \\{[\\s\\S]*?\\n\\t\\t\\};)` + ) + const objectMatch = contents.match(objectRegex) + + if (!objectMatch || objectMatch[1].includes(item)) { + return contents + } + + const objectBlock = objectMatch[1] + const listRegex = new RegExp( + `(\\n\\t\\t\\t${propertyName} = \\(\\n)([\\s\\S]*?)(\\t\\t\\t\\);)` + ) + const listMatch = objectBlock.match(listRegex) + const nextBlock = listMatch + ? objectBlock.replace(listRegex, `$1$2\t\t\t\t${item},\n$3`) + : objectBlock.replace( + /\n\t\t};$/, + `\n\t\t\t${propertyName} = (\n\t\t\t\t${item},\n\t\t\t);\n\t\t};` + ) + + return contents.replace(objectBlock, nextBlock) +} + +function stablePbxId(value: string) { + let hash = 0 + for (let index = 0; index < value.length; index += 1) { + hash = (hash * 31 + value.charCodeAt(index)) >>> 0 + } + return hash.toString(16).toUpperCase().padStart(24, '0').slice(0, 24) +} + +export { addSwiftPackageProducts } + +export default createRunOncePlugin( + withCoreAISwiftPackage, + pkg.name, + pkg.version +) diff --git a/packages/core-ai/src/index.ts b/packages/core-ai/src/index.ts new file mode 100644 index 00000000..20aee593 --- /dev/null +++ b/packages/core-ai/src/index.ts @@ -0,0 +1,29 @@ +export type { CoreAILanguageSession } from './ai-sdk' +export { + coreAI, + CoreAIImageGenerationModel, + CoreAILanguageModel, + CoreAITextEmbeddingModel, + CoreAITranscriptionModel, + createCoreAIProvider, +} from './ai-sdk' +export { default as CoreAI } from './NativeCoreAI' +export type { + CoreAICapabilities, + CoreAIEmbeddingResult, + CoreAIGenerationOptions, + CoreAIGenerationPart, + CoreAIImageGenerationOptions, + CoreAIImageGenerationResult, + CoreAILoadedModel, + CoreAIMessage, + CoreAIModelConfig, + CoreAIModelInfo, + CoreAIModelSource, + CoreAIModelTask, + CoreAIPlatform, + CoreAITaskInput, + CoreAITaskResult, + CoreAITranscriptionResult, +} from './types' +export { toNativeModelConfig } from './types' diff --git a/packages/core-ai/src/types.ts b/packages/core-ai/src/types.ts new file mode 100644 index 00000000..77cf91ce --- /dev/null +++ b/packages/core-ai/src/types.ts @@ -0,0 +1,202 @@ +import type { UnsafeObject } from 'react-native/Libraries/Types/CodegenTypes' + +export type CoreAIPlatform = 'iOS' | 'macOS' + +export type CoreAIModelTask = + | 'language' + | 'embedding' + | 'asr' + | 'diffusion' + | 'segmentation' + | 'object-detection' + | 'depth' + | 'super-resolution' + | 'classification' + | 'encoding' + | 'raw' + | 'unknown' + +export type CoreAIModelSource = + | { + type: 'file' + uri: string + } + | { + type: 'bundle' + name: string + extension?: string + subdirectory?: string + } + +export interface CoreAIModelConfig { + id: string + source: CoreAIModelSource + task?: CoreAIModelTask + family?: string + variant?: CoreAIPlatform +} + +export interface NativeCoreAIModelConfig { + id: string + sourceType: string + sourceUri?: string + bundleName?: string + bundleExtension?: string + bundleSubdirectory?: string + task?: string + family?: string + variant?: string +} + +export interface CoreAITensorDescriptor { + name?: string + dataType?: string + shape?: number[] +} + +export interface CoreAIModelInfo { + id?: string + family?: string + task: CoreAIModelTask + platforms: CoreAIPlatform[] + functions: { + name: string + inputs: CoreAITensorDescriptor[] + outputs: CoreAITensorDescriptor[] + }[] + maxContextLength?: number + modelSizeBytes?: number + metadata?: UnsafeObject +} + +export interface CoreAILoadedModel { + modelHandle: string + info: CoreAIModelInfo +} + +export interface CoreAILanguageSession { + sessionHandle: string +} + +export interface CoreAIMessage { + role: 'assistant' | 'system' | 'tool' | 'user' + content: string +} + +export interface CoreAIGenerationOptions { + temperature?: number + maxTokens?: number + topP?: number + topK?: number + schema?: UnsafeObject + tools?: UnsafeObject +} + +export type CoreAIGenerationPart = + | { type: 'text'; text: string } + | { type: 'reasoning'; text: string } + | { type: 'tool-call'; toolName: string; input: string } + | { type: 'tool-result'; toolName: string; output: string } + +export interface CoreAIStreamUpdateEvent { + streamId: string + content: string +} + +export interface CoreAIStreamCompleteEvent { + streamId: string +} + +export interface CoreAIStreamErrorEvent { + streamId: string + code?: string + error: string +} + +export interface CoreAIEmbeddingResult { + embeddings: number[][] + metadata?: UnsafeObject +} + +export interface CoreAITranscriptionSegment { + text: string + startSecond: number + endSecond: number +} + +export interface CoreAITranscriptionResult { + text: string + segments: CoreAITranscriptionSegment[] + language?: string + durationInSeconds?: number + metadata?: UnsafeObject +} + +export interface CoreAIImageGenerationOptions { + n?: number + size?: `${number}x${number}` + aspectRatio?: `${number}:${number}` + negativePrompt?: string + seed?: number + stepCount?: number + guidanceScale?: number + schedulerType?: string + decodeResolution?: 'auto' | 'full' | 'half' | 'tiled' + strength?: number + lazyModelLoading?: boolean +} + +export interface CoreAIImageGenerationResult { + images: string[] + metadata?: UnsafeObject +} + +export interface CoreAITaskInput { + imageUri?: string + audioUri?: string + text?: string + values?: string[] + data?: UnsafeObject +} + +export interface CoreAITaskResult { + task: CoreAIModelTask + output: UnsafeObject + metadata?: UnsafeObject +} + +export interface CoreAICapabilities { + isCoreAIRuntimeAvailable: boolean + isCoreAILMAvailable: boolean + isCoreAIDiffusionAvailable: boolean + isCoreAISegmentationAvailable: boolean + isCoreAIObjectDetectionAvailable: boolean + supportedPlatform: boolean + missingProducts: string[] +} + +export function toNativeModelConfig( + config: CoreAIModelConfig +): NativeCoreAIModelConfig { + if (config.source.type === 'file') { + return { + id: config.id, + sourceType: 'file', + sourceUri: config.source.uri, + task: config.task, + family: config.family, + variant: config.variant, + } + } + + return { + id: config.id, + sourceType: 'bundle', + bundleName: config.source.name, + bundleExtension: config.source.extension, + bundleSubdirectory: config.source.subdirectory, + task: config.task, + family: config.family, + variant: config.variant, + } +} diff --git a/packages/core-ai/tsconfig.build.json b/packages/core-ai/tsconfig.build.json new file mode 100644 index 00000000..50db0b50 --- /dev/null +++ b/packages/core-ai/tsconfig.build.json @@ -0,0 +1,9 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib/typescript", + "noEmit": false, + "emitDeclarationOnly": true + } +} diff --git a/packages/core-ai/tsconfig.json b/packages/core-ai/tsconfig.json new file mode 100644 index 00000000..d2fecb28 --- /dev/null +++ b/packages/core-ai/tsconfig.json @@ -0,0 +1,7 @@ +{ + "extends": "../../tsconfig.base.json", + "include": ["src"], + "compilerOptions": { + "noEmit": true + } +} diff --git a/website/src/docs/_meta.json b/website/src/docs/_meta.json index e30bf51a..9ec5cb02 100644 --- a/website/src/docs/_meta.json +++ b/website/src/docs/_meta.json @@ -21,6 +21,13 @@ "collapsible": true, "collapsed": false }, + { + "type": "dir", + "name": "core-ai", + "label": "Core AI", + "collapsible": true, + "collapsed": false + }, { "type": "dir", "name": "llama", diff --git a/website/src/docs/core-ai/_meta.json b/website/src/docs/core-ai/_meta.json new file mode 100644 index 00000000..75f9cbd0 --- /dev/null +++ b/website/src/docs/core-ai/_meta.json @@ -0,0 +1,5 @@ +[ + { "type": "file", "name": "getting-started", "label": "Getting Started" }, + { "type": "file", "name": "api", "label": "API Surface" }, + { "type": "file", "name": "models", "label": "Starter Models" } +] diff --git a/website/src/docs/core-ai/api.md b/website/src/docs/core-ai/api.md new file mode 100644 index 00000000..5c1570a2 --- /dev/null +++ b/website/src/docs/core-ai/api.md @@ -0,0 +1,85 @@ +# API Surface + +Core AI has two API layers. + +## AI SDK Adapters + +Use these when you want portable AI SDK app code: + +```typescript +import { coreAI } from '@react-native-ai/core-ai' +import { embed, experimental_transcribe, generateImage, generateText } from 'ai' + +const language = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +await generateText({ model: language, prompt: 'Hello' }) +``` + +AI SDK adapter status: + +| Capability | AI SDK API | Core AI API | Status | +| ----------------- | --------------------------------------------- | -------------------------------- | -------------------------------- | +| Language text | `generateText`, `streamText` | `coreAI.languageModel(...)` | Native wrapper implemented | +| Structured output | `generateText` / `streamText` response format | `coreAI.languageModel(...)` | Routed through language sessions | +| Image generation | `generateImage` | `coreAI.imageModel(...)` | Native diffusion wrapper | +| Text embeddings | `embed`, `embedMany` | `coreAI.embeddingModel(...)` | Explicit unsupported error | +| Transcription | `experimental_transcribe` | `coreAI.transcriptionModel(...)` | Explicit unsupported error | + +The embedding and transcription adapters are present because AI SDK has matching interfaces and Apple’s registry has starter export recipes. They reject until this package gets task-specific native runners for CLIP/CLAP and wav2vec2/Whisper outputs. + +Reranking, speech generation, and video generation are intentionally not exposed as working Core AI adapters yet. AI SDK has model interfaces for reranking and speech, but Apple’s current `coreai-models` registry does not provide a Core AI reranker, TTS model, or video-generation starter. Use `@react-native-ai/apple` for system speech synthesis. + +## Direct Native APIs + +Use direct APIs when Core AI has no AI SDK equivalent or when you need Core AI-specific lifecycle control: + +```typescript +import { CoreAI, coreAI, toNativeModelConfig } from '@react-native-ai/core-ai' + +const config = { + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +} as const + +await CoreAI.inspectModel(toNativeModelConfig({ ...config, task: 'language' })) + +const model = coreAI.languageModel({ + ...config, +}) + +await model.prepare() + +const session = await model.createSession({ + instructions: 'Keep answers short.', +}) + +await session.respond('Create a vocabulary card for "flower".') +``` + +Native-only API status: + +| Capability | API | Status | Why native-only | +| -------------------- | ----------------------------------------- | -------------------------- | -------------------------------------------------------------------------- | +| Model inspection | `CoreAI.inspectModel(...)` | Implemented | AI SDK does not manage Core AI asset metadata. | +| Specialization | `CoreAI.specializeModel(...)` | Metadata stub | Core AI-specific preparation step. | +| Persistent sessions | `model.createSession(...)` | Implemented for LLMs | AI SDK owns message history; Core AI sessions own native transcript state. | +| Segmentation | `CoreAI.runTask('segmentation', ...)` | Implemented | AI SDK image APIs do not model segmentation masks. | +| Object detection | `CoreAI.runTask('object-detection', ...)` | Implemented | AI SDK has no detection primitive. | +| Depth estimation | `CoreAI.runTask('depth', ...)` | Explicit unsupported error | AI SDK has no depth primitive. | +| Super-resolution | `CoreAI.runTask('super-resolution', ...)` | Explicit unsupported error | AI SDK has no image-to-image upscaling primitive. | +| Raw `.aimodel` calls | `CoreAI.runRawFunction(...)` | Explicit unsupported error | Raw tensor APIs need Core AI-specific ownership and performance rules. | + +## Model Management TBD + +Core AI still needs a model-management layer for remote assets. Apple’s Core AI APIs load and specialize local `.aimodel` or `.aimodelc` URLs, so remote models need to be downloaded into app storage before they can be used. + +The intended follow-up is to add APIs that can download a model bundle, return a local `source: { type: 'file', uri }`, optionally call Core AI specialization ahead of first use, and persist bookmark data for specialized cached assets. This should reuse shared download and storage helpers with other runtimes where practical, instead of creating a separate downloader for every provider. + +## Starter Models + +The starter model list in these docs is informational only. The package does not ship a static model catalog; apps should load available models dynamically from their own bundle, download state, or registry integration. diff --git a/website/src/docs/core-ai/getting-started.md b/website/src/docs/core-ai/getting-started.md new file mode 100644 index 00000000..498d90a5 --- /dev/null +++ b/website/src/docs/core-ai/getting-started.md @@ -0,0 +1,109 @@ +# Getting Started + +`@react-native-ai/core-ai` lets React Native apps run app-provided Apple Core AI model bundles. Use it when you export models from [`apple/coreai-models`](https://github.com/apple/coreai-models) and ship or download those bundles in your app. + +Use `@react-native-ai/apple` for Apple system models. Use `@react-native-ai/core-ai` for custom Core AI model assets. + +## Installation + +```bash +npm install @react-native-ai/core-ai +``` + +If you use the AI SDK, install `ai` v6 and the required polyfills: + +```bash +npm install ai +``` + +## Requirements + +- React Native New Architecture +- iOS 27 or newer for Core AI runtime APIs +- Xcode 27 or newer for building apps with Core AI +- Exported `.aimodel` bundles from `apple/coreai-models` +- The Apple Core AI Models Swift Package linked to the host app target + +## Swift Package Setup + +Add `https://github.com/apple/coreai-models` to the iOS app as a Swift Package dependency, then link the products you use: + +This is the intended integration path, but it still needs full verification in an Xcode 27 / iOS 27 host app with Apple Core AI products linked into the app target. + +- `CoreAILM` for `coreAI.languageModel(...)` and sessions +- `CoreAIDiffusion` for `coreAI.imageModel(...)` +- `CoreAISegmentation` for `CoreAI.runTask('segmentation', ...)` +- `CoreAIObjectDetection` for `CoreAI.runTask('object-detection', ...)` + +React Native installs this package through CocoaPods, but CocoaPods will not automatically link those Swift Package products into your app target. Treat SPM as a host-app integration step. + +For Expo, add the config plugin. It links all known Apple Core AI Swift Package products by default: + +```json +{ + "expo": { + "plugins": ["@react-native-ai/core-ai"] + } +} +``` + +Pass `products` only when you want to link a smaller set: + +```json +{ + "expo": { + "plugins": [["@react-native-ai/core-ai", { "products": ["CoreAILM"] }]] + } +} +``` + +The plugin patches the generated Xcode project with `XCRemoteSwiftPackageReference` and links the selected products. For bare React Native, add the Swift Package in Xcode or use a project patching script that modifies `project.pbxproj` predictably. + +The native module checks for missing products and returns clear setup errors instead of failing silently. + +The current native implementation wires language sessions, diffusion image generation, image segmentation, and object detection. Starter-model docs only list those exposed feature families; embeddings, transcription, depth, super-resolution, and classification still need follow-up native wrappers. + +## First iOS Model + +Start with Qwen3 0.6B because Apple’s registry includes an iOS preset: + +```bash +uv run coreai.llm.export Qwen/Qwen3-0.6B --platform iOS --output-dir ./models +``` + +Then load it from React Native: + +```typescript +import { coreAI } from '@react-native-ai/core-ai' +import { streamText } from 'ai' + +const model = coreAI.languageModel({ + id: 'qwen3-0.6b', + source: { type: 'file', uri: qwen3ModelDirectory }, + variant: 'iOS', +}) + +const result = streamText({ + model, + prompt: 'Explain Core AI in one sentence.', +}) + +for await (const delta of result.textStream) { + console.log(delta) +} +``` + +For a persistent native session: + +```typescript +await model.prepare() + +const session = await model.createSession({ + instructions: 'Answer in one short paragraph.', +}) + +const response = await session.respond('What is on-device AI?') + +await session.close() +await model.unload() +``` diff --git a/website/src/docs/core-ai/models.md b/website/src/docs/core-ai/models.md new file mode 100644 index 00000000..851de553 --- /dev/null +++ b/website/src/docs/core-ai/models.md @@ -0,0 +1,26 @@ +# Starter Models + +Apple's repo publishes export recipes, not official ready-made `.aimodel` downloads. This table links the prebuilt Core AI bundles we found online; remaining starter models should use the Apple export recipe until we publish our own hosted bundles, for example from a Callstack Hugging Face org. + +| Type | Model | Platform | Prebuilt Core AI bundle | Apple export recipe | +| ------------------ | -------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| Language | [`qwen3-0.6b`](https://huggingface.co/Qwen/Qwen3-0.6B) | iOS + macOS | [iOS `.aimodel`](https://huggingface.co/mlboydaisuke/qwen3-0.6b-CoreAI-official/tree/main/ios/qwen3_0_6b_mixed_4bit_8bit_static.aimodel), [macOS `.aimodel`](https://huggingface.co/mlboydaisuke/qwen3-0.6b-CoreAI-official/tree/main/macos/qwen3_0_6b_4bit_dynamic.aimodel) | [Qwen3 recipe](https://github.com/apple/coreai-models/tree/main/models/qwen3) | +| Language | [`qwen2.5-1.5b-instruct`](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) | iOS + macOS | Planned Callstack-hosted bundle | [Qwen2.5 recipe](https://github.com/apple/coreai-models/tree/main/models/qwen2) | +| Language | [`qwen3-4b`](https://huggingface.co/Qwen/Qwen3-4B) | iOS + macOS | [iOS `.aimodel`](https://huggingface.co/mlboydaisuke/qwen3-4b-CoreAI-official/tree/main/ios/qwen3_4b_mixed_4bit_8bit_static.aimodel), [macOS `.aimodel`](https://huggingface.co/mlboydaisuke/qwen3-4b-CoreAI-official/tree/main/macos/qwen3_4b_4bit_dynamic.aimodel) | [Qwen3 recipe](https://github.com/apple/coreai-models/tree/main/models/qwen3) | +| Language | [`gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b) | macOS | [macOS `.aimodel`](https://huggingface.co/mlboydaisuke/gpt-oss-20b-CoreAI-official/tree/main/macos/gpt_oss_20b_dynamic.aimodel) | [GPT-OSS recipe](https://github.com/apple/coreai-models/tree/main/models/gpt_oss) | +| Language | [`gemma3-4b-it`](https://huggingface.co/google/gemma-3-4b-it) | macOS | [macOS `.aimodel`](https://huggingface.co/mlboydaisuke/gemma-3-4b-it-CoreAI-official/tree/main/macos/gemma_3_4b_it_4bit_dynamic.aimodel) | [Gemma 3 recipe](https://github.com/apple/coreai-models/tree/main/models/gemma3) | +| Image generation | [`sd-1.5`](https://huggingface.co/runwayml/stable-diffusion-v1-5) | iOS + macOS | Planned Callstack-hosted bundle | [Stable Diffusion recipe](https://github.com/apple/coreai-models/tree/main/models/stable-diffusion) | +| Image generation | [`flux2-klein-4b`](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B) | iOS + macOS | [pipeline bundle](https://huggingface.co/mlboydaisuke/FLUX.2-klein-4B-CoreAI/tree/main) | [FLUX.2 recipe](https://github.com/apple/coreai-models/tree/main/models/flux2) | +| Image segmentation | [`efficient-sam-vitt`](https://huggingface.co/merve/EfficientSAM/blob/main/efficient_sam_vitt.pt) | iOS + macOS | Planned Callstack-hosted bundle | [EfficientSAM recipe](https://github.com/apple/coreai-models/tree/main/models/efficient-sam) | +| Image segmentation | [`sam3`](https://huggingface.co/facebook/sam3) | iOS + macOS | [float16 `.aimodel`](https://huggingface.co/lenitas/coreai-artifacts/tree/main/artifacts/sam3/float16/sam3_float16.aimodel), [w6 palettized `.aimodel`](https://huggingface.co/lenitas/coreai-artifacts/tree/main/artifacts/sam3/w6palett-fp16/sam3_w6palett_fp16.aimodel) | [SAM 3 recipe](https://github.com/apple/coreai-models/tree/main/models/sam3) | +| Object detection | [`yolos-tiny`](https://huggingface.co/hustvl/yolos-tiny) | iOS + macOS | Planned Callstack-hosted bundle | [YOLOS recipe](https://github.com/apple/coreai-models/tree/main/models/yolo) | +| Object detection | [`yolos-base`](https://huggingface.co/hustvl/yolos-base) | iOS + macOS | Planned Callstack-hosted bundle | [YOLOS recipe](https://github.com/apple/coreai-models/tree/main/models/yolo) | + +Example exports for listed models: + +```bash +uv run coreai.llm.export Qwen/Qwen3-0.6B --platform iOS --output-dir ./models +uv run coreai.diffusion.export runwayml/stable-diffusion-v1-5 +uv run models/efficient-sam/export.py --model efficient_sam_vitt +uv run models/yolo/export.py --model hustvl/yolos-tiny +``` diff --git a/website/src/docs/index.md b/website/src/docs/index.md index 07dd6091..9f37fdb7 100644 --- a/website/src/docs/index.md +++ b/website/src/docs/index.md @@ -30,6 +30,17 @@ Native integration with Apple's on-device AI capabilities through `@react-native Production-ready with instant availability on supported iOS devices. +### Apple Core AI + +Run app-provided Core AI model bundles through `@react-native-ai/core-ai`: + +- **Language Sessions** - Custom Core AI language models with persistent native sessions +- **AI SDK Adapters** - Language and image generation today, with embedding and transcription surfaces kept explicit until native runners land +- **Native-only Tasks** - Segmentation, object detection, model inspection, specialization, and model lifecycle calls +- **Starter Models** - iOS-first examples sourced from Apple’s `coreai-models` registry, kept as docs/example data rather than a package catalog + +Requires exported Core AI assets and Apple’s `coreai-models` Swift Package products linked into the host app target. + ### Llama Engine Run any GGUF model from HuggingFace locally using `llama.rn` through `@react-native-ai/llama`: