Skip to content

Commit 3f29c83

Browse files
committed
soda-core support
1 parent 717ba36 commit 3f29c83

15 files changed

Lines changed: 2424 additions & 48 deletions

File tree

README.md

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,13 @@ The CLI is functional for core workflows. Here's where things stand:
1717
| Auth (login, logout, status, profiles) | Working |
1818
| Datasource (list, get, create, delete, onboard) | Working |
1919
| Dataset (list, get, update, delete, profiling, permissions, onboard) | Working |
20-
| Contract (list, push, pull, diff, create, verify via cloud) | Working |
20+
| Contract (list, push, pull, diff, create, lint, verify via cloud or local) | Working |
2121
| Monitor (list, config, add column/custom, update, delete) | Working |
2222
| Results (list with filtering, sorting, date ranges) | Working |
2323
| Runner (list, get, create, delete) | Working |
2424
| IAM (user list, group CRUD, role list) | Working |
2525
| Job logs | Working |
26-
| Contract verify (local via soda-core) | Planned |
26+
| Contract verify (local via soda-core) | Working |
2727
| Incidents | Planned |
2828
| Notifications, Secrets | Planned |
2929
| Dashboard | Planned |
@@ -104,9 +104,12 @@ sodacli datasource onboard <datasource-id> --monitoring --profiling --contracts
104104
### 3. Verify a contract
105105

106106
```bash
107-
# Run checks against your data
107+
# Run checks via Soda Cloud Runner
108108
sodacli contract verify orders.yml
109109

110+
# Or run locally via soda-core (no cloud needed)
111+
sodacli contract verify orders.yml --local --datasource datasource.yml
112+
110113
# Check results
111114
sodacli results list --status failing
112115
sodacli job logs <scan-id>
@@ -155,10 +158,12 @@ sodacli contract create --dataset ds/db/schema/table --mode copilot # AI-ge
155158
sodacli contract pull ds/db/schema/table # download from cloud
156159
sodacli contract push my_table.yml # upload to cloud
157160
sodacli contract diff my_table.yml # local vs cloud diff
158-
sodacli contract lint my_table.yml # validate syntax
161+
sodacli contract lint my_table.yml # validate syntax (offline)
162+
sodacli contract lint contracts/*.yml # lint multiple files
159163
sodacli contract verify my_table.yml # run checks via cloud Runner
160164
sodacli contract verify my_table.yml --no-wait # fire and forget
161-
sodacli contract verify my_table.yml --runner soda-core --datasource config.yml # run locally (planned)
165+
sodacli contract verify my_table.yml --local --datasource config.yml # run locally via soda-core
166+
sodacli contract verify my_table.yml --local --datasource config.yml --push # run locally + push results to cloud
162167
```
163168

164169
### Monitors
@@ -212,9 +217,12 @@ sodacli auth login \
212217
--api-key-secret "$SODA_API_KEY_SECRET" \
213218
--no-interactive
214219

215-
# Run contract checks
220+
# Run contract checks (via cloud Runner)
216221
sodacli contract verify contracts/orders.yml --no-interactive --output json
217222

223+
# Or run locally (no cloud auth needed, just soda-core on PATH)
224+
sodacli contract verify contracts/orders.yml --local --datasource datasource.yml
225+
218226
# Exit codes
219227
# 0 = all checks passed
220228
# 1 = one or more checks failed → fail the pipeline
@@ -288,8 +296,6 @@ The CLI code is written for these. They'll work as soon as the API endpoints shi
288296

289297
### Planned Features
290298

291-
- **Local contract execution.** `sodacli contract verify --runner soda-core` runs checks locally via the soda-core Python engine, no Soda Cloud needed.
292-
- **Real contract linting.** `sodacli contract lint` using soda-core for full schema validation.
293299
- **Dashboard.** Org-level overview of datasets, results, and incidents.
294300
- **Contract proposals.** PR-style review flow for contract changes.
295301

@@ -306,3 +312,15 @@ The goal is one CLI that covers the full data quality lifecycle:
306312
7. **Govern.** `sodacli iam` and `sodacli dataset permissions` control who can do what.
307313

308314
All of this works the same way for humans typing commands and for AI agents calling them programmatically. Same interface, same exit codes, same JSON output.
315+
316+
## Soda CLI vs soda-core
317+
318+
| | Soda CLI (`sodacli`) | soda-core (`soda`) |
319+
|---|---|---|
320+
| **Language** | Go (single binary, no dependencies) | Python (requires pip + DB connectors) |
321+
| **Execution** | Cloud via Soda Runner, or local via `--local` | Local only |
322+
| **Scope** | Full platform: datasources, datasets, contracts, monitors, results, IAM, incidents | Contract verification and data source testing |
323+
| **Contract generation** | `contract create --mode copilot` (AI) or `skeleton` | Manual authoring only |
324+
| **CI/CD** | `--no-interactive`, `--output json`, structured exit codes | Basic exit codes |
325+
326+
**Why use Soda CLI?** If you only need to run checks locally, soda-core is enough. If you want to manage your entire data quality lifecycle from one tool — generate contracts with AI, monitor anomalies, track results, control permissions, and integrate with CI/CD — use sodacli. It shells out to soda-core for local execution when needed (`--local`), so you get both.

command_tree.txt

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,10 @@ sodacli
9393
│ │ [--no-wait] # start generation and return immediately (copilot mode only)
9494
│ │ # skeleton → async: POST /contracts/actions/createSkeleton → poll status → fetch contract
9595
│ │ # copilot → async: POST /contracts/actions/generate → poll status → fetch contract (requires license)
96-
│ ├── lint (alias: validate) [<file>] # validate contract YAML via soda-core 🔌 [requires soda-core on PATH]
97-
│ │ # uses soda-core for real schema validation; falls back to basic YAML parse if not installed
96+
│ ├── lint (alias: validate) [<file...>] # validate contract YAML against JSON schema 🏠
97+
│ │ # validates structure, properties, and types against embedded contract schema
98+
│ │ # supports multiple files and glob patterns; defaults to contracts/*.yml
99+
│ │ # no auth or network required
98100
│ ├── push [<file>] # push contract definition to cloud (upsert) ✅
99101
│ │ # reads 'dataset:' field from file to find/create the contract
100102
│ ├── pull <identifier> # pull contract from cloud → <table>.yml ✅
@@ -108,15 +110,13 @@ sodacli
108110
│ │ # --no-interactive → fails with clear error describing what's missing
109111
│ │ --output <file>
110112
│ ├── verify <file> # verify contract checks against data
111-
│ │ [--datasource <file>] # explicit datasource config override
112-
│ │ [--runner <name>] # execution target:
113-
│ │ │ # (omitted) → push to Soda Cloud, verify via cloud Runner ✅
114-
│ │ │ # soda-core → run locally via soda-core engine 🔌 [requires soda-core on PATH]
115-
│ │ [--push] # push results to Soda Cloud (useful with --runner soda-core)
113+
│ │ [--datasource <file>] # datasource config file (required with --local)
114+
│ │ [--local] # run locally via soda-core 🔌 [requires soda-core on PATH]
115+
│ │ [--push] # push results to Soda Cloud (useful with --local)
116116
│ │ [--no-wait] # start verification and return immediately (cloud mode only) ✅
117117
│ │ [--set key=value] # runtime variable overrides (repeatable)
118-
│ │ # cloud mode: push contract → POST /contracts/{id}/verify → poll GET /scans/{id} → results
119-
│ │ # local mode: shell out to soda-core, requires --datasource or soda.yml, prompts install if missing
118+
│ │ # default: push contract → POST /contracts/{id}/verify → poll GET /scans/{id} → results
119+
│ │ # --local: shell out to soda-core, requires --datasource, prompts install if missing
120120
│ │ # exit codes: 0=pass 1=checks failed 2=error 3=auth error
121121
│ └── proposal # PR flow for published contracts ❌ [NOT YET IMPLEMENTABLE - API work required]
122122
│ ├── list [--status open|done|all]

go/cmd/contract.go

Lines changed: 203 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,10 @@ import (
1212
"gopkg.in/yaml.v3"
1313

1414
"github.com/soda-data-inc/soda-cli/internal/api"
15+
"github.com/soda-data-inc/soda-cli/internal/config"
16+
"github.com/soda-data-inc/soda-cli/internal/lint"
1517
"github.com/soda-data-inc/soda-cli/internal/output"
18+
"github.com/soda-data-inc/soda-cli/internal/sodacore"
1619
)
1720

1821
var contractCmd = &cobra.Command{
@@ -276,20 +279,99 @@ var contractDiffCmd = &cobra.Command{
276279
// ── contract lint ─────────────────────────────────────────────────────────────
277280

278281
var contractLintCmd = &cobra.Command{
279-
Use: "lint [file]",
282+
Use: "lint [file...]",
280283
Aliases: []string{"validate"},
281284
Short: "Validate contract syntax (no network required)",
285+
Long: `Validate contract YAML files against the Soda data contract schema.
286+
287+
Checks structure, property names, and value types without connecting to Soda Cloud.
288+
Supports multiple files and glob patterns.
289+
290+
Exit codes: 0=valid, 2=validation errors found`,
282291
RunE: func(cmd *cobra.Command, args []string) error {
283-
file := "contracts/*.yml"
284-
if len(args) > 0 {
285-
file = args[0]
292+
files := resolveLintFiles(args)
293+
if len(files) == 0 {
294+
return output.Errorf(2, "no contract files found — provide file paths or place contracts in a contracts/ directory")
286295
}
287-
fmt.Println(output.Dim.Render(" Linting " + file + "..."))
288-
output.PrintSuccess("Contract syntax is valid.", GCtx)
289-
return nil
296+
297+
results, err := lint.LintFiles(files)
298+
if err != nil {
299+
return output.Errorf(2, "lint error: %v", err)
300+
}
301+
302+
return displayLintResults(results)
290303
},
291304
}
292305

306+
// resolveLintFiles expands arguments (which may contain globs) into file paths.
307+
// Falls back to contracts/*.yml then *.yml in the current directory.
308+
func resolveLintFiles(args []string) []string {
309+
if len(args) > 0 {
310+
var files []string
311+
for _, arg := range args {
312+
matches, err := filepath.Glob(arg)
313+
if err != nil || len(matches) == 0 {
314+
// Treat as literal file path
315+
files = append(files, arg)
316+
} else {
317+
files = append(files, matches...)
318+
}
319+
}
320+
return files
321+
}
322+
// Default: contracts/*.yml, then *.yml
323+
if matches, _ := filepath.Glob("contracts/*.yml"); len(matches) > 0 {
324+
return matches
325+
}
326+
if matches, _ := filepath.Glob("*.yml"); len(matches) > 0 {
327+
return matches
328+
}
329+
return nil
330+
}
331+
332+
func displayLintResults(results []*lint.LintResult) error {
333+
if output.EffectiveFmt(GCtx) == "json" {
334+
s, err := lint.ResultsJSON(results)
335+
if err != nil {
336+
return output.Errorf(2, "failed to encode results: %v", err)
337+
}
338+
fmt.Println(s)
339+
for _, r := range results {
340+
if !r.Valid {
341+
return output.Errorf(2, "")
342+
}
343+
}
344+
return nil
345+
}
346+
347+
hasErrors := false
348+
totalFiles := len(results)
349+
validFiles := 0
350+
351+
for _, r := range results {
352+
if r.Valid {
353+
validFiles++
354+
if !GCtx.Quiet {
355+
fmt.Println(" " + output.Green.Render("✓") + " " + r.File)
356+
}
357+
} else {
358+
hasErrors = true
359+
fmt.Println(" " + output.Red.Render("✗") + " " + r.File)
360+
for _, e := range r.Errors {
361+
fmt.Printf(" %s: %s\n", output.Dim.Render(e.Path), e.Message)
362+
}
363+
}
364+
}
365+
366+
fmt.Println()
367+
if hasErrors {
368+
invalidFiles := totalFiles - validFiles
369+
return output.Errorf(2, "%d file(s) checked, %d valid, %d with errors", totalFiles, validFiles, invalidFiles)
370+
}
371+
output.PrintSuccess(fmt.Sprintf("All %d contract file(s) are valid.", totalFiles), GCtx)
372+
return nil
373+
}
374+
293375
// ── contract create ───────────────────────────────────────────────────────────
294376

295377
var contractCreateCmd = &cobra.Command{
@@ -570,13 +652,23 @@ var contractVerifyCmd = &cobra.Command{
570652
Short: "Run contract checks against your data",
571653
Long: `Execute data quality checks defined in a contract file.
572654
573-
Pushes the contract to Soda Cloud and triggers verification via a Runner.
655+
By default, pushes the contract to Soda Cloud and triggers verification via a Runner.
574656
Polls for results and displays a summary.
575657
658+
With --local, runs verification locally via soda-core (must be on PATH).
659+
In local mode, --datasource <config.yml> is required.
660+
Use --push to publish local results to Soda Cloud.
661+
576662
Exit codes: 0=all passing, 1=checks failed, 2=error, 3=auth error`,
577663
Args: cobra.ExactArgs(1),
578664
RunE: func(cmd *cobra.Command, args []string) error {
579665
file := args[0]
666+
local, _ := cmd.Flags().GetBool("local")
667+
668+
if local {
669+
return runContractVerifyLocal(cmd, file)
670+
}
671+
580672
noWait, _ := cmd.Flags().GetBool("no-wait")
581673

582674
client, err := newAPIClient()
@@ -748,6 +840,107 @@ done:
748840
}
749841
}
750842

843+
// ── contract verify --local ───────────────────────────────────────────────────
844+
845+
func runContractVerifyLocal(cmd *cobra.Command, contractFile string) error {
846+
// Validate contract file exists
847+
if _, err := os.Stat(contractFile); err != nil {
848+
return output.Errorf(2, "could not read file %s: %v", contractFile, err)
849+
}
850+
851+
// --datasource is required in local mode
852+
datasourceFile, _ := cmd.Flags().GetString("datasource")
853+
if datasourceFile == "" {
854+
return output.Errorf(2, "--datasource <config-file> is required in local mode\n Example: sodacli contract verify orders.yml --local --datasource datasource.yml")
855+
}
856+
if _, err := os.Stat(datasourceFile); err != nil {
857+
return output.Errorf(2, "datasource config file not found: %s", datasourceFile)
858+
}
859+
860+
// --no-wait doesn't apply in local mode
861+
if noWait, _ := cmd.Flags().GetBool("no-wait"); noWait {
862+
fmt.Println(output.Yellow.Render(" Warning:") + " --no-wait is ignored in local mode (soda-core runs synchronously)")
863+
}
864+
865+
// Find soda-core binary
866+
binPath, err := sodacore.FindBinary()
867+
if err != nil {
868+
return err
869+
}
870+
871+
// Show version in verbose mode
872+
if GCtx.Verbose {
873+
version := sodacore.CheckVersion(binPath)
874+
fmt.Println(output.Dim.Render(" soda-core version: " + version))
875+
}
876+
877+
// Handle --push: build temp soda-cloud config
878+
push, _ := cmd.Flags().GetBool("push")
879+
var cloudConfigPath string
880+
var cleanup func()
881+
if push {
882+
creds, err := config.LoadCredentials()
883+
if err != nil {
884+
return output.Errorf(2, "could not read credentials for --push: %v", err)
885+
}
886+
profile, err := config.GetProfile(GCtx.Profile, creds)
887+
if err != nil {
888+
return output.Errorf(3, "--push requires authentication: %v", err)
889+
}
890+
cloudConfigPath, cleanup, err = sodacore.WriteTempCloudConfig(profile.Host, profile.APIKeyID, profile.APIKeySecret)
891+
if err != nil {
892+
return output.Errorf(2, "could not create soda-cloud config: %v", err)
893+
}
894+
defer cleanup()
895+
}
896+
897+
// Build args
898+
setVars, _ := cmd.Flags().GetStringArray("set")
899+
opts := sodacore.VerifyOpts{
900+
ContractFile: contractFile,
901+
DatasourceFile: datasourceFile,
902+
SetVars: setVars,
903+
Verbose: GCtx.Verbose,
904+
Publish: push,
905+
SodaCloudFile: cloudConfigPath,
906+
}
907+
cliArgs := sodacore.BuildVerifyArgs(opts)
908+
909+
// Print what we're about to do
910+
if !GCtx.Quiet {
911+
fmt.Println(output.Dim.Render(" Running locally via soda-core..."))
912+
if GCtx.Verbose {
913+
fmt.Println(output.Dim.Render(" Command: soda " + strings.Join(cliArgs, " ")))
914+
}
915+
}
916+
917+
// Execute
918+
stream := output.EffectiveFmt(GCtx) != "json"
919+
result, err := sodacore.Run(binPath, cliArgs, stream)
920+
if err != nil {
921+
return output.Errorf(2, "failed to execute soda-core: %v", err)
922+
}
923+
924+
// JSON output mode: wrap soda-core output
925+
if !stream {
926+
fmt.Printf(`{"local": true, "exit_code": %d, "output": %q}`+"\n", result.ExitCode, result.Stdout)
927+
}
928+
929+
// Map exit code
930+
mapped := sodacore.MapExitCode(result.ExitCode, result.Stderr)
931+
if mapped == 0 {
932+
if !GCtx.Quiet && stream {
933+
output.PrintSuccess("Local verification passed.", GCtx)
934+
}
935+
return nil
936+
}
937+
if mapped == 1 {
938+
// soda-core already printed the failure summary; just set the exit code.
939+
return output.Errorf(1, "")
940+
}
941+
return output.Errorf(2, "soda-core exited with error (exit code: %d)", result.ExitCode)
942+
}
943+
751944
// ── contract proposal ─────────────────────────────────────────────────────────
752945

753946
var contractProposalCmd = &cobra.Command{
@@ -904,8 +1097,8 @@ func init() {
9041097
contractCopilotCmd.Flags().String("dataset", "", "Dataset FQN to generate from")
9051098
contractCopilotCmd.Flags().String("output", "", "Output file path")
9061099

907-
contractVerifyCmd.Flags().String("datasource", "", "Datasource config file override")
908-
contractVerifyCmd.Flags().Bool("runner", false, "Delegate execution to Soda Runner")
1100+
contractVerifyCmd.Flags().String("datasource", "", "Datasource config file (required with --local)")
1101+
contractVerifyCmd.Flags().Bool("local", false, "Run verification locally via soda-core (requires soda-core on PATH)")
9091102
contractVerifyCmd.Flags().Bool("push", false, "Push results to Soda Cloud after verification")
9101103
contractVerifyCmd.Flags().Bool("no-wait", false, "Start verification and return immediately without waiting for results")
9111104
contractVerifyCmd.Flags().StringArray("set", nil, "Runtime variable overrides (key=value)")

go/go.mod

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,11 @@ require (
66
github.com/charmbracelet/huh v0.6.0
77
github.com/charmbracelet/lipgloss v1.0.0
88
github.com/olekukonko/tablewriter v0.0.5
9+
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
910
github.com/spf13/cobra v1.8.1
1011
golang.org/x/term v0.27.0
12+
golang.org/x/text v0.18.0
13+
gopkg.in/yaml.v3 v3.0.1
1114
)
1215

1316
require (
@@ -34,6 +37,4 @@ require (
3437
github.com/spf13/pflag v1.0.5 // indirect
3538
golang.org/x/sync v0.8.0 // indirect
3639
golang.org/x/sys v0.28.0 // indirect
37-
golang.org/x/text v0.18.0 // indirect
38-
gopkg.in/yaml.v3 v3.0.1 // indirect
3940
)

0 commit comments

Comments
 (0)