Validating yang structure data

[ahassany]

I've tried to validate the example in RFC 8791 using yanglint but without a success, is yang structures fully supported for data and schema or schema only?

This the examples I tried:

module example-module {
  yang-version 1.1;
  namespace "urn:example:example-module";
  prefix exm;

  import ietf-yang-structure-ext {
    prefix sx;
  }

  sx:structure address-book {
    list address {
      key "last first";
      leaf last {
        type string;
      }
      leaf first {
        type string;
      }
      leaf street {
        type string;
      }
      leaf city {
        type string;
      }
      leaf state {
        type string;
      }
    }
  }
}

and input data:

{
  "example-module:address-book": {
    "example-module:address": [
      {
        "city": "Bedrock",
        "example-module-aug:zipcode": "70777",
        "first": "Fred",
        "last": "Flintstone",
        "street": "301 Cobblestone Way"
      },
      {
        "city": "Bedrock",
        "example-module-aug:zipcode": "70777",
        "first": "Charlie",
        "last": "Root",
        "street": "4711 Cobblestone Way"
      }
    ]
  }
}

witht the command:

yanglint  example-module.yang  example.json -f json

libyang err : Node "address-book" not found in the "example-module" module. (line 2)
YANGLINT[E]: Failed to parse input data file "example.json".

[michalvasko]

It seems yanglint(1) does not support this but libyang does. You would have to call lyd_parse_ext_data() yourself in a small app. We may add the support for this in yanglint later.

[ahassany]

Thank you @michalvasko, I will check with my team if we can allocate capacity to contribute back to yanglint for this feature.

[mxyns]

Hi @michalvasko,
I'm not a Yang expert, first time having a look at libyang and yanglint. I've had a look at the state of Yang structure extension in both.

The RFC8791 for the Yang data structure extension allows for structures to be at any level of the tree

There is no assumption that a YANG
   data structure can only be used as a top-level abstraction, and it
   may also be nested within some other data structure.

It seems like structures are supported in libyang as a top-level node only structure.c@63: /* structure can appear only at the top level of a YANG module or submodule */ am I understanding that correctly?

It looks like if you want to validate using an extension (here a structure), you need to tell the json/xml/... parser which extension instance (singular) to use to parse & validate before the file is even read.
This is because the api for "normal" parsing is

LIBYANG_API_DECL LY_ERR lyd_parse_ext_data(const struct lysc_ext_instance *ext, struct lyd_node *parent, struct ly_in *in,
        LYD_FORMAT format, uint32_t parse_options, uint32_t validate_options, struct lyd_node **tree);

while the api for "normal" w/o extension parsing is

static LY_ERR
lyd_parse(const struct ly_ctx *ctx, const struct lysc_ext_instance *ext, struct lyd_node *parent, struct lyd_node **first_p,
        struct ly_in *in, LYD_FORMAT format, uint32_t parse_opts, uint32_t val_opts, struct lyd_node **op);

It seems like "normal" parsing you just pass the context, and the json/xml parser infers the module to parse with from there using the context. It also looks like the parsing w/ extension requires you to pass the extension instance directly (no inference here).

So, from what I understood of the yanglint code, there is currently no way of doing the validation of a structure with yanglint at the moment.

Assuming, the structure is always the top-level, and is the only structure of the tree, one way to fix that would be to make the user pass as CLI parameter the exact module name to validate the data against, on top of the list of modules to load. Something like yanglint module-1.yang module-2.yang message.json --top-level module-1.yang. From there we can programmatically find the exact extension instance used for the top-level and pass it to the parser.

If this is not acceptable, we would need to change the parsers so that they first determine if the top-level node uses an extension or not, and parse based on that.

Am I on the right track here? If so, is the first idea of adding a parameter to yanglint acceptable?

[michalvasko]

We actually have a student working on this but it may take some time.

Regarding the nested structure support, I am confused because in the YANG module there is

          This extension is only valid as a top-level statement, i.e.,
          given as a substatement to 'module' or 'submodule'.

Also, you are correct that yanglint will need some parameters for top-level structures, something like --yang-structure <name>, you need to know both that it is a structure and which exactly.

[mxyns]

Thank you @michalvasko

Regarding the nested structure support, I am confused because in the YANG module there is

    This extension is only valid as a top-level statement, i.e.,
    given as a substatement to 'module' or 'submodule'.

That is indeed confusing 😅
I've asked a colleague of mine. To them, this means that the structure has to be top-level in a (sub-)module definition. However, that structure might not be top-level in whatever data you try to validate. That can be the case if the data you have has a structure contained in an anydata node of another module. This makes sense to me.

This would mean that we need to change the parsers to detect if any node of the tree is a structure and not only at top-level. Maybe we can begin with just the --yang-structure <module-name.yang> and enhance that later?

[mxyns]

Also, I'm not sure just the structure name or just the module name is enough. We also need the name of the module in which the structure is defined since multiple modules can define a structure with identical names, right?

[michalvasko]

I have asked about the possible placement of structure data on the netmod mailing list. But regardless of that, the feature will not be changed for now and the yanglint support is actually almost implemented and should be merged into devel today. I will reply to this issue once it is.

[mxyns]

okay that's great thanks

[mxyns]

linking the PR to the issue for future reference #2410

[michalvasko]

PR has been merged, you can test the feature.

[mxyns]

hi @michalvasko @Ardival
i'm trying to validate the example json given by @ahassany with the example-module as described in the updated README.md.

I've removed the structure augmentation from example-module-aug for now so which means I'm validating example.json:

{
  "example-module:address-book": {
    "example-module:address": [
      {
        "city": "Bedrock",
        "first": "Fred",
        "last": "Flintstone",
        "street": "301 Cobblestone Way"
      },
      {
        "city": "Bedrock",
        "first": "Charlie",
        "last": "Root",
        "street": "4711 Cobblestone Way"
      }
    ]
  }
}

with example-module.yang:

module example-module {
  yang-version 1.1;
  namespace "urn:example:example-module";
  prefix exm;

  import ietf-yang-structure-ext {
    prefix sx;
  }

  sx:structure address-book {
    list address {
      key "last first";
      leaf last {
        type string;
      }
      leaf first {
        type string;
      }
      leaf street {
        type string;
      }
      leaf city {
        type string;
      }
      leaf state {
        type string;
      }
    }
  }
}

here is what i get

max@max-tk:~/CLionProjects/libyang$ yanglint 
> clear
> add /home/max/yang/example-module.yang
> list
List of the loaded models:
    i ietf-yang-metadata@2016-08-05
    I yang@2025-01-29
    i ietf-inet-types@2013-07-15
    i ietf-yang-types@2013-07-15
    I ietf-yang-schema-mount@2019-01-14
    i ietf-yang-structure-ext@2020-06-17
    I example-module
> data -t ext -k example-module:structure:address-book /home/max/yang/example.json
YANGLINT[E]: Nothing to validate in the extension input data.
YANGLINT[E]: Failed to parse input data file "/home/max/yang/example.json".
> 

do you see what I'm doing wrong? seems like the json parser returns a NULL tree after the lyd_parse_ext_data but I'm not sure why

[mxyns]

I've also tried with

{
  "example-module:address": [
    {
      "city": "Bedrock",
      "first": "Fred",
      "last": "Flintstone",
      "street": "301 Cobblestone Way"
    },
    {
      "city": "Bedrock",
      "first": "Charlie",
      "last": "Root",
      "street": "4711 Cobblestone Way"
    }
  ]
}

and I get the following error

YANGLINT[E]: Yanglint does not support more than one top-level node in extension data.
YANGLINT[E]: Failed to parse input data file "/home/max/yangshi/example.json".