Skip to content

Latest commit

 

History

History

tools

README.rst

Tools to Create the Conformance Test Suite

The conformance test files originate from the https://github.com/google/cel-spec repository. They are all Protobuf messages, serialized into textproto, like this:

test {
  name: "self_eval_int_zero"
  expr: "0"
  value: { int64_value: 0 }
}

The gherkinize.py script translates these into Gherkin scenarios.

Scenario: self_eval_zeroish/self_eval_int_zero

    When CEL expression '0' is evaluated
    Then value is celpy.celtypes.IntType(source=0)

The behave tool expands the various Gherkin steps into Python functions to arrange a fixture, take an action on the fixture, and assert the actual result is the expected value.

Usage

Gherkin generation is controlled by a Makefile in the features directory which provides two commands:

  • make all checks the cel-spec repository for .textproto files, copies them to the local features directory, and converts them to .features files.
  • make clean removes the .feature and .textproto files.

The underlying command runs the gherkinize.py script for each .textproto file:

% python ../tools/gherkinize.py basic.textproto -o basic.feature

This parses a source .textproto file and generates an equivalent .feature file.

A good way to use this is to do a checkout from https://github.com/google/cel-spec into an adjacent directory. By default, the Makefile looks for <repo>/../../google/cel-spec but the location can be overridden with the CEL_SPEC_PATH environment variable.

refresh_spec.py

This script executes a number of git commands to pull the most recent tag for the google/cel-spec project.

gherkinize.py

Each .textproto file follows the SimpleTestFile schema.

These are deserialized into Python objects using an SDK generated from the schemas.

These Python objects that represent the Protobuf data and then loaded into classes that represented Gherkin concepts (features with comment-delineated sections, and scenarios with given, when and then clauses) and CEL types native to this library.

The classes representing CEL types overload __repr__() with the Python code needed to instantiate the actual CEL types in src/celpy/celtypes.py — this code is used when rendering the Gherkin clauses.

Finally, the classes representing features, sections, and scenarios are rendered to Gherkin tests using the gherkinize.feature template.

Tests with unimplemented features (notably, enums) generate a warning but do not result in scenarios. Tests which do not currently pass are listed in tags.toml. Presence in this file results in a @wip tag being added to the scenario. In general, it's expected that scenarios will be removed from this list over time and once passing, scenarios should never be added back to this file.

Finally, the complete, generated .feature files are all tagged with @conformance.