🎶 pop-test manual

• Frees you and allows you to refactor and test your systems more easily. You can for example completely rewrite a program, even in a different language, and stay certain your implementation works as expected.
• Catches mistakes early – Helps find problems before real users do since you test with real dependencies.
• Test coverage with a real meaning - test your systems from head to toe
• Saves time – No need to test every little part manually, test on higher level.
• Easy to use – Just write simple rules, in a comfy DSL and pop-test does the rest!
• Fast and reliable – Built with Rust, a programming language known for speed and safety.

🚧 stands for work-in-progress and will be used when describing and documenting parts of pop-test which are still under development

🧪 stands for experimental and will be used when describing and documenting parts of pop-test which are not completely tested yet

💡 stands for todo/idea and will be used when describing and documenting future improvements or TODO's

For example tests and learning demos, refer to the scenarios directory: ../../resources/scenarios

Here follows a trivial example of pop-tests.

In this example we spin up a REST API written in Scala, with SBT, and poll its /health endpoint for 120 seconds or until healthy.

If the service is deemed healthy, then we perform some checks on the response of certain HTTP endpoints. These checks are performed using a jql inspired query format, and this allows us to check deeply nested attributes of the JSON response.

Reminder: the tool supports YAML, TOML and JSON, but YAML is as of now the preferred format.

More checks are possible (and encouraged), including PostgreSQL rows existence, Kafka offsets and more. For example's sake, we keep it simple:

pop_services:
  - id: 'green-energy-mix'
    dir: '$HOME/Ontwikkeling/Werk'
    base_url: 'http://localhost:50000'
    start_cmd_args:
      - 'sbt "greenEnergyMix/run"'
    env_vars:
      SOME_TOKEN: '0027809ce9'
    health_check:
      path: '/health'
      patience_seconds: 120

pop_actions:
  - name: 'Check that the forecast endpoint returns a 200 with a correct data, with load and solar'
    assert_url: '/api/v1/forecast'
    service_id: 'green-energy-mix'
    json_patterns:
      - pattern: 'meta.forecastDate'
        condition: 'non-empty-str'
      - pattern: 'data.[0].load'
        condition: '> 0'
      - pattern: 'data.[0].solar'
        condition: '>= 0'

If you downloaded or received a binary from a friend, in most systems you can just add it to ~/.local/bin/ or /usr/local/bin or somewhere in your $PATH.

I also provide container images frequently: https://hub.docker.com/repository/docker/jjba23/pop-test/general

I prefer running directly from the Rust source code, and use Nix as my package manager and development environment.

It is recommended to run with TESTCONTAINERS_COMMAND=keep and RUST_LOG=debug as environment variables.

Also since pop-test will do the resource management for you, we don't need to rely on Ryuk or on testcontainers-rs to do the cleanup.

See the Makefile for more examples of how to run it.

When installed and available as binary, simply run:

TESTCONTAINERS_COMMAND=keep pop-test -p ./path/to/file.yaml

-p or --path (required, string) - path to file or directory that contains scenarios

-h or --help for help

-v or --version for version

This is how I usually run in development mode:

TESTCONTAINERS_COMMAND=keep RUST_LOG=debug nix develop -c cargo run -- --path ./resources/scenarios

Imagine you're building a toy robot, but you don't need to understand all the small parts inside the robot to know if it works. You just need to press a button and see if it performs the task correctly.

A big advantage of this is that you test based on the output, not the code itself or implementation.

Of course this doesn't mean you shouldn't add some unit/property-based tests when useful. But these shouldn't be the bulk of your tests.

It's great when you change or improve your code (refactor), that you don’t need to worry about breaking anything as long as the system gives the same results as before.

It's kind of like saying: "As long as pressing the button still works the same, everything’s fine."

These tests are a safety net for changes to systems (specially if you aren't too familiar with the system yet). It gives you confidence when making changes to the code.

How Does This "Free You"? You can change the internal code without fear because you already have pop-tests that check if everything works as expected. So, you can keep improving the system without worrying about accidentally breaking things.

This also allows for faster development, since you don’t have to check every single little part of the code manually.

As your system grows, you don’t need to rewrite tests every time. You just keep running your pop-tests, and you know that new features won’t break old ones.

In short, pop-tests make engineers happier, business people more certain of what they sell and overall, it makes refactoring and maintaining things easier.

In order to provide an expressive and powerful way to describe test scenarios, a lot of care has gone into creating a good domain-specific language, which remains flexible but concise.

Here follows an explanation of every individual field and parameter possible, including default or optional ones, as well as some recommendations.

Unless otherwise stated, the YAML representation of the Scenario DSL will be used. This is the author's preferred file format, as it can be more readable for complex testing scenarios.

The tool supports YAML, TOML and JSON file formats seamlessly.

I highly recommend that you use the JSON schema for scenarios (PopScenario). This will help you write correct YAML/JSON, and catch errors early.

You can find it in the resources folder, with latest here:

https://codeberg.org/jjba23/pop-test/raw/branch/trunk/resources/json-schema/pop-test.json

Since scenarios are plain data, one can opt to write scenarios in one's favourite programming language, and then serialize that to JSON/YAML and run with pop-test from there.

See ../../src/pop_model.rs for more. Below follows a detailed, as up-to-date as possible overview of pop-test's own data models.

Software testing is an essential part of modern engineering, and pop-test takes it to the next level by making end-to-end testing simple, powerful, and reliable.

Whether you’re refactoring legacy code, maintaining a complex system, or ensuring business continuity, pop-test provides the confidence and efficiency your team needs.

pop-test is designed for modern development teams who need robust testing without complexity. Its expressive DSL, seamless service orchestration, and extensibility make it a long-term solution for ensuring software quality across any technology stack.

With pop-test, you gain more than just a testing tool—you gain a strategic advantage in software development. Engineering teams become more productive, businesses gain confidence in their products, and maintaining large systems becomes easier than ever.

pop-test is not just about testing—it’s about enabling better engineering.

pop-test - your test orchestration master of puppets

Aside:

With the power of pop-test, and taking profit of the rise of LLMs (Large Language Models) , having tests be declarative and expressed in tree like structures we can also open the gates to automated generation of test cases that test your systems as a user would.

pop-test: A Rust-based test orchestration tool designed for integration, end-to-end (e2e), black-box, and other testing methodologies, using a DSL for defining test scenarios.

DSL (Domain-Specific Language): A structured way of defining test scenarios in YAML, TOML, or JSON, enabling automated environment setup and testing.

Integration Testing: Testing that ensures different parts of a system (e.g., services, databases, APIs) work together as expected.

End-to-End (e2e) Testing: Testing the entire flow of an application from start to finish, mimicking real-world usage scenarios.

Black-Box Testing: Testing software functionality without knowledge of its internal implementation.

Golden Test: A test that verifies output against an expected, pre-recorded "golden" reference value.

pop-service: A service defined within a test scenario, representing an application or component under test.

pop-action: A step in a test scenario that performs an assertion, validation, or setup action.

Health Check: A mechanism to verify that a service is running and operational before proceeding with tests.

jql (JSON Query Language): A query format used within pop-test for asserting conditions in JSON responses.

SQL Assertion: A database query within a test scenario that checks expected conditions in a PostgreSQL database.

Kafka Action: A test action that publishes or validates messages in a Kafka topic.

Keycloak Integration: A feature allowing pop-test to authenticate requests using Keycloak-generated access tokens.

Testcontainers: A library for managing containers during test execution, used by pop-test to start dependent services dynamically.

Patience Seconds: The maximum time pop-test waits for a service to become healthy before proceeding with a test.

Orchestrated Service: A service managed by pop-test, such as Kafka, PostgreSQL, or a simulated API.

Environment Variables: Key-value pairs injected into services to configure their behavior dynamically.

Base URL: The root URL of a service under test, used to construct HTTP requests.

Scenario File: A YAML, TOML, or JSON file defining a test case, specifying services, actions, and assertions.

pop-server: A mock web server used in tests to simulate external system responses.

Assertion: A validation step that checks if a condition is met, such as HTTP response codes, database states, or JSON values.

Delay Seconds: A time-based delay before executing an action in a test scenario.

Expected Code: The HTTP status code expected in an assertion (e.g., 200 for success).

Row DSL: A syntax used to express database query assertions within a test scenario.

Quantifiers: Keywords like `[first]`, `[all]`, `[last]`, or `[n]` used in Row DSL to specify which rows should be checked.

pop Kafka: The built-in Kafka container service used in tests, providing an event-driven messaging system.

pop PostgreSQL: The built-in PostgreSQL container service used in tests for verifying database-related behavior.

pop Keycloak: The built-in Keycloak container service used in tests for handling authentication and authorization.

pop Docker Image: A pre-built container image for running pop-test within a containerized environment.

Inline Script: A shell command or script embedded directly within a test scenario to perform additional setup, teardown, or validation tasks.

pop-storage: A storage abstraction in pop-test that allows interacting with temporary file systems, databases, or cloud storage during tests.