Negative testing¶

Generate negative test cases that verify your API rejects invalid input according to your OpenAPI spec.

What gets tested¶

Running and verifying¶

Choose an output type¶

• Executable tests: use the template generator (Gradle plugin recommended). Generated sources are compiled and run by ./gradlew test.
• Data-driven suites: use the test-suite-writer generator (CLI or Gradle). Output is .json / .yaml consumed by your own runner.

CLI: generate JSON suites¶

Prerequisite: have openapi-testgen available (install it or run via npx). See Installation.

For the full CLI command and output walkthrough, see Getting started. To target a specific operation, see Include operations.

Provide security values (avoid placeholders)¶

validSecurityValues keys are security scheme names (from components.securitySchemes), not header names:

openapi-testgen \
  --spec-file ./openapi.yaml \
  --output-dir ./generated \
  --generator test-suite-writer \
  --generator-option format=json \
  --generator-option outputFileName=test-suites.json \
  --setting 'validSecurityValues.ApiKeyAuth=test-api-key-123'

Verify output¶

After generating, verify the output using jq. See Getting started for a full output example.

jq 'keys' generated/test-suites.json
jq '[.[] | .testCases[]] | length' generated/test-suites.json

Gradle plugin¶

For Gradle plugin setup and configuration, see Gradle integration.

To use environment variables for security values and base URLs in CI, see CI/CD integration.

Path parameters¶

Path parameters are always required in OpenAPI (they're part of the URL path). The generator creates test cases for:

• Schema violations: Invalid values that don't match the parameter's schema constraints (wrong type, out of range, invalid format, pattern mismatches)
• Format violations: Values that don't match specified formats (uuid, email, date, etc.)

The ParameterSchemaValidationTestProvider orchestrates path parameter test generation, delegating to individual SchemaValidationRule implementations.

Example OpenAPI spec¶

paths:
    /users/{userId}:
        get:
            operationId: getUser
            parameters:
                -   name: userId
                    in: path
                    required: true
                    schema:
                        type: string
                        pattern: '^[a-z0-9_]+$'

Invalid path parameter test cases¶

Using the samples/openapi.yaml spec, the generator produces:

For UUID-formatted path parameters (e.g., format: uuid), the core fixtures include:

Inspecting output¶

{
    "name": "Invalid Path userId parameter: Invalid Pattern",
    "method": "GET",
    "path": "/users/{userId}",
    "pathParams": { "userId": "AE." },
    "headers": [{ "key": "X-API-Key", "value": "test-api-key-123" }],
    "body": null,
    "expectedStatusCode": 400,
    "rule": "art.galushko.openapi.testgen.pattern.support.InvalidPatternSchemaValidationRule"
}

Key fields: pathParams (the invalid value), expectedStatusCode (400), rule (FQCN of the rule).

Format violations¶

When a path parameter uses format: uuid, the generator emits a wrong-UUID case:

{
    "name": "Invalid Path userId parameter: Wrong UUID Format",
    "method": "DELETE",
    "path": "/users/{userId}",
    "pathParams": { "userId": "8e258b27-c787-49ef-9539-11461b251ffg" },
    "expectedStatusCode": 400,
    "rule": "art.galushko.openapi.testgen.rules.schema.WrongUuidFormatSchemaValidationRule"
}

The invalid UUID 8e258b27-c787-49ef-9539-11461b251ffg has a g character (invalid hex).

Pattern-based validation¶

When path parameters have a pattern constraint, the pattern-support module generates invalid values using InvalidPatternSchemaValidationRule.

For the pattern ^[a-z0-9_]+$ (lowercase alphanumeric with underscores):

• Valid: user_123, abc
• Invalid: AE. (uppercase and dot violate the pattern)

Applicable rules (path parameters)¶

Query parameters¶

The generator produces two categories of query parameter tests:

Missing required query parameter¶

When a query parameter has required: true, the generator creates a test case that omits the parameter entirely, expecting the API to return a 400 Bad Request.

paths:
    /persons:
        get:
            operationId: listPersons
            parameters:
                -   name: person
                    in: query
                    required: true
                    schema:
                        $ref: '#/components/schemas/Person'

Generated test case:

{
    "name": "Missed Required Query Parameter person",
    "method": "GET",
    "path": "/persons",
    "queryParams": {},
    "expectedStatusCode": 400,
    "rule": "art.galushko.openapi.testgen.providers.parameter.MissedRequiredParameterTestProvider"
}

Complex object query parameters¶

When a query parameter references an object schema with required properties, the generator produces tests for each missing required property. For the Person schema:

components:
    schemas:
        Person:
            type: object
            required: [ name, age ]
            properties:
                name:
                    type: string
                    minLength: 1
                    maxLength: 100
                age:
                    type: integer
                    minimum: 0
                    maximum: 150
                address:
                    $ref: '#/components/schemas/Address'

The generator creates these test cases:

• Invalid Query person parameter: Missed Required Object Properties age
• Invalid Query person parameter: Missed Required Object Properties name
• Invalid Query person parameter: Object Property address Missed Required Object Properties city
• Invalid Query person parameter: Object Property address Missed Required Object Properties street

Invalid query parameter values¶

Object property constraints generate invalid-value tests. For the Person schema above:

• Invalid Query person parameter: Object Property age Integer Breaking - Non-integer value
• Invalid Query person parameter: Object Property age Invalid Type - Wrong type
• Invalid Query person parameter: Object Property age Out Of Maximum Boundary Number - Exceeds max
• Invalid Query person parameter: Object Property age Out Of Minimum Boundary Number - Below min
• Invalid Query person parameter: Object Property name Out Of Maximum Length String - Too long
• Invalid Query person parameter: Object Property name Out Of Minimum Length String - Too short

Fixture-backed example (invalid type for age):

{
    "name": "Invalid Query person parameter: Object Property age Invalid Type",
    "method": "GET",
    "path": "/persons",
    "queryParams": {
        "person": { "age": "abc", "name": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }
    },
    "expectedStatusCode": 400,
    "rule": "art.galushko.openapi.testgen.rules.composed.ObjectItemSchemaValidationRule"
}

Header parameters¶

The generator produces different test cases depending on the header type:

Security header validation (missing or invalid values) is generated by auth rules, not schema rules.

Missing required header (non-security)¶

paths:
    /users:
        get:
            operationId: listUsers
            parameters:
                -   name: X-Request-ID
                    in: header
                    required: true
                    schema:
                        type: string
                        format: uuid

Missing required header:

{
    "name": "Missed Required Header Parameter X-Request-ID",
    "method": "GET",
    "path": "/users",
    "headers": [{ "key": "authorization", "value": "<valid_BearerAuth_placeholder>" }],
    "expectedStatusCode": 400,
    "rule": "art.galushko.openapi.testgen.providers.parameter.MissedRequiredParameterTestProvider"
}

Invalid UUID format:

{
    "name": "Invalid Header X-Request-ID parameter: Wrong UUID Format",
    "method": "GET",
    "path": "/users",
    "headers": [
        { "key": "authorization", "value": "<valid_BearerAuth_placeholder>" },
        { "key": "X-Request-ID", "value": "8e258b27-c787-49ef-9539-11461b251ffg" }
    ],
    "expectedStatusCode": 400,
    "rule": "art.galushko.openapi.testgen.rules.schema.WrongUuidFormatSchemaValidationRule"
}

Security headers (auth)¶

Security headers defined in securitySchemes are handled by AuthTestCaseProviderForOperation and produce different expected status codes.

components:
    securitySchemes:
        BearerAuth:
            type: http
            scheme: bearer
            bearerFormat: JWT
        ApiKeyAuth:
            type: apiKey
            in: header
            name: X-API-Key
security:
    -   BearerAuth: [ ]
    -   ApiKeyAuth: [ ]

No security values provided:

{
    "name": "No security values provided",
    "method": "GET",
    "path": "/users",
    "headers": [{ "key": "X-Request-ID", "value": "d5a5495b-cbdc-4237-a66e-000000000000" }],
    "expectedStatusCode": 401,
    "rule": "art.galushko.openapi.testgen.rules.auth.AllSecurityMissedAuthValidationRule"
}

Missing one required scheme (AND security):

{
    "name": "Missing Authorization header security",
    "method": "GET",
    "path": "/secure-and",
    "headers": [{ "key": "X-API-Key", "value": "<valid_apiKeyHeader_api_key_placeholder>" }],
    "expectedStatusCode": 401,
    "rule": "art.galushko.openapi.testgen.rules.auth.MissingSecurityValuesAuthValidationRule"
}

Invalid Authorization header:

{
    "name": "Invalid Authorization header security",
    "method": "GET",
    "path": "/users",
    "headers": [
        { "key": "X-Request-ID", "value": "d5a5495b-cbdc-4237-a66e-000000000000" },
        { "key": "authorization", "value": "bearer some_really_invalid_authorization_header" }
    ],
    "expectedStatusCode": 401,
    "rule": "art.galushko.openapi.testgen.rules.auth.InvalidSecurityValuesAuthValidationRule"
}

Distinguishing header types in output¶

Request body schema¶

The generator produces two categories of request body tests:

Example OpenAPI spec¶

paths:
    /orders:
        post:
            operationId: createOrder
            requestBody:
                required: true
                content:
                    application/json:
                        schema:
                            $ref: '#/components/schemas/NewOrder'

components:
    schemas:
        OrderItem:
            type: object
            required: [ sku, quantity, price ]
            additionalProperties: false
            properties:
                sku: { type: string }
                quantity: { type: integer, minimum: 1 }
                price: { type: number, minimum: 0 }
        NewOrder:
            type: object
            required: [ userId, items ]
            additionalProperties: false
            properties:
                userId: { type: string }
                items:
                    type: array
                    minItems: 1
                    items: { $ref: '#/components/schemas/OrderItem' }

Missing required request body¶

When requestBody.required: true, the generator emits a single test case that omits the body entirely:

{
    "name": "Required Request Body is missing",
    "method": "POST",
    "path": "/orders",
    "headers": [{ "key": "X-API-Key", "value": "test-api-key-123" }],
    "body": null,
    "requestBodyMediaType": "application/json",
    "expectedBody": { "code": "bad_request", "message": "Invalid input" },
    "responseBodyMediaType": "application/json",
    "expectedStatusCode": 400,
    "rule": "art.galushko.openapi.testgen.providers.body.MissedRequiredRequestBodyTestProvider"
}

Schema constraint violations¶

RequestBodySchemaValidationTestProvider applies schema rules to the resolved body schema. Fixture-backed examples for createOrder:

• Incorrect Request Body: Missed Required Object Properties items --- Body: {"userId":"a"} --- Rule: MissedRequiredObjectPropertiesSchemaValidationRule
• Incorrect Request Body: Missed Required Object Properties userId --- Body: {"items":[{"price":0,"quantity":1,"sku":"a"}]} --- Rule: MissedRequiredObjectPropertiesSchemaValidationRule
• Incorrect Request Body: Object Property items Below Min Items Array --- Body: {"items":[],"userId":"a"} --- Rule: ObjectItemSchemaValidationRule
• Incorrect Request Body: Object Property items Array Item Object Property price Out Of Minimum Boundary Number --- Body: {"items":[{"price":-1,"quantity":1,"sku":"a"}],"userId":"a"} --- Rule: ObjectItemSchemaValidationRule

Excerpt showing a nested array item violation:

{
    "name": "Incorrect Request Body: Object Property items Array Item Object Property price Out Of Minimum Boundary Number",
    "method": "POST",
    "path": "/orders",
    "headers": [{ "key": "X-API-Key", "value": "test-api-key-123" }],
    "body": { "items": [{ "price": -1, "quantity": 1, "sku": "a" }], "userId": "a" },
    "requestBodyMediaType": "application/json",
    "expectedBody": { "code": "bad_request", "message": "Invalid input" },
    "responseBodyMediaType": "application/json",
    "expectedStatusCode": 400,
    "rule": "art.galushko.openapi.testgen.rules.composed.ObjectItemSchemaValidationRule"
}

Media type behavior¶

Request body schema validation processes all supported media types in requestBody.content that have schemas. Supported media types include application/json, text/json, application/jwt, application/xml, text/xml, application/x-www-form-urlencoded, and +json / +jwt / +xml suffixes. Unsupported media types are skipped and logged as warnings.

Budget controls for complex schemas¶

Deep or highly nested schemas can generate many test cases:

testGenerationSettings:
    maxSchemaDepth: 50
    maxMergedSchemaDepth: 50
    maxSchemaCombinations: 100
    maxTestCasesPerOperation: 1000

See Budget controls for details.

Applicable rules (request bodies)¶

Request bodies can trigger the same schema rules as parameters. For nested objects and arrays, the rule field contains a composed rule class that wraps the underlying simple rule.

Filtering and targeting tests¶

All filtering options apply across parameter and request body test categories.

Target specific operations¶

Use includeOperations to generate tests for specific paths before generation:

testGenerationSettings:
    includeOperations:
        "/users/{userId}": [ GET ]
        "/orders": [ POST ]

Exclude specific test cases¶

Use ignoreTestCases to exclude tests by exact name (no wildcards):

testGenerationSettings:
    ignoreTestCases:
        "/users/{userId}":
            "GET":
                - "Invalid Path userId parameter: Invalid Pattern"
        "/orders":
            "POST":
                - "Required Request Body is missing"

Disable schema validation rules¶

testGenerationSettings:
    ignoreSchemaValidationRules:
        - art.galushko.openapi.testgen.pattern.support.InvalidPatternSchemaValidationRule

Disable auth validation rules¶

testGenerationSettings:
    ignoreAuthValidationRules:
        - "art.galushko.openapi.testgen.rules.auth.AllSecurityMissedAuthValidationRule"

See Ignore rules for the full ignore configuration reference.

Inspecting generated output¶

# Path parameter tests
jq '.. | objects | select(.name? | strings | startswith("Invalid Path"))' generated/test-suites.json

# Missing required query/header parameter tests
jq '.. | objects | select(.name? | strings | contains("Missed Required"))' generated/test-suites.json

# Request body schema violations
jq '.. | objects | select(.name? | strings | startswith("Incorrect Request Body"))' generated/test-suites.json

Related docs¶

• Configuration - YAML config, ignore rules, security values
• Providers catalog - Provider details
• Rules catalog - All schema and auth validation rules
• Test-suite-writer - Output format details
• CLI reference
• Gradle plugin reference