DeepTraq YAML Configuration — Overview

The DeepTraq YAML configuration file defines how an authenticated web application scan should be executed. It controls the scan scope, request headers, authentication behavior, and authentication verification logic used during the scan.

This configuration is used by the DeepTraq scanning engine to dynamically generate a ZAP automation plan and perform a fully authenticated security scan.

At a high level, the YAML file contains four main sections:

Scope Configuration – defines which URLs belong to the application being scanned
Request Headers Configuration – defines headers required for authenticated requests
Authentication Configuration – defines how login is performed
Authentication Verification – defines how the system confirms authentication succeeded

Each section is explained below.

1. Scan Scope Configuration

The scope section defines which URLs belong to the application and should be included during scanning.

This section ensures that the scanner only interacts with intended application endpoints and avoids crawling unrelated domains or destructive endpoints such as logout.

The scope block typically contains three components:

Entry URLs

Entry URLs define the starting points for crawling and spidering.

These URLs are the first locations the scanner visits to begin discovering application endpoints.

For modern applications such as React, Angular, or Vue, the entry URL should usually be the base domain of the application rather than a client-side route.

For example:

Correct entry URL:

https://example.com

Incorrect entry URL:

https://example.com/#/dashboard

Client-side routes containing # fragments are handled entirely inside the browser and never reach the server, meaning traditional crawlers cannot discover them.

If the application consists of multiple domains (for example a frontend domain and an API domain), the order of entry URLs becomes important.

The first domain listed is treated as the primary domain for authentication reporting. If the login API resides on a backend domain, that domain should be listed first.

Include Paths

Include paths define the URL patterns that must be scanned.

These patterns support regular expressions and are used to ensure that both frontend and backend application endpoints are included in the scan.

For modern SPA architectures, the backend API domain is usually more important than the frontend domain, since the actual application functionality resides in API endpoints.

Include paths should therefore include patterns for both domains when applicable.

Exclude Paths

Exclude paths define URLs that must never be scanned.

Certain endpoints should always be excluded from automated scanning, particularly:

Logout endpoints
Session termination endpoints
Destructive administrative actions

If logout endpoints are scanned, the scanner may unintentionally terminate its own authenticated session, causing the remainder of the scan to behave as if the user is logged out.

Static assets such as images, stylesheets, and fonts are automatically ignored by the scanner and typically do not need to be excluded manually.

2. Request Headers Configuration

Many modern applications require specific HTTP headers to be present on every request. If these headers are missing, the application may silently reject requests even when authentication is valid.

The headers section allows you to define all headers that should be automatically attached to every request generated by the scanner.

Headers typically fall into two categories.

Dynamic Authentication Headers

Dynamic headers contain values that change during login, such as tokens or session identifiers.

For example:

JWT tokens
Bearer tokens
Access tokens
Session cookies

These values are usually extracted automatically from the login response and injected into subsequent requests during the scan.

A common example is the Authorization header used for JWT authentication.

Example — Token-Based Authentication

Many modern applications return a token in the login response and expect it in the Authorization header.

Example request header used after login:

Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

In this case, the token is extracted from the login response and dynamically inserted into the Authorization header for every request.

Example configuration:

Authorization: "Bearer {%json:accessToken%}"

Some applications maintain authentication using session cookies instead of tokens.

After login, the server sends a session identifier using the Set-Cookie response header, and the browser includes it in future requests.

Example authenticated request header:

Cookie: SESSIONID=abc123xyz789

The session value can be dynamically extracted from the login response and reused in scan requests.

Example configuration:

Cookie: "SESSIONID={%cookie:SESSIONID%}"

For detailed explanations, configuration patterns, and additional examples for both token and cookie-based authentication, refer to the Authentication Headers section.

Static Application Headers

Some applications require additional headers that remain constant for a specific user, tenant, or client. These headers are not generated dynamically during login and must therefore be explicitly defined in the configuration.

Typical examples include:

tenant identifiers
client identifiers
application version headers
organization identifiers
role indicators

These values are often part of the application's business logic or multi-tenant architecture and must be included in requests for the backend to process them correctly.

Below are examples of static headers that may be required by some applications:

Role: "admin"
Version: "1.1.0"
X-Client-ID: "web"

These headers remain constant and can be defined directly in the request configuration.

How to Identify Required Static Headers

The easiest way to identify required headers is by inspecting a successful authenticated request in the browser.

Steps:

Login to the application normally in your browser.
Open Developer Tools → Network tab.
Select any authenticated request.
Inspect the Request Headers section.
Identify any application-specific headers that are not related to standard HTTP headers.

If the application includes custom headers such as tenant identifiers or client identifiers, they should be replicated in the configuration.

Important Notes

Missing Business Headers Can Cause Authentication Verification Failures

In some applications, authenticated endpoints require additional business headers even after login succeeds.

If these headers are missing:

the authentication credentials may still be correct
session tokens or cookies may be extracted successfully
but the verification URL may fail

This can cause the scanner to report authentication failure, even though the login itself succeeded.

Typical symptoms include:

verification URL returning 403 Forbidden
verification URL returning 401 Unauthorized
verification URL returning an application error message

When this occurs, check whether the authenticated requests in the browser contain additional headers that must also be included in the scan configuration.

Some applications also require custom headers during the login request itself. If these headers are missing, the login request may fail entirely.

In such cases, custom headers must be provided using the authentication.headers configuration.

Example use cases include:

applications requiring a client identifier during login
APIs that require version headers
gateways requiring tenant identifiers before authentication

If the login request fails unexpectedly, inspect the login request in the browser to determine whether additional headers are required.

Debugging Missing Headers Using cURL

When authentication or verification fails, the easiest way to debug the issue is to replicate the request using curl.

Start by copying the request and sending it with the minimum required headers.

Example workflow:

Copy the request URL and basic headers from the browser.
Run the request using curl.
Observe the server response.

If the response returns:

401 Unauthorized
403 Forbidden
application-specific error messages

this often indicates that required headers are missing.

Identifying the Correct Headers

To determine which headers are required:

Open Chrome Developer Tools → Network tab.
Locate the working request in the browser.
Inspect all headers present in that request.
Test them incrementally using curl.

Important: Do not copy every header from the browser.

Many headers such as:

User-Agent
Accept
Accept-Language
Connection

are not required for authentication.

Instead, identify application-specific headers and add them gradually until the request succeeds.

This trial-and-error approach helps identify the minimal set of headers required for the application to process authenticated requests correctly.

3. Authentication Configuration

The authentication section defines how the scanner performs login.

Different applications implement authentication in different ways, so the configuration supports several authentication modes.

Authentication Type	Required Fields
browser	type, loginPageUrl
form	type, loginPageUrl, loginBackendUrl, bodyTemplate
json	type, loginPageUrl, loginBackendUrl, bodyTemplate

NOTE: The full authentication block can be omitted when using pre-injected Cookie or Authorization headers. Optionally, you can provide authentication.pollUrl to verify that the injected headers are functioning correctly.

NOTE: If the login request requires custom headers, provide them using authentication.headers to ensure the login succeeds.

JSON Authentication

Required fields:

type
loginPageUrl
loginBackendUrl
bodyTemplate

JSON authentication is used when the application exposes a login API endpoint that accepts credentials and returns a token. Set type="json"

authentication:
    type: "json"

This method is common for:

React applications
Angular applications
Vue applications
REST APIs
Mobile backend services

In this mode the scanner sends a POST request to the login API endpoint and extracts the authentication token from the response.

The request body template must match the exact payload expected by the login API.

The placeholders {identifier} and {password} are automatically replaced with the credentials provided when creating the scan.

Example

type: json
loginPageUrl: [https://example.com/login](https://example.com/login)
loginBackendUrl: [https://example.com/api/auth/login](https://example.com/api/auth/login)
bodyTemplate: {"email":"{username}","password":"{password}"}

Form Authentication

Required fields:

type
loginPageUrl
loginBackendUrl
bodyTemplate

Form authentication is used when the application relies on a traditional HTML login form. Set type="form"

authentication:
    type: "form"

This is typical for server-rendered frameworks such as:

Django
Ruby on Rails
Laravel
Spring MVC
older PHP applications

In this case the scanner first loads the login page, detects the HTML form fields, and submits credentials in the same way a browser would.

The server typically returns a session cookie, which is then reused for subsequent requests.

Example

type: form
loginPageUrl: [https://example.com/login](https://example.com/login)
loginBackendUrl: [https://example.com/login](https://example.com/login)
bodyTemplate: username={username}&password={password}

Browser Authentication

Required fields:

type
loginPageUrl

Browser authentication launches a headless browser to perform login exactly as a real user would. Set type="browser"

authentication:
    type: "browser"

This method is required when the login process involves:

JavaScript-driven authentication flows
SSO authentication
OAuth redirects
MFA or OTP steps
dynamically generated login forms

Because a real browser is used, the system can automatically capture additional application traffic generated during the login process, improving URL discovery.

Example

type: browser
loginPageUrl: [https://example.com/login](https://example.com/login)

Some applications require additional headers during the login request itself. If these headers are missing, the login request may fail even if the credentials and request body are correct.

Typical scenarios include:

APIs requiring a client identifier
gateways requiring a tenant identifier
applications expecting a version header
platforms enforcing custom authentication gateways

In these cases, custom headers must be provided using the authentication.headers configuration.

Example configuration:

type: json
loginPageUrl: [https://example.com/login](https://example.com/login)
loginBackendUrl: [https://example.com/api/login](https://example.com/api/login)
bodyTemplate: {"email":"{username}","password":"{password}"}
headers:
    X-Client-ID: "web"
    X-Tenant-ID: "tenant-123"
    Version: "1.1.0"

These headers will be included only during the login request.

If authentication fails unexpectedly, inspect the login request in the browser to identify any headers that must be replicated.

Login normally using the browser.
Open Developer Tools → Network tab.
Locate the login request.
Inspect the Request Headers section.
Identify any application-specific headers used during login.

Only replicate relevant application headers, not standard browser headers.

In some scenarios, no login flow is needed because the scanner uses pre-existing credentials, such as a session cookie or a Bearer token.

This is common when:

Authentication is already handled externally
Tokens or cookies are manually injected into requests
No username/password login is required

How It Works

When using pre-injected authentication:

The authentication block in the YAML can be omitted entirely
Or only the pollUrl can be provided to verify authentication

Example:

authentication:
  pollUrl: "https://cvi.techvedika.com/gateway/organization/getAll"

The scanner will not perform a login request
Instead, it will hit the pollUrl (a protected endpoint) to verify if the injected cookie or token is valid
If pollUrl returns a valid response, authentication is considered successful

Example Usage

Optional Poll URL Verification:

authentication:
  pollUrl: "https://api.example.com/user/profile"

Optional Custom Headers:

headers:
  X-Client-ID: web
  tenant-id: 1

This configuration allows the scanner to perform an authenticated scan without using login credentials, relying solely on injected authentication data.

See Special Case: Verification for Pre-Injected Authentication for details.

4. Authentication Verification

Performing a login request does not guarantee that authentication actually succeeded.

Applications may return HTTP 200 responses even when credentials are incorrect, making it necessary to explicitly verify authentication status.

Verification is configured using the following keys:

pollUrl
loggedInRegex

These keys control how the scanner confirms that the session is authenticated.

Configuration	Verification Behavior
pollUrl only	Verifies authentication by calling a protected endpoint
loggedInRegex only	Verifies authentication by matching the login response
pollUrl + loggedInRegex	Verifies protected endpoint and response content
neither defined	Uses automatic detection (not reliable)

The YAML configuration supports three verification approaches.

NOTE: When Cookies or Bearer tokens are directly injected instead of performing a credentials-based login, use only pollUrl for verification. In this case, pollUrl must be a GET endpoint that returns HTTP 200 to confirm successful authentication.

In some cases, authentication can be verified directly using the login API response.

This works when the login endpoint returns a clear indicator of success, such as:

a success message
a token
a user object
a predictable field in the response

If only loggedInRegex is configured and pollUrl is not provided, the scanner verifies authentication by applying the regex check directly to the login response.

Example concept:

loggedInRegex: "accessToken"

In this case:

The scanner sends the login request.
It inspects the login response body.
If the response matches the regex pattern, authentication is considered successful.

This approach is commonly used for JSON API authentication flows where the login endpoint returns tokens or user information.

Verification Using a Protected Endpoint

The most reliable method is verifying authentication by accessing a protected endpoint that is only available after login.

This is done using the pollUrl parameter along with an optional loggedInRegex check.

Example configuration concept:

pollUrl: https://example.com/api/user/profile loggedInRegex: "email"

In this setup:

The scanner logs in using the configured credentials.
It sends a request to the pollUrl.
If the endpoint returns a successful response, the scanner assumes the session is authenticated.
If loggedInRegex is defined, the response body must also match the regex pattern.

This method verifies authentication using an actual authenticated application endpoint, making it the most reliable approach.

Automatic Verification (Not Recommended)

If both pollUrl and loggedInRegex are omitted, the scanner attempts to automatically detect authentication success.

This auto-detection method relies on observing request/response behavior during login.

However, this approach is often unreliable, especially for:

Single Page Applications (SPA)
API-based authentication
token-based authentication systems
applications without redirects

Because of these limitations, auto-detection may incorrectly report:

authentication failures
invalid credentials
verification failures

Special Case: Verification for Pre-Injected Authentication

When using pre-injected authentication headers (Bearer token, session cookie, or custom headers) instead of performing a login flow, authentication verification works slightly differently.

How It Works

Since there is no login request, the scanner cannot rely on a login response for verification.
Verification can be performed using the pollUrl key alone.
The scanner will send a GET request to the pollUrl using the injected headers.
The authentication is considered successful if the response returns HTTP 200.
loggedInRegex is optional in this case, but can still be used if the endpoint returns a predictable response body.

Example Configuration

authentication:
  pollUrl: "https://api.example.com/user/profile"

headers:
  Authorization: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
  X-Client-ID: web
  tenant-id: 1

Scanner sends a GET request to https://api.example.com/user/profile
Injected headers are automatically included
HTTP 200 → authentication verified ✅
HTTP 401 / 403 / other errors → authentication failed ❌

Best Practices

When configuring the DeepTraq YAML file, the following best practices can improve scan reliability:

Ensure entry URLs include the correct domains and are ordered properly for multi-domain applications.
Replicate all required headers from a successful authenticated request in your browser.
Match the login request body exactly with the payload expected by the application's login API.
Use a protected API endpoint for authentication verification whenever possible.
Avoid using client-side SPA routes for verification checks.
Dynamic headers (Authorization, Cookie) or custom static headers can be provided in the headers section.
Username and password are not required in this scenario.
PollUrl is optional — it can be used to verify that the injected credentials are working. If omitted, the scan assumes that the credentials are valid.
This method is particularly useful for API testing or authenticated scans where the authentication mechanism cannot easily be automated.

Following these practices helps ensure that the scanner maintains a valid authenticated session throughout the entire scan.

1. Scan Scope Configuration

Entry URLs​

Include Paths​

Exclude Paths​

2. Request Headers Configuration

Dynamic Authentication Headers​

Example — Token-Based Authentication​

Example — Cookie-Based Authentication​

Static Application Headers​

How to Identify Required Static Headers​

Important Notes​

Custom Headers During Login​

Debugging Missing Headers Using cURL​

Identifying the Correct Headers​

3. Authentication Configuration

JSON Authentication​

Form Authentication​

Browser Authentication​

Special Case — Custom Headers Required During Login​

How to Identify Required Login Headers​

Special Case: Using Pre-Injected Authentication (No Login Flow)​

How It Works​

Example Usage​

4. Authentication Verification

Verification Using Login Response​

Verification Using a Protected Endpoint​

Automatic Verification (Not Recommended)​

Special Case: Verification for Pre-Injected Authentication​

How It Works​

Example Configuration​

Best Practices

Entry URLs

Include Paths

Exclude Paths

Dynamic Authentication Headers

Example — Token-Based Authentication

Example — Cookie-Based Authentication

Static Application Headers

How to Identify Required Static Headers

Important Notes

Custom Headers During Login

Debugging Missing Headers Using cURL

Identifying the Correct Headers

JSON Authentication

Form Authentication

Browser Authentication

Special Case — Custom Headers Required During Login

How to Identify Required Login Headers

Special Case: Using Pre-Injected Authentication (No Login Flow)

How It Works

Example Usage

Verification Using Login Response

Verification Using a Protected Endpoint

Automatic Verification (Not Recommended)

Special Case: Verification for Pre-Injected Authentication

How It Works

Example Configuration