VTL Engine Documentation

The VTL Engine is a Python library that allows you to validate, format and run VTL scripts.

It is a Python-based library around the VTL Language 2.1

Note

The data types and type casting is based on the VTL 2.2 specification (preview).

The vtlengine library provides full SDMX compatibility:

• Direct SDMX file loading: Load SDMX-ML, SDMX-JSON, and SDMX-CSV files directly in the run() and semantic_analysis() functions

• pysdmx integration: Work with pysdmx objects (Schema, DataStructureDefinition, Dataflow, PandasDataset)

• Automatic format detection: SDMX files are automatically detected by extension

• Flexible mappings: Map SDMX URNs to VTL dataset names using the sdmx_mappings parameter

Check the pysdmx documentation for more information about working with SDMX data.

Useful links

• MeaningfulData: who we are

• Source Code

• PyPI link

• Bug Tracker

• New features Tracker

Installation

Requirements

The VTL Engine requires Python 3.9 or higher.

Install with pip

To install the VTL Engine on any Operating System, you can download it from pip:

pip install vtlengine

Important

It is recommended to install any python package in a virtual environment. Please follow these steps

S3 URI support is provided by the DuckDB execution engine; see DuckDB Engine.

Getting Started

• 10 minutes to VTL Engine
  • Semantic Analysis
    • Example 1: Correct VTL
    • Example 2: Incorrect VTL
    • Example 2b: Using SDMX Structures
  • Run VTL Scripts
    • Example 3: Simple run
    • Example 3b: Run using the DuckDB engine
    • Example 4: Run from SDMX Dataset
    • Example 4b: Run with SDMX Files
    • Example 4c: Comparing two datasets sharing one SDMX Dataflow
    • Example 5: Run with multiple Value Domains and External Routines as dictionaries.
    • Example 6: Run using Paths
  • Run with Scalar Values
    • Example 7: Run with Scalar Values
  • Prettify

Using the Engine

• API
  • create_ast_with_comments()
  • generate_sdmx()
  • prettify()
  • run()
  • run_sdmx()
  • semantic_analysis()
  • validate_dataset()
  • validate_external_routine()
  • validate_value_domain()
• Data Types
  • Type Hierarchy
  • Data Types Reference
    • String
    • Integer
    • Number
    • Boolean
    • Date
    • Time_Period
    • Time (TimeInterval)
    • Duration
  • Null Handling
  • Type Casting
    • Implicit Casting (Automatic)
    • Explicit Casting (cast operator)
      • Supported conversions without mask
      • Supported conversions with mask
      • Cast on datasets
• Extra Inputs
  • Value Domains
    • Definition Format
    • Supported Types
    • Input Options
    • VTL Usage
    • Example
    • Validation
  • External Routines
    • Definition Format
    • Input Options
    • VTL Syntax
    • Security
    • Examples
      • Using a dictionary
    • Validation

Engines

• Pandas Engine
  • Overview
  • When to use it
  • Limitations
  • Configuration
  • When to switch to DuckDB
• DuckDB Engine
  • Overview
  • When to prefer it over pandas_engine
  • Enabling
  • In-memory vs. file-backed database
  • Configuration
  • S3 URI support
  • Behavioural notes

Configuration

• Environment Variables
  • Number Handling
    • COMPARISON_ABSOLUTE_THRESHOLD
    • OUTPUT_NUMBER_SIGNIFICANT_DIGITS
  • DuckDB Engine
    • VTL_MEMORY_LIMIT
    • VTL_THREADS
    • VTL_TEMP_DIRECTORY
    • VTL_MAX_TEMP_DIRECTORY_SIZE
    • VTL_USE_IN_MEMORY_DB
    • VTL_DUCKDB_DECIMAL_WIDTH
    • VTL_SKIP_LOAD_VALIDATION
  • S3 Configuration
    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • AWS_SESSION_TOKEN
    • AWS_DEFAULT_REGION
    • AWS_ENDPOINT_URL
  • Examples
    • Setting comparison threshold
    • Controlling numeric precision
    • Tuning the DuckDB engine
    • Using S3 with environment variables
    • Using a custom S3 endpoint

Error Messages

• Error Messages