This activity is undertaken under the work programme of the High-Level Group for Modernisation of Official Statistics. See original activity proposal, which defines the scope of the work
Interoperability between SDMX to DDI-CDI - Work plan
Concrete deliverables are centered on creating an automated data integration workflow using DDI-CDI as a central link. The primary deliverables is a set of mappings and Python scripts designed to automate the transformation of data from different source models into a unified DDI-CDI representation and vice-versa. These scripts will cover three specific use cases:
- A simplified "proto-SDMX" example
- A full SDMX XML message from the ILO
- An example using Google Data Commons's .mcf files (based on schema.org)
Accompanying these scripts, the working stream has produced a "DDI-CDI dimensional data integration profile" which is a subset of the full DDI-CDI classes and properties that are relevant for describing and integrating dimensional data.
These files will serve as tangible examples of the desired DDI-CDI output for each use case, conforming to the specified DDI-CDI profile and extensions. Additionally, the project will deliver supporting documentation, including a summary document, a checklist of deliverables, and an analysis of open issues, priorities, and next steps. A key part of the work also involves identifying and brainstorming methods to codify the necessary data mappings and transformations
For more information see Discussion on model elements and overall approach and DDI-CDI and other standards.
Interoperability between DDI Codebook to CDI
- EOSC Metadata Schema and Crosswalk Registry was not successful as it reported an error during the import of the DDI-CDI xsd file before to broke completely.
- Mapping elements for DDI-C to DDI-CDI can be found in these two tables
- Mapping of DDI Codebook metadata at the variable level which is suitable for producing DDI-CDI was embedded also in the Python tools developed by Pascal Heus, which were documented with Copilot, based on the codebook_to_ddi methods
- DDI-Codebook file developed by Knut Wenzig (DIW Berlin), with 2 files/datasets, each with 2 variables. One variable also has missing categories/codes. The content was inspired by this working paper: Publishing Fine-Grained Standardized Metadata – Lessons Learned from Three Research Data Centers - KonsortSWD This could be used with Pascals tool to produce DDI-CDI. Afterwards one would have the basis to present different snippets of DDI-C (like in the working paper above) and compare them with DDI-CDI an approach which seems to be more promising that finding 1 to 1 relations between the two standards’ elements.
Problem statement: Metadata mappings for data integration
Main proposal: combined use of technical standards to meet different mapping needs. This includes the use of RML to handle structural transformations (schema alignment and implicit identifiers) while SSSOM to handle semantic negotiation (value and unit alignment).
Benefits
-
Bridges the structural gap: semantic mapping alone is insufficient for end-to-end data integration; distinguishing between "schema lifting" (RML) and "semantic alignment" (SSSOM).
-
Enables machine-actionable ETL: By moving from procedural code to declarative mapping files, we pave the way for automated discovery and execution of cross-domain data integration pipelines.
-
Promotes scalable interoperability: support of "data mesh" approach, allowing data providers to publish authoritative mappings that consumers can reuse, significantly reducing the manual effort required for integration.
-
For more information see extensive report on Metadata Mapping and Data Integration
-
SDMX-XMLto DDI-CDI structural mapping specification
- SDMX 3 in relation to microdata, and gaps identified, focused on survey microdata (30mins - Edgardo)
- Variable cascade and microdata, at high-level (30mins - Dan)
- SDMX Content constraints (Edgardo)
- [Use of concepts at different granularities in CDI] (conceptual variables, conceptual domains, etc.) (Dan)
- DDI Codebook to SDMX (Edgardo)
- Data description in DDI-CDI (Flavio)
- Presentation "diagrams 2" (Luis)
- Presentation "Recap and update" (Flavio)
- CodeList mapping (Flavio)
- DDI products mappings (Wendy)
- Tentative SDMX-DDI fragment for data (Flavio)
- A proposal for attempting a mapping (Flavio)
SDMX Global Conference: Towards Practical Interoperability: Mapping SDMX and DDI for Data Integration