@@ -11,14 +11,15 @@
Join us on [](https://tiny.ydata.ai/dcai-ydata-synthetic)
# YData Synthetic
-A package to generate synthetic tabular and time-series data leveraging the state of the art generative models.
-
-## 🎊 The exciting features:
-> These are must try features when it comes to synthetic data generation:
- > - A new streamlit app that delivers the synthetic data generation experience with a UI interface. A low code experience for the quick generation of synthetic data
- > - A new fast synthetic data generation model based on Gaussian Mixture. So you can quickstart in the world of synthetic data generation without the need for a GPU.
- > - A conditional architecture for tabular data: CTGAN, which will make the process of synthetic data generation easier and with higher quality!
-
+`YData-Synthetic` is an open-source package developed in 2020 with the primary goal of educating users about generative models for synthetic data generation.
+Designed as a collection of models, it was intended for exploratory studies and educational purposes.
+However, it was not optimized for the quality, performance, and scalability needs typically required by organizations.
+
+!!! note "Update"
+ Even though the journey was fun, and we have learned a lot from the community it is now time to upgrade `ydata-synthetic`.
+ Heading towards the future of synthetic data generation we recommend users to transition to `ydata-sdk`, which provides a superior experience with enhanced performance,
+ precision, and ease of use, making it the preferred tool for synthetic data generation and a perfect introduction to Generative AI.
+
## Synthetic data
### What is synthetic data?
Synthetic data is artificially generated data that is not collected from real world events. It replicates the statistical components of real data without containing any identifiable information, ensuring individuals' privacy.
@@ -32,68 +33,50 @@ Synthetic data can be used for many applications:
> **Looking for an end-to-end solution to Synthetic Data Generation?**
-
+Make sure your Google Colab is running Python versions `>=3.9, <=3.12`. Learn how to configure Python versions on Google Colab [here](https://stackoverflow.com/questions/68657341/how-can-i-update-google-colabs-python-version/68658479#68658479).
diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md
deleted file mode 100644
index 3814bb39..00000000
--- a/docs/getting-started/quickstart.md
+++ /dev/null
@@ -1,78 +0,0 @@
-# Quickstart
-
-`ydata-synthetic` is equipped to handle both **tabular** (comprising numeric and categorical features) and sequential, **time-series** data. In this section we explain how you can **quickstart the synthesization** of tabular and time-series datasets.
-
-## Synthesizing a Tabular Dataset
-The following example showcases how to synthesize the [Adult Census Income](https://www.kaggle.com/datasets/uciml/adult-census-income) dataset with CTGAN:
-=== "Tabular Data"
- ```python
- # Import the necessary modules
- from pmlb import fetch_data
- from ydata_synthetic.synthesizers.regular import RegularSynthesizer
- from ydata_synthetic.synthesizers import ModelParameters, TrainParameters
-
- # Load data
- data = fetch_data('adult')
- num_cols = ['age', 'fnlwgt', 'capital-gain', 'capital-loss', 'hours-per-week']
- cat_cols = ['workclass','education', 'education-num', 'marital-status',
- 'occupation', 'relationship', 'race', 'sex', 'native-country', 'target']
-
- # Define model and training parameters
- ctgan_args = ModelParameters(batch_size=500, lr=2e-4, betas=(0.5, 0.9))
- train_args = TrainParameters(epochs=501)
-
- # Train the generator model
- synth = RegularSynthesizer(modelname='ctgan', model_parameters=ctgan_args)
- synth.fit(data=data, train_arguments=train_args, num_cols=num_cols, cat_cols=cat_cols)
-
- # Generate 1000 new synthetic samples
- synth_data = synth.sample(1000)
- ```
-
-## Synthesizing a Time-Series Dataset
-The following example showcases how to synthesize the [Yahoo Stock Price](https://www.kaggle.com/datasets/arashnic/time-series-forecasting-with-yahoo-stock-price) dataset with TimeGAN:
-=== "Time-Series Data"
- ```python
- # Import the necessary modules
- import pandas as pd
- from ydata_synthetic.synthesizers.timeseries import TimeSeriesSynthesizer
- from ydata_synthetic.synthesizers import ModelParameters, TrainParameters
-
- # Define model parameters
- gan_args = ModelParameters(batch_size=128,
- lr=5e-4,
- noise_dim=32,
- layers_dim=128,
- latent_dim=24,
- gamma=1)
-
- train_args = TrainParameters(epochs=50000,
- sequence_length=24,
- number_sequences=6)
-
- # Read the data
- stock_data = pd.read_csv("stock_data.csv")
-
- # Training the TimeGAN synthesizer
- synth = TimeSeriesSynthesizer(modelname='timegan', model_parameters=gan_args)
- synth.fit(stock_data, train_args, num_cols=list(stock_data.columns))
-
- # Generating new synthetic samples
- synth_data = synth.sample(n_samples=500)
- ```
-
-## Running the Streamlit App
-Once the package is [installed](installation.md) with the "streamlit" extra, the app can be launched as:
-
-=== "Streamlit App"
- ```python
- from ydata_synthetic import streamlit_app
-
- streamlit_app.run()
- ```
-
-The console will then output the URL from which the app can be accessed.
-
-:fontawesome-brands-youtube:{ style="color: #EE0F0F" } Here's a [quick example](https://www.youtube.com/watch?v=6Lzi26szKNo&t=4s) of how to synthesize data with the Streamlit App – :octicons-clock-24: 5min
-
-
\ No newline at end of file
diff --git a/docs/index.md b/docs/index.md
index a95ff9ac..57ebd27b 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -3,7 +3,7 @@
[](https://pypi.org/project/ydata-synthetic)
-
+
[](https://pepy.tech/project/ydata-synthetic)
@@ -13,21 +13,15 @@
[](https://discord.com/invite/mw7xjJ7b7s)
## Overview
-`ydata-synthetic` is the go-to Python package for **synthetic data generation for tabular and time-series data**. It uses the latest Generative AI models to learn the properties of real data and create realistic synthetic data. This project was created to educate the community about synthetic data and its applications in real-world domains, such as data augmentation, bias mitigation, data sharing, and privacy engineering. To learn more about Synthetic Data and its applications, [check this article](https://ydata.ai/resources/10-most-frequently-asked-questions-about-synthetic-data).
+`YData-Synthetic` is an open-source package developed in 2020 with the primary goal of educating users about generative models for synthetic data generation.
+Designed as a collection of models, it was intended for exploratory studies and educational purposes.
+However, it was not optimized for the quality, performance, and scalability needs typically required by organizations.
-## Current Functionality
-- 🤖 **Create Realistic Synthetic Data using Generative AI Models:** `ydata-synthetic` supports the state-of-the-art generative adversarial networks for data generation, namely Vanilla GAN, CGAN, WGAN, WGAN-GP, DRAGAN, Cramer GAN, CWGAN-GP, CTGAN, and TimeGAN. Learn more about the use of [GANs for Synthetic Data generation](https://medium.com/ydata-ai/generating-synthetic-tabular-data-with-gans-part-1-866705a77302).
+!!! tip "We are now ydata-sdk!"
+ Even though the journey was fun, and we have learned a lot from the community it is now time to upgrade `ydata-synthetic`.
-- 📀 **Synthetic Data Generation for Tabular and Time-Series Data:** The package supports the synthesization of tabular and time-series data, covering a wide range of real-world applications. Learn how to leverage `ydata-synthetic` for [tabular](https://ydata.ai/resources/gans-for-synthetic-data-generation) and [time-series](https://towardsdatascience.com/synthetic-time-series-data-a-gan-approach-869a984f2239) data.
-
-- 💻 **Best Generation Experience in Open Source:** Including a guided UI experience for the generation of synthetic data, from reading the data to visualization of synthetic data. All served by a slick Streamlit app.
-:fontawesome-brands-youtube:{ style="color: #EE0F0F" } Here's a [quick overview](https://www.youtube.com/watch?v=ep0PhwsFx0A) – :octicons-clock-24: 1min
-
-!!! question
-
- **Looking for an end-to-end solution to Synthetic Data Generation?**
-
- [YData Fabric](https://ydata.ai/products/synthetic_data) enables the generation of high-quality datasets within a full UI experience, from data preparation to synthetic data generation and evaluation. Check out the [Community Version](https://ydata.ai/ydata-fabric-free-trial).
+ Heading towards the future of synthetic data generation we recommend users to transition to `ydata-sdk`, which provides a superior experience with enhanced performance,
+ precision, and ease of use, making it the preferred tool for synthetic data generation and a perfect introduction to Generative AI.
## Supported Data Types
@@ -41,11 +35,35 @@
=== "Time-Series Data"
**Time-series data** exhibit a sequencial, **temporal dependency** between records, and may present a wide range of patterns and trends, including **seasonality** (patterns that repeat at calendar periods -- days, weeks, months -- such as holiday sales, for instance) or **periodicity** (patterns that repeat over time).
- Read more about generating time-series data in this [article](https://ydata.ai/resources/synthetic-time-series-data-a-gan-approach) and check this [quickstart guide](getting-started/quickstart.md#synthesizing-a-time-series-dataset) to get started with time-series data synthesization.
-
+ Read more about generating [time-series data in this article](https://ydata.ai/resources/synthetic-time-series-data-a-gan-approach) and check this [quickstart guide](getting-started/quickstart.md#synthesizing-a-time-series-dataset) to get started with time-series data synthesization.
+
+=== "Multi-Table Data"
+ **Multi-Table data** or databases exhibit a referential behaviour between and database schema that is expected to be replicated and respected by the synthetic data generated.
+ Read more about database [synthetic data generation in this article]() and check this [quickstart guide for Multi-Table synthetic data generation]()
+ **Time-series data** exhibit a sequential, **temporal dependency** between records, and may present a wide range of patterns and trends, including **seasonality** (patterns that repeat at calendar periods -- days, weeks, months -- such as holiday sales, for instance) or **periodicity** (patterns that repeat over time).
+
+## Validate the quality of your synthetic data generated
+
+Validating the quality of synthetic data is essential to ensure its usefulness and privacy. YData Fabric provides tools for comprehensive synthetic data evaluation through:
+
+1. **Profile Comparison Visualization:**
+Fabric delivers side-by-side visual comparisons of key data properties (e.g., distributions, correlations, and outliers) between synthetic and original datasets, allowing users to assess fidelity at a glance.
+
+2. **PDF Report with Metrics:**
+Fabric generates a PDF report that includes key metrics to evaluate:
+
+- Fidelity: How closely synthetic data matches the original.
+- Utility: How well it performs in real-world tasks.
+- Privacy: Risk assessment of data leakage and re-identification.
+
+These tools ensure a thorough validation of synthetic data quality, making it reliable for real-world use.
## Supported Generative AI Models
-The following architectures are currently supported:
+With the upcoming update of `ydata-synthetic`to `ydata-sdk`, users will now have access to a single API that automatically selects and optimizes
+the best generative model for their data. This streamlined approach eliminates the need to choose between
+various models manually, as the API intelligently identifies the optimal model based on the specific dataset and use case.
+
+Instead of having to manually select from models such as:
- [GAN](https://arxiv.org/abs/1406.2661)
- [CGAN](https://arxiv.org/abs/1411.1784) (Conditional GAN)
@@ -58,4 +76,8 @@ The following architectures are currently supported:
- [TimeGAN](https://papers.nips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf) (specifically for *time-series* data)
- [DoppelGANger](https://dl.acm.org/doi/pdf/10.1145/3419394.3423643) (specifically for *time-series* data)
+The new API handles model selection automatically, optimizing for the best performance in fidelity, utility, and privacy.
+This significantly simplifies the synthetic data generation process, ensuring that users get the highest quality output without
+the need for manual intervention and tiring hyperparameter tuning.
+
\ No newline at end of file
diff --git a/docs/integrations/gx_integration.md b/docs/integrations/gx_integration.md
index fb87614f..585f1616 100644
--- a/docs/integrations/gx_integration.md
+++ b/docs/integrations/gx_integration.md
@@ -15,6 +15,9 @@ Great Expectations renders Expectations to clean, human-readable documentation c
## Validating your Synthetic Data with Great Expectations
+!!! note `Outdated`
+ From ydata-synthetic vx onwards this example will no longer work. Please check `ydata-sdk` and [synthetic data generation examples](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_tabular_data/).
+
#### 1. Install the required libraries:
We recommend you create a virtual environment and install ydata-synthetic and great-expectations by running the following command on your terminal.
diff --git a/docs/reference/api/index.md b/docs/reference/api/index.md
deleted file mode 100644
index e69de29b..00000000
diff --git a/docs/reference/api/preprocessing/base.md b/docs/reference/api/preprocessing/base.md
deleted file mode 100644
index f8bbddb0..00000000
--- a/docs/reference/api/preprocessing/base.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.preprocessing.base_processor.BaseProcessor
\ No newline at end of file
diff --git a/docs/reference/api/preprocessing/regular/ctgan_preprocessor.md b/docs/reference/api/preprocessing/regular/ctgan_preprocessor.md
deleted file mode 100644
index 1978007d..00000000
--- a/docs/reference/api/preprocessing/regular/ctgan_preprocessor.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.preprocessing.regular.ctgan_processor.CTGANDataProcessor
\ No newline at end of file
diff --git a/docs/reference/api/preprocessing/regular/preprocessor.md b/docs/reference/api/preprocessing/regular/preprocessor.md
deleted file mode 100644
index 1fcd4076..00000000
--- a/docs/reference/api/preprocessing/regular/preprocessor.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.preprocessing.regular.processor.RegularDataProcessor
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/cgan.md b/docs/reference/api/synthesizers/cgan.md
deleted file mode 100644
index b6f9a01b..00000000
--- a/docs/reference/api/synthesizers/cgan.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.CGAN
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/gan.md b/docs/reference/api/synthesizers/gan.md
deleted file mode 100644
index 7263103a..00000000
--- a/docs/reference/api/synthesizers/gan.md
+++ /dev/null
@@ -1,4 +0,0 @@
-
-
-
-::: ydata_synthetic.synthesizers.base.BaseGANModel
diff --git a/docs/reference/api/synthesizers/regular/cgan.md b/docs/reference/api/synthesizers/regular/cgan.md
deleted file mode 100644
index b6f9a01b..00000000
--- a/docs/reference/api/synthesizers/regular/cgan.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.CGAN
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/regular/cramergan.md b/docs/reference/api/synthesizers/regular/cramergan.md
deleted file mode 100644
index 4a6349be..00000000
--- a/docs/reference/api/synthesizers/regular/cramergan.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.CRAMERGAN
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/regular/ctgan.md b/docs/reference/api/synthesizers/regular/ctgan.md
deleted file mode 100644
index 92fa365b..00000000
--- a/docs/reference/api/synthesizers/regular/ctgan.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.CTGAN
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/regular/cwgangp.md b/docs/reference/api/synthesizers/regular/cwgangp.md
deleted file mode 100644
index c1ebe0e2..00000000
--- a/docs/reference/api/synthesizers/regular/cwgangp.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.CWGANGP
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/regular/dragan.md b/docs/reference/api/synthesizers/regular/dragan.md
deleted file mode 100644
index 99f4d684..00000000
--- a/docs/reference/api/synthesizers/regular/dragan.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.DRAGAN
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/regular/gan.md b/docs/reference/api/synthesizers/regular/gan.md
deleted file mode 100644
index 4a9e607f..00000000
--- a/docs/reference/api/synthesizers/regular/gan.md
+++ /dev/null
@@ -1,5 +0,0 @@
-
-
-
-::: ydata_synthetic.synthesizers.base.BaseGANModel
-
diff --git a/docs/reference/api/synthesizers/regular/vanilllagan.md b/docs/reference/api/synthesizers/regular/vanilllagan.md
deleted file mode 100644
index cf8f9975..00000000
--- a/docs/reference/api/synthesizers/regular/vanilllagan.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.VanilllaGAN
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/regular/wgan.md b/docs/reference/api/synthesizers/regular/wgan.md
deleted file mode 100644
index 7f07d94b..00000000
--- a/docs/reference/api/synthesizers/regular/wgan.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.WGAN
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/regular/wgan_gp.md b/docs/reference/api/synthesizers/regular/wgan_gp.md
deleted file mode 100644
index 8148d36d..00000000
--- a/docs/reference/api/synthesizers/regular/wgan_gp.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.regular.model.WGAN_GP
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/timeseries/doppelganger.md b/docs/reference/api/synthesizers/timeseries/doppelganger.md
deleted file mode 100644
index 7a8c1696..00000000
--- a/docs/reference/api/synthesizers/timeseries/doppelganger.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.timeseries.doppelganger.model.DoppelGANger
\ No newline at end of file
diff --git a/docs/reference/api/synthesizers/timeseries/timegan.md b/docs/reference/api/synthesizers/timeseries/timegan.md
deleted file mode 100644
index 31b0ed9e..00000000
--- a/docs/reference/api/synthesizers/timeseries/timegan.md
+++ /dev/null
@@ -1,2 +0,0 @@
-
-::: ydata_synthetic.synthesizers.timeseries.timegan.model.TimeGAN
\ No newline at end of file
diff --git a/docs/reference/changelog.md b/docs/reference/changelog.md
deleted file mode 100644
index e69de29b..00000000
diff --git a/docs/support/analytics.md b/docs/support/analytics.md
deleted file mode 100644
index 991ade44..00000000
--- a/docs/support/analytics.md
+++ /dev/null
@@ -1,44 +0,0 @@
-
-# Analytics & Telemetry
-
-## Overview
-
-`ydata-synthetic` is a powerful library designed to generate synthetic data.
-As part of our ongoing efforts to improve user experience and functionality, `ydata-synthetic`
-includes a telemetry feature. This feature collects anonymous usage data, helping us understand
-how the library is used and identify areas for improvement.
-
-The primary goal of collecting telemetry data is to:
-
-- Enhance the functionality and performance of the ydata-synthetic library
-- Prioritize new features based on user engagement
-- Identify common issues and bugs to improve overall user experience
-
-### Data Collected
-The telemetry system collects non-personal, anonymous information such as:
-
-- Python version
-- `ydata-synthetic` version
-- Frequency of use of `ydata-synthetic` features
-- Errors or exceptions thrown within the library
-
-## Disabling usage analytics
-
-We respect your choice to not participate in our telemetry collection.
-If you prefer to disable telemetry, you can do so by setting an environment
-variable on your system. Disabling telemetry will not affect the functionality
-of the ydata-profiling library, except for the ability to contribute to its usage analytics.
-
-### Set an Environment Variable
-In your notebook or script make sure to set YDATA_SYNTHETIC_NO_ANALYTICS
-environment variable to `True`.
-
-````python
- import os
-
- os.environ['YDATA_SYNTHETIC_NO_ANALYTICS']='True'
-````
-
-
-
-
diff --git a/docs/support/contribute.md b/docs/support/contribute.md
deleted file mode 100644
index d85cc367..00000000
--- a/docs/support/contribute.md
+++ /dev/null
@@ -1,22 +0,0 @@
-## Contribute
-
-You are always welcome to contribute to this incredible ecosystem for synthetic data generation. We have several areas in that we are always looking for an extra pair of hands to help us get things going:
-
-- **Documentation:** we all love it, but keeping it up-to-date can be challenging! But guess what? This is also the fastest way for you to get to know ydata-synthetic and start contributing! Even if it is missing, like a new example that could help the community go from zero to hero with synthetic data!
-- **Getting started:** Issues that use this tag are usually the most friendly for someone that has just begun the journey into open-source contributions. So don't be shy; assign the task to you and introduce yourself in the GitHub issue! We will be there to guide you.
-
-But we always look for contributions that go beyond documentation and fixes
-
-- **Synthetic data for NLP:** If you are an expert or just someone that would like to dive into this topic, we welcome you to participate in a small project around the generation of synthetic text.
-- **Synthetic data for images:** if you are a computer vision and you like to share, feel free to add some examples for images!
-- Any other research around synthetic data you would like to share with the community is more than welcome! If you want your research to be part of a rich ecosystem, open a PR!
-
-### PR name and convention
-For the commit messages, if you could please read this [commit-message guidelines](https://juhani.gitlab.io/go-semrel-gitlab/commit-message/) or [conventional commit messages](https://www.conventionalcommits.org/en/v1.0.0-beta.3/#summary) and follow these conventions, it would be awesome.
-
-### Issues and where I can find more info about contributing
-If you need help figuring out where to start or want to learn what other contributors are doing, go to the [Data-Centric AI community Discord](https://discord.com/invite/mw7xjJ7b7s) channel and introduce yourself in the #ydata-synthetic channel.
-
-If you can't find an issue that interests you and wants to add an improvement or change, create a new one.
-
-
\ No newline at end of file
diff --git a/docs/support/help-troubleshooting.md b/docs/support/help-troubleshooting.md
index c576cf13..4d7f7a4b 100644
--- a/docs/support/help-troubleshooting.md
+++ b/docs/support/help-troubleshooting.md
@@ -2,9 +2,9 @@
## Troubleshooting
-- Having a hard time installing **ydata-synthetic** on your laptop?
+- Having a hard time installing **ydata-sdk** on your laptop?
:fontawesome-brands-youtube:{ style="color: #EE0F0F" }
-[Installing ydata-synthetic](https://www.youtube.com/watch?v=aESmGcxtBdU) – :octicons-clock-24:
+[Installing and generating synthetic data with ydata-sdk](https://www.youtube.com/watch?v=AUiGjCLIYzE&t=53s) – :octicons-clock-24:
- You still have questions about python versions or how to get started? Check this [blogpost](https://ydata.ai/resources/10-most-asked-questions-on-ydata-synthetic)!
diff --git a/docs/synthetic_data/faqs.md b/docs/synthetic_data/faqs.md
index e541bfee..a6a23a87 100644
--- a/docs/synthetic_data/faqs.md
+++ b/docs/synthetic_data/faqs.md
@@ -7,28 +7,6 @@ Depending on your use case, the downstream application of your synthetic data, a
For a use-case oriented UI experience, try [YData Fabric](https://ydata.ai/ydata-fabric-free-trial). From an interactive and complete data profiling to an efficient synthetization, your data preparation process will be seamlessly adjusted to your data characteristics.
-## How can I run the Streamlit app?
-
-To try `ydata-synthetic` using the streamlit app, you need to install it using the `[]` notation that encodes the extras that the package incorporates. In this case, you can simply create your virtual environment and install `ydata-synthetic` as:
-
-```bash
-pip install ydata-synthetic[streamlit]
-```
-
-Note that Jupyter or Colab Notebooks are not yet supported, so you need to work it out in your Python environment. Once the package is installed, you can use the following snippet to start the app:
-
-```python
-from ydata_synthetic import streamlit_app
-
-streamlit_app.run()
-```
-
-And that's it! After running the command, the console will output the URL from which you can access the app!
-
-!!! example
- For a step-by-step installation guide, [check this 5-min video](https://www.youtube.com/watch?v=jj9X1_cKRwI&t=2s) that will help you get started!
-
-
## What is the best way to evaluate the quality of my synthetic data?
The most appropriate metrics to evaluate the quality of your synthetic data are also dependent on the goal for which synthetic data will be used. Nevertheless, we may define three essential pillars for synthetic data quality: privacy, fidelity, and utility:
@@ -51,7 +29,7 @@ Most issues with installations are usually associated with unsupported Python ve
Let’s see how you can get both right:
### Python Versions
-Note that `ydata-synthetic` currently requires Python >=3.9, < 3.11 so if you're trying to run our code in Google Colab, then you need to [update your Google Colab’s Python version](https://stackoverflow.com/questions/68657341/how-can-i-update-google-colabs-python-version/68658479#68658479) accordingly. The same goes for your development environment.
+Note that `ydata-sdk` currently requires Python >=3.9, < 3.13 so if you're trying to run our code in Google Colab, then you need to [update your Google Colab’s Python version](https://stackoverflow.com/questions/68657341/how-can-i-update-google-colabs-python-version/68658479#68658479) accordingly. The same goes for your development environment.
### Virtual Environments
A lot of troubleshooting arises due to misalignments between environments and package requirements.
@@ -60,9 +38,9 @@ Virtual Environments isolate your installations from the "global" environment so
Using conda, creating a new environment is as easy as running this on your shell:
```
-conda create --name synth-env python==3.9 pip
+conda create --name synth-env python==3.12 pip
conda activate synth-env
-pip install ydata-synthetic
+pip install ydata-sdk
```
Now you can open up your Python editor or Jupyter Lab and use the synth-env as your development environment, without having to worry about conflicting versions or packages between projects!
diff --git a/docs/synthetic_data/index.md b/docs/synthetic_data/index.md
index 8bc25fc4..6065b0d0 100644
--- a/docs/synthetic_data/index.md
+++ b/docs/synthetic_data/index.md
@@ -4,16 +4,13 @@
take the place of real-world data. The data can be used as an alternative or supplement to real-world data when real-world
data is not readily available. It can also be used as a Machine Learning performance booster.
-The ydata-synthetic package is an open-source Python package developed by YData’s team that allows users to experiment
-with several generative models for synthetic data generation. The main goal of the package is to serve as a way for data
+The ydata-sdk package is a Python package developed by YData’s team that allows users to easily benefit from Generative AI
+and generate synthetic data. The main goal of the package is to serve as a way for data
scientists to get familiar with synthetic data and its applications in real-world domains, as well as the potential of **Generative AI**.
-The *ydata-synthetic* package provides different methods for generating synthetic tabular and time-series data,
-such as Variational Auto Encoders (VAE), [Gaussian Mixture Models (GMM)](single_table/gmm_example.md), and [Conditional Generative Adversarial Networks (CTGAN)](single_table/ctgan_example.md).
-The package also includes a user-friendly UI interface that guides users through the steps and inputs to generate synthetic data
-samples.
+The *ydata-sdk* package provides different methods for generating synthetic tabular, time-series data as well as databases.
-The package also aims to facilitate the exploration and understanding of synthetic data generation methods and their limitations.
+The package also aims to facilitate the exploration and understanding of synthetic data generation methods!
-### 📄Get started with synthetic data for tabular data with CTGAN
-### 📈 Get started with synthetic data for time-series with TimeGAN
\ No newline at end of file
+### 📄Get started with synthetic data for tabular data
+### 📈 Get started with synthetic data for time-series
\ No newline at end of file
diff --git a/docs/synthetic_data/single_table/cgan_example.md b/docs/synthetic_data/single_table/cgan_example.md
index 1014339b..0a2250f6 100644
--- a/docs/synthetic_data/single_table/cgan_example.md
+++ b/docs/synthetic_data/single_table/cgan_example.md
@@ -1,5 +1,10 @@
# Synthesize tabular data
+!!! note "Outdated"
+ Note that this example won't work with the latest version of `ydata-synthetic`.
+
+ Please check `ydata-sdk` to see [how to generate conditional synthetic data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_with_conditional_sampling/).
+
**Using *CGAN* to generate tabular synthetic data:**
Real-world domains are often described by **tabular data** i.e., data that can be structured and organized in a table-like format, where **features/variables** are represented in **columns**, whereas **observations** correspond to the **rows**.
diff --git a/docs/synthetic_data/single_table/cramer_gan_example.md b/docs/synthetic_data/single_table/cramer_gan_example.md
index 1d021889..02e453fe 100644
--- a/docs/synthetic_data/single_table/cramer_gan_example.md
+++ b/docs/synthetic_data/single_table/cramer_gan_example.md
@@ -1,5 +1,10 @@
# Synthesize tabular data
+!!! note "Outdated"
+ Note that this example won't work with the latest version of `ydata-synthetic`.
+
+ Please check `ydata-sdk` to see [how to generate synthetic data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_tabular_data/).
+
**Using *CRAMER GAN* to generate tabular synthetic data:**
Real-world domains are often described by **tabular data** i.e., data that can be structured and organized in a table-like format, where **features/variables** are represented in **columns**, whereas **observations** correspond to the **rows**.
diff --git a/docs/synthetic_data/single_table/ctgan_example.md b/docs/synthetic_data/single_table/ctgan_example.md
index 6010ddac..8158ed8c 100644
--- a/docs/synthetic_data/single_table/ctgan_example.md
+++ b/docs/synthetic_data/single_table/ctgan_example.md
@@ -1,5 +1,10 @@
# Synthesize tabular data
+!!! note "Outdated"
+ Note that this example won't work with the latest version of `ydata-synthetic`.
+
+ Please check `ydata-sdk` to see [how to generate synthetic data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_tabular_data/).
+
**Using *CTGAN* to generate tabular synthetic data:**
Real-world domains are often described by **tabular data** i.e., data that can be structured and organized in a table-like format, where **features/variables** are represented in **columns**, whereas **observations** correspond to the **rows**.
diff --git a/docs/synthetic_data/single_table/cwgangp_example.md b/docs/synthetic_data/single_table/cwgangp_example.md
index 0e471953..bc518d9c 100644
--- a/docs/synthetic_data/single_table/cwgangp_example.md
+++ b/docs/synthetic_data/single_table/cwgangp_example.md
@@ -1,5 +1,10 @@
# Synthesize tabular data
+!!! note "Outdated"
+ Note that this example won't work with the latest version of `ydata-synthetic`.
+
+ Please check `ydata-sdk` to see [how to generate conditional synthetic data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_with_conditional_sampling/).
+
**Using *CWGAN-GP* to generate tabular synthetic data:**
Real-world domains are often described by **tabular data** i.e., data that can be structured and organized in a table-like format, where **features/variables** are represented in **columns**, whereas **observations** correspond to the **rows**.
diff --git a/docs/synthetic_data/single_table/dragan_example.md b/docs/synthetic_data/single_table/dragan_example.md
index e7235c32..b96a95f4 100644
--- a/docs/synthetic_data/single_table/dragan_example.md
+++ b/docs/synthetic_data/single_table/dragan_example.md
@@ -1,5 +1,10 @@
# Synthesize tabular data
+!!! note "Outdated"
+ Note that this example won't work with the latest version of `ydata-synthetic`.
+
+ Please check `ydata-sdk` to see [how to generate synthetic data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_tabular_data/).
+
**Using *DRAGAN* to generate tabular synthetic data:**
Real-world domains are often described by **tabular data** i.e., data that can be structured and organized in a table-like format, where **features/variables** are represented in **columns**, whereas **observations** correspond to the **rows**.
diff --git a/docs/synthetic_data/single_table/gmm_example.md b/docs/synthetic_data/single_table/gmm_example.md
index 56a2ec29..0bd82997 100644
--- a/docs/synthetic_data/single_table/gmm_example.md
+++ b/docs/synthetic_data/single_table/gmm_example.md
@@ -1,5 +1,10 @@
# Synthesize tabular data
+!!! note "Outdated"
+ Note that this example won't work with the latest version of `ydata-synthetic`.
+
+ Please check `ydata-sdk` to see [how to generate synthetic data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_tabular_data/).
+
**Using *GMMs* to generate tabular synthetic data:**
Real-world domains are often described by **tabular data** i.e., data that can be structured and organized in a table-like
diff --git a/docs/synthetic_data/single_table/wgan_example.md b/docs/synthetic_data/single_table/wgan_example.md
index 99db864a..97591903 100644
--- a/docs/synthetic_data/single_table/wgan_example.md
+++ b/docs/synthetic_data/single_table/wgan_example.md
@@ -1,5 +1,10 @@
# Synthesize tabular data
+!!! note "Outdated"
+ Note that this example won't work with the latest version of `ydata-synthetic`.
+
+ Please check `ydata-sdk` to see [how to generate synthetic data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_tabular_data/).
+
**Using *WGAN* to generate tabular synthetic data:**
Real-world domains are often described by **tabular data** i.e., data that can be structured and organized in a table-like format, where **features/variables** are represented in **columns**, whereas **observations** correspond to the **rows**.
diff --git a/docs/synthetic_data/single_table/wgangp_example.md b/docs/synthetic_data/single_table/wgangp_example.md
index 367d0980..4955655d 100644
--- a/docs/synthetic_data/single_table/wgangp_example.md
+++ b/docs/synthetic_data/single_table/wgangp_example.md
@@ -1,5 +1,10 @@
# Synthesize tabular data
+!!! note "Outdated"
+ Note that this example won't work with the latest version of `ydata-synthetic`.
+
+ Please check `ydata-sdk` to see [how to generate synthetic data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_tabular_data/).
+
**Using *WGAN-GP* to generate tabular synthetic data:**
Real-world domains are often described by **tabular data** i.e., data that can be structured and organized in a table-like format, where **features/variables** are represented in **columns**, whereas **observations** correspond to the **rows**.
diff --git a/docs/synthetic_data/streamlit_app.md b/docs/synthetic_data/streamlit_app.md
deleted file mode 100644
index 4863790d..00000000
--- a/docs/synthetic_data/streamlit_app.md
+++ /dev/null
@@ -1,46 +0,0 @@
-# The UI guided experience for Synthetic Data generation
-
-´ydata-synthetic´ offers a UI interface to guide you through the steps and inputs to generate structure tabular data.
-The streamlit app is available from *v1.0.0* onwards, and supports the following flows:
-
-- Train a synthesizer model for a single table dataset
-- Generate & profile the generated synthetic samples
-
-- -
- -## Installation - -pip install ydata-synthetic[streamlit] - -## Quickstart - -Use the code snippet below in a python file: - -!!! warning "Use python scripts" - - I know you probably love Jupyter Notebooks or Google Colab, but make sure that you start your - synthetic data generation streamlit app from a python script as notebooks are not supported! - -``` py - from ydata_synthetic import streamlit_app - streamlit_app.run() -``` - -Or use the file streamlit_app.py that can be found in the [examples folder](). - -``` py - python -m streamlit_app -``` - -The below models are supported: - -- [ydata-sdk Synthetic Data generator](https://docs.sdk.ydata.ai/0.6/examples/synthesize_tabular_data/) -- CGAN -- WGAN -- WGANGP -- DRAGAN -- CRAMER -- CTGAN - diff --git a/docs/synthetic_data/time_series/doppelganger_example.md b/docs/synthetic_data/time_series/doppelganger_example.md index b418879b..2cd125fa 100644 --- a/docs/synthetic_data/time_series/doppelganger_example.md +++ b/docs/synthetic_data/time_series/doppelganger_example.md @@ -1,5 +1,11 @@ # Synthesize time-series data +!!! note "Outdated" + Note that this example won't work with the latest version of `ydata-synthetic`. + + Please check `ydata-sdk` to see [how to generate synthetic time-series data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_timeseries_data/). + + **Using *DoppelGANger* to generate synthetic time-series data:** Although tabular data may be the most frequently discussed type of data, a great number of real-world domains — from traffic and daily trajectories to stock prices and energy consumption patterns — produce **time-series data** which introduces several aspects of complexity to synthetic data generation. diff --git a/docs/synthetic_data/time_series/timegan_example.md b/docs/synthetic_data/time_series/timegan_example.md index b4c36c3f..ec2fbe87 100644 --- a/docs/synthetic_data/time_series/timegan_example.md +++ b/docs/synthetic_data/time_series/timegan_example.md @@ -1,6 +1,20 @@ # Synthesize time-series data -**Using *TimeGAN* to generate synthetic time-series data:** +!!! note "Outdated" + Note that this example won't work with the latest version of `ydata-synthetic`. + + Please check `ydata-sdk` to see [how to generate synthetic time-series data](https://docs.fabric.ydata.ai/latest/sdk/examples/synthesize_timeseries_data/). + +## Why YData Fabric vs TimeGAN for time-series data +YData Fabric offers advanced capabilities for time-series synthetic data generation, surpassing TimeGAN in terms of flexibility, +scalability, and ease of use. With YData Fabric, users can generate high-quality synthetic time-series data while benefiting from built-in data profiling tools +that ensure the integrity and consistency of the data. Unlike TimeGAN, which is a single model for time-series, YData Fabric offers a solution that is suitable for different types of datasets and behaviours. +Additionally, YData Fabric is designed for scalability, enabling seamless handling of large, complex time-series datasets. Its guided UI makes it easy to adapt to different time-series scenarios, +from healthcare to financial data, making it a more comprehensive and flexible solution for time-series data generation. + +For more on [YData Fabric vs Synthetic data generation with TimeGAN read this blogpost](https://ydata.ai/resources/the-best-generative-ai-model-for-time-series-synthetic-data-generation). + +## Using *TimeGAN* to generate synthetic time-series data Although tabular data may be the most frequently discussed type of data, a great number of real-world domains — from traffic and daily trajectories to stock prices and energy consumption patterns — produce **time-series data** which introduces several aspects of complexity to synthetic data generation. diff --git a/docs/synthetic_data/ydata_fabric_app.md b/docs/synthetic_data/ydata_fabric_app.md new file mode 100644 index 00000000..ddf2ea44 --- /dev/null +++ b/docs/synthetic_data/ydata_fabric_app.md @@ -0,0 +1,28 @@ +# The UI guided experience for Synthetic Data generation + +[YData Fabric provides a robust, guided user interface (UI) specifically designed to streamline synthetic data generation](https://ydata.ai/products/fabric). +This interface is tailored to support users at every level, ensuring that both novice users and experienced data scientists can efficiently generate +synthetic datasets while adhering to best practices. + +## Step-by-Step Workflow +The YData Fabric UI organizes the synthetic data generation process into a structured, step-by-step workflow. +Each stage of the process is clearly defined and supported by guidance within the interface, helping users navigate tasks like data profiling, +metadata and synthesizer configuration and synthetic data quality evaluation. + +- **Data Upload and Profiling:** Users start by uploading their datasets directly into the platform. YData Fabric’s profiling tool automatically scans +the data, generating insights into key attributes such as data distributions, correlations, and missing values. +These insights are presented in an intuitive, visual format, ensuring users can quickly assess the quality and structure of their data. +- **Alerts for Data Issues:** The UI will alert users to potential issues such as data imbalances, outliers, or incomplete fields that may affect the +quality of the synthetic data. +- **Synthetic Data Generation Model Configuration:** Once the data is profiled, the UI supports metadata configuration (categorical, numerical, dates, etc), +anonymization integration. +- **Model Performance Insights:** During the model training phase, YData Fabric monitors key performance indicators (KPIs) like fidelity, utility and privacy. +These KPIs, such as data fidelity and privacy scores, are displayed on the dashboard, allowing users to evaluate how closely the synthetic data aligns with the original dataset. +- **Customization and Advanced Controls:** For more experienced users, YData Fabric provides customization options within the guided UI. +Users have access to advanced settings, such as [conditional synthetic data generation](https://ydata.ai/resources/conditional-synthetic-data-generation-for-robust-machine-learning-applications) or business rules. +- **Preserving Data Integrity:** For datasets requiring strict adherence to structural patterns (e.g., time-series data, healthcare records or databases). + +### Getting started with YData Fabric (Community version) +YData Fabric’s Community Version offers users a free, accessible entry point to explore synthetic data generation. +To get started, users can [sign up for the Community Version and access the guided UI directly](https://ydata.ai/register). +Once registered, users are provided with a range of features, including data profiling, synthetic data generation, pipelines and access to YData’s proprietary models for data quality! \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index c8c15442..f5db0ca8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,10 +9,9 @@ nav: - 'index.md' - Overview: 'index.md' - Installation: 'getting-started/installation.md' - - Quickstart: 'getting-started/quickstart.md' - - Synthetic Data Generation: + - Synthetic Data Generation: - "synthetic_data/index.md" - - UI interface - Streamlit app: "synthetic_data/streamlit_app.md" + - UI interface - YData Fabric: "synthetic_data/ydata_fabric_app.md" - Generate Single Table: - GMM: "synthetic_data/single_table/gmm_example.md" - CGAN: "synthetic_data/single_table/cgan_example.md" @@ -32,31 +31,6 @@ nav: - Great Expectations: "integrations/gx_integration.md" - Support: - Help & Troubleshooting: 'support/help-troubleshooting.md' - - Contribution Guidelines: 'support/contribute.md' - - Contribution Guidelines: 'support/contribute.md' - - Analytics: 'support/analytics.md' - - Reference: - - Changelog: 'reference/changelog.md' - - API: - - Synthesizers: - - GAN: 'reference/api/synthesizers/regular/gan.md' - - Regular: - - CGAN: 'reference/api/synthesizers/regular/cgan.md' - - CRAMERGAN: 'reference/api/synthesizers/regular/cramergan.md' - - CTGAN: 'reference/api/synthesizers/regular/ctgan.md' - - CWGANGP: 'reference/api/synthesizers/regular/cwgangp.md' - - DRAGAN: 'reference/api/synthesizers/regular/dragan.md' - - VANILLAGAN: 'reference/api/synthesizers/regular/vanillagan.md' - - WGAN_GP: 'reference/api/synthesizers/regular/wgan_gp.md' - - WGAN: 'reference/api/synthesizers/regular/wgan.md' - - Timeseries: - - TimeGAN: 'reference/api/synthesizers/timeseries/timegan.md' - - DoppelGANger: 'reference/api/synthesizers/timeseries/doppelganger.md' - - Preprocessing: - - BaseProcessor: 'reference/api/preprocessing/base.md' - - Regular: - - RegularDataProcessor: 'reference/api/preprocessing/regular/preprocessor.md' - - CTGANDataProcessor: 'reference/api/preprocessing/regular/ctgan_preprocessor.md' theme: name: material language: en diff --git a/setup.py b/setup.py index 05c2b92b..baa72e29 100644 --- a/setup.py +++ b/setup.py @@ -42,19 +42,10 @@ keywords='data science ydata', url='https://github.com/ydataai/ydata-synthetic', license="https://github.com/ydataai/ydata-synthetic/blob/master/LICENSE", - python_requires=">=3.9, <3.12", + python_requires=">=3.9, <3.13", packages=find_namespace_packages('src'), package_dir={'':'src'}, include_package_data=True, options={"bdist_wheel": {"universal": True}}, install_requires=requirements, - extras_require={ - "streamlit": [ - "streamlit==1.29.0", - "typing-extensions>=3.10.0", - "streamlit_pandas_profiling==0.1.3", - "ydata-profiling<5", - "ydata-sdk>=0.2.1", - ], - }, )