Skip to content

Commit 00b4d08

Browse files
deorendeoren
authored
Merge pull request WhyAskWhy#1 from deoren/initial-dev-env-work
Initial version of mysql2sqlite-dev project
2 parents 14e28a5 + 0aaa331 commit 00b4d08

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+2880
-1
lines changed

‎.gitignore

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# https://github.com/WhyAskWhy/mysql2sqlite
2+
# https://github.com/WhyAskWhy/mysql2sqlite-dev
3+
4+
# Real data that I don't want to accidentally commit
5+
sample_data
6+

‎README.md

+97-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,98 @@
11
# mysql2sqlite-dev
2-
Development environment for mysql2sqlite work
2+
3+
Development environment for
4+
[mysql2sqlite](https://github.com/WhyAskWhy/mysql2sqlite) work. This repo may
5+
be folded into that repo at a later time.
6+
7+
## Requirements
8+
9+
In order to reduce setup requirements on the part of the developer, the scripts
10+
used by this project make multiple assumptions that the main project does not:
11+
12+
- Ubuntu 16.04
13+
- clean, without any packages already installed aside from OS and
14+
integration tools (e.g., Hyper-V, VirtualBox, VMware Tools)
15+
- Internet access (needed for installation and setup of packages)
16+
17+
## Repo layout
18+
19+
File/Path | Purpose | Notes
20+
--------- | ------- | -----
21+
`bin` | Scripts meant to be used directly by developer| Most are used directly within the guest/VM, but some from Host system
22+
`bin/create_db.sh` | Run after the `bin/setup_dev_environment.sh` script in order to create SQLite database file | Usually run once
23+
`bin/rebuild_test_db.sh` | Run as-needed to reset db state after tinkering with the content. Convenience script only
24+
`bin/setup_dev_environment.sh` | Main setup script to kickstart a new dev environment | Usually run just once per new dev env
25+
`bin/sync_files.ps1` | Copy files from Hyper-V Host to Hyper-V guest | Alternative to first cloning this repo to get access to `bin/setup_dev_environment.sh` script
26+
`bin/validate_dbs.py` | Run after the `bin/create_db.sh` script to perform basic validation of newly created SQLite db. | WIP, future script that I hope to refactor and use as pre and post CI tools to confirm valid conversion results. This script may be moved to the main project in the near future.
27+
`bin/validate_dbs.sh` | Run after the `bin/create_db.sh` script to perform basic validation of newly created SQLite db. | DEPRECATED. This script will probably be removed in the near future as most of the functionality has been replicated by the `bin/validate_dbs.py` script
28+
`etc` | Configuration files for various dependencies/tools used within dev environment | Deployed as part of running the `bin/setup_dev_environment.sh` script
29+
`keys` | Local copy of upstream repo package signing keys | May be phased out if not proven useful
30+
`sql` | Import files used to setup database used by main project | Imported as part of running the `bin/setup_dev_environment.sh` script
31+
`LICENSE` | License for this collection of content | Intended to match the license used for main project
32+
`README.md` | Main doc file for this project | Please submit a PR or bug report for any missing or incorrect coverage
33+
34+
## Dependencies
35+
36+
### Gitea (or similar local Git mirror)
37+
38+
To keep from abusing GitHub, VSTS and other Git "remotes", I run a local Gitea
39+
instance. This is entirely optional and can be overriden by adjusting the
40+
`MAIN_PROJECT_GIT_REPO_URL` and `THIS_DEV_ENV_GIT_REPO_URL` variables
41+
within the various build scripts to reference a different Git remote.
42+
Alternatives are provided, but commented out in case you wish to use them.
43+
44+
You will find evidence of Gitea specified by the `local-mirror:3000` Git repo
45+
URLs in the various build scripts.
46+
47+
### Squid proxy server
48+
49+
Used much for the same reason as with the local Gitea instance, I run a local
50+
Squid VM to cache remote content for local rebuilds.
51+
52+
Not yet included here, this proxy has been enabled by way of the
53+
`/etc/environment` and `/etc/apt/apt.conf` conf files. Those files were
54+
modified by way of the `Network proxy` configuration UI within the test VM.
55+
See the link below in the References section for details.
56+
57+
Using this UI (or directly modifying the files mentioned) allows our Ubuntu
58+
16.04 test box to use the Squid proxy server for system-wide
59+
proxy use. This is of particular help with speeding up package installation
60+
for repeat builds. I like to also think that using local mirrors for builds
61+
is appreciated by upstream package mirror maintainers/owners.
62+
63+
### Postfix
64+
65+
As part of the setup work, Postfix is installed and configured to use local
66+
MySQL and SQLite databases for many of its lookup tables. This is intended
67+
to serve as a means of verifying that mysql2sqlite was successful in exporting
68+
entries from the MySQL tables to the local SQLite database.
69+
70+
The idea is that if we get consistent, duplicate results when running
71+
postmap queries against both the MySQL and SQLite database tables then
72+
mysql2sqlite is working as intended.
73+
74+
Listens on localhost only.
75+
76+
### MariaDB
77+
78+
The latest upstream version of MariaDB 10.0.x is installed and configured in an
79+
insecure manner for production purposes, but sufficient for our local testing
80+
needs. No password is set for the `root` account.
81+
82+
Listens on localhost only.
83+
84+
## Tools
85+
86+
The following list of tools are installed as part of running the
87+
`bin/setup_dev_environment.sh` script during the initial test VM setup:
88+
89+
- DB Browser GUI application
90+
- MySQL Workbench (or an alternative)
91+
92+
These tools are meant to assist with taking a "hands-on" approach to
93+
troubleshooting issues with source and destination databases.
94+
95+
## References
96+
97+
- the local [docs/references.md](docs/references.md) file.
98+
- the main project [docs/references.md](https://github.com/WhyAskWhy/mysql2sqlite/blob/master/docs/references.md) file.

‎bin/create_db.sh

+59
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/bin/bash
2+
3+
# https://github.com/WhyAskWhy/mysql2sqlite
4+
# https://github.com/WhyAskWhy/mysql2sqlite-dev
5+
6+
# Purpose:
7+
#
8+
# Run primary script to generate output SQLite db file for testing
9+
10+
#MAIN_PROJECT_GIT_REPO_URL="https://github.com/deoren/mysql2sqlite"
11+
#MAIN_PROJECT_GIT_REPO_URL="https://github.com/WhyAskWhy/mysql2sqlite"
12+
MAIN_PROJECT_GIT_REPO_URL="http://local-mirror:3000/mirror/mysql2sqlite"
13+
MAIN_PROJECT_GIT_REPO_BASENAME="$(basename ${MAIN_PROJECT_GIT_REPO_URL})"
14+
MAIN_PROJECT_GIT_REPO_BRANCH="master"
15+
16+
# THIS_DEV_ENV_GIT_REPO_URL="https://github.com/deoren/mysql2sqlite-dev"
17+
# THIS_DEV_ENV_GIT_REPO_URL="https://github.com/WhyAskWhy/mysql2sqlite-dev"
18+
THIS_DEV_ENV_GIT_REPO_URL="http://local-mirror:3000/mirror/mysql2sqlite-dev"
19+
THIS_DEV_ENV_GIT_REPO_BASENAME="$(basename ${THIS_DEV_ENV_GIT_REPO_URL})"
20+
THIS_DEV_ENV_GIT_REPO_BRANCH="master"
21+
22+
23+
# Get updated repo contents
24+
if [[ ! -d /tmp/${MAIN_PROJECT_GIT_REPO_BASENAME} ]]; then
25+
cd /tmp
26+
git clone ${MAIN_PROJECT_GIT_REPO_URL}
27+
cd ${MAIN_PROJECT_GIT_REPO_BASENAME}
28+
git checkout ${MAIN_PROJECT_GIT_REPO_BRANCH}
29+
else
30+
cd /tmp/${MAIN_PROJECT_GIT_REPO_BASENAME}
31+
git reset HEAD --hard
32+
git checkout ${MAIN_PROJECT_GIT_REPO_BRANCH}
33+
git pull --ff-only
34+
fi
35+
36+
# Check for required modules
37+
if ! python3 -c "import mysql.connector"
38+
then
39+
# Looks like we're missing some modules: install them
40+
pip3 install -r /tmp/${MAIN_PROJECT_GIT_REPO_BASENAME}/requirements.txt --user
41+
fi
42+
43+
# Go ahead and pre-create the output dir for SQLite database
44+
sudo mkdir -vp /var/cache/mysql2sqlite
45+
sudo chown -vR ${USER}: /var/cache/mysql2sqlite
46+
47+
48+
# TODO: What command-line options are needed?
49+
# - path to main config file?
50+
# - path to output db file?
51+
python3 /tmp/${MAIN_PROJECT_GIT_REPO_BASENAME}/mysql2sqlite.py
52+
53+
if [[ $? -eq 0 ]]; then
54+
echo "Successfully generated SQLite db file."
55+
echo "Run python3 /tmp/${THIS_DEV_ENV_GIT_REPO_BASENAME}/bin/validate_dbs.py next to confirm db was created properly."
56+
else
57+
echo "FAILURE to generate SQLite db file!"
58+
exit 1
59+
fi

‎bin/rebuild_test_db.sh

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/bash
2+
3+
# https://github.com/WhyAskWhy/mysql2sqlite
4+
# https://github.com/WhyAskWhy/mysql2sqlite-dev
5+
6+
# Purpose: Quickly/easily drop and recreate test MySQL database
7+
8+
if [[ ! -d "$PWD/sql" ]]
9+
then
10+
echo "Incorrect working directory. Move into mysql2sqlite-dev repo and try again."
11+
else
12+
13+
echo "* Dropping MySQL mailserver database ..."
14+
mysql -u root -e "drop database mailserver;"
15+
16+
echo "* Recreating MySQL mailserver database ..."
17+
mysql -u root < $PWD/sql/mysql_db_schema.sql ||
18+
{ echo "Failed to import db schema database. Aborting!"; exit 1; }
19+
20+
echo "* Setting up MySQL mailserver database user, access ..."
21+
mysql -u root < $PWD/sql/setup_mysql_database.sql ||
22+
{ echo "Failed to setup database. Aborting!"; exit 1; }
23+
24+
echo "* Importing MySQL mailserver database test data ..."
25+
mysql -u root < $PWD/sql/mysql_test_data.sql ||
26+
{ echo "Failed to import test data. Aborting!"; exit 1; }
27+
28+
fi

0 commit comments

Comments
 (0)