deltasql - Manual

A strong version control system for databases saves time to developers, who would like to keep their schema updated while developing with minimal effort. With a version control for databases, the number of mistakes dued to different schemas or missing scripts is considerably reduced. Critical errors in production due to schema inconsistencies disappear, too.

deltasql is an Open Source tool to synchronize databases with source code. While developing middle size or big applications, developers undertake changes to the data model which go along with changes to the source code. From time to time, branches of source code are done to stabilize the code which will go to production. A sort of data model branch is also needed.

deltasql provides a simple set of php scripts to be executed on an apache server backed by a mySQL database to collect all scripts which change the data model, and means to handle data model branches. The trick is to number the sql scripts and to create on each database instance a table which keeps the number of the latest executed script (the table deltasql uses is named TBSYNCHRONIZE).

A form allows the user to enter data from the synchronization table and thereafter the needed chain of datamodel updates is computed and shown to the user. The user has to manually execute all scripts. It is possible to update development schemas (the HEADs) and production schemas (the branches), to transform a production schema into a development schema and vice versa. Also in case of a schema dumped and imported into another database, it is still upgradeable as the synchronization table is contained into the copyed schema. However, it is not possible to downgrade a schema back to a previous version.

There are deltasql clients (listed in the table above), which automatically collect synchronization data from a given database schema. Though deltasql works best with Oracle, PostgreSQL and mySQL schemas, any other database type can use most of deltasql functionality.

Purpose

deltasql Server allows developers to submit their SQL scripts in a central place. Each time a script is submitted, deltasql assigns a number to it (the script's version). Database administrators can check at which version the database table is, by checking the last inserted row in the table TBSYNCHRONIZE. They also can look at the project name in this table. They can then ask the deltasql Server for the currently missing scripts.

With deltasql: A developer sends the script to the deltasql server only, the exact database state is always known and can be synchronized to any state.

When scripts are sent, a database module has to be chosen. A project can consist of one or several modules, as in some companies, they tipically use a three-level structure for their source code modules. The first database module is the lowest level and represents tables which are common for all projects in a company. The middle database module represents the tables of the application which is sold. The highest level module contains tables which are needed for the customization of one particular customer, and which are not needed for other customers.

For simple applications, it is enough to specify one module for each project, although deltasql was developed as help in a much more complex development environment.

deltasql also supports database branches, which are similar to source code branches. It is possible to choose, if a script belongs also to a particular branch. And it is possible to update a branch database to a newer branch, or to the development schema represented by the branch named HEAD.

deltasql Server is written in such a way, that clients can interface to it and query the database automatically. Deltasql is shipped with several clients which easy the interface to database schemas and sql clients, which show latest available scripts and which can perform continouus database integration tasks. People in the Open Source community are encouraged to write their own clients, or to partecipate in improvements of the server itself.

Without deltasql: A developer has to send his/her scripts to everyone, the exact database state is known only to the database owner.

If you find the project useful, or if you use deltasql for managing your Open Source project, you are encouraged to add this button to your homepage. Link it to the project page at http://deltasql.sourceforge.net.This might bring some additional traffic and maybe involve others in the project who will improve the quality of this software. Thanks!

Install Guides

Install steps for the Server

deltasql runs on Apache webserver with the PHP module backed by a mySQL database.

If you want to run it on windows, you should first download from the Apachefriends.org website XAMPP, a package which bundles Apache, mySQL and phpMyAdmin (and more... :-) for Linux and Windows. deltasql was developed with XAMPP 3.0.12, but any newer version might work as well. Then you should install XAMPP. Using the XAMPP Control Panel start Apache and mySQL.

Of course, XAMPP is not a strict requirement, just a good entry point for people who would like to get the software stack required by deltasql installed in few minutes. Almost all Linux distributions come with options to install Apache, Apache PHP module and a mySQL database. deltasql works with them as well.

As next step, download the deltasql PHP scripts, unzip them and copy them to the htdocs directory created by XAMPP or by your Linux distribution, so that you get a htdocs/deltasql directory full of PHP scripts.

If you visit the main page index.php, there is a link called Please Setup Deltasql.... Click on this link and follow the instructions there. Set a password for the admin user of deltasql. If you installed XAMPP, you do not need to define a mySQL password for the root account, as XAMPP sets the empty password for root by default. Else please use the correct root password of mySQL. Pressing the "Submit" button will install everything you need to run your deltasql server.

This movie shows the steps described above, too.

If something should go wrong, you can read also through the next section.

Manual install steps for the server

Go to the directory deltasql/conf, take example_config.inc.php, copy it to config.inc.php. Edit the variables in config.inc.php depending on your hardware and network configuration.

As next step, you should visit http://localhost/phpmyadmin, to access phpMyAdmin, an interface that speaks with the mySQL database. In phpMyAdmin, create a user delta_user with a password you can set in htdocs/deltasql/conf/config.inc.php. If the deltasql server is published on the Internet, you should also modify the constant $dns_name with the IP number of your computer or with its DNS name.

Now take deltasql/db/script.sql and go back to http://localhost/phpmyadmin. Execute the SQL script with the phpMyAdmin interface. It will create a deltasql database.

If you did all steps right, you should get a working deltasql Server as in this webpage if you visit http://localhost/deltasql

Log as Administrator with username admin and password testdbsync, so that you can create your own users. Do not forget to change the admin password shortly after with the Change password form.

Install steps for the deltaclient on Windows (optional step)

Download first deltaclient_windows.zip from the main page and unzip it in some directory. Locate deltaclient.exe in the bin subfolder and launch it. Continue with the steps described here.

Picture: deltaclient running on Windows 7.

How to install deltasql Server on a Raspberry Pi

I acquired a Raspberry Pi 3, a wonderful and cheap ARM machine you get for less than 70$, here the steps to install deltasql on it. I use Raspbian GNU/Linux 8 jessie as operating system. Install steps should be very similar also for an Ubuntu or Debian distribution.

First, I followed this tutorial to install Apache with PHP and mySQL.

Then install also phpmyadmin, which could be useful later on, with:


pi@rasputin:~ $ sudo apt-get install phpmyadmin

pi@rasputin:~ $ sudo cp /etc/phpmyadmin/apache.conf  /etc/apache2/sites-available/phpmyadmin.conf

pi@rasputin:~ $ cd /etc/apache2/sites-available

pi@rasputin:~ $ sudo a2ensite phpmyadmin.conf

pi@rasputin:~ $ sudo service apache2 reload

Now checkout the latest deltasql from git repository. To do so, you first need to install git with


pi@rasputin:~ $ sudo apt-get install git



pi@rasputin:~ $ cd /var/www/html

pi@rasputin:~ $ sudo git clone git://git.code.sf.net/p/deltasql/code deltasql

Temporarily give full access rights to the "conf" folder:


pi@rasputin:~ $ cd deltasql

pi@rasputin:~ $ sudo chmod 777 conf

pi@rasputin:~ $ sudo service apache2 restart

Now you can visit http:// [Raspberry address] /deltasql

Choose the link "Setup deltasql" and carefully fill the fields according to the provided instructions. Hit the "Install" button at the bottom.

The install script will run and you should have now a running deltasql instance on http:// [Raspberry address] /deltasql

Remember to change the permissions on the conf folder back to correct defaults with


pi@rasputin:~ $ sudo chmod 755 conf

pi@rasputin:~ $ sudo chmod 644 conf/config.inc.php

pi@rasputin:~ $ sudo service apache2 restart

If you did everything correctly, you will have a running deltasql instance at: http:// [Raspberry address] /deltasql

POST SCRIPTUM: If after installing deltasql, the install_execution.php complains that it can not write the config.inc.php because of the permissions on the directory. You can then do this:


pi@rasputin:~ $ cd /var/www/html/deltasql/conf

pi@rasputin:~ $ sudo cp example_config config.inc.php

Modify config.inc.php, it will contain the same fields you were filling before


pi@rasputin:~ $ sudo nano config.inc.php

Hit Ctrl+O to save the changes. Then set the permissions to the file:


pi@rasputin:~ $ sudo chmod 644 config.inc.php

And restart apache with


pi@rasputin:~ $ sudo service apache2 restart

Install steps for the deltaclient on Linux (optional step)

Download first deltaclient_linux.tar.gz for Gnome or Unity and unpack it with gunzip deltaclient_linux.tar.gz and tar -xf deltaclient_linux.tar. Change in the bin subfolder with cd bin and launch deltaclient with ./deltaclient.

Picture: deltaclient running on Ubuntu Unity.

If deltaclient launches as in the screenshot above, you can read the next section. If you get an error message about missing libraries, you can try to compile deltaclient on Linux.

Install steps for the deltaclient (general steps)

Deltaclient can copy to clipboard the query to retrieve the current schema version and the generated script as well. The user still needs to paste the queries and the generation script into his/her own SQL client (TOAD, SQL Developer, SQL Server Management Studio, SquirrelSQL etc.).

At startup, deltaclient will download all projects and branches which are currently available on Deltasql Server.

Based on the Project selection, deltaclient will show only branches which belong to the selected project. Similarly, when selecting the From: field, it will show only valid branches and tags into the To: field.

The version field should be filled with the result retrieved from the query which gets copyed when pressing the 'Copy SQL script to retrieve version number' button. Please note that the query works only on a prepared database.

The branches and tags checkboxes are useful to filter branches and tags from the From: and To: comboboxes. The special branch HEAD will always appear, even if the branches checkbox is unchecked.

When running the first time, click on 'Settings...' button and configure the communication settings. The proxy parameter is important, if you run deltaclient in an enterprise behind a proxy. If the proxy parameter is set incorrectly, deltasql will hang the next time is running. In such a case, launch restart.bat on Windows (or restart.sh on Linux) and correct the proxy parameter.

The url setting should point to your inhouse deltasql server. In such case, you should leave the proxy parameter empty. For testing purposes, you can leave the predefined "deltasql.sourceforge.net/deltasql" value, just to see how deltaclient behaves.

Instead of notepad, you can also define another editor to view SQL scripts such as Notepad++, Ultraedit or similar.

The deltaclient user parameter is not very important: it is used by deltasql server to collect statistics on how many times deltaclient was used to retrieve the synchronization script.

Install steps for the dbredactor client (optional step)

The dbredactor client is a deltasql client which can retrieve from the database the current schema version, and using this information it retrieves from deltasql server the synchronization script. Said script is shown to the user either in pretty printed HTML format or in text format on your editor of choice.

Download first dbredactor.zip from the deltasql main page (at the bottom!). Unzip the file in the Eclipse workspace. Add the build.xml into the Ant window by right clicking and choosing 'Add build files...'. Copy sample-build.properties to build.properties and set the correct URL for the deltasql server in the deltasql.server.url property. Create in the /config directory a configuration for the database schema as they are done under /config/examples. Next, update config.set with the path to the directory you created.

Picture: the two Ant targets of dbredactor

If everything is configured correctly, you'll be able to check if a schema update is needed with a double click on the ant target dbsync info and to get the script needed to update with dbsync update.

Install steps for the ant client into Eclipse (optional step)

The dbredactor client is a lightweight deltasql client like dbredactor. It integrates best into your build.xml script for the Eclipse/Java development environment.

Download first ant-client.zip from the deltasql main page (at the bottom!) and unzip it somewhere. This zip file contains only two tiny files: deltasql-build.xml and sample-deltasql.properties.

Picture: the target of the Ant Client

Copy deltasql-build.xml into the Eclipse project where you are working (at best in the root directory of the Eclipse project). Add the deltasql-build.xml file into the Ant window by right clicking and choosing 'Add build files...'. Copy sample-deltasql.properties to deltasql.properties. Read through deltasql properties and adjust all properties. Create a lib folder in the Eclipse project (if not already there) and copy all the .jars you can find in the clients\java\dbredactor\lib directory of this package (or only the ones you enabled in deltasql.properties, at your wish).

Et voil\E0, now you should have a working ant client. By pressing on "RetrieveUpdates" you will receive the script with the latest db updates. The ant client is also easy to integrate in an existing build.xml!

How to install the bash client (optional)

Picture: Bash client performing a database synchronization. Automating the bash client leads to continouus database integration!

The bash client allows in combination with deltasql server to setup a continouus database integration.

Download first the Bash client (bash_client.tar.gz) from the deltasql main page (at the bottom!) onto your favourite GNU/Linux server.

Unpack it with gunzip bash_client.tar.gz and tar -xf bash_client.tar. Make sure all shell scripts (also in subdirectories) have executable rights with chmod 775 *.sh. Open deltasql.conf, and configure each variable of the file.

You can test the connection to the deltasql server by running ./deltaclient.sh 1. To test the full cycle (retrieving schema version, contacting deltasql server and retrieving synchronization script, executing the synchronization script) please launch ./continouusintegration.sh.

Then with crontab -e register continouusintegration.sh as a cron job. From time to time, read the logfile sync.log to check that everything is running as expected. syncscript.log contains the concatenation of all scripts executed on the database schema.

How to install the Python deltasql client (optional)

The Python client is a simple deltasql client which helps to synchronize mySQL databases. Unlike other clients, it does not support other database types.

Download first the Python client (pyclient.tar.gz) from the deltasql main page (at the bottom!) onto your favourite GNU/Linux server.

Unpack it with gunzip pyclient.tar.gz and tar -xf pyclient.tar. Make sure all Python scripts have executable rights with chmod 775 *.py. Open config.ini, and configure each variable of the file.

Python does not come with a preconfigured mysql connector. To get pyclient working, you need either to install MySQLdb or Oracle's mysql connector. Then select the connector you installed in the variable pydbadapter of the config.ini file.

Depending on the option executeupdate in config.ini, the Python client will either show the script into your editor of choice for later execution, or it will execute it directly into the database schema. The output of the last execution is stored in script.out, and you should monitor this file for possible errors. The concatenated output of all executions is stored in fullscript.out.

The script itself is robust and does several checks before automatically executing an update in the database schema.

To test the full cycle (retrieving schema version, contacting deltasql server and retrieving synchronization script, executing the synchronization script) please launch ./pyclient.py.

Usage

User Management

There are four types of users: Guests, Developers, Project Managers and Administrators. Guests can only browse through project's scripts, search among them and compute synchronizations for schemas. Developers can do everything Guests can do, but they also can submit new scripts, and edit old ones. Project Managers can create projects and add modules. They can create branches and tags as well. Administrators are Project Managers which can add new users to the deltasql database. They manage permissions as well.

This movie shows how user management works, too.

Quick Guide

This movie should be a good introduction tutorial.

Define a module 'mymodule' with Create DB Module. Then define a project 'myproject' with Create a project. Under List projects it is possible to add the module 'mymodule' to the project 'myproject' with the link 'Add Module'.

Now choose 'Table' under List projects. Specify your database type. This will generate a table which needs to be created in all involved database schemas, you would like to keep under version control with deltasql. Create this table in these database schemas with a sql client.

Developers can now submit scripts with this form. They have to choose to which module the script belongs. If branches are created, they can also choose if a script is only a script for development schemas (a HEAD script), or if it needs to be applied to a previously created branch. They can list all previously submitted scripts. In this list, they can also modify a submitted script, if it contains errors, or if it is incomplete.

To update a database schema with the submitted scripts, you should first query the database schema with SELECT * FROM TBSYNCHRONIZE ORDER BY VERSIONNR DESC. Retrieve in the first row the project name (in PROJECTNAME column), its current version (in VERSIONNR) and its source branch value (in BRANCHNAME). The source branch value HEAD is used for development schemas. For production databases (typically installed to a customer), the source branch value can be the value set when a branch is created. Just look at the first row and disregard the other rows, they are kept so that you have a little bit history of what happened to that database schema (it can be helpful in case you run a script intended for another schema by mistake).

Once the three parameters are retrieved from the database schema, it is possible to ask for a database synchronization with the form Synchronize database schema. The field 'From:' should be filled with the value found in TBSYNCHRONIZE.BRANCHNAME. The field 'To:' (the so called target branch) should be decided as follows: choose HEAD if you would like to update a development schema (= 'From: HEAD') or if you would like to turn a production schema (= 'From:'= name of a created branch) into a development schema. Choose defined branch name for the 'To:' field, if you would like to update a production schema with scripts sent to the production branch, or if you would like to update an older development schema to a newer production branch.

Synchronize database schema will return a page with the missing SQL scripts for that given database schema. Choose 'Edit->Select All' from your browser and then 'Edit->Copy'. Paste the script into your favourite SQL client, which is connected to the database schema and execute it.

The script from Synchronize database schema will create at the end an UPDATE statement, which will update the information in TBSYNCHRONIZE for the next time.

Please note, that if something goes wrong, deltasql server reports an error according to the following list.

This is deltasql in short. Have fun!

How to define projects and modules

Modules are collections of scripts and projects are collections of modules. On the main page of deltasql, you can create a module and a project. Linking modules to projects is done on the List Projects page by using the Add module link.

At best you should model your projects and modules to follow your source code structure. If your source code is simple, you can start with one project containing one module. It is not enough to define the deltasql module alone, as the synchronization works only at project level.

Deltasql is known to work best if you use it in this configuration: assume your company sells software A to customer X and Y and software B to customer Z.

You should define three projects called Project_X, Project_Y and Project_Z as in the table below:

Project_X	Project_Y	Project_Z
`module_utils`	`module_utils`	`module_utils`
`module_A`	`module_A`	`module_B`
`module_Cust_X`	`module_Cust_Y`	`module_Cust_Z`

module_utils contains scripts which are of support of all applications. Typically, scripts which relate to tables for user management, groups and roles belong here.

module_A contains scripts which belong to software A. All scripts which belong to the core of sotware A belong here.

module_B contains scripts which belong to software B. All scripts which belong to the core of sotware B belong here.

module_Cust_X contains scripts which belong to software A, and are of use only for customer X. This is a customization module.

module_Cust_Y contains scripts which belong to software A, and are of use only for customer Y. This is a customization module.

module_Cust_Z contains scripts which belong to software B, and are of use only for customer Z. This is a customization module.

This is just an example on how deltasql works in some companies where their source code is modeled as in the table above.

Other ways to define a project might be to separate scripts which alter database structure and scripts which change content in tables.

Additionally, it is possible to define modules without linking them to a project, so that they can be used as collections for maintenance tasks.

On projects and modules, there is also a FAQ.

When should one branch?

Branching is something difficult and powerful. In the beginning, do not branch any project. Just submit all scripts to HEAD and learn how deltasql synchronizes in this scenario.

Later on, you can branch a project in the List Projects page, if you click on the Branch link (visible only if your user has at least Project manager permissions).

In general, you should branch a project at the same time when the source code is branched. You can then submit scripts both to HEAD and to the branch you created, only to HEAD or only to the new branch, according to how the source code HEAD and branch were modified.

If you master deltasql, you will then be able to transform a developer schema into a production schema or to dump an existing production schema and upgrade it to the latest HEAD for further development.

This movie explains how branching works.

When should one tag?

Tagging is easier than branching. Simply create a new tag in the List Branches and Tags page, when the software you are developing is packed into a new release. Just be careful to tag the correct branch: If the release is for development (or if you are not using branches at all) simply tag HEAD. If the release is on a production schema, you must tag the branch which is followed by the production schema.

This movie explains how tagging works.

How synchronization works

Synchronization can get difficult if branches are involved. To warm up, imagine the following scenario:

Scenario 1: all scripts are submitted to HEAD

   Submit script 1 to HEAD to deltasql server
   Submit script 2 to HEAD to deltasql server
   Submit script 3 to HEAD to deltasql server
   Submit script 4 to HEAD to deltasql server
   Submit script 5 to HEAD to deltasql server
   Submit script 6 to HEAD to deltasql server

Example 1.1

Imagine we have an old database schema used for development, if we launch the query to retrieve the database version, it returns 3 for the version, and HEAD for branchname. The fact that the field branchname is set to HEAD indicates that the old database is a development database.

Please make sure you know the difference between a production and development database in deltasql before continuing to read.

If we ask deltasql to synchronize with the form where we enter version=3 and branchname=HEAD, deltasql will generate a synchronization script as follows:

   Synchronization scripts contains scripts 4, 5, 6 and updates version of old database schema to 6

This example is very easy to understand. If your project is simple and if you do not need to switch between development and production environments, then you do not need to branch.

Scenario 2: scripts are submitted to HEAD and/or BRANCH_1

Here, we have a more complex scenario. A branch is created, and some scripts belong to the branch, or to the branch and the HEAD

   Submit script 1 to HEAD
   Submit script 2 to HEAD
   Create new branch BRANCH_1
   Submit script 3 to HEAD
   Submit script 4 to BRANCH_1
   Submit script 5 to HEAD and BRANCH_1
   Submit script 6 to HEAD

Example 2.1

   Version in old database schema: 1 
   Branch name in old database schema: HEAD 
   Update to: HEAD in synchronization form

This example is very similar to example 1.1. The old database is still a development database. Only the branch complicates it a little bit.

   Synchronization scripts contains scripts 2, 3, 5, 6 and updates version of old database schema to 6

Script 4 is missing because it belongs to the branch only.

Example 2.2

   Version in old database schema: 2 
   Branch name in old database schema: BRANCH_1 
   Update to: BRANCH_1 in synchronization form

This time, the old database is a production database, as branchname is set to BRANCH_1. We just would like deltasql to update the database and to keep it in production. Therefore, we say in the synchronization form to update to BRANCH_1.

   Synchronization scripts contains scripts 4, 5 and updates version of old database schema to 6

The synchronization mechanism considered only scripts which were flagged with BRANCH_1.

Example 2.3

   Version in old database schema: 2 
   Branch name in old database schema: BRANCH_1 
   Update to: HEAD in synchronization form

The old database is a production database. We would like deltasql to transform it into a development database. Therefore we set Update to: HEAD into the synchronization form.

    Synchronization scripts contains scripts 3, 4, 5, 6 and updates version of old database schema to 6
	Additionally the synchronization script updates branchname to HEAD

The synchronization script executed all missing scripts to create a development schema and also changed the definition of the database from BRANCH_1 to HEAD, making it effectively a development schema.

This movie is an introduction tutorial on how synchronization works.

deltasql Concepts

Developers submit scripts which belong to a module. A project consists of one or several modules. A database schema is always related to a single project. To each script a unique version number is assigned. If a database shows a given version in TBSYNCHRONIZE, deltasql can compute which scripts are necessary to reach another version.

Normally, database schemas and in particular development ones, need the entire chain of module-related scripts to be updated. This chain, as in the everywhere known Control Version System (cvs), is called HEAD. However, deltasql allows to create and name branches for a project. Once a branch is created, developers can choose if a particular scripts belongs only to HEAD, or if it belongs also to the newly created branch. Internally, deltasql simply stores the version number when the branch was created.

When updating with the form, one has to choose the target branch. The target branch can be HEAD, if all scripts need to be executed, and therefore if the database schema has to get a development one. Specifying a new target branch which is not HEAD will update the database schema to the version number assigned to this branch (by inserting a new row in TBSYNCHRONIZE with the version number), no matter if the source branch is the same branch, or if it is a previous one. In principle, it is also possible to update a long forgotten development schema (source branch = HEAD), to another target branch.

Tips

Define script's collections for maintenance tasks

With deltasql it is also possible to keep a collection of scripts which are occasionally used for maintenance tasks. It is sufficient to create a database module with a name like collection of my scripts without adding the module to any project.

Modules which are not added to a project do not partecipate in the synchronization process. So you can submit maintenace scripts to this module without fear that they will appear in the synchronization sequence. Typically, you will change the default title db update to what the script effectively does. deltasql underlines titles which are not default and allows to search also in the title field of a script.

By using the function Search among scripts, you can then specifically select the special module to search among the maintenance SQL scripts.

You can see how your scripts improve over time in the Show Script page, if you click on the History button. You will be presented with a history of revisions. You can see how old revisions looked like or diff them to see what changed between them.

Filter script - output particular synchronization scripts

When listing scripts (in List Scripts), it is possible to filter the scripts (e.g. by date, module, author and more) by using the command Search among scripts on top of the page. Normally, the scripts are filtered and shown as a list. By checking the checkbox Output as text list before clicking on Submit, the scripts will be shown as a normal text file that can be copy pasted into your SQL client of choice. This can be sometimes useful, if you need to repair a database schema by applying some subset of the recorded scripts.

There are also users that like this feature as substitution to the database synchronization form.

Advanced Topics

Insights into the deltasql Algorithm

Picture: Deltasql timeline with source and target

The answer to this FAQ question describes in general the steps performed by the deltasql algorithm.

The main synchronization logic is contained in the file dbsync_update.inc.php. If you want to gain complete understanding of what deltasql does, you could study this file.
The first algorithm step is to traverse back from target to source the pictured tree of development, and to record each segment of the traversal in a table called TBSCRIPTGENERATION. In case there is no direct backward path from target to source, Deltasql summons error 13.

If you want to see the content of the table TBSCRIPTGENERATION, you can enable the checkbox 'Include debug information' in the Synchronization form.
With the information in TBSCRIPTGENERATION, deltasql walks now from source to target, and in each segment of its walk, it issues the standard query explained in the next point.
The main query that drives the collection of scripts on a tree segment is: (we call it the "standard query")
```
      SELECT DISTINCT s.* from tbscript s, tbscriptbranch sb where 
       (s.versionnr>$fromversionnr) and (s.versionnr<=$toversionnr) and 
       (s.module_id in 
              (select module_id from tbmoduleproject where project_id=$projectid) 
               and (s.id=sb.script_id) and (sb.branch_id =$tobranchid)))
```
The query shows that the table tbscriptbranch contains a relation between a given script (stored in tbscript) and the branches (stored in tbbranch) to which it was submitted. tbbranch contains a particular row named HEAD, too. Therefore also the HEAD is represented as a row in tbbranch and can be considered logically as a branch.

$fromversionnr is the value as it is retrieved in the table TBSYNCHRONIZE, and represents the current synchronization point of the database schema. $toversionnr is the version number of the latest script on a tree segment.

The queries issued by deltasql to retrieve scripts can be inspectioned as well by enabling the checkbox 'Include debug information' in the Synchronization form.
The generated scripts for each segment are collated together in what becomes the synchronization script.
Upgrades from production schemas (following branches) to development schemas (following HEAD) are handled by dividing the algorithm in two steps with the division set to the current schema version number: a tree traversal back to the root of the entire tree on which a modified query is launched (can be seen by enabling 'Include debug information' in the Synchronization form), so that scripts belonging to HEAD but not to the branch are retrieved for the backward part, and a HEAD to HEAD upgrade for the forward part. The two steps can be seen looking at the table TBSCRIPTGENERATION included in the debug information. exclbranch is set to 1 in the backward part, and set to 0 in the forward part. The value in exclbranch drives deltasql to launch the standard query if set to 0, and the modified query if set to 1.

If you install deltasql with the option Test System with Test data set, you can verify the deltasql correctness with this file.

How to write your own deltasql client

deltasql Server is written in PHP and can run whenever Apache runs. You might want to write your own deltasql client that integrates better into your own development environment. There are already several client available. They can be downloaded at the bottom of the main page.

A deltasql client should perform the following operations:

0. Retrieve the valid project and branches currently defined on the deltasql server.
To perform this step, the deltasql client will touch the following two scripts: dbsync_list_projects.php and dbsync_list_branches.php. The format of these files is a comma separated text value. The header line describes the retrieved columns. Advanced deltasql clients can adjust dynamically the branches based on the project name, as it is done with the freepascal deltaclient.
```
Examples:
http://deltasql.sourceforge.net/deltasql/dbsync_list_projects.php
http://deltasql.sourceforge.net/deltasql/dbsync_list_branches.php
```
1. Connect to the database schema and retrieve project name, version and branch

This can be simply done by executing the query SELECT * FROM TBSYNCHRONIZE ORDER BY VERSIONNR DESC. The query is so simple as the table has one row per update which was already done. The latest database update has the highest version number. The important columns in this row are PROJECTNAME, VERSIONNR and BRANCHNAME. As you can imagine, PROJECTNAME is the name of the project of this database schema and VERSIONNR is the number of the latest script executed in this database schema. BRANCHNAME tells if the database schema is a development schema (if the value is HEAD), or if it is a production branch. If it is a production branch, the value will be the name of the branch.

If you prefer a query that returns only the row of interest, you might use this one:

 select * from TBSYNCHRONIZE where versionnr = (select max (versionnr) from TBSYNCHRONIZE);

2. Check on deltasql Server if there is a newer version for this project

The client should touch the URL below and store the resulting page in a file called projectversion.properties. The parameter project in the URL is the value PROJECTNAME retrieved in step one.

 /dbsync_automated_currentversion.php?project=TestProject

In the projectversion.properties file, there will be a line project.version= which contains the current latest version on deltasql server for this project. The client should now compare the latest version on deltasql and the VERSIONNR value in the database schema and decide if it is necessary to continue with step 3. If not, the user should be informed that there is no new version, that the datamodel did not change since last time.

3. Retrieve Script from deltasql Server

The client should now ask the deltasql server for the script to be executed in the database schema. It should touch the URL below and store the result as text in a file script.sql, for example.

 /dbsync_automated_update.php?project=TestProject&version=22&frombranch=TST1&tobranch=HEAD

project is the PROJECTNAME column retrieved in step one and version is the column VERSIONNR. frombranch is the value stored in BRANCHNAME. tobranch should be HEAD if you want to upgrade a schema (branch or development) to the latest development schema. It should be set to the value contained in BRANCHNAME if the production schema is updated without becoming a development schema. If the parameters frombranch or tobranch are not specified, they default to HEAD.

The parameters described in the previous paragraph are the most important ones, but there are more described in the next paragraph, to allow customization and even XML export. If something goes wrong, deltasql server reports an error according to the following list.

4. Visualize or execute script in database schema

Depending if your client includes drivers that are capable of executing complex scripts like packages and stored procedures (JDBC drivers are not), you can either directly execute script.sql in the database schema or show the script to the user and ask him/her to execute it.

The script always contains an UPDATE statement, that sets the new version and branch into the table queried in step one.

Client URL parameters

These are all possible parameters which can be passed by a client to the dbsync_automated_update.php script. String values need to be URL encoded.

project= the Project name as retrieved from the colum PROJECTNAME in TBSYNCHRONIZE
version= the version number as retrieved from the colum VERSIONNR in TBSYNCHRONIZE
frombranch= the current branch as retrieved from the colum BRANCHNAME in TBSYNCHRONIZE
tobranch= the target branch that will be reached after applying the synchronization script
user= the user name that requested the synchronization script
client= the name of the deltasql client that requests the synchronization script
htmlformatted= set to 1 to get a pretty printed HTML version of the synchronization script (usually it is a text file without any particular formatting)
xmlformatted= set to 1 to get an XML version of the synchronization script, see below for the schema definition
dbtype= the type of the database, it should be one of the constants defined in utils/constants.inc.php

XML examples as retrieved from clients

deltasql Server answers with two XML types, either an Error message or a synchronization script, if the option xmlformatted=1 is set. Click on the links below to see examples

Synchronization Script XML example ...
Error Message XML example...

List of error codes for deltasql server

The entire list is here.

Directory structure of the deltasql_1.x.y package

The structure itself is described in this text file.

Source code walkthrogh

While most of the .php files deal with creating, inserting, updating or deleting data, there are some units which lie outside this schema and that are worth a look:

synchronization_table.php: contains the preparation script for any database schema which will be put under deltasql version control
dbsync_update.inc.php: contains the logic of retrieving scripts for the synchronization step. The heart of deltasql is here. Everything is built around this unit.
dbsync.php: synchronization via Form.
dbsync_automated_update.php: synchronization for deltasql clients.
utils/verification_scripts.inc.php: calls to the verification script for the different database types.

How to compile deltaclient on Linux, Windows or MacOSX

To compile deltaclient, you need a working copy of Lazarus, an IDE for fast software prototyping in Pascal, installed on your operating system. You can download the lazarus package from the Lazarus Homepage. Lazarus is packaged to run on several architectures, including Linux, Windows and MacOSX. If you run Linux, you can also try to install it via your packet manager, e.g. apt-get install lazarus on Debian, emerge -av lazarus on Gentoo, yum install lazarus on Red Hat, etc.

Once Lazarus is installed, launch it. Choose File -> Open from the menu, navigate to the directory clients/freepascal/deltaclient and open the file deltaclient.lpr. Once you opened the file, simply click on the green play button. This will compile deltaclient on your favorite operating system. You will find the deltaclient executable in the clients/freepascal/bin subfolder.

Picture: Press the green play button to get deltaclient compiled on your favourite operating system!

If you should run into trouble, please contact us on the forum.

Feedback on this document

If you need help in setting up your own deltasql server, you can contact us at the deltasql mailing list. We appreciate your feedback.

Have fun!

Back to Introduction

deltasql - Database Evolution under Control

Main

Resources

Follow us

Manual for deltasql 1.8.2

Summary

Database Evolution Under Control

(how deltasql works)