SQL-File Technology for Transact-SQL

This article introduces so-called statically parameterized SQL language, proposes use of enhanced SQL-code residing in specially organized project of SQL-files, that are located in corresponding configured tree of files/subfolders. Connected SQL-files (static scripts) are stored together with simple definition files of values named as SQL-settings, containing parameters for the static SQL. (It’s a short formulation.)

Index:

TO INTRODUCTION (SCROLL TO)
I. What is SQL-file technology (approximate explanation)
II. Simple script (Hello) sample at Handicraft-CODE pages
III. What is a gain from SQL-file use?
IV. CMD is well suitable for lightweight superstructure with user commands and settings.
V. Multiplatform settings represent powerful approach for programming in two or more languages, if you are able to access configured values (settings) with help of language preprocessor or in a task of source code generation.
VI. SQL-file technology at Handicraft-CODE web-pages
VII. My work, universal constructor named SQL-file
VIII. SQL-Query-Bridge Outline (abstract idea, imaginary)
TO [END OF ARTICLE CONTENT] (SCROLL TO BLOG AUTHOR)

Introduction

My realized idea was to pick up a set of tools as an alternative IDE, which is able for operation with parameterized SQL in files: command panel with text editor, file tree, commands etc. Along with this — to develop additional building utilities, in order to support in a certain degree an alternative approach for database interaction from perspective of constructing and/or programming (from developer’s or administrator’s machine, client of SQL Server), that is based primarily on some editor of source text in files (as the central location of code sources). One of the way for such approach is to use old-styled traditional file commander, which contains simple but universal text editor, with possibility to compose command line scripts (not only sole SQL) for different cases where complex SQL-processing is needed. Also as one of intentions of SQL-file creation, writing of some auxiliary SQL-code was made, suite of helpers in form of a library (client side definitions, plus auxiliary routines for server side).

My small invention is called SQL-file technology (for MSSQL and T-SQL language). In spite of a modest implementation of the SQL-file, I think that the idea has certain value and sense.

fill_with_data.jpg

I. What is SQL-file technology (approximate explanation)

There is an official utility from Microsoft, SQLCMD. It is what SQL-file technology is similar to, at first view. Indeed it is strongly based on this powerful and helpful file oriented utility. At time of SQL-file creation (from its early versions) it was not reasonable to organize a fully separated really autonomous script processor (at client side), whereas it’s desirable to have a good integration and compatibility with other MSSQL features in your approach. SQLCMD is a base file translator (dev.time client utility for the SQL Server), but by no means it’s a complete program environment, it doesn’t suggest something like file project integration etc. It does support environment variables (properties) as file script parameters (SQLCMD insertions to SQL-code like $(<VariableName>)), but there is no effective and unified approach where we can configure a complex collection of properties like we can do it for instance in “.props”-file(s) (XML, MSBuild). From the other side SQL Server Management Studio (which is the default IDE) is not also so well for advanced working (programming) with files. Indeed this graphical console does not really (completely) work with files, but instead performs SQL-script corrections in the database (which is the primary script location for this dev.tool). So scripts in files are supported rather as illusion in such GUI (DB-console), there is no advanced processing of file input in this tool. A lack in special client tools exists for working (programming) in DB with files, as primary storage of SQL scripts. SQL-file technology is a modest attempt to create (organize) corresponding client constituent (tools/utilities) as additional, at least experimental, suite for DBMS, for administrator and/or developer, that is specially oriented to operate with SQL source code in files (as primary location of SQL-scripts).

Numerous technical details are included into this article: file names, names of variables, micro-fragments of configurations etc. It doesn’t mean all such things have to be clear for the reader in one moment. The details are relatively understandable though, their purpose here is to meet you with a spirit of applied techniques and to introduce partially some approaches. For normal acquaintance, a set of references exist, web-links to corresponding resource, containing a lot of named screenshots and minimal textual information. (Article perception depends on applied images from that pages.)

SQL-files are applied (translated) to database so to speak from a location of source code in file project (in grouping folder): (1) In the editor; (2) By activation of single file from list (file items) in UI (file commander or from another panel); (3) From specially organized command line script (e.g. complex translation in several steps). File groups may be processed (translated to database) in one step (as a single operation): (1) One chosen SQL-file (plus some additional special files); (2) Simple group of scripts, such as “*.sql” or “<asterisk_pattern>.sql” (with possibility to specify some item names in “$sql_order.list”, to correct the order of file sequence for the processing); (3) Subdirectories by name pattern (all at once), plus local SQL-files (like “*.sql”), that is a big group (for single translation, is performed in one connection). Each script in SQL-file project is located properly in the tree, correct location of source file is important here. See pictures ($SQLTRANS examples): Table structure creation ($SqlTrans, SQLCMD), Data filling ($SqlTrans, SQLCMD), SQLAUX-functions in one go, App.procedures translation ($SqlTrans, SQLCMD), Incorrect syntax at line (SQL-file screenshots at Handicraft-CODE pages; a little of Cyrillic on two last images is due to localization of the author’s machine, nothing important is in these messages, these are simply confirmation and ending pauses for SQL-translation command).

The main user interaction tool in SQL-file program environment is Far Manager 3 (widely known and powerful file commander), it is a rich console-mode application, and this is here the primary IDE (prompt editor and command console). Some other tools are possible to use with SQL-file: (1) Windows Explorer (as scarce command panel); (2) SSMS official IDE (with some restrictions and constraints however); (3) Autonomous external source text editor (GUI).

If a part of application code is inside database: SP, functions, views etc., it is reasonable to store such sources of “programmatics” (active DB-objects with SQL-code) in corresponding SQL-file project. Stored procedures and functions should have proper name prefix, in order to be able effectively destroy entire DB constituent of the application (it’s corresponding subsystem in DB). Configuring file tree of our scripts, thinking carefully about dependencies of the programmatics, determining their translation order, we hereby prepare further recreation of app.component(s) in DB (entirely, partially or by single object) to be an easy task. Usually target program code (at server side) of correct programmatics (files) is generated quickly in SQL-DB (at one, two, three, four, five, … not so long), such translation(s) are made often in time of development. It is also a normal thing to correct some objects (through file editing + translation) on working application or service (at some pause interval), even not interrupting its connection to SQL. (Complex translation of SQLAUX scripting library, for instance, include translation of 173 objects in DB, with their primary destruction, and it normally takes not more than 10 seconds for a local database, processed by SQL Server installed at the same machine.)

SQL-FILE AND HANDICRAFT-SDK:

SQL-File Technology is a general notion. In abstratc it may be considered as some approach for organization of support for Enhanced T-SQL Language in Files, or even for some another SQL-dialect in files (for corresponding DBMS). (In the last part of the artice, modified approach is mentioned, SQL-File App., that is a full abstraction.) However in this article I talk mainly about my concrete implementation for MSSQL, with Far Manager as primary IDE (FAR and SSMS), and I use a special name connected with this: Handicraft-SDK. This small SDK is provided by the article’s author. For SQL-file, it represents a set of command-line utilities, in “CMD-utilities” folder, SQLAUX scripting library (in “SQL\SQLAUX” subfolder) and some other interesting things (not only support for SQL-file). So SQL-File Technology for Transact-SQL is based on the Handicraft-SDK, this key dependency makes Enhanced T-SQL in Files runnable. The other names related to Handicraft-SDK, which are used in this article are: “Handicraft Toolkit” (full code package) and “Handicraft-CODE”, they mean things that are resembling to the SDK, but imply some additionals (like SDK + advanced samples etc.).

SQL-file technology, $SQLTRANS-utility, SQLAUX scripting lib.

SQL-file technology suggests some components for advanced programming in MSSQL database (on file source-code-base):

  1. $SQLTRANS utility (see: $SqlTrans command, $SQLTRANS syntax print), which is inalienable part of SQL-file, it is based on SQLCMD (official command line utility, that is equipped with preprocessor from MS), and also on SqlCatPP.exe (this subprogram, as main function, concatenates source SQL files for the translation; also it has some functions of simple preprocessor); $SqlTrans command is realized in $sqltrans.cmd utility script (“CMD-utilities” folder of Handicraft-SDK);
  2. SQLAUX library settings for preprocessor (helpful keywords and constants for so-called Enhanced T-SQL), are accessed as $(<VariableName>_AUX), i.e. macro helpers, see: #SQL_extensions.AUX.sql, #Constants.AUX.sql, #Data_types.AUX.sql (SQLAUX API – headers files, in code view);
  3. SQLAUX library objects in DB (functions and stored procedures: [AUX:<ObjectName>], [AUX::<ObjectName>], database helpers), see: SQLAUX programmatics and tables (listing);
  4. Collection of templates and samples (SQL, CMD, TXT, $sql_order.list), this is needed to derive new SQL-file project (configured folder with connected files).

WARNING!
SQL-file was tested under the following system locales (default code pages for single-byte apps):
1. English (USA), i.e. CP-1252 (Latin alphabet) / OEM: CP437;
2. Russian (Russia), i.e. CP-1251 (Cyrillic script) / OEM: CP866.
For the system languages (default locales) SQL-translation date-time stamp may be formed with some mistakes, probably in its date part. I’m sorry! (Sometimes it may be non-critical because time order inside the day is then preserved.) For locales different from the two above, following environment variables may be helpful: YYYY_START_IN_DATE (optional), MM_START_IN_DATE, DD_START_IN_DATE. Zero-based positions have to be specified through the variables, according to corresponding user locale, concerning the DATE environment variable (CMD). Also they are needed to account any nonstandard settings of date-time format in your system, concerning the DATE variable. (This problem is connected to limitations of legacy CMD-processor, which is used for implementation of $SQLTRANS command.)

All my utilities (EXE) are supported with corresponding source code, in order to be able (as for a user of the SQL-file) to rebuild them. Small projects (C#, C++) are located under “HandicraftSDK\Auxiliary programs (sources)” folder, in SDK download package(s). The sources are provided with multiple build commands (as CMD-files in project root) with corresponding operation names. So it’s not too difficult to adjust such compilation if some corrections will be needed. (Also official build tools are required then.)

Folders in Handicraft-SDK that are essential for SQL-file technology

HandicraftSDK” — root folder

MAIN FOLDERS UNDER THE ROOT (SQL-FILE FOUNDATION DETAILS):

Auxiliary programs (sources)” — little projects of utilities (C#, C++) >>

  1. Command line” — C# projects for command line utilities located in “CMD-utilities” folder;
  2. Far plugins” — tiny CPP-project of CmdCpAuto.dll (Far Manager plugin in the “Utilities” folder).

CMD-utilities” — $-commands (command-line): $sqltrans.cmd, $sqlupload.cmd, $sqltocsvexp.cmd, … >>

  1. Assemblies” — binary subprograms for $-commands: SqlCatPP.exe, SqlToCsvExport.exe, ConfirmationPause.exe, TypeInColor.exe, … ;
  2. Shell” — auxiliary CMD (+instructions), for integration with Far Manager etc.

SQL” — SQLAUX library + samples (SQL-file) >>

  1. Programming samples (SQL-file)” — six simple samples in Enhanced T-SQL (SQL-projects – demos), are provided with corresponding “!ReadMe.txt”;
  2. SQLAUX” — scripting library (SQL, CMD) that is usually imported and consumed by typical SQL-file project;
  3. “Extralight-ORM test legacy sample (En+Ru)\MyFlat app. (MVC_Razor+TS+ADO+TSQL)” — advanced bilingual sample (EN/RU) with SQL-file use: ASP queries are in English (C#), and SQL-files are in Russian (SP in Enhanced T-SQL, with Cyrillic, contain English comments); (SQL-project is located in “SQL (DB-modelling)” subfolder); this is additional, advanced sample.

Utilities” — key utilities and miscellaneous auxiliaries >>

  1. “Far plugins\CmdCpAuto plugin, build 100 (Far v3)” — CmdCpAuto, Far Manager plugin for automatic code-page selection in the embedded Far Editor, when CMD/BAT file is being opened for editing (normally by F4-key).

(For deeper acquaintance with the SDK explore “HandicraftSDK\**\*” file/folder-tree.)

SQL-translation ($SQLTRANS-command organization)

SQL-translation ($SQLTRANS command) is a compound operation. Its main phases (internal steps) are listed below, but this is a very brief explanation. For deeper understanding in details it is recommended to look into CMD script-code of the utility (“HandicraftSDK\CMD-utilities\$sqltrans.cmd”, inside Handicraft-SDK).

Stages of SQL-translation (internal steps):
  1. $sqltrans.cmd receives SQL file names (or pattern) as parameter(s);
  2. $sql_settings.cmd is called (if exists) from current translation folder, for the sake of environment preparation;
  3. SqlCatPP.exe is invoked in order to form concatenated and lightly preprocessed SQL-source(s), resulting SQL source is placed into “%TEMP%\$SQLTRANS\.sql”, in user profile temp. file store; date and time stamp is used for temporary file naming;
  4. SQLCMD.EXE (SQL Server command-line utility) is invoked by $sqltrans.cmd, to execute earlier prepared SQL-source from %TEMP% folder, SQL output TXT is places near into the same folder with same base name (as for the temporary SQL-script), or into the current folder (if output report filename is specified for “$sqltrans.cmd” call).

Also $SQLTRANS may only prepare settings environment (SQL-settings variables generation), being called as:

  • call $SQLTRANS “(SettingsOnly)” (in user defined command of complex SQL-translation);
  • call $SQLTRANS “[SettingsOnly]” (invocation from “SSMS.cmd”, IDE starter).

In addition to user defined commands, in settings-only mode it is used in SSMS.cmd, that is advanced starter of the official IDE (SQL Server Management Studio), so it is launched with a set of prepared environment variables, from the SQL-project (for parameterized SQL). Environment preparation is initiated by $sql_settings.cmd (from the same folder as for SSMS.cmd), which in it’s turn usually invokes @<project_name>.cmd (SQL-project settings in UTF-8 encoding, without BOM).

CMD-processor’s role

CMD-processor (Command Prompt, legacy command-line interpreter in MS Windows, it is joined with Windows Console) plays an important role in this “handicraft” implementation of SQL-file. The batches are well adopted for SQL-file (they are equipped with abundant set of auxiliary subroutines). This handicraft IDE is native in CMD and console. Specially applied command line format acts as a glue between SQL-settings (environmet variables), SQL-scripts and command scripts (SQL-translation scenarious). This is not a modern way, but it works stably. Another known command line processors may not occur so much successive in this mission. Simple operations in command scripts must look shortly and simply and must not look bulky and complicated; this is strictly required because of necessity in lot of user defined commands, managed plain files of short and medium size. (To make user experience more attractive, invention of a special script/configuration format will be necessary, for SQL-project and SQL-commands construction, with desirable support of general scripting possibilities, like JS-engine out of browser, to process command instructions in imperative style.)

Resuming the above, SQL-file technology is an approach to organize SQL-file project, in order to code inside database in so-called Enhanced T-SQL, with SQL-settings of different levels (SQLAUX macro keywords; SQL-project central settings; modified SQL-settings of file subtree), with possibility to invoke some additional functions and procedures of universal character (general helpers in DB). This technology implementation (for MSSQL) is targeted for time of development and/or administration. This means, that your project environment is not directly for user application. Of cause the application may access all the objects in the database (SP, functions, views, tables, triggers) that are created with help of the SQL-file. But application queries with embedding of SQL-settings (via SQL-preprocessor, in Enhanced T-SQL) are not possible in this implementation (such possibility was not realized in my Handicraft-CODE).

$SQLTRANS utility is the a key subprogram in the SQL-file. It is written in CMD and acts as intermediary between SqlCatPP.exe (internal subprogram with function of SQL-scripts concatenator and also of additional preprocessor) and SQLCMD.EXE (powerful command line utility from MS), which interacts with SQL Server. The simplest way to translate SQL-file (execute its content on SQL Server) is following command: $sqltrans "<Base name>.sql", or $sqltrans "<Base name>.sql" "<Base name>.txt" (to save SQL execution report into corresponding TXT). There are many variants of invocation. ($sqltrans-command without parameters prints its syntax, shows some hints about how to call it with files.)

Preprocessor function in SqlCatPP

There is own simple preprocessor in SqlCatPP subprogram (with two different modes), besides functionality of the SQLCMD (that is the official PP from MS). So-called strong preprocessing mode exists in SQL-file, supported by SqlCatPP.exe (in addition to so-called preprocessor weak mode, which in its turn is used in order to eliminate incorrectness of error diagnostics connected with ingenious removal of original source line with :SetVar directive, by the SQLCMD). Strong preprocessing mode is intended for the sake of possibility to reconstruct preprocessing result (preprocessed and concatenated scripts) in order to transfer (e.g. send by the e-mail) such target script to so-called database dealer. DB-dealer (human who operates in DB) applies SQL-scripts received from the developer, executes them on the target DB. He or she is not able to prepare developer’s environments (like SQL-project does at developer’s machine). All he or she is needed is to have already prepared target SQL-script (as result of strong preprocessing under influence of SQL-project environment at computer of the developer). For instance, SQLAUX library itself is exported to the outside through “sqlaux_library.sql” file, compound script from “HandicraftSDK\SQL\SQLAUX\Source\$OUTPUT” folder, 173 DB-objects are destroyed-created in it (it can be executed from the SSMS, with default settings). However, strong preprocessing mode should be used with care (only if it’s necessary). It has not been tested enough and it may pose some risk. (As a matter of fact it’s hard to achieve exactly same behavior in SqlCatPP as MS-preprocessor has in SQLCMD, regarding interpreting of comments, literals, PP-directives etc.)

Dual character of CMD-batches organization for SQL-commands

Most part of SQL-commands (CMD-scripts) supports so-called dual invocation. In case of normal call the console is not usually closed at the end until you press correspondent key(s). Also at the beginning, request of confirmation to proceed the command there may be appeared. If the command fails (some error is encountered), it normally prints error message (for SQL-translation failure it is colored in red) and plays bell-signal. This is a normal behaviour for the error. In case of so-called nested invocation (it is applied in complex batch for invocation like following: call $SQLTRANS.cmd … or some other stock or user defined commands), in such a call the above mentioned measures are avoided: no confirmation, no completion pause, no bell-signal in case of error. If you click on a command (properly named command file) in Windows Explorer, then standalone console appears, you are able to read the output (text and messages printed to the console) when the command succeeds or fails. This is due to special measures inherent to normal call. In case of the nested call root command is probably invoked in normal way (so it’s all right with confirmation, pause, messages etc.). Invocation duality is helpful therefore. Not only SQL-commands, C#-project building commands (containing MSBUILD / dotnet build etc.) and others, many CMD-batches are organized in dual invocation manner.

Such is a rough sketch of my production around SQL Server. Its essential constituent is original program environment at client-side, albeit it is rather an integration with its key component, official utilities: SQLCMD, BCP (key programs of MSSQL Command Line Utilities). The other part is T-SQL scripting library named SQLAUX, that is a server-side targeted component.

Console natural environment (i.e. pseudographical commander) and only some GUIs

SQL-file primary IDE is severe. Text editor in Far Manager does not provide you with function of code autocompletion. Though Intellisence for SQL is available in SSMS, if you run IDE-process via SSMS.cmd starter, command file in SQL-project root (folder). But restriction exists for SSMS+Environment. Variables from SQL-setting CMDs then are frozen for whole session of the IDE (Ssms.exe, OS process).

Instead, Far editor is prompt and native for programming in SQL-file. It is suitable for creation/editing of SQL script as well as for editing of CMD and many other file types. Code coloring is nice (dark theme in VS Code looks similarly). Vertical block selection exists in the editor, characters from OEM code page are displayed properly, including pseudographics, series of unicode encodings is supported well, etc. Far Manager is valuable because it is universal enough.

Anyway SQL-commands (project CMDs) are autonomous, such CMD may be launched in separated console window (via <Shift+Enter> in Far Manager or from folder of the Windows Explorer).

II. Simple script (Hello) sample at Handicraft-CODE pages

There is a modest web-resource named Handicraft-CODE. It is dedicated to so-called handicraft programming, especial autonomous manner/style in software development. SQL-file technology is based on Handicraft-SDK, this key dependency makes Enhanced T-SQL runnable. The SDK in its turn is one of two large-scale divisions in Handicraft-CODE public pages. At the sixth part of this article there is a list of useful Handicraft links connected to SQL-file technology. The central SQL-file page is: https://handicraft.remelias.ru/sdk/sql_file.html.
Lack of description texts have to be compensated by numerous screenshots (scripts, settings, configurations, execution output, result sets etc.).
Simple script (Hello) sample path in the SDK (folder):
“HandicraftSDK\SQL\Programming samples (SQL-file)\Simple script (Hello)”.

Simple script (Hello) sample is that from where screenshot pages begin (https://handicraft.remelias.ru/sdk/sql/screenshots_1.html, 4 items in bold at the beginning). In the download packages it is one of six simple examples, six subfolders of “Programming samples (SQL-file)” folder. The sample consists of 13 files of following types (extensions): SQL, CMD, TXT, and one of the files is without the extension: %simple_script%; it is so-called SQL microscript, is included by “:r”-directive as a head of “simple_script.sql”, which is here a main SQL-script. The files are dependent one from the other. So indeed this simple script is not trivial, several features are gathered in it. The first of three T-SQL batches, portions of instructions continuing to GO-separator (but not including it), do following (see below):

The first batch is in %simple_script% file, it prints report header as text message: *** SIMPLE SCRIPT (<date and time>): ***. The other two batches are in simple_script.sql.

The second batch produces five primitive result sets, they are:

  1. General greeting (message);
  2. Extended (additional) greeting, in Russian;
  3. $(str_UnicodeString)-property value (from SQL-project settings) that are letters from modern Greek alphabet (24 capitals);
  4. SQLAUX system variables (output of three names-values is performed): is_sqlaux_imported, sql_translation_timestamp_aux, sql_settings_script_aux;
  5. sql_translation_dir_aux variable view (file translation folder path, is ended with backslash).

The third batch prints output termination decorator: — — — (three m-dashes through the space character). That’s all (this is a simple script).

The pictures are on the first page with the screenshots (10 images under the first 4 captions-groups, SEE ALL 10 PICTURES).

Simple script sample screenshots (at 1st page with pictures):

  1. Simple script (Hello) / T-SQL in files (simple_script.sql and %simple_script% in SQL-project directory);
  2. Simple script settings (SQL-file project defs: $sql_settings.cmd, @simple_script.cmd);
  3. Simple script output (TXT-report in simple_script.txt, Console, SSMS);
  4. Simple script run (elementary simple_script.cmd and complex run_simple_script.cmd).

(Date in decorative header of SQL-output that is produced in the 1st batch, contains Cyrillic month name, because of default Russian language of connected user, for SQL Server author’s login.)

PURPOSES OF CONNECTED SAMPLE FILES (SQL-PROJECT):

  1. !ReadMe.txt — simple explanation of sample program intentions;
  2. $ide_root.sql — it is intended for SSMS.cmd to designate project root folder for the IDE;
  3. $sql_settings.cmd — predefined settings batch name, is activated by $SQLTRANS, it attaches SQLAUX import definitions and project definitions from @simple_script.cmd (conventional name);
  4. $sql_settings.sql.bak — alternative way for specification of SQL-settings (it is not compatible with SSMS, that is why it’s not recommended), $sql_settings.sql is inserted at the beginning of any SQL-translation (automatically by the $SQLTRANS), if it is not blocked by $sql_settings_script variable (hyphen value ‘-’ in the SQLAUX) or if it is referenced in $sql_settings.cmd (via $sql_settings_script variable);
  5. %simple_script% — this microscript is included in the first line of simple_script.sql (:r "%simple_script%"), its purpose is to print a caption of the report;
  6. @simple_script.cmd — project definitions, SQL-settings in the project root, imported by $sql_settings.cmd (some of them may be overridden in subfolders if they are), this is conventional name for SQL-settings non-trivial configuration in UTF-8 (text content is stored without signature, i.e. no BOM-prefix, but CmdCpAuto plugin for Far Manager “understands” the special filename);
  7. exec_no_confirm.cmd — as opposed to simple_script.cmd which is in a normal mode, this command (batch-file) avoids involving of user confirmation step (there is nothing unsafe in simple_script.sql), SQL-output goes to the console;
  8. exec_simple_script.cmd — normal execution with SQL-output (result print) to the console;
  9. run_simple_script.cmd — minimal example of so-called complex command (SQL-output goes to the console);
  10. simple_script.cmd — the most natural command form for saving SQL-ouput to corresponding TXT-report, with obligatory user confirmation (in general SQL-script may be dangerous for the database);
  11. simple_script.sql — main SQL-script producing compound report;
  12. simple_script.txt — TXT-report (SQL-output as result of simple_script.sql execution);
  13. SSMS.cmd — so-called IDE-starter, it prepares SQL-project environment (SQL-settings as environment variables) and launches the IDE process (Ssms.exe), $ide_root.sql is then opened in SQL Server Management Studio (as single SQL-file tab).

Strange control flow in Transact-SQL language (default mode for error checking)

In T-SQL language we have some strange control flow effective by default (here I imply code without exception catching/guard) that obliges us to check @@ERROR variable almost after each instruction, elsewhere there is no guarantee that execution is interrupted correctly (goes to a panic branch or terminates) if SQL-instruction fails. This is a legacy behaviour (it’s a default mode of control flow in T-SQL) in SQL Server, inherited from the very earliest versions (in that time while without exception handling support). This fact (unreliable or inconvenient control flow in T-SQL, in default mode) was a main reason, among other serious circumstances, due to SQL-file approach was invented, that then allowed (to the author) to use parameterized SQL files with special macro keywords (folded sequences of instructions for repeating use).

Such control flow in Transact-SQL that we do have there by default (with checks of @@ERROR variable after each potentially dangerous instruction) is suitable, for instance, for a code that is intended to execute somewhere under OS kernel-mode (like function driver in pure C-language for example). But it wasn’t a good idea to state this regime by default (i.e. in connection with code without specially organized exception capture) for a high-level language with transactions. May be it’s still looking normally for CMD scripts (to check errorlevel variable everywhere), but it’s definitely that such default regime is not happy for SQL(s).

Main block for exception capturing, script embracing macros: $(BEGIN_AUX), $(END_AUX)

Batch content in SQL-file is usually wrapped between $(BEGIN_AUX) and $(END_AUX) macros (SQLAUX library keywords). Only one such protection code block per T-SQL batch is permissible. Below you can see how main error protection is unfolded. If an error occurs inside a block, it is caught as exception to be thrown at the end of the catch section, after possible rollback of opened transaction (if there is) which was especially initiated (if it was) earlier by $(BEGIN_TRANSACTION_AUX) macro in the block.

SOURCE SCRIPT IN SQL-FILE:

$(BEGIN_AUX)
-- QUERY BODY:
: : : : :
$(END_AUX)
GO

SCRIPT FOR EXECUTION AT SQL SERVER:

begin begin try set xact_abort off; set ansi_nulls,ansi_null_dflt_on,quoted_identifier,arithabort,nocount on; set implicit_transactions off; declare @initial_tran_count_aux int, @is_local_tran_open_aux bit; select @initial_tran_count_aux=@@trancount, @is_local_tran_open_aux=0; end try begin catch throw; end catch; begin try
-- QUERY BODY:
: : : : :
if @is_local_tran_open_aux=1 raiserror ('Not closed local transaction was detected, previously opened by BEGIN_TRANSACTION_AUX macro.',11,1); end try begin catch if @is_local_tran_open_aux=1 and (@@trancount>@initial_tran_count_aux or xact_state()=-1) rollback transaction; set @is_local_tran_open_aux=0; throw; end catch; end

VERSION WITH LINES WRAPPING:

begin begin try set xact_abort off; set ansi_nulls,ansi_null_dflt_on,quoted_identifier,arithabort,nocount on; set implicit_transactions off; declare @initial_tran_count_aux int, @is_local_tran_open_aux bit; select @initial_tran_count_aux=@@trancount, @is_local_tran_open_aux=0; end try begin catch throw; end catch; begin try
— QUERY BODY:
: : : : :
if @is_local_tran_open_aux=1 raiserror (‘Not closed local transaction was detected, previously opened by BEGIN_TRANSACTION_AUX macro.’,11,1); end try begin catch if @is_local_tran_open_aux=1 and (@@trancount>@initial_tran_count_aux or xact_state()=-1) rollback transaction; set @is_local_tran_open_aux=0; throw; end catch; end

GO-separator is excluded by SQLCMD. $(BEGIN_AUX) and $(END_AUX) insertions are unfolded to the above instruction sequences by SQLCMD, or may be processed by SqlCatPP if strong preprocessing mode is switched on (set SqlCatPP.ProcessDirectives=1 in $sql_settings.cmd, but this possibility should be used with caution).

Also look at this SQL-code, stored procedure DB-object (shown in SSMS):
$(BEGIN_AUX) macro in DB (preprocessing result).

Other samples (plain and advanced)

Simple script (Hello) sample is one of six elementary examples provided with SQL-file.
This is a list of elementary samples from “HandicraftSDK\SQL\Programming samples (SQL-file)” folder (so-called Plain Samples):

  1. GUID-list generation” – generates a list of 50 GUIDs into “guid_list.txt” plain file;
  2. Lists downloading (sys-info)” – demonstrates how to use $SqlToCsvExp command;
  3. Nested transaction test” – amusing example with nested transactions (SP in files);
  4. Simple script (Hello)” – configured project of connected SQL-files (no subfolders);
  5. Trigger test” – simple experiment with triggers (in files), for table with odd numbers;
  6. Uploading to DB (used buildings)” – sample usage of $SqlUpload command.

Each elementary sample (its folder) is provided with corresponding minimal instructions in “!ReadMe.txt”. All sample projects (above mentioned and others) are listed in “SQL-projects-sum.txt”, in root folder of Handicraft Toolkit. As for the Simple script (Hello), this SQL-project does not have subfolders. For the majority of other samples, they have. SQL-settings in subfolder are inherited from the above level, by specially organized “$sql_settings.cmd” (it usually attaches settings from the upper directory level). Some new variables may be added, some existing values can be corrected. Even in named user commands (batches) we can specify something peculiar (related to the utilities). Usually utility behaviour is configured (in “$sql_settings.cmd”) by special variables like $sqltrans.*, SqlToCsvExport.* (base setting name is implied in place of the asterisk). For instance, in exec_simple_script.cmd: rem set $sqltrans.ContinueOnBatchError=1. (If you uncomment this line, all three GO-batches in the Simple Script Hello will probably work in spite of possible batch errors.)

The other two samples in SQL-file (advanced):

  1. “HandicraftSDK\SQL\Extralight-ORM test legacy sample (En+Ru)\MyFlat app. (MVC_Razor+TS+ADO+TSQL)\SQL (DB-modelling)” —
    application constituent in DB (MyFlat app. API in SP + one base SP), in Russian (with some comments in English);
  2. “BookRegistry\SQL (DB-modelling)”, “BookRegistry\SHARED\$ClientServer\!DB-queries (application SP)” —
    two connected SQL-projects for BookRegistry Sample App. (table structure, data filling, SP, C#/SQL side-by-side), this is an extra sample.

III. What is a gain from SQL-file use?

SQL-file technology can be applied with your scripts with maximal intensity or in some insignificant degree. You may only use file translation or other utilities (like $SqlUpload or $SqlToCsvExp commands). For instance, somebody may decide to depend on helpful macros from SQLAUX library (its unfoldable keywords), but it doesn’t mean you need to use its database helper objects (i.e. SQLAUX SP, DB-function etc.). If you, for instance, have a folder with SQL-files, it is enough to place $sql_settings.cmd in it to be able to translate them by $SQLTRANS, by one or all together (however processing of partial file group is available only through corresponding subfolder). You even may be able to translate your files without simple $sql_settings.cmd, if you define some system or user variables: $sql_server, $sql_database (or, if needed, $sql_user, $sql_password_var and correspondingly named variable as SQL-password container). Temporary variables specification is possible in Far Manager, in its command prompt or via user menu, for OS process of the commander. Thus, user CMD-batches are not strictly obligated in SQL-file.

Anyway (as for the above etc.) use of $sql_settings.cmd is recommended. The other very recommended thing is not to avoid SQLAUX library import, which is performed in $sql_settings.cmd (this file also imports @<project_name>.cmd, SQL-settings in UTF-8). (This is similarly to some important include-files in C-language or in RC-file, not for the sake of function headers but mainly in order to have required preprocessor-level defs.)

Far Manager keyboard shortcuts etc.

Far Manager keyboard shortcuts are listed below, for translation of single file with “.sql”-extension.

Keyboard shortcuts for SQL-translation from Far Manager panels (assigned in Main menu :: Commands :: File associations :: A file mask or several file masks: *.sql):

  • <Enter> — initiate SQL-translation in the commander window (i.e. output to parent console), with safety confirmation;
  • <Ctrl+PageDown> — initiate SQL-translation in standalone window (maximized console), without safety confirmation.

Keyboard shortcuts for SQL-translation in Far Manager embedded editor (activated by LUA-macros Editor_CtrlEnter.lua and Editor_CtrlF12.lua):

  • <Ctrl+Enter> — save SQL-file, initiate SQL-translation in the commander window (i.e. output to parent console), with safety confirmation;
  • <Ctrl+F12> — save SQL-file, initiate SQL-translation in standalone window (maximized console), without safety confirmation.

Far Manager keyboard shortcuts for SQL-command initiation, for execution of “<Command name>.cmd”:

  • <Enter> or <MouseLeft::DoubleClick> — execute CMD in parent console window of the commander;
  • <Shift+Enter> or <Shift + MouseLeft::DoubleClick> — execute CMD in standalone console window.

Windows Explorer folder may be opened from the Far Manager by <Shift+Enter> or <Shift + MouseLeft::DoubleClick>, on “..”-sign (the upper dir.).

Also Editor_CtrlR.lua macro is intended for possibility to quickly comment command line with “rem ” (without quotes), in the embedded editor.

SOME USEFUL PARAMETERS FOR THE EDITOR (FAR MANAGER “CONFMN”/F9 :: OPTIONS :: EDITOR SETTINGS):
  • Expand tabs: Expand all tabs to spaces;
  • 3Tab size (this is an optimal tab size recommended by Handicraft-CODE);
  • xShow scrollbar (vertical scroll in the embedded FAR editor).

Also see: SQL-file installation — how to install the programming environment (required steps).

THERE ARE TWO MAIN DIRECTIONS IN SQL-FILE USE:

= I.=
Utility application for needs in DB-administration

= II.=
Developer’s use for creation of application (service) part in DB: SP, scalar and table functions, views, triggers etc;
such objects are called programmatics in SQL-file terminology (i.e. objects with some activity, and even views)

IV. CMD is well suitable for lightweight superstructure with user commands and settings.

It sounds strange, but in spite of its legacy character CMD-processor is not easily replaceable in some cases. All that is needed for SQL-file intention is that, that CMD does naturally. I.e.: decorated output, invocation of another batch, satisfactory error checking (even bell signal is enabled there), simple parameters support, organization of subroutines etc. (In SQL-file missing base functionality of Cmd.exe and related commands has been filled in by corresponding tiny subprograms, EXE from “CMD-utilities\Assemblies” subfolder, e.g. $CONFIRMPAUSE and $TYPEINCOLOR commands.)

Even UTF-8 encoding is partially available, in SQL-settings CMD “@<project_name>.cmd” (conventional name), it is used there without BOM-prefix. (The other CMDs are in OEM code-page, like CP437, CP866.) Special Far Manager plugin CmdCpAuto (little subprogram with source-code: “HandicraftSDK\Auxiliary programs (sources)\Far plugins\CmdCpAuto”, “HandicraftSDK\Utilities\Far plugins\CmdCpAuto plugin, build 100 (Far v3)”) is used with SQL-file to switch code-page automatically for some file extensions and file names, when you open embedded file editor (normally by F4 key).

CMD is not perfect of cause, but it is viable with SQL-file. (This is a handicraft implementation of the technology.)

V. Multiplatform settings represent powerful approach for programming in two or more languages, if you are able to access configured values (settings) with help of language preprocessor or in a task of source code generation.

It would be fine to be able to specify a set of definitions, in some universal preprocessor targeted more than one language, so-called multilingual preprocessor. Certain values (especially constants) may be needed simultaneously: in the database (in SQL objects stored in DB), in server process’s EXE (in some language with OOP), and at the client side on user’s computer (another language with OOP). For example, we want to know string length capacities, minimums and maximums etc., they are required at several sides (in DB, in server process, in user input filed etc.). Imagine yourself, that we describe (in some textual or binary format) a lot of values (sizes and not only), dependent one on another, and all these names and values become wonderfully available as code autocompletion function in every of the above mentioned three languages (places), with some little corresponding differences of cause, with possibility of the conditional compilation etc. It could be a magic glue with a static character (i.e. for just before the compilation/interpretation stage) for uniting of very different environments, such fabulous preprocessor. But the reality is a contrary as a rule, and we do not have such static joining (except some rare situations like in C/C++).

SQL-settings defined in “@<project_name>.cmd” are mainly for parameterized SQL files, but not only. In extremis some important settings can be used (involved into) in source code generation, via corresponding building task. In MyFlat app. and BookRegistry app. WriteLinesToFile MSBUILD-task performs such generation — of C# class, with valuable constants for use in related projects (classes: MyFlat.DB.DBConstants, BookRegistry.Server.DB.DBConstants and BookRegistry.Client.DB.DBConstants, in DBConstants.auto.cs). Scripts: generate_metadata.cmd, generate_metadata.props, generate_metadata.targets and DBConstants.cs.props (in “SQL (DB-modelling)” SQL-project folder). This is a labour-intensive organization of minimal static integration of two programming languages (T-SQL/C#).

VI. SQL-file technology at Handicraft-CODE web-pages

To understand the subject, the following web-pages exist for your help in https://handicraft.remelias.ru/ domain:

In addition to samples available with Handicraft-SDK, one more advanced SQL-project exists, suggested by the auther to explope it, — BookRegistry app., extra example of SQL-file use. (All code is in English; however data in tables are in Russian, but this is not an obstacle for understending of T-SQL code.)

THERE ARE TWO CONNECTED SQL-PROJECTS (AMONG OTHER FOLDERS WITH CODE IN C#), FOLDERS:
  • “BookRegistry\SQL (DB-modelling)”: SQL-project base (tables, app.invisible subqueries);
  • “BookRegistry\SHARED\$ClientServer\!DB-queries (application SP)”: application queries, C#/SQL side-by-side.

These are so-called separated SQL-projects, two parts are dependent one from other. Stored procedures with unified parameters are represented (to be consumed by ASP-server process). Besides result code also @Status (GUID) and @Message (string) parameters are used for error reporting, that may be used to report error targeted to web-client directly from SP. (And other interesting features there are.)

BookRegistry application is described in short in my former article at C# Corner:
>> Client-Server WASM-Application In C#, TypeScript And Transact-SQL (from Sep 29, 2020, by Sergei Y.Kitáev).

VII. My work, universal constructor named SQL-file

SQL-file technology is, of cause, experimental production. Never the less this approach has been proved (as earlier product incarnations) at least in two real-world databases, in a system intended for calculation of payment for a series of communal and other living services (processed in flat owner database).

SQL-File App. as conceptual transformation of SQL-file technology

Potentially, another technology, SQL-File App. (i.e. proposed name) can be created as analogical approach, for writing of application queries (not only SP and functions in DB) in parameterized Enhanced T-SQL (app.time queries), and in the same time, which have to be suitable also for a dev.time queries (in developer’s environment). This day we (I) only have the handicraft implementation of SQL-file technology (for MSSQL). This is in fact peculiar constructor for dev.time only, for determining of settings for parameterized SQL-files, and for building of our own SQL-translation commands (adopted scripts for command line processor).

However I believe that the way with parameterized SQL files is potentially effective, and it is not impossible to transform simple SQL-file into something more flexible, such as the SQL-file app. Also I suppose there is no good way to globally unify today’s SQL-languages (it’s a contradictory idea), but there is an approach for partial unification of SQL dialects, which have to be based, I propose, on identical query headers (for all supported SQL dialects) and unique implementations of query bodies for the each dialect (DBMS). These are examples of relatively unified query headers (in some approximation): GetServiceUser.sql (side-by-side files: SQL), UpdateBook.sql (side-by-side files: SQL), with result @Status and @Message output parameters (@RStatus and @RMessage names in newer unpublished version of corresponding DB-library), returnable integer value, specification of one or more result sets (if provided) etc.

Such an application (DB accessing program) may be organized for interaction with SQL-queries by dividing each query into two heterogeneous parts, that is formed as so-called side-by-side files, i.e. located near in the same folder at the developer, with identical base name. For instance: C# and SQL (“.cs” for execution at SQL client-side, and “.sql” for SQL server-side). Only fundamental DB-mapping functionality is required, to transmit SQL-query input data (like parameters) and to obtain result data from SQL-queries (returnable integer value, output parameters, result sets). No big ORM is needed because this idea is contrary to classical ORM concept. It’s like writing code for OS kernel-mode in C, providing the application with user mode library in C# and/or C++ to access the driver. Here I imply so-called paradigm of IRPs (Input/Output Request Packets, in KMDF/WDK). Special auxiliary tools will be needed of cause for effectiveness of such development (with dual SQL-queries), in order to automatically generate pieces of source code for access to “Database IRP”: SQL/C#/… (Examples in my experiments are labour-intensive.)

Also it seems to me that some modifications in DBMS are required, if we want to have a fluent support of efficient client queries from the server (SQL execution) side, by providing a worth-while analog of temporary stored procedure as prepared client query (from corresponding SQL-file or by a piece of SQL-code received from the client). Moving this way we (or somebody) can invent a lot of useful and entertaining.

At the end

Sincerely it is not likely you’ll suddenly have an admiration examining SQL-file. It is because it may look knotty at the first view and also giving you a doubtful gain. My opinion is that it’s not too complex indeed and is worthwhile, at least from experimental perspective. I’ve created this constructor and it have worked a lot for me. But general idea is more valuable than particular implementation. I hope it’ll be interesting for you to acquaint yourself with SQL-file technology.

VIII. SQL-Query-Bridge Outline (abstract idea, imaginary)

There is a download at Handicraft-CODE root page (https://handicraft.remelias.ru/): “SQL-Query-Bridge Outline YYYY-MM-DD.7z”. It contains outline of imaginary SQL-project, for translation of its query sources by abstract utility-preprocessor named SQL-Query-Bridge-PP. We assume that there is a special format (structure) intended for writing of SQL-query (files: “*.q.sql”, and also settings: “@*.*”), on base of what the utility generates T-SQL script for creation of stored procedure, and also C#-code is generated. It is available for extension by user (with help of partial classes), CS-code for interaction with corresponding SQL-query, together with CSharp data models (are also created by the preprocessor). Example contains only one query (T-SQL / C#): SelectBooks, analog of such query from BookRegistry app. sample (from Handicraft Toolkit), with same query name. This sketch (query format, project, settings) is a fantasy thing (pure abstraction).

Blog author

Sergei Y.Kitáev
Author of the Handicraft-CODE resource
https://handicraft.remelias.ru/ (page passage, root)
https://handicraft.remelias.ru/sdk/sql_file.html (to SQL-file)

Leave a Reply

Your email address will not be published. Required fields are marked *