On the Delphi *.dcp files

2

A *.dcp file is created when Delphi builds a package. It always bothered me that the default *.dcp file location does not take into account the build configuration. For example, the default location for Win32 platform is $(BDSCOMMONDIR)\DCP; if you build a package in DEBUG configuration and then in RELEASE configuration, the release *.dcp overwrites the debug *.dcp.

But the point is: *.dcp files are needed to develop dependent packages. If you develop a single package, say PackageA, you can forget about the PackageA.dcp file and where it is located; it is simply not needed.

If you develop 2 packages, say PackageA and PackageB, and the PackageB depends on the PackageA (requires PackageA), then you can’t even fully create PackageB without specifying the location of PackageA.dcp file. In this case the solution is: rely on the default *.dcp file location and don’t change it. The build workflow should be as follows:

  • Build PackageA in DEBUG configuration
  • Build PackageB in DEBUG configuration
  • Build PackageA in RELEASE configuration
  • Build PackageB in RELEASE configuration

The release versions of *.dcp files has overwritten the debug versions, but the *.dcp files served their purpose: the compiled debug version of PackageB depends on compiled debug version of PackageA, the compiled release version of PackageB depends on compiled release version of PackageA; the *.dcp files are not needed anymore.

Introduction to Python for Delphi programmers

3

Why Delphi programmer needs Python at all? The main reason is: Python ecosystem is much bigger and much more active than Delphi ecosystem; there are many useful and actively developing projects in Python, much more than you can find in Delphi. Currently I am dabbling in a nice MkDocs documentation generator and planning to use it for documenting my project.

The first thing you need to understand about Python is project isolation. In Delphi you can add units, packages, etc to a project and it does not affect other projects; the Delphi projects are well isolated. If you simply install Python and start developing projects, you quickly discover that there is no project isolation at all; if you add python package for one project, the package becomes available for all projects. It is quite stunning when you encounter it for the first time, but there is a solution: you need a separate Python installation for each project. These separate installations are called environments. The idea is: you have one global Python installation with the sole purpose of creating environments; you never use the global installation for developing projects. When you start a new project you create a new environment, and when later on you add packages to the environment it does not affect global installation or other environments.

There are several ways to create environments, the way I use is Conda package manager.

I am using Python on Linux Mint, and Linux Mint already has Python installed (on Windows you probably need to install Python first). But this Python belongs to the system; the system installed it for its own purposes, and trying to modify the system Python is bad idea. Good news are: Python is Conda installation requirement, and you have it.

Go to Miniconda download page and download Miniconda for your system. There are two installer versions, based on Python 2 and Python 3 – use the one based on Python 3; it does not matter much; by default Python 3 version creates Python 3 environments. Don’t install Anaconda – if you want to play with Anaconda install it later into environment.

Open Terminal window, go to the download folder and run the downloaded installer; accept the default settings and answer “yes” to the installer questions. After the installation is completed close the Terminal window and open it again. Now you have Python installed in your home folder; to check it run which python command:

  serg@TravelMate ~ $ which python
  /home/serg/miniconda3/bin/python

This is global Python installation that will be used to create environments.

To play with MkDocs I’ve created environment named mkdocs:

  conda create -n mkdocs pip

pip is Python package manager that will be included in the new environment. Conda documentation may say that you need not pip because you can install everything using Conda itself; I believe this is too good to be true, and prefer to have pip in any environment, and it is better to install pip right in the environment create command.

Now we need to activate the newly created environment; after the activation check that we have different python installed in the environment:

  serg@TravelMate ~ $ source activate mkdocs
  (mkdocs) serg@TravelMate ~ $ which python
  /home/serg/miniconda3/envs/mkdocs/bin/python

The next step is to upgrade pip in the environment:

  pip install --upgrade pip

and finally install MkDocs package into the environment:

  pip install mkdocs

Check that mkdocs is installed:

  (mkdocs) serg@TravelMate ~ $ mkdocs --version
  mkdocs, version 1.0.2 from /home/serg/miniconda3/envs/mkdocs/lib/python3.7/site-packages/mkdocs (Python 3.7)

If you are not going to do more for now, deactivate the environment

  (mkdocs) serg@TravelMate ~ $ source deactivate
  serg@TravelMate ~ $ 

or just close the Terminal window.

Installing Lazarus 1.8 on Linux Mint 18.3

4

Yesterday I come across my old 32-bit Celeron laptop with 2Gb memory and broken battery and decided to install Linux on it. First I tried Centos 7.5, then Ubuntu 16.04, and finally settled down on Mint 18.3 Cinnamon which I liked most. After playing a little with the OS I decided to install the brand new Lazarus 1.8. I spent a couple of hours searching for a clear installation guide in Internet; I scanned tons of outdated nonsense before I finally found what I was looking for.

My Linux Mint installation was fresh, and I need not purge it from previous fpc/lazarus installations. So I started from downloading three 32-bit .deb packages from sourceforge.

After downloading my Downloads folder looked so:

Downloads

The next recommended step is to check hashes of the files. I launched Terminal and changed to Downloads directory:

hashes2

Now time to install the packages. As recommended in the linked guide, I typed in Terminal

sudo apt install ./fpc_3.0.4-1_i386.deb
sudo apt install ./fpc-src_3.0.4-1_i386.deb
sudo apt install ./lazarus-project_1.8.0.0_i386.deb
sudo apt-mark hold fpc fpc-src lazarus lazarus-project

And that is all! After opening the Cinnamon menu I’ve found this:

menu

Yet another surprise from Delphi compiler

5

Build and run the following console app:

program ItWorks;

{$APPTYPE CONSOLE}

uses
  SysUtils;

type
  TTest = record
    FData: Integer;
  end;

procedure DoTest(const S: string);
begin
  Writeln(TTest(S).FData);
end;

begin
  try
    DoTest('123');
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
  Readln;
end.

I tested in Delphi XE and Delphi 10.1 Berlin, it compiles and even works!

TStreamCipher

0

TForge 0.76 is released.

The release introduces a new class TStreamCipher. The name TStreamCipher does not mean that the class supports stream cipher algorithms only, such as Salsa20 or RC4 – it supports all cipher algorithms currently implemented in TForge, like TCipher class does; block ciphers are converted into stream ciphers using CTR mode of operation.

The initial motivation to write TStreamCipher class was to generate “solid” byte-wise keystream. TCipher class generates block-wise keystream; to obtain solid byte-wise keystream you need to maintain additional state, and that’s what the TStreamCipher class is doing. Here is the code sample illustrating the above:

const
  Nonce = 42;

var
  Cipher: TCipher;
  StreamCipher: TStreamCipher;
  Key: ByteArray;
  KeyStream, KeyStream1, KeyStream2: ByteArray;

begin
// create 128-bit AES key:
  Key:= ByteArray.AllocateRand(16);

// generate 16 bytes of keystream:
  KeyStream:= TCipher.AES.ExpandKey(Key, CTR_ENCRYPT, Nonce).KeyStream(16);
  Writeln(KeyStream.ToHex);

// generate 8 + 8 bytes of keystream:
  Cipher:= TCipher.AES.ExpandKey(Key, CTR_ENCRYPT, Nonce);
// Warning - KeyStream1:= Cipher.KeyStream(8) + Cipher.KeyStream(8) is wrong
//   because the compiler does not evaluate the summands in order
  KeyStream1:= Cipher.KeyStream(8);
  KeyStream1:= KeyStream1 + Cipher.KeyStream(8);
  Writeln(KeyStream1.ToHex);
  Assert(KeyStream <> KeyStream1);

// generate 8 + 8 bytes of keystream using TStreamCipher instance:
  StreamCipher:= TStreamCipher.AES.ExpandKey(Key, Nonce);
  KeyStream2:= StreamCipher.KeyStream(8);
  KeyStream2:= KeyStream2 + StreamCipher.KeyStream(8);
  Writeln(KeyStream2.ToHex);
  Assert(KeyStream = KeyStream2);
end;

The Cipher instance generates 8 + 8 bytes of keystream by taking the first 8 bytes from the first block and the second 8 bytes from the second block (AES block size is 16 bytes), that is why the generated keystream is different.

The code also illustrates the use of Nonce. The purpose of nonce is to encrypt multiple messages (ex files) with the same secret key – each message should be encrypted with unique non-secret nonce. Modern cryptoalgorithms such as Salsa20 have built-in support for nonce, and TForge supported the feature since version 0.71; old block cipher algorithms such as AES allow to implement the same feature using IV (initialization vector), and this is the second thing that TStreamCipher class is doing; TStreamCipher API does not expose IV of the underlying cryptoalgorithm, it exposes Nonce only.

Nonces are now supported by both TStreamCipher and TCipher classes, as the above code sample shows; there is a limitation though: the block size of an algorithm should be 128 bits at least; that means you can’t use nonces with legacy 64-bit block ciphers such as DES or Triple DES.

TForge 0.76 includes new demo project FileEncryptDemo (Demos\Ciphers folder) which shows how to use TStreamCipher. The idea of the project is to implement secure encryption of multiple files with a single key (and multiple non-secret nonces) and transparent reading of the encrypted files.

On the “out” parameter specifier in Delphi

8

I’ve posted before what the “out” parameter specifier is actually doing in a Delphi code. Now let us consider when the “out” keyword should be used instead of “var”.

There is only one case when the “out” keyword should be used. It is closely related to COM technology support, but it is not really limited to COM. The case is importing a function with a parameter of interface type which violates Delphi contract on reference counting. To illustrate the point consider the following DLL:

library DemoDll;

uses
  SysUtils;

type
  PMyUnknown = ^TMyUnknown;
  TMyUnknown = record
    FVTable: PPointer;
    FRefCount: Integer;
    class function QueryIntf(Inst: Pointer; const IID: TGUID;
                             out Obj): HRESULT; stdcall; static;
    class function AddRef(Inst: PMyUnknown): Integer; stdcall; static;
    class function Release(Inst: PMyUnknown): Integer; stdcall; static;
  end;

const
  VTable: array[0..2] of Pointer = (
    @TMyUnknown.QueryIntf,
    @TMyUnknown.Addref,
    @TMyUnknown.Release);

function ReleaseInstance(Inst: Pointer): Integer;
type
  TVTable = array[0..2] of Pointer;
  PVTable = ^TVTable;
  PPVTable = ^PVTable;

  TRelease = function(Inst: Pointer): Integer; stdcall;

begin
  Result:= TRelease(PPVTable(Inst)^^[2])(Inst);
end;

class function TMyUnknown.QueryIntf(Inst: Pointer; const IID: TGUID;
  out Obj): HRESULT;
begin
  Result:= E_NOINTERFACE;
end;

class function TMyUnknown.Addref(Inst: PMyUnknown): Integer;
begin
  Inc(Inst.FRefCount);
  Result:= Inst.FRefCount;
end;

class function TMyUnknown.Release(Inst: PMyUnknown): Integer;
begin
  Dec(Inst.FRefCount);
  Result:= Inst.FRefCount;
  if Result = 0 then
    FreeMem(Inst);
end;

procedure GetIUnknown1(var Inst: PMyUnknown);
var
  P: PMyUnknown;

begin
  New(P);
  P^.FVTable:= @VTable;
  P^.FRefCount:= 1;
  if Inst <> nil then ReleaseInstance(Inst);
  Inst:= P;
end;

procedure GetIUnknown2(var Inst: PMyUnknown);
begin
  New(Inst);
  Inst^.FVTable:= @VTable;
  Inst^.FRefCount:= 1;
end;

exports
  GetIUnknown1,
  GetIUnknown2;

{$R *.res}

begin
  ReportMemoryLeaksOnShutdown:= True;
end.

The DLL exports 2 procedures which create instances implementing IUnknown. The first procedure (GetIUnknown1) follows Delphi rules on reference counting, the second (GetIUnknown2) does not. To use the procedures in an executable we need an import unit:

unit DemoImport;

interface

procedure GetIUnknown1(var I: IUnknown);
//procedure GetIUnknown2(var I: IUnknown); // memory leak
procedure GetIUnknown2(out I: IUnknown);
//                     ^^^!
implementation

procedure GetIUnknown1(var I: IUnknown); external 'DemoDLL' name 'GetIUnknown1';
//procedure GetIUnknown2(var I: IUnknown); external 'DemoDLL' name 'GetIUnknown2';
procedure GetIUnknown2(out I: IUnknown); external 'DemoDLL' name 'GetIUnknown2';

end.

Note that the second procedure is imported with out parameter specifier; the result of replacing “out” by “var” is a memory leak which can be shown by the test application:

program DemoEXE;

{$APPTYPE CONSOLE}

uses
  SysUtils,
  DemoImport in 'DemoImport.pas';

procedure Test;
var
  I: IUnknown;

begin
  GetIUnknown1(I);
  GetIUnknown2(I);
end;

begin
  Test;
  Readln;
end.

Class Methods vs Instance Methods

10

C# implements BigInteger.Parse as a static class method, and there is obvious reason for it: you can call a static class methods in a variable’s declaration:

  BigInteger N = BigInteger.Parse(stringToParse);

Delphi/Pascal does not support the above syntax, so the equivalent code is

var
  N: BigInteger;
begin
  N:= BigInteger.Parse(stringToParse);

But now it looks like implementing BigInteger.Parse as a static class method is nothing but additional typing; using an instance method looks better:

var
  N: BigInteger;
begin
  N.Parse(stringToParse);

So, what is the right “Delphi way” of implementing BigInteger.Parse – as a static class method or as an instance method?

Integer division by constant

4

If divisor is fixed, the division operation can be replaced by multiplication by a reciprocal; the idea is very simple: x / d = x * (1/d); since d is fixed we can precompute 1/d and use faster multiplication operation. This trick works for integer division, with some additional complications. For example, the next function is identical to division by 5 for all 32-bit dividends:

function Div5(Dividend: LongWord): LongWord;
const
  Mult = $CCCCCCCD;
  PostSh = 2;

asm
        MOV     EDX,Mult
        MUL     EDX
        MOV     EAX,EDX
        SHR     EAX,PostSh
end;

It turns out that 32-bit multiplication constant does not exists for every divisor; in this case it is possible to use 33-bit constant with hidden senior 1 bit:

function Div5_2(Dividend: LongWord): LongWord;
const
  Mult = $9999999A;
  PostSh = 3;

asm
        MOV     ECX,EAX     // save dividend
        MOV     EDX,Mult
        MUL     EDX
        MOV     EAX,ECX     // restore dividend
        ADD     EAX,EDX
        RCR     EAX,1       // because addition could produce carry
        SHR     EAX,PostSh-1
end;

Naively, the multiplication constant ($9999999A) and postshift (3) can be obtained using the following steps:

1. Write 1/5 in binary form: 1/5 = 0.0011 0011 0011 0011 0011 ..

2. Count leading zeroes after decimal point and add 1; this gives the postshift

3. Write the leading significant bits:

1100 1100 1100 1100 1100 1100 1100 1100 1100 1100 ..

4. Round to 33 bits; rounding is not necessarily based on the value of 34th bit only (later we will use a different approach which is free from rounding problem)

1100 1100 1100 1100 1100 1100 1100 1100 1101 0

5. Remove the leading bit and write as hexadecimal to get the multiplication constant:

99 99 99 9A

The second version is more complicated, but good news is that it works for all divisors. Generally, it can be proven that for N-bit division there exists (N+1) bit multiplication constant – for the details see the key article Division by Invariant Integers using Multiplication. The rest of the post is a review of the results obtained in the article cited. I will use 32-bit arithmetic, but the results can be generalized for any bitness.

The multiplication constant and postshift can be found using the following procedure:

procedure GetParams(Divisor: LongWord);
var
  L, N: LongWord;
  Tmp: LongWord;
  M: UInt64;
  Mult: LongWord;

begin
  Assert(Divisor > 1);
  Tmp:= Divisor;
  L:= 0;
  repeat            // count number of significant bits in Divisor
    Tmp:= Tmp shr 1;
    Inc(L);
  until Tmp = 0;
  N:= 1 shl L;      // N = 2^L
  M:= $100000000;   // 2^32
  M:= (M * (N - Divisor)) div Divisor + 1;
  Mult:= LongWord(M);
  Writeln('Mult  = ' + IntToHex(Mult, 8));
  Writeln('Shift = ' + IntToStr(L));
end;

The multiplication constant is not unique. If the constant obtained from the above procedure is odd, it makes sense to increment it by 1 (provided the incremented constant is correct too). The correctness can be checked by

function CheckMult(Divisor, Mult: LongWord): Boolean;
var
  L, N: LongWord;
  Tmp: LongWord;
  P32, MD: UInt64;
  Min, Max: UInt64;

begin
  P32:= $100000000;   // 2^32;
  MD:= (P32 + Mult) * Divisor;
  L:= 0;
  Tmp:= Divisor;
  repeat            // count number of significant bits in Divisor
    Tmp:= Tmp shr 1;
    Inc(L);
  until Tmp = 0;
  N:= 1 shl L;
  Min:= P32 * N;
  Max:= Min + N;
  Result:= (Min <= MD) and (MD <= Max);
end;

The above function gives sufficient condition for correct multiplication constant; it is probably possible that some constant fails the check but still correct, as can be shown by the full search over the dividend range; we will not consider this case.

Optimization 1. If the multiplication constant is even, we can shift it right and fit into 32-bit range, thus obtaining shorter division algorithm; this also decrements the postshift.

As an example consider division by 641. The GetParams procedure returns

Mult = 98F603FF
Shift = 10

Since the multiplication constant is odd, we try incremented constant 98F60400. CheckMult shows that it is correct too; the constant ends with 10 zero bits, which is equal to postshift, so we eliminate the postshift:

function Div641(Dividend: LongWord): LongWord;
const
  Mult = $663D81;  // = $198F60400 shr 10

asm
        MOV     EDX,Mult
        MUL     EDX
        MOV     EAX,EDX
end;

Optimization 2. If the divisor is even, we can preshift both dividend and divisor right, thus decreasing the required precision and obtaining shorter division algorithm again (with the dividend preshift step). For details see the article cited above.