DEB build scripts: several enhancements as follow:

- major rework of update_repositories.sh script, which now supports
multiple branches and provide better feedback to user.
- added support for pre-built extra dependency packages. A user can now
use Slavek Banko's binary packages instead of building the extra
dependency locally. This removes a quite tedious process during the
setup of the building environment.
- bug fixes and more user friendly folder names.
- improved README to cover the complete process until TDE installation.

Special thanks to Gregory Guy <zadig83@laposte.net> for testing and
feedback done so far.

Signed-off-by: Michele Calgaro <michele.calgaro@yahoo.it>

1 files changed, 248 insertions, 0 deletions

diff --git a/debian/_buildscripts/local/README.txt b/debian/_buildscripts/local/README.txt
new file mode 100644
index 000000000..cbe44130a
--- /dev/null
+++ b/debian/_buildscripts/local/README.txt
@@ -0,0 +1,248 @@
+-----
+Index
+-----
+A) Environment preparation
+B) Notes about scripts
+C) How to use the scripts
+
+
+--------------------------
+A) Environment preparation
+--------------------------
+0) Initial notes
+   - this guide has been prepared based on a clean debian testing netinstall system without any other DE.
+     It should work for other debian/ubuntu distros as well, but eventually you may run into small differences
+     here and there.
+   - you can change folder names below, as long as you update the "_config.sh" file accordingly.
+
+1) Install following packages: bc, cdbs, git, pbuilder, rsync and required dependencies.
+   NOTE: sudo should already be installed. If not, install sudo as well.
+
+2) Create a base folder for TDE, hereafter referred to as TDE_DIR (for example $HOME/tde_src)
+
+3) Create the following folders [ see NOTE at point 0) about folder names ]
+   - in TDE_DIR: 0_logs : contains log files for repo update and global builds (more on this later)
+                 1_git  : contains the git repo clones and build hook scripts
+                 2_build: folder used for build preparation and for local builds
+                 3_repo : local repo for package installation (to be configured in /etc/apt/sources.list)
+                 buildscripts: contains a local copy of the build scripts, which can be modified as required
+   - in TDE_DIR/1_git: 
+                 edeps  : contains extra dependency modules necessary to build TDE
+                 hooks  : contains build hook scripts to execute ad-hoc code before and after the building process.
+                          Mainly used to apply patches automatically during the building process
+                          
+4) Clone TDE git repositories
+   - TDE main repo
+     cd "TDE_DIR/1_git"
+     git clone https://mirror.git.trinitydesktop.org/gitea/TDE/tde
+     cd tde
+     git submodule init -- scripts
+     git submodule update -- scripts
+     ./scripts/switch_all_submodules_to_head_and_clean anonymous
+   - TDE packaging repo 
+     cd "TDE_DIR/1_git"
+     git clone https://mirror.git.trinitydesktop.org/gitea/TDE/tde-packaging
+
+5) Add your user to the sudo group (not required if you are root)
+     su -c "adduser <username> sudo"
+   Optionally, consider extending your sudo timeout interval to avoid having to type your password too often
+   (this could be quite painful especially during long builds, which would not be able to complete if unattended).
+   To do this, type "sudo visudo" and then add "Defaults timestamp_timeout=<new timeout>", where the value is in
+   minutes. Use a value of -1 to set an infinite timeout.    
+   Logout and login again to make sure the new settings are applied.
+
+6) Copy the files from "TDE_DIR/1_git/tde-packaging/debian/_buildscripts/local/additional_files" folder
+   to the respectivily named folders.
+
+7) Create pbuilder base package with the command:
+     sudo pbuilder --create
+   If you wish to build for a different distro or architecture, use the command:
+     sudo DIST=<target distro> ARCH=<target architecture> pbuilder --create
+   
+8) Setup the build scripts locally:
+  - copy build scripts from "TDE_DIR/1_git/tde-packaging/debian/_buildscripts/local" to "TDE_DIR/buildscripts"
+    and make sure all shell scripts are executable
+  - cd "TDE_DIR/buildscripts"
+  - cp ./internals/_config_template.sh _config.sh
+  - edit "_config.sh" to set your preferences and check that the various folder names match the structure on your disk.
+    Make sure to set the following variables:
+     * TDE_DIR to the correct path.
+     * DISTRO, DISTRO_NAME and ARCHITECTURE to match the distro and architecture you want to build for.
+     * UPDATE_BRANCHES to the branches you want to keep updated from the main repositories. 
+     * DEFAULT_REPO_BRANCH to the branch to check out at the end of the repositories update process.
+
+9) Some additional packages (referred to as extra dependencies) are required to build and install TDE in debian/ubuntu. 
+   These modules can be built locally, but the procedure is a bit cumbersome. Alternatively, Slavek Banko's
+   pre-built binaries can be used.
+     
+   9.1) Using pre-built extra dependencies (recommended option)
+     - open "_config.sh" and set the variable USE_PREBUILD_EXTRA_DEPS to "y" (this is the default initial setting)
+     - add the following lines to the /etc/apt/sources.list file. This will setup Slavek's repositories in apt for 
+       installing the extra dependency packages when TDE is installed.
+				* For R14.1.x series (repository branch "master")
+					# --- SLAVEK BANKO'S EXTRA DEPENDENCIES REPOSITORY ---
+					deb http://mirror.xcer.cz/trinity-tb <YOUR DISTRO> deps
+				* For R14.0.x series (repository branch "r14.0.x")
+					# --- SLAVEK BANKO'S EXTRA DEPENDENCIES REPOSITORY ---
+					deb http://mirror.xcer.cz/trinity-sb <YOUR DISTRO> deps-r14
+	     For example:
+		     deb http://mirror.xcer.cz/trinity-tb buster deps 
+		 - install package dirmngr if required
+		 - import Slavek's GPG key into your apt keyring
+		   sudo apt-key adv --keyserver pool.sks-keyservers.net --recv-key A04BE668
+		  
+   9.2) Building extra dependencies locally
+     - open "_config.sh" and set the variable USE_PREBUILD_EXTRA_DEPS to "n"
+     - install the following packages: links2 and required dependencies.
+		 - using links2 (or another browser), download the source code for the extra dependency modules from 
+		   Slavek's PPA	(usually more recently updated):
+				 https://quickbuild.pearsoncomputing.net/~slavek-banko/+archive/deps-r14/+packages
+			 or from the official PPA:
+				 https://quickbuild.pearsoncomputing.net/~trinity/+archive/trinity-nightly-build-dependencies/+packages
+			 NOTES:
+			 * Once the page open in your browser, select your distro in the combobox near the "Filter" button and click
+				 the button to update. You should get a list of 10-15 modules, depending on the distro.
+			 * Click on one module at a time, it will expand (or open another page) and show a list of .deb and other files.
+			 * Save those files (exclude .deb files. Only .orig.tar.xz, .dsc, .debian.tar.xz are required) and store
+				 them to TDE_DIR/1_git/edeps/<MODULE NAME> folder (one folder per module).
+
+10) Add the following lines to the /etc/apt/sources.list file. This will setup your local repo in apt.
+		  # --- LOCAL TDE REPOSITORY ---
+		  deb [trusted=yes] file:TDE_DIR/3_repo <YOUR DISTRO> main
+	  For example:
+		  deb [trusted=yes] file:/home/tde_src/3_repo buster main 
+ 
+----------------------
+B) Notes about scripts
+----------------------
+1) * Notes *
+Modules are built using the build_module.sh script. After the build is completed, the installation .deb files are located in TDE_DIR/2_build/debs/<MODULE NAME>/ and the source code and build reports in TDE_DIR/2_build/debs/<MODULE NAME>/src/
+
+The source code can be either the git repo or a local copy in TDE_DIR/2_build/build/<MODULE NAME>.
+A module can be built in a clean chroot environment using pbuilder (default option) or locally using dpkg-buildpackage (useful for quick debugging/developing).
+When using pbuilder, a hook can be used to invoke a shell when the build fails.
+
+Build logs are automatically stored to files, but can also be displayed during the build process.
+The default location of a module build log is TDE_DIR/2_build/debs/<MODULE NAME>/src/__build__.log
+
+When building sets of modules or the whole TDE, a global build summary is automatically stored to TDE_DIR/0_logs/build_result.log to quickly check what built and what failed. It is recommended to delete that file before starting a new TDE build (if not, build results will be appended at the end of the file).
+
+
+2) * Scripts description *
+- scripts in 'internals' folder
+  Scripts used intenally by other scripts. No need for invoking these directly.
+
+- update_git_repository.sh:
+  Script used to update the local clone of the git repositories.
+  It is possible to update multiple branches as specified by the variable UPDATE_BRANCHES in the configuration
+  file. After the update is completed, the local repositories will be switched to the branch specified by the
+  DEFAULT_REPO_BRANCH variable. The default branch can be overridden by using OVERRIDE_DEFAULT_REPO_BRANCH when
+  invoking the script. 
+  Usage:
+    [OVERRIDE_DEFAULT_REPO_BRANCH="<branch name>"] update_git_repository.sh [options]
+  Options:
+    -i  (Incremental) : in case the previous update was interrupted, continue from the last known updated module.
+                        (useful on slow/unstable internet connections). If the previous update had completed, 
+                        this option is ignored.
+    -v  (Verbose)     : display and log more output. Useful for troubleshooting.
+
+- build_module.sh
+  Build a single module.
+  Usage:
+    build_module.sh [options] module_name
+  Options:
+    -g  (Git)         : build from git repo sources. If missing, build from the local copy in build folder.
+    -l  (Local)       : build the module locally. If missing, build in a clean chroot environment
+    -sl (Show Log)    : output the building logs to terminal while the build is ongoing
+    -lr (Log Result)  : log (append) build result (OK, FAILED) to TDE_DIR/0_logs/build_result.log file
+    -sh (Shell Hook)  : use a shell hook for failing builds, only valid if building using pbuilder (clean chroot environment)
+    -po (Prepare Only): only prepare the source folder but do not build the module. Useful to prepare the source code before
+                        doing local changes/development. The module can then be built from the modified local folder
+    -d  (Debug)       : enable debug symbols if possible (debian/rules file must contain "RelWithDebInfo" for this to work)
+    -ip (Internal Pbuilder): build using internal pbuilder mode (experimental)
+  
+- <dd>_<set_name>.sh
+  A number of scripts used to build sets of modules. Each script builds an individual set.
+  Modules are built from the git sources and the build result is automatically appended to 
+  TDE_DIR/0_logs/build_result.log.
+  Usage:
+    <dd>_<set_name>.sh [options] set_name
+  Options:
+    -s N: if specified, skip first N modules from the set of modules
+
+	The sets are logically grouped as "base system", "applications" and "others"
+  00_extradeps.sh                             : extra dependencies modules
+  01_base_01.sh        - 03_base_03.sh        : TDE base system
+  04_application_01.sh - 09_application_06.sh : application modules
+  10_others_01.sh      - 11_others_02.sh      : other modules that are somehow less used
+  91_build_base.sh                            : whole TDE base system in 00_extradeps.sh - 03_base_03.sh
+  92_build_applications.sh                    : all TDE applications in 04_application_01.sh - 09_application_06.sh
+  93_build_others.sh                          : all modules in 10_others_01.sh - 11_others_02.sh
+
+- 99_build_TDE.sh
+  Script used to build the complete TDE at once. Calls the dd_setname.sh set scripts in order.
+  Usage:
+    build_TDE.sh
+    
+- create_repo.sh
+  Creates a local repository from the .deb files currently stored in TDE_DIR/2_build/debs. 
+  Usage:
+    [sudo] create_repo.sh [options]
+  Options:
+    -b (Backup) : create a backup of the existing repository in TDE_DIR/CFG_REPO_DIR.backup
+
+  
+3) * Building hooks *
+Hooks are available to execute ad-hoc code before and after the build process. For example this is very useful to automatically apply patches.
+There are two type of hooks: 
+- pre_build : applied before switching the module to quilt format and build
+- post_build: applied after the build (dpkg-buildpackage or pbuilder) has terminated 
+To use a hook, create an executable script (pre_build.sh and/or post_build.sh) in the TDE_DIR/1_git/hooks/<MODULE NAME> folder. The scripts are executed in the build_module.sh environment, so have access to all the variables defined in that file. See the files in the "hook examples" folder for real usage samples.
+
+
+
+-------------------------
+C) How to use the scripts
+-------------------------
+1) Follow the steps in section "A) Environment preparation" (only required the first time).
+2) cd "TDE_DIR/buildscripts"
+3) Update to latest git repository using
+     ./update_repositories.sh
+4) Run "sudo pbuilder update" at the beginning of the day to update the base package to the latest version. This will speed up
+   the process when building several modules in sequence.
+5) Build modules as per your needs. You don't need to use "sudo" directly since the scripts will do that automatically if
+   required. Just type your sudo password when prompted to do so.
+6) Create a local repository from the packages you have just built, to be used as installation repository. 
+     [sudo] ./create_repo.sh
+7) Install TDE as you usually do. For example as follow:
+     - sudo apt-get update
+     - sudo aptitude install tdebase-trinity   (for a minimalistic TDE environment)
+         or
+       sudo aptitude install tde-trinity       (for a standard TDE environment)
+
+  
+  
+Examples of real usage:
+
+1) build a single module
+   - ./build_module.sh -g -sl "dependencies/libr"       -> build libr package. This is a good test to check
+                                                           whether everything is working fine
+   - ./build_module.sh -g "tdelibs"       -> build "tdelibs" from git sources in a clean chroot environment
+   - ./build_module.sh -g -l -sl "applications/amarok"  -> build "amarok" locally from git sources and 
+                                                           display building logs during building
+   - ./build_module.sh -sh -lr "tdebase"  -> build "tdebase" from the local sources (in TDE_DIR/2_build/build/tdebase)
+                                             in a clean chroot environment and launch a shell in case of building failure.
+                                             Append the build result (OK, FAILED) to TDE_DIR/scripts/logs/build_result.log
+   - ./build_module.sh -g -po "tdelibs"   -> prepare "tdelibs" for building from git sources. Source code will be available
+                                             in TDE_DIR/2_build/build/tdelibs. After you have made changes to the source and
+                                             want to build the modified package, run './build_module.sh "tdelibs"'
+                                          
+2) build a single set
+  (optional) delete the TDE_DIR/0_logs/build_result.log file
+  ./01_base_01.sh           -> build this set. 
+  ./03_base_03.sh -s 3      -> build this set but skip the first 3 modules of the set.
+
+3) build all TDE
+  ./99_build_TDE.sh         -> build all TDE
+
+4) [sudo] ./create_repo.sh  -> build or update your local TDE repository
\ No newline at end of file