From eeaccbeddbaa2974406eb20cbb7beca3fb4a59f0 Mon Sep 17 00:00:00 2001 From: Michele Calgaro Date: Wed, 11 Jul 2018 23:38:32 +0900 Subject: DEB build scripts: several enhancements as follow: - major rework of update_repositories.sh script, which now supports multiple branches and provide better feedback to user. - added support for pre-built extra dependency packages. A user can now use Slavek Banko's binary packages instead of building the extra dependency locally. This removes a quite tedious process during the setup of the building environment. - bug fixes and more user friendly folder names. - improved README to cover the complete process until TDE installation. Special thanks to Gregory Guy for testing and feedback done so far. Signed-off-by: Michele Calgaro --- debian/_buildscripts/local/README.txt | 248 ++++++++++++++++++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 debian/_buildscripts/local/README.txt (limited to 'debian/_buildscripts/local/README.txt') diff --git a/debian/_buildscripts/local/README.txt b/debian/_buildscripts/local/README.txt new file mode 100644 index 000000000..cbe44130a --- /dev/null +++ b/debian/_buildscripts/local/README.txt @@ -0,0 +1,248 @@ +----- +Index +----- +A) Environment preparation +B) Notes about scripts +C) How to use the scripts + + +-------------------------- +A) Environment preparation +-------------------------- +0) Initial notes + - this guide has been prepared based on a clean debian testing netinstall system without any other DE. + It should work for other debian/ubuntu distros as well, but eventually you may run into small differences + here and there. + - you can change folder names below, as long as you update the "_config.sh" file accordingly. + +1) Install following packages: bc, cdbs, git, pbuilder, rsync and required dependencies. + NOTE: sudo should already be installed. If not, install sudo as well. + +2) Create a base folder for TDE, hereafter referred to as TDE_DIR (for example $HOME/tde_src) + +3) Create the following folders [ see NOTE at point 0) about folder names ] + - in TDE_DIR: 0_logs : contains log files for repo update and global builds (more on this later) + 1_git : contains the git repo clones and build hook scripts + 2_build: folder used for build preparation and for local builds + 3_repo : local repo for package installation (to be configured in /etc/apt/sources.list) + buildscripts: contains a local copy of the build scripts, which can be modified as required + - in TDE_DIR/1_git: + edeps : contains extra dependency modules necessary to build TDE + hooks : contains build hook scripts to execute ad-hoc code before and after the building process. + Mainly used to apply patches automatically during the building process + +4) Clone TDE git repositories + - TDE main repo + cd "TDE_DIR/1_git" + git clone https://mirror.git.trinitydesktop.org/gitea/TDE/tde + cd tde + git submodule init -- scripts + git submodule update -- scripts + ./scripts/switch_all_submodules_to_head_and_clean anonymous + - TDE packaging repo + cd "TDE_DIR/1_git" + git clone https://mirror.git.trinitydesktop.org/gitea/TDE/tde-packaging + +5) Add your user to the sudo group (not required if you are root) + su -c "adduser sudo" + Optionally, consider extending your sudo timeout interval to avoid having to type your password too often + (this could be quite painful especially during long builds, which would not be able to complete if unattended). + To do this, type "sudo visudo" and then add "Defaults timestamp_timeout=", where the value is in + minutes. Use a value of -1 to set an infinite timeout. + Logout and login again to make sure the new settings are applied. + +6) Copy the files from "TDE_DIR/1_git/tde-packaging/debian/_buildscripts/local/additional_files" folder + to the respectivily named folders. + +7) Create pbuilder base package with the command: + sudo pbuilder --create + If you wish to build for a different distro or architecture, use the command: + sudo DIST= ARCH= pbuilder --create + +8) Setup the build scripts locally: + - copy build scripts from "TDE_DIR/1_git/tde-packaging/debian/_buildscripts/local" to "TDE_DIR/buildscripts" + and make sure all shell scripts are executable + - cd "TDE_DIR/buildscripts" + - cp ./internals/_config_template.sh _config.sh + - edit "_config.sh" to set your preferences and check that the various folder names match the structure on your disk. + Make sure to set the following variables: + * TDE_DIR to the correct path. + * DISTRO, DISTRO_NAME and ARCHITECTURE to match the distro and architecture you want to build for. + * UPDATE_BRANCHES to the branches you want to keep updated from the main repositories. + * DEFAULT_REPO_BRANCH to the branch to check out at the end of the repositories update process. + +9) Some additional packages (referred to as extra dependencies) are required to build and install TDE in debian/ubuntu. + These modules can be built locally, but the procedure is a bit cumbersome. Alternatively, Slavek Banko's + pre-built binaries can be used. + + 9.1) Using pre-built extra dependencies (recommended option) + - open "_config.sh" and set the variable USE_PREBUILD_EXTRA_DEPS to "y" (this is the default initial setting) + - add the following lines to the /etc/apt/sources.list file. This will setup Slavek's repositories in apt for + installing the extra dependency packages when TDE is installed. + * For R14.1.x series (repository branch "master") + # --- SLAVEK BANKO'S EXTRA DEPENDENCIES REPOSITORY --- + deb http://mirror.xcer.cz/trinity-tb deps + * For R14.0.x series (repository branch "r14.0.x") + # --- SLAVEK BANKO'S EXTRA DEPENDENCIES REPOSITORY --- + deb http://mirror.xcer.cz/trinity-sb deps-r14 + For example: + deb http://mirror.xcer.cz/trinity-tb buster deps + - install package dirmngr if required + - import Slavek's GPG key into your apt keyring + sudo apt-key adv --keyserver pool.sks-keyservers.net --recv-key A04BE668 + + 9.2) Building extra dependencies locally + - open "_config.sh" and set the variable USE_PREBUILD_EXTRA_DEPS to "n" + - install the following packages: links2 and required dependencies. + - using links2 (or another browser), download the source code for the extra dependency modules from + Slavek's PPA (usually more recently updated): + https://quickbuild.pearsoncomputing.net/~slavek-banko/+archive/deps-r14/+packages + or from the official PPA: + https://quickbuild.pearsoncomputing.net/~trinity/+archive/trinity-nightly-build-dependencies/+packages + NOTES: + * Once the page open in your browser, select your distro in the combobox near the "Filter" button and click + the button to update. You should get a list of 10-15 modules, depending on the distro. + * Click on one module at a time, it will expand (or open another page) and show a list of .deb and other files. + * Save those files (exclude .deb files. Only .orig.tar.xz, .dsc, .debian.tar.xz are required) and store + them to TDE_DIR/1_git/edeps/ folder (one folder per module). + +10) Add the following lines to the /etc/apt/sources.list file. This will setup your local repo in apt. + # --- LOCAL TDE REPOSITORY --- + deb [trusted=yes] file:TDE_DIR/3_repo main + For example: + deb [trusted=yes] file:/home/tde_src/3_repo buster main + +---------------------- +B) Notes about scripts +---------------------- +1) * Notes * +Modules are built using the build_module.sh script. After the build is completed, the installation .deb files are located in TDE_DIR/2_build/debs// and the source code and build reports in TDE_DIR/2_build/debs//src/ + +The source code can be either the git repo or a local copy in TDE_DIR/2_build/build/. +A module can be built in a clean chroot environment using pbuilder (default option) or locally using dpkg-buildpackage (useful for quick debugging/developing). +When using pbuilder, a hook can be used to invoke a shell when the build fails. + +Build logs are automatically stored to files, but can also be displayed during the build process. +The default location of a module build log is TDE_DIR/2_build/debs//src/__build__.log + +When building sets of modules or the whole TDE, a global build summary is automatically stored to TDE_DIR/0_logs/build_result.log to quickly check what built and what failed. It is recommended to delete that file before starting a new TDE build (if not, build results will be appended at the end of the file). + + +2) * Scripts description * +- scripts in 'internals' folder + Scripts used intenally by other scripts. No need for invoking these directly. + +- update_git_repository.sh: + Script used to update the local clone of the git repositories. + It is possible to update multiple branches as specified by the variable UPDATE_BRANCHES in the configuration + file. After the update is completed, the local repositories will be switched to the branch specified by the + DEFAULT_REPO_BRANCH variable. The default branch can be overridden by using OVERRIDE_DEFAULT_REPO_BRANCH when + invoking the script. + Usage: + [OVERRIDE_DEFAULT_REPO_BRANCH=""] update_git_repository.sh [options] + Options: + -i (Incremental) : in case the previous update was interrupted, continue from the last known updated module. + (useful on slow/unstable internet connections). If the previous update had completed, + this option is ignored. + -v (Verbose) : display and log more output. Useful for troubleshooting. + +- build_module.sh + Build a single module. + Usage: + build_module.sh [options] module_name + Options: + -g (Git) : build from git repo sources. If missing, build from the local copy in build folder. + -l (Local) : build the module locally. If missing, build in a clean chroot environment + -sl (Show Log) : output the building logs to terminal while the build is ongoing + -lr (Log Result) : log (append) build result (OK, FAILED) to TDE_DIR/0_logs/build_result.log file + -sh (Shell Hook) : use a shell hook for failing builds, only valid if building using pbuilder (clean chroot environment) + -po (Prepare Only): only prepare the source folder but do not build the module. Useful to prepare the source code before + doing local changes/development. The module can then be built from the modified local folder + -d (Debug) : enable debug symbols if possible (debian/rules file must contain "RelWithDebInfo" for this to work) + -ip (Internal Pbuilder): build using internal pbuilder mode (experimental) + +-
_.sh + A number of scripts used to build sets of modules. Each script builds an individual set. + Modules are built from the git sources and the build result is automatically appended to + TDE_DIR/0_logs/build_result.log. + Usage: +
_.sh [options] set_name + Options: + -s N: if specified, skip first N modules from the set of modules + + The sets are logically grouped as "base system", "applications" and "others" + 00_extradeps.sh : extra dependencies modules + 01_base_01.sh - 03_base_03.sh : TDE base system + 04_application_01.sh - 09_application_06.sh : application modules + 10_others_01.sh - 11_others_02.sh : other modules that are somehow less used + 91_build_base.sh : whole TDE base system in 00_extradeps.sh - 03_base_03.sh + 92_build_applications.sh : all TDE applications in 04_application_01.sh - 09_application_06.sh + 93_build_others.sh : all modules in 10_others_01.sh - 11_others_02.sh + +- 99_build_TDE.sh + Script used to build the complete TDE at once. Calls the dd_setname.sh set scripts in order. + Usage: + build_TDE.sh + +- create_repo.sh + Creates a local repository from the .deb files currently stored in TDE_DIR/2_build/debs. + Usage: + [sudo] create_repo.sh [options] + Options: + -b (Backup) : create a backup of the existing repository in TDE_DIR/CFG_REPO_DIR.backup + + +3) * Building hooks * +Hooks are available to execute ad-hoc code before and after the build process. For example this is very useful to automatically apply patches. +There are two type of hooks: +- pre_build : applied before switching the module to quilt format and build +- post_build: applied after the build (dpkg-buildpackage or pbuilder) has terminated +To use a hook, create an executable script (pre_build.sh and/or post_build.sh) in the TDE_DIR/1_git/hooks/ folder. The scripts are executed in the build_module.sh environment, so have access to all the variables defined in that file. See the files in the "hook examples" folder for real usage samples. + + + +------------------------- +C) How to use the scripts +------------------------- +1) Follow the steps in section "A) Environment preparation" (only required the first time). +2) cd "TDE_DIR/buildscripts" +3) Update to latest git repository using + ./update_repositories.sh +4) Run "sudo pbuilder update" at the beginning of the day to update the base package to the latest version. This will speed up + the process when building several modules in sequence. +5) Build modules as per your needs. You don't need to use "sudo" directly since the scripts will do that automatically if + required. Just type your sudo password when prompted to do so. +6) Create a local repository from the packages you have just built, to be used as installation repository. + [sudo] ./create_repo.sh +7) Install TDE as you usually do. For example as follow: + - sudo apt-get update + - sudo aptitude install tdebase-trinity (for a minimalistic TDE environment) + or + sudo aptitude install tde-trinity (for a standard TDE environment) + + + +Examples of real usage: + +1) build a single module + - ./build_module.sh -g -sl "dependencies/libr" -> build libr package. This is a good test to check + whether everything is working fine + - ./build_module.sh -g "tdelibs" -> build "tdelibs" from git sources in a clean chroot environment + - ./build_module.sh -g -l -sl "applications/amarok" -> build "amarok" locally from git sources and + display building logs during building + - ./build_module.sh -sh -lr "tdebase" -> build "tdebase" from the local sources (in TDE_DIR/2_build/build/tdebase) + in a clean chroot environment and launch a shell in case of building failure. + Append the build result (OK, FAILED) to TDE_DIR/scripts/logs/build_result.log + - ./build_module.sh -g -po "tdelibs" -> prepare "tdelibs" for building from git sources. Source code will be available + in TDE_DIR/2_build/build/tdelibs. After you have made changes to the source and + want to build the modified package, run './build_module.sh "tdelibs"' + +2) build a single set + (optional) delete the TDE_DIR/0_logs/build_result.log file + ./01_base_01.sh -> build this set. + ./03_base_03.sh -s 3 -> build this set but skip the first 3 modules of the set. + +3) build all TDE + ./99_build_TDE.sh -> build all TDE + +4) [sudo] ./create_repo.sh -> build or update your local TDE repository \ No newline at end of file -- cgit v1.2.3