From acdbeb6d9e62570f95a18c512797d57ee2936923 Mon Sep 17 00:00:00 2001 From: "Heiko Schlittermann (HS12-RIPE)" Date: Tue, 1 Nov 2016 23:40:22 +0100 Subject: [PATCH 1/1] Adjustments to be location independent --- Home.md | 12 ++++---- Installation.md | 79 ++++++++++++++++--------------------------------- 2 files changed, 32 insertions(+), 59 deletions(-) diff --git a/Home.md b/Home.md index 7e9e1d4..6904bab 100644 --- a/Home.md +++ b/Home.md @@ -1,16 +1,16 @@ Welcome to the exim-build-farm-client wiki! -The Exim Build Farm Client software is run on machines to build and test the [Exim](http://www.exim.org) MTA software. It uses the current development tree and rebuilds/tests Exim when new commits are added to the [Exim source repository](https://github.com/Exim/exim). The result are recorded and reported back to the [Exim Build Farm](http://eximbuild.mrball.net) website where developers and interested parties can review builds and tests across various distros and versions. +The Exim Build Farm Client software is run on machines to build and test the [Exim](http://www.exim.org) MTA software. It uses the current development tree and rebuilds/tests Exim when new commits are added to the [Exim source repository](https://github.com/Exim/exim). The result are recorded and reported back to the [Exim Build Farm](http://buildfarm.exim.org) website where developers and interested parties can review builds and tests across various distros and versions. -View the [Installation](https://github.com/mrballcb/exim-build-farm-client/wiki/Installation) page for info on how to install the Build Farm Client. There is an additional separate page on preparing your system to [build documentation](https://github.com/mrballcb/exim-build-farm-client/wiki/BuildingDocs). +View the [Installation](./Installation.html) page for info on how to install the Build Farm Client. There is an additional separate page on preparing your system to [build documentation](./BuildingDocs.html). ## Overview -In the default configuration, a full rebuild of Exim will occur when a new commit to the monitored branches occurs or every 7 days, and it will also build and run the test suite. You can choose to have it also build the docs, but there are several extra package requirements for [building the documentation](https://github.com/mrballcb/exim-build-farm-client/wiki/BuildingDocs). +In the default configuration, a full rebuild of Exim will occur when a new commit to the monitored branches occurs or every 7 days, and it will also build and run the test suite. You can choose to have it also build the docs, but there are several extra package requirements for [building the documentation](./BuildingDocs.html). -Review your enabled features in the Makefile and make adjustments to your *build-farm.conf* to enable those features, using either the makefile_set, makefile_add, or makefile_regex. You may also receive requests from the developers to enable specific features on your build. Documentation for enabling these in [build-config.conf](https://github.com/mrballcb/exim-build-farm-client/wiki/BuildConfigConf) is elsewhere. +Review your enabled features in the Makefile and make adjustments to your *build-farm.conf* to enable those features, using either the makefile_set, makefile_add, or makefile_regex. You may also receive requests from the developers to enable specific features on your build. Documentation for enabling these in [build-config.conf](./BuildConfigConf.html) is elsewhere. -The default build configuration will build all branches specified in a text file on the server. Most of the time this will only be HEAD, but occasionally the developers may have an interest in building some experimental code on a different branch. The above configuration will do all of it automatically for you. The *run_cron.sh* will also check the master repo at github and update itself (the Exim Build Farm Client software). +The default build configuration will build all branches specified in a text file on the server. Most of the time this will only be HEAD, but occasionally the developers may have an interest in building some experimental code on a different branch. The above configuration will do all of it automatically for you. The *run_cron* will also check the master repo at of the Exim Build Farm Client Software and update itself. ## Credits -This code is derived from the excellent [PostgreSQL Build Farm](http://www.pgbuildfarm.org/) client and server code. Many thanks to them for a fantastic collection which serves such a useful function. \ No newline at end of file +This code is derived from the excellent [PostgreSQL Build Farm](http://www.pgbuildfarm.org/) client and server code. Many thanks to them for a fantastic collection which serves such a useful function. diff --git a/Installation.md b/Installation.md index 26a7f13..5c637f8 100644 --- a/Installation.md +++ b/Installation.md @@ -1,7 +1,7 @@ # Installing Exim BuildFarm Client So you want to give back to the Exim project but don't know how? Being a member of the Exim BuildFarm is one way you can help. If your distro and version is not on the list that is currently being built, or if your build configuration is drastically different than others with your distro and version, then please consider submitting a request to join the farm. I would also like to point out that the Debian project has excellent Exim coverage on their Experimental [Build Farm](https://buildd.debian.org/status/package.php?p=exim4&suite=experimental). We're not discouraging you from joining the Exim BuildFarm if you're Debian or Debian derivative, but merely want to acknowledge the excellent job the Debian project already does with it. -You will need to submit an [Exim BuildFarm Application](http://eximbuild.mrball.net/cgi-bin/register-form.pl) and the BuildFarm administration has sent you an email with your machine alias (aka _animal_) and secret password, which you will enter in step 9 below. If you just want to run the build farm client and never submit the results, you call simply run everything with the --test option and it will go through all defined build/test steps without saving the state anywhere. In that case, if you accidentally run it without --test, the build process will still work, but the build state will be locally recorded, and when the stated is submitted to the server, it will reject the feedback because it's from an unknown _animal_. +You will need to submit an [Exim BuildFarm Application](http://buildfarm.exim.org/cgi-bin/register-form.pl) and the BuildFarm administration has sent you an email with your machine alias (aka _animal_) and secret password, which you will enter in step 9 below. If you just want to run the build farm client and never submit the results, you call simply run everything with the --test option and it will go through all defined build/test steps without saving the state anywhere. In that case, if you accidentally run it without --test, the build process will still work, but the build state will be locally recorded, and when the stated is submitted to the server, it will reject the feedback because it's from an unknown _animal_. If you want to run more than one instance of the build farm to test multiple different configurations, such as one with OpenSSL and one with GnuTLS, make a copy of the *build-farm.conf* template for each subsequent animal simply start at step 9. More details on the extra configuration required to keep it separate from the main build are at the end of this page. @@ -11,15 +11,18 @@ If you want to run more than one instance of the build farm to test multiple dif 3. Create a user that will run the Build Farm. I suggest the user **farm**. Due to some assumptions that are made in the test suite, the group name must also be **farm**. It is ok if you make it a secondary group or the primary group. You also must assign a geckos name, or some tests will fail. I suggest "Exim BuildFarm". An example command that should work for most distributions: `useradd -m -c 'Exim BuildFarm' farm` 4. Add **exim** as a secondary group to the **farm** user. (You should **NOT** add **farm** as a secondary group to the exim user; this will cause test errors.) Here is a sample user/group configuration: -
farm@buildclient:~/code$ id farm
-uid=1001(farm) gid=1001(farm) groups=1001(farm),137(exim)
-farm@buildclient:~/code$ id exim
-uid=480(exim) gid=137(exim) groups=137(exim)
+ + ```` + farm@buildclient:~/code$ id farm + uid=1001(farm) gid=1001(farm) groups=1001(farm),137(exim) + farm@buildclient:~/code$ id exim + uid=480(exim) gid=137(exim) groups=137(exim) + ```` 5. Configure sudo to allow user **farm** to elevate to root without a password. Run `visudo` and add: `farm ALL=(ALL) NOPASSWD: ALL` 6. Change to the **farm** user. 7. Checkout the client code: -`git clone https://github.com/mrballcb/exim-build-farm-client.git code` +`git clone git://git.exim.org/buildfarm-client.git code` This will create the repo checkout in the directory *~/code/*. 8. Change into the *~/code/* directory and copy the config template to the default filename: `cp build-farm.conf.template build-farm.conf` @@ -29,52 +32,22 @@ This will create the repo checkout in the directory *~/code/*. **CAUTION:** Do not ever change anything in the git repo under that is created in *$HOME/buildfarm/HEAD/exim/*. It will cause the build to fail in the Git-Checkout phase. 11. Directory permissions must be lax enough for the **exim** user running the test suite to be able to access the files that user farm has checked out. One recommendation that we know works is to make the **farm** user's home directory be both group and world readable and group and world searchable: `chmod og+rx $HOME`. Additionally, every subdirectory between */* and *$HOME/buildfarm* should have this permissive access. Some users have reported they must set *$HOME/buildfarm* to 0775 for the test suite to run. What do these open permissions fix? One thing that tighter permissions will break, for example, is test 0005, because the sudo'd exim process in the test suite will be unable to read that test config file. -12. Make sure that your umask is 0022: `umask 0022`. This also will need to be set in any script you call the run_build.pl script. -13. Test the configure process by running `./run_build.pl --test --verbose=2 --only-steps=configure`. If there are errors, you'll need to correct them until the process succeeds (ends with OK). You can repeat this as many times as necessary because test mode does not store the status of the git repo or the status of each stage of the build. -14. Test the build process by running `./run_build.pl --test --verbose=2 --only-steps=configure,make`. If there are build errors, make adjustments, install additional packages, etc, and repeat the test. -15. Test the test suite by running `./run_build.pl --test --verbose=2 --only-steps=configure,make,test --override range_num_tests='1 2'`. There will be one error on a standard system: the Exim test suite uses ifconfig to obtain the IP address, and this binary is frequently not in a regular user's path. Temporarily fix this by running `export PATH="/usr/sbin:/sbin:$PATH"` to prepend the probably directories where that command is located to your regular user path. (Instructions on how to do this automatically in the cron job are below). If there are build errors when building the test suite, or runtime errors trying to run the test suite, you may need to install additional packages (you shouldn't though). -16. If you enabled the documentation building process in the *build-farm.conf*, then you can try to build it with `./run_build.pl --test --verbose=2 --only-steps=configure,make-doc`. For documentation generation to succeed, it will require extra packages to be installed to support xml, xslt, pdf, and a few other things. Please see the comments on the [Building Documentation](https://github.com/mrballcb/exim-build-farm-client/wiki/BuildingDocs) page. +12. Make sure that your umask is 0022: `umask 0022`. This also will need to be set in any script you call the run_build script. +13. Test the configure process by running `./run_build --test --verbose=2 --only-steps=configure`. If there are errors, you'll need to correct them until the process succeeds (ends with OK). You can repeat this as many times as necessary because test mode does not store the status of the git repo or the status of each stage of the build. +14. Test the build process by running `./run_build --test --verbose=2 --only-steps=configure,make`. If there are build errors, make adjustments, install additional packages, etc, and repeat the test. +15. Test the test suite by running `./run_build --test --verbose=2 --only-steps=configure,make,test --override range_num_tests='1 2'`. There will be one error on a standard system: the Exim test suite uses ifconfig to obtain the IP address, and this binary is frequently not in a regular user's path. Temporarily fix this by running `export PATH="/usr/sbin:/sbin:$PATH"` to prepend the probably directories where that command is located to your regular user path. (Instructions on how to do this automatically in the cron job are below). If there are build errors when building the test suite, or runtime errors trying to run the test suite, you may need to install additional packages (you shouldn't though). +16. If you enabled the documentation building process in the *build-farm.conf*, then you can try to build it with `./run_build --test --verbose=2 --only-steps=configure,make-doc`. For documentation generation to succeed, it will require extra packages to be installed to support xml, xslt, pdf, and a few other things. Please see the comments on the [Building Documentation](./BuildingDocs.html) page. 17. If you can get past each of these steps, then your build farm system meets the minimum requirements. 18. The official process can be kicked off by running `/home/farm/code/run_cron --run-all`. This will run the default build configuration, keep track of the git repository status, and upload the build results to the server. -19. Once that command runs with no complaints, add it to the **farm** user crontab. You can run it at whatever frequency you choose, I suggest 2 hours. If a previous instantiation is still running, the script will detect the lockfile and exit so as not to step on each other. I had a problem running the *run_cron.sh* script in that cron gives a highly sanitized path to the script when it runs it. I made a second script to call the first one so I could insert path elements in that were needed: -
$ more /home/farm/bin/build_farm.sh
-#!/bin/bash
-export PATH="/usr/local/bin:/sbin:/usr/sbin:$PATH"
-$HOME/code/run_cron.sh --run-all $@
-
-Then I make my cronjob call: `6 */2 * * * $HOME/bin/build_farm.sh` .... but there was a big problem ... -20. My cronjob ran great for a couple weeks. Then another problem popped up running the cron job in that the test portion suddenly started failing with an odd error: -
** runtest error: Failed to open /dev/tty: No such device or address
-This is not a sudo issue. This is happening because the cron daemon does not give a tty to the cronjob that it starts. (How the heck did it ever work?) The runtest script needs a tty in normal operation. To fix this, you have two options: a) scripting magic to start screen and run the build inside of screen b) an old trick of ssh'ing to localhost with the *-tt* option to force allocation of a local tty when starting the *build_farm.sh* script. -

-**Option A**: To use the screen based solution, I made a simple script:
-

-$ more $HOME/bin/build_farm_screen.sh
-#!/bin/bash
-TITLE="BuildFarm"
-COUNT=`screen -list | grep $TITLE | wc -l`
-if [ $COUNT -eq 0 ]; then
-  echo "Screen not running, start a new one"
-  screen -d -m -S $TITLE
-fi
-screen -S $TITLE -p 0 -X stuff 'for DACONF in $HOME/code/build-farm.conf
-  do
-    $HOME/bin/build_farm.sh --config=$DACONF $@ 2>/dev/null
-  done
-'
-

-Note that the lone single quote on the last line is required, and it must be on the next line, not at the end of the previous line because it emits a newline, causing the command to be executed in the screen session.  Once that works properly, then the cron command changes to:
-
`6 */2 * * * $HOME/bin/build_farm_screen.sh`

-**Option B**: To use the ssh based solution, you need to configure the **farm** user to use key-based authentication with its own key.  Assuming you have not generated an ssh key yet:
-
# Press Enter to use defaults for all questions in
-# next command, including no password
-ssh-keygen -t dsa
-cat .ssh/id_dsa.pub >> .ssh/authorized_keys
-# Do the following command once to accept the new host key
-ssh farm@localhost

-Once that works properly, then the cron command changes to:
-
`6 */2 * * * ssh -tt farm@localhost $HOME/bin/build_farm.sh`

-21. The default tests that are run are a limited set, from 1 to 999.  This covers basic Exim functions, but does not exercise a lot of the advanced functions.  Once a few cronjob runs complete successfully, increase the range of tests to run.  Edit the *build-farm.conf* file and change the `range_num_tests => '1 999',` to `range_num_tests => '1 5999',` and it will run more advanced tests.
+19. Once that command runs with no complaints, add it to the **farm** user crontab.  You can run it at whatever frequency you choose, I suggest 2 hours.  If a previous instantiation is still running, the script will detect the lockfile and exit so as not to step on each other.  I had a problem running the *run_cron* script in that cron gives a highly sanitized path to the script when it runs it.  I made a second script to call the first one so I could insert path elements in that were needed:
+
+    ````
+    $ more /home/farm/bin/build_farm.sh
+    #!/bin/bash
+    export PATH="/usr/local/bin:/sbin:/usr/sbin:$PATH"
+    $HOME/code/run_cron.sh --run-all $@
+    ````
+20. The default tests that are run are a limited set, from 1 to 999.  This covers basic Exim functions, but does not exercise a lot of the advanced functions.  Once a few cronjob runs complete successfully, increase the range of tests to run.  Edit the *build-farm.conf* file and change the `range_num_tests => '1 999',` to `range_num_tests => '1 5999',` and it will run more advanced tests.
 
 
 ## Multiple build clients on one machine
@@ -87,6 +60,6 @@ As mentioned above, you can start at step 9 for each additional build you'll do
 5. Add a new cron job for this second build farm client in the same way you chose to do the first one, just make sure to specify the --config option with it.
 
 ## Further documentation
-* Details of options in [build-farm.conf](https://github.com/mrballcb/exim-build-farm-client/wiki/BuildConfigConf)
-* Details of [potential testing commandlines](https://github.com/mrballcb/exim-build-farm-client/wiki/TestingBuilds)
-* Details of [building documentation](https://github.com/mrballcb/exim-build-farm-client/wiki/BuildingDocs)
+* Details of options in [build-farm.conf](./BuildConfigConf.html)
+* Details of [potential testing commandlines](./TestingBuilds.html)
+* Details of [building documentation](./BuildingDocs.html)
-- 
2.30.2