猿教程 Logo

如何在Ubuntu 16.04上安装Buildbot (How To Install Buildbot on Ubuntu 16.04)


Introduction

Buildbot is a Python-based continuous integration system for automating software build, test, and release processes. It uses Python's Twisted library to handle asynchronous communication between a buildmaster and one or more workers to facilitate testing builds on multiple platforms. Buildbot is highly configurable and makes few assumptions about how the build process should work, making it suitable for complex build processes or projects that require their tools to grow with the unique needs of the project.

In this tutorial, we'll install and configure a Buildbot buildmaster and worker on the same machine.


Prerequisites

To follow this tutorial, you will need:

  • An Ubuntu 16.04 server with at least 1 GB of RAM, configured with a non-root sudo user and a firewall as described in the Ubuntu 16.04 initial server setup guide.

When the server is set up, you're ready to follow along.


Step 1 — Installing Buildbot

The Buildbot project recommends using the Python Package Index, pip, to install Buildbot in order to get the most recent version, which is often several releases ahead of what is available in the Ubuntu packages.

We’ll begin as our sudo user, and use apt-get update to ensure we have the latest list of packages:

$ sudo apt-get update
sudo apt-get update

Then, we’ll install pip itself:

$ sudo apt-get install python-pip
sudo apt-get install python-pip

Once pip is available, we'll use it to install the Buildbot bundle, which includes the master and worker as well as other dependencies, including those required by the web interface. Pip creates .cache files in the home directory of the user who executes it. We'll use sudo's -H flag to put these files in the right location:

$ sudo -H pip install 'buildbot[bundle]'
sudo -H pip install 'buildbot[bundle]'

Depending on the speed of your server, this may take a little bit to complete. The end of the output from a successful installation should look something like the following:

Output
       Successfully installed Automat-0.5.0 Jinja2-2.9.6 MarkupSafe-1.0 
PyJWT-1.5.0 Tempita-0.5.2 Twisted-17.1.0 attrs-16.3.0 autobahn-17.5.1 
buildbot-0.9.6 buildbot-console-view-0.9.6 buildbot-waterfall-view-0.9.6 
buildbot-worker-0.9.6 buildbot-www-0.9.6 constantly-15.1.0 
decorator-4.0.11 future-0.16.0 incremental-16.10.1 pbr-3.0.0 
python-dateutil-2.6.0 six-1.10.0 sqlalchemy-1.1.9 
sqlalchemy-migrate-0.11.0 sqlparse-0.2.3 txaio-2.7.1 zope.interface-4.4.0

It may also display a recommendation to upgrade pip itself:

Output
       . . .
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

While this won't affect our Buildbot installation, we'll take a moment to upgrade to pip's latest release:

$ sudo -H pip install --upgrade pip
sudo -H pip install --upgrade pip
Output
       Collecting pip
 Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
   100% |████████████████████████████████| 1.3MB 768kB/s
Installing collected packages: pip
 Found existing installation: pip 8.1.1
   Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-9.0.1

Finally, we’ll verify the installation of Buildbot by checking the version:

$ buildbot --version
buildbot --version
Output
       Buildbot version: 0.9.6
Twisted version: 17.1.0

In the tutorial prerequisite, we configured a UFW firewall to allow SSH traffic only. We'll verify the status:

$ sudo ufw status
sudo ufw status
Output
       Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)

Buildbot uses port 8010 for the web interface, which isn't allowed, so we'll open that now.

$ sudo ufw allow 8010
sudo ufw allow 8010

Then, we'll add a dedicated system user and group to run the Buildbot services:

$ sudo addgroup --system buildbot
$ sudo adduser buildbot --system --ingroup buildbot --shell /bin/bash
sudo addgroup --system buildbot

        sudo adduser buildbot --system --ingroup buildbot --shell /bin/bash

Finally, we'll log in as our new user to install Buildbot:

$ sudo --login --user buildbot
sudo --login --user buildbot

This will log us in as the buildbot user and place us in the /home/buildbot directory, where we'll configure our master and worker:


Step 2 — Configuring the Master

We'll use the the buildbot command create-master followed by the value of the base directory:

buildbot$ buildbot create-master master
buildbot create-master master
Output
       mkdir /home/buildbot/master
creating /home/buildbot/master/master.cfg.sample
creating database (sqlite:///state.sqlite)
buildmaster configured in /home/buildbot/master

Next we'll copy master.cfg.sample to master.cfg and leave the original in place for reference:

buildbot$ cp ~/master/master.cfg.sample ~/master/master.cfg
cp ~/master/master.cfg.sample ~/master/master.cfg

Then, we'll edit the file to allow us to reach the web interface from a local machine.

buildbot$ nano ~/master/master.cfg
nano ~/master/master.cfg

In order to access the web interface from a desktop or other device, we'll change the buildbotURL from localhost to the IP address or domain name of the server. Other important configuration values are set in master.cfg, but we're going to keep the rest of the defaults for now.

Near the bottom of the file, locate the buildbotURL line and replace localhost with the IP address or domain name of your site:

c['buildbotURL'] = "http://IP_or_site_domain:8010/"
Note: The master.cfg also pre-defines a worker in the "Workers" section. ~/master/master.cfg . . . ####### WORKERS # The 'workers' list defines the set of recognized workers. Each element is # a Worker object, specifying a unique worker name and password. The same # worker name and password must be configured on the worker. c['workers'] = [worker.Worker("example-worker", "pass")] . . . Later in the tutorial, we’ll create a worker with these credentials.

When you’ve modified the 'buildbotURL', save and exit the file, then start the master:

buildbot$ buildbot start ~/master
buildbot start ~/master
Output
       Following twistd.log until startup finished..
The buildmaster appears to have (re)started correctly.

Finally, let’s visit the site in a web browser on port 8010 at the buildbotURL we configured:

http://IP_or_site_domain:8010/

Now that we have the master running and have verified that we can access the web interface, we’ll create the example worker.


Step 3 — Configuring a Worker

The relationship between a master and a worker is established when a worker's name and password in the master.cfg file matches the name and password of a worker configured to use the master.

In this step, we'll create and configure a worker by calling buildbot-worker's create-worker command and passing in four settings:

  • worker is the name of the directory where the worker's settings will be stored
  • localhost is the address where the worker's master is running
  • example-worker is the name of the worker and must uniquely identify the worker in the ~/master/master.cfg file.
  • pass is the worker's password and this password must match the value in ~master/master.cfg.
buildbot$ buildbot-worker create-worker worker localhost example-worker pass
buildbot-worker create-worker worker localhost example-worker pass
Output
       mkdir /home/buildbot/worker
mkdir /home/buildbot/worker/info
Creating info/admin, you need to edit it appropriately.
Creating info/host, you need to edit it appropriately.
Not creating info/access_uri - add it if you wish
Please edit the files in /home/buildbot/worker/info appropriately.
worker configured in /home/buildbot/worker

When the worker first connects, it will send files in the info directory to the buildmaster where it runs. They'll be displayed in the web interface to give developers more information about test failures.

We'll configure these now. First, open the file containing the administrator's e-mail, delete the example line, Your Name Here and replace it with your name and email address.

buildbot$ nano ~/worker/info/admin
nano ~/worker/info/admin
Sammy Shark 

When you're done, save and exit the file.

The info/host file, by convention, provides the OS, version, memory size, CPU speed, versions of relevant libraries installed, and finally the Buildbot version running on the worker.

Open the file and paste in the relevant information, updating the sample content as needed for your system:

buildbot$ nano ~/worker/info/host
nano ~/worker/info/host

Update the information you use to reflect the specifics of your system:

Ubuntu 16.04.2 2GB Droplet - Buildbot version: 0.9.6 - Twisted version: 17.1.0

When you're done, save and exit. Finally, start the worker:

buildbot$ buildbot-worker start worker
buildbot-worker start worker
Output
       Following twistd.log until startup finished..
The buildbot-worker appears to have (re)started correctly.

Now that both the master and worker are configured and running, we'll execute a test build.


Step 4 — Running a Test Build

To run a test build, we'll open the "Builds" menu in the web interface, then select "Workers". The example worker and the information we set in info/admin and info/host should be displayed. From here, we can click on the default builder, "runtests" to force a build.

The "runtests" screen will have little information until the first build request is made. We'll force one now by clicking "force" button in the upper right of the screen:

This will bring up a dialog which allows you to enter information about the forced build.

For this test build, we'll leave the fields blank and click the "Start Build" button in the popup window. Note that if you do enter a value in the "Your name" field, it must contain a valid email address.

In a few seconds, the build should complete successfully:

You can explore the details of each step in the build by clicking the number or arrow next to its name:

You may have noticed that we weren't required to log in to execute this build. By default, anyone can access administrative functions, so before we finish, we'll take a moment to lock that down and create a user account. You can learn more about the available options ins [Buildbot's Authorization documentation].(http://docs.buildbot.net/current/developer/authz.html).

Open the master.cfg file again:

buildbot$ nano ~/master/master.cfg
nano ~/master/master.cfg

At the bottom of the file, add the following lines, changing the username and password.

. . .
c['www']['authz'] = util.Authz(
       allowRules = [
           util.AnyEndpointMatcher(role="admins")
       ],
       roleMatchers = [
           util.RolesFromUsername(roles=['admins'], usernames=['Sammy'])
       ]
)
c['www']['auth'] = util.UserPasswordAuth([('Sammy','Password')])
buildbot$ buildbot restart ~/master
buildbot restart ~/master

When we reload the web interface, a link should appear in the upper-right that says Anonymous and access to the administrative functions is no longer available.

We'll test the credentials we just added by clicking "Anonymous", which will bring up a login box where we can enter the username and password we configured. When we log in, we should see that while "Anonymous" no longer has access to start a build, our "Sammy" user does.

At this point, our installation of Buildbot is complete and we've taken a minimal step to secure the interface. The username and password, however, are being transmitted in plain text. We recommend as a next step and before using Buildbot in earnest that you secure the web interface with a reverse proxy.


Conclusion

In this tutorial, we've installed and configured the Buildbot master and a local Buildbot worker on the same machine. If you're evaluating Buildbot, you might want to take the project's Quick Tour.

Otherwise, proceed to the next tutorial, How To Create Systemd Unit Files for Buildbot, to allow the server's init system to manage the Buildbot processes.

介绍

Buildbot是一个基于Python的持续集成系统,用于自动化软件构建,测试和发布流程。 它使用Python的Twisted库来处理构建主管和一个或多个工作人员之间的异步通信,以便于在多个平台上测试构建。 Buildbot是高度可配置的,并且对构建过程应如何工作做出很少的假设,使其适合复杂的构建过程或需要其工具随项目的独特需求而增长的项目。

在本教程中,我们将在同一台机器上安装和配置Buildbot buildmaster和worker。


先决条件

要遵循本教程,您将需要:

  • 具有至少1 GB RAM的Ubuntu 16.04服务器,配置有非根sudo用户和防火墙,如Ubuntu 16.04初始服务器设置指南中所述。

当服务器设置完成后,您就可以随时随地。


步骤1 - 安装Buildbot

Buildbot项目建议使用Python软件包索引pip来安装Buildbot以获取最新版本,这通常是Ubuntu软件包中提供的几个版本。

我们将作为我们的sudo用户开始,并使用apt-get update来确保我们拥有最新的软件包列表:

$ sudo apt-get update
sudo apt-get update

然后,我们将安装pip本身:

$ sudo apt-get install python-pip
sudo apt-get install python-pip

一旦可用,我们将使用它来安装Buildbot软件包,其中包括master和worker以及其他依赖项,包括Web界面所需的依赖项。 Pip在执行它的用户的主目录中创建.cache文件。 我们将使用sudo的-H标志将这些文件放在正确的位置:

$ sudo -H pip install 'buildbot[bundle]'
sudo -H pip install 'buildbot[bundle]'

根据服务器的速度,这可能需要一点时间才能完成。 成功安装的输出结束应如下所示:

Output
       Successfully installed Automat-0.5.0 Jinja2-2.9.6 MarkupSafe-1.0 
PyJWT-1.5.0 Tempita-0.5.2 Twisted-17.1.0 attrs-16.3.0 autobahn-17.5.1 
buildbot-0.9.6 buildbot-console-view-0.9.6 buildbot-waterfall-view-0.9.6 
buildbot-worker-0.9.6 buildbot-www-0.9.6 constantly-15.1.0 
decorator-4.0.11 future-0.16.0 incremental-16.10.1 pbr-3.0.0 
python-dateutil-2.6.0 six-1.10.0 sqlalchemy-1.1.9 
sqlalchemy-migrate-0.11.0 sqlparse-0.2.3 txaio-2.7.1 zope.interface-4.4.0

它也可能会显示一个升级点的建议:

Output
       . . .
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

虽然这不会影响我们的Buildbot安装,我们会花一点时间升级到pip的最新版本:

$ sudo -H pip install --upgrade pip
sudo -H pip install --upgrade pip
Output
       Collecting pip
 Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
   100% |████████████████████████████████| 1.3MB 768kB/s
Installing collected packages: pip
 Found existing installation: pip 8.1.1
   Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-9.0.1

最后,我们将通过检查版本来验证Buildbot的安装:

$ buildbot --version
buildbot --version
Output
       Buildbot version: 0.9.6
Twisted version: 17.1.0

在教程的前提下,我们配置了一个UFW防火墙,仅允许SSH流量。 我们将验证状态:

$ sudo ufw status
sudo ufw status
Output
       Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)

Buildbot使用端口8010作为Web界面,这是不允许的,所以我们现在打开它。

$ sudo ufw allow 8010
sudo ufw allow 8010

然后,我们将添加专门的系统用户和组来运行Buildbot服务:

$ sudo addgroup --system buildbot
$ sudo adduser buildbot --system --ingroup buildbot --shell /bin/bash
sudo addgroup --system buildbot

        sudo adduser buildbot --system --ingroup buildbot --shell /bin/bash

最后,我们将以新用户身份登录Buildbot:

$ sudo --login --user buildbot
sudo --login --user buildbot

这将作为buildbot用户登录,并将我们放在/ home / buildbot目录中,我们将配置我们的主人和工作人员:


步骤2 - 配置主站

我们将使用buildbot命令create-master,后跟基本目录的值:

buildbot$ buildbot create-master master
buildbot create-master master
Output
       mkdir /home/buildbot/master
creating /home/buildbot/master/master.cfg.sample
creating database (sqlite:///state.sqlite)
buildmaster configured in /home/buildbot/master

接下来,我们将master.cfg.sample复制到master.cfg,并将原始文件保留在适当位置以供参考:

buildbot$ cp ~/master/master.cfg.sample ~/master/master.cfg
cp ~/master/master.cfg.sample ~/master/master.cfg

然后,我们将编辑文件,以便我们从本地机器访问Web界面。

buildbot$ nano ~/master/master.cfg
nano ~/master/master.cfg

为了从桌面或其他设备访问Web界面,我们将将buildbotURL从localhost更改为服务器的IP地址或域名。 master.cfg中设置了其他重要的配置值,但现在我们将保留其余的默认值。

在文件底部,找到buildbotURL行,并将localhost替换为您网站的IP地址或域名:

c['buildbotURL'] = "http://IP_or_site_domain:8010/"
注意:master.cfg还在“Workers”部分中预定义了一个worker。 〜/ master / master.cfg。。。 ####### 工人 #“工人名单”定义了一组公认的工人。 每个元素都是 #一个Worker对象,指定唯一的工作人员名称和密码。 一样 必须在worker上配置#worker名称和密码。 c ['workers'] = [worker.Worker(“example-worker”,“pass”)] 。。。 在本教程的后面,我们将使用这些凭据创建一个工作。

修改'buildbotURL'后,保存并退出该文件,然后启动主控:

buildbot$ buildbot start ~/master
buildbot start ~/master
Output
       Following twistd.log until startup finished..
The buildmaster appears to have (re)started correctly.

最后,我们通过我们配置的buildbotURL在8010端口的Web浏览器中访问该站点:

HTTP:// IP_or_site_domain:8010/

现在我们已经掌握了主程序,并且已经验证了我们可以访问Web界面,我们将创建一个示例worker。


步骤3 - 配置工作人员

当master.cfg文件中的工作人员的名称和密码与配置为使用主服务器的工作人员的名称和密码匹配时,建立主服务器和工作人员之间的关系。

在此步骤中,我们将通过调用buildbot-worker的create-worker命令并传递四个设置来创建和配置一个worker:

  • worker是将存储工作者设置的目录的名称
  • localhost是工作人员的主人所在的地址
  • example-worker是工作者的名称,并且必须唯一标识〜/ master / master.cfg文件中的worker。
  • pass是工作人员的密码,此密码必须与〜master / master.cfg中的值相匹配。
buildbot$ buildbot-worker create-worker worker localhost example-worker pass
buildbot-worker create-worker worker localhost example-worker pass
Output
       mkdir /home/buildbot/worker
mkdir /home/buildbot/worker/info
Creating info/admin, you need to edit it appropriately.
Creating info/host, you need to edit it appropriately.
Not creating info/access_uri - add it if you wish
Please edit the files in /home/buildbot/worker/info appropriately.
worker configured in /home/buildbot/worker

当工作人员第一次连接时,它会将信息目录中的文件发送到运行的buildmaster。 它们将显示在Web界面中,为开发人员提供有关测试失败的更多信息。

我们现在将配置它们。 首先,打开包含管理员电子邮件的文件,删除示例行,您的姓名这里,并将其替换为您的姓名和电子邮件地址。

buildbot$ nano ~/worker/info/admin
nano ~/worker/info/admin
Sammy Shark 

完成后,保存并退出文件。

info / host文件按惯例提供操作系统,版本,内存大小,CPU速度,安装的相关库的版本,最后在工作程序上运行的Buildbot版本。

打开文件并粘贴相关信息,根据需要更新示例内容:

buildbot$ nano ~/worker/info/host
nano ~/worker/info/host

更新您用于反映系统细节的信息:

Ubuntu 16.04.2 2GB Droplet - Buildbot version: 0.9.6 - Twisted version: 17.1.0

完成后,保存并退出。 最后,启动工人:

buildbot$ buildbot-worker start worker
buildbot-worker start worker
Output
       Following twistd.log until startup finished..
The buildbot-worker appears to have (re)started correctly.

既然master和worker都配置和运行,我们将执行一个测试构建。


步骤4 - 运行测试版本

要运行测试版本,我们将在Web界面中打开“Builds”菜单,然后选择“Workers”。 应该显示示例工作人员和我们在info / admin和info / host中设置的信息。 从这里,我们可以点击默认构建器“runtests”来强制构建。

直到建立第一个请求,“runtests”屏幕才会有很少的信息。 现在我们将通过点击屏幕右上方的“强制”按钮强制一个:

这将弹出一个对话框,允许您输入有关强制构建的信息。

对于这个测试版本,我们将把这些字段留空,点击弹出窗口中的“开始生成”按钮。 请注意,如果您在“您的姓名”字段中输入值,则必须包含有效的电子邮件地址。

几秒钟内,构建应成功完成:

您可以通过点击其名称旁边的数字或箭头来浏览构建中每个步骤的详细信息:

您可能已经注意到,我们不需要登录才能执行此构建。 默认情况下,任何人都可以访问管理功能,所以在完成之前,我们将花一点时间将其锁定并创建一个用户帐户。 您可以在[Buildbot的授权文档](http://docs.buildbot.net/current/developer/authz.html)中了解有关可用选项的更多信息。

再次打开master.cfg文件:

buildbot$ nano ~/master/master.cfg
nano ~/master/master.cfg

在文件的底部,添加以下行,更改用户名和密码。

. . .
c['www']['authz'] = util.Authz(
       allowRules = [
           util.AnyEndpointMatcher(role="admins")
       ],
       roleMatchers = [
           util.RolesFromUsername(roles=['admins'], usernames=['Sammy'])
       ]
)
c['www']['auth'] = util.UserPasswordAuth([('Sammy','Password')])
buildbot$ buildbot restart ~/master
buildbot restart ~/master

当我们重新加载Web界面时,右上方会出现一个链接,表示匿名,对管理功能的访问不再可用。

我们将通过单击“匿名”来测试我们刚添加的凭据,这将打开一个登录框,我们可以输入我们配置的用户名和密码。 当我们登录时,我们应该看到,当“匿名”不再有权启动构建时,我们的“Sammy”用户就可以了。

在这一点上,我们安装的Buildbot是完整的,我们已经采取了一个最小的步骤来保护界面。 然而,用户名和密码正在以纯文本形式发送。 我们建议作为下一步,在使用Buildbot认真地使用反向代理来保护Web界面之前。


结论

在本教程中,我们已经在同一台机器上安装并配置了Buildbot master和本地Buildbot worker。 如果您正在评估Buildbot,您可能需要参加项目的快速浏览。

否则,请继续下一个教程“如何为Buildbot创建Systemd单元文件”,以允许服务器的init系统管理Buildbot进程。