EDITING BOARD
RO
EN
×
▼ BROWSE ISSUES ▼
Issue 46

Configuration management with Chef

Andrei Ivanov
DevOps Engineer @Endava
PROGRAMMING
Building large scale applications is a very rewarding thing. However, the maintenance and scaling process/part of such apps can be quite challenging sometimes. The applications and the operating system can each have different, possible configurations to fit the environment they are deployed in. Moreover, they have to be consistent across the development, the QA and the production stages, and point out to the information relevant at each stage. For example, the information contained in a database URL will be different than that of a URL used in the development stage, which is obviously different than the one used in production. Some of this information is not available for all the people involved in the development process, because they should not care about it either. There are many configuration management tools to configure applications across environments and one of them is *Chef*. The definition on official website reads: "Chef is a configuration management tool written in Ruby and Erlang. It uses a pure-Ruby, domain-specific language (DSL) for writing system configuration "recipes"." ### Chef concepts Some of **Chef**'s essential components are: \- **Nodes** - A node can be anything from a physical machine, a virtual one, a network device that is managed by **Chef** \- **Chef server** - This is the central part of the product, and it is the place where the configuration data is stored. The nodes use **Chef**-client to get the proper configuration and deploy it accordingly. \- **Workstation** - This is a computer that is used to configure **Chef**-related tasks via a set of command line tools, a computer that interacts with the **Chef** server and nodes, also allowing developers to write cookbooks. \- **Chef client** - This is an agent that sits on every node that **Chef** is managing and which will handle the actual deployment part. \- **Cookbooks** - These can be defined as the fundamental unit of configuration and policy distribution. A cookbook defines a scenario and contains everything required to support that scenario. \- **[Supermarket](https://supermarket.chef.io)** - This is the community site for cookbooks. Some cookbooks in the supermarket are developed and maintained by **Chef**. For some simple tasks, like installing **Apache** or **iptables**, it is better not to reinvent the wheel and get a tested, maintained ready to go cookbook which you can find in the supermarket. \- **Others** (compliance, analytics, delivery) - These are just some other components, used in **Chef**, which we are going to leave aside for now. The **Chef** architecture overview: [](images/articles/tsm46/c1.png) ### The practical scenario In this article, we will get straight into the practical part, by setting up a very simple scenario. We will take 2 virtual machines in the **aws** cloud. Then, with the help of a basic **Chef** cookbook, we will be deploying a simple **http** server and a static webpage, both of them displaying some server and environment information. Then, we will update the cookbook to dynamically search the environment and generate an html image tag pointing to the http location of the second web server. At the end we will simulate an environment change (from dev to prod). Sounds pretty simple, doesn't it? ### Prerequisites: This example will be done using: \- **Linux**, **Vagrant**, **virtualbox/aws** \- **Chef** server, **chefdk** installed on the workstation ### Implementation First, let's create a Chef environment called **chefdemodev** in which we're going to deploy some services. ``` knife environment create chefdemodev ``` When issuing the previous command, **Chef** will be filling in some defaults for us. Note that all the configuration in **Chef** is expressed in **json** format. ``` { "name": "chefdemodev", "description": "", "cookbook_versions": { }, "json_class": "Chef::Environment", "chef_type": "environment", "default_attributes": { }, "override_attributes": { } } ``` For this example we will provision 2 machines in **aws** using **Vagrant**. **Vagrant** will undertake the task of setting up these instances, everything from talking to the cloud provider to setting up the **Chef** bootstrapping (installing **Chef**-client on the node and adding that node to the **Chef** server). The relevant **Vagrant** file contents are listed below. Notice that the **Chef** environment has the same value for both instances. ``` ENVIRONMENT = "chefdemodev" VMNAME1 = "webserver01" VMNAME2 = "webserver02" VIRTUAL_MACHINES={ "#{ENVIRONMENT}-#{VMNAME1}" => { :hostname => "#{ENVIRONMENT}-#{VMNAME1}.altidev.net", :subnetid => "#{SUBNET}", :instancetype => "#{VMTYPE}", :ami => "#{VMAMI}", #centos image :securitygroup => "#{SECGROUP}", :role => ['all'], :environment => "#{ENVIRONMENT}", :recipe => ['iptables::disabled'], :tags => { :owner => "#{AWS_TAG_OWNER}", :group => "#{AWS_TAG_GROUP}", :cost_center => "#{AWS_TAG_COST_CENTER}", :environment => "#{ENVIRONMENT}" } }, "#{ENVIRONMENT}-#{VMNAME2}" => { :hostname => "#{ENVIRONMENT}-#{VMNAME2}.altidev.net", :subnetid => "#{SUBNET}", :instancetype => "#{VMTYPE}", :ami => "#{VMAMI}", #centos image :securitygroup => "#{SECGROUP}", :role => ['all'], :environment => "#{ENVIRONMENT}", :recipe => ['iptables::disabled'], :tags => { :owner => "#{AWS_TAG_OWNER}", :group => "#{AWS_TAG_GROUP}", :cost_center => "#{AWS_TAG_COST_CENTER}", :environment => "#{ENVIRONMENT}" } } } ``` We are going to bring those 2 vm's up: ``` vagrant up Bringing machine 'chefdemodev-webserver01' up with 'aws' provider... Bringing machine 'chefdemodev-webserver02' up with 'aws' provider... ``` We can also verify the status of those vm's: ``` vagrant status Current machine states: chefdemodev-webserver01 running (aws) chefdemodev-webserver02 running (aws) ``` Let's come back to the **Chef** part. Having those machines provisioned for us by **Vagrant**, we can also see them added to the **Chef** server, in the **chefdemodev** environment: ``` knife node list --environment=chefdemodev chefdemodev-webserver01.altidev.net chefdemodev-webserver02.altidev.net ``` Next, we are going to set up the **Chef** environment with some basic settings, so that we can deploy a basic **Apache** service. ``` Knife environment edit chefdemodev ``` And we will add ``` "apache": { "default_site_enabled": true } ``` Then, we are going to create our cookbook called **chefdemoweb**. In order to do this, we are going to use a tool called **Berkshelf**. You can read more about **Berkshelf** at , since I will not go into the details here. **Berkshelf** is a dependency manager for cookbooks and it will also create a basic cookbook structure for us. ``` berks cookbook chefdemoweb create chefdemoweb/files/default create chefdemoweb/templates/default …… ``` If we inspect the **chefdemoweb** directory we'll see that **Berkshelf** has created this structure for us: attributes Berksfile CHANGELOG.md chefignore files default Gemfile libraries LICENSE metadata.rb providers README.md recipes default.rb resources templates default test integration default Thorfile Vagrantfile Next, we'll add the following code to the default recipe ( recipe/default.rb), note that we are using a community cookbook named *apache2* to do the heavy lifting for us. ``` include_recipe 'apache2' template "#{node['apache']['docroot_dir']}/index.html" do source "index.html.erb" owner 'root' group node['apache']['root_group'] mode '0644' variables( :environment => node.chef_environment ) end service "httpd" do action :restart end ``` We will also add a template for our webpage in templates/default/index.html.erb ``` Chefdemo

Hi

This is <%= node['fqdn'] %>

This is the <%= @environment %> environment

``` We'll also add the apache2 dependency in the metadata.rb file. This will tell chef that this cookbook depends on the apache2 cookbook. ``` depends 'apache2' ``` Next, we'll use **Berkshelf** to resolve the dependencies between cookbooks and also upload the cookbook to the **Chef** server, so that all **Chef** nodes will be able to access it. ``` berks install && berks upload Resolving cookbook dependencies... Fetching 'chefdemoweb' from source at . Using chefdemoweb (0.1.0) from source at . Using apache2 (3.2.0) Uploaded apache2 (3.2.0) to: 'https://chef.altidev.net:443/' Uploaded chefdemoweb (0.1.0) to: 'https://chef.altidev.net:443/' …. ``` Now, into the previously created chefdemodev-webserver01, we will run our newly created **chefdemoweb** cookbook. ``` chef-client -o "recipe[chefdemoweb]" ``` A **Chef**-client run output will be quite large, so I am not going to include it here. However, we notice a few important lines that will tell us that the cookbook has configured a basic webpage, and restarted the webserver: ``` create new file /etc/httpd/sites-available/default.conf update content in file /etc/httpd/sites-available/default.conf from none to 15a87b update content in file /var/www/html/index.html restart service service[apache2] ``` Opening the address in a browser will test out if everything worked as intended. [](images/articles/tsm46/c2.png) We will repeat the deployment for the second vm in the same way as above with a small change. We will set a *boolean* attribute on the node, **is_cdn** with the value **true**. ``` knife node edit chefdemodev-webserver02.altidev.net "name": "chefdemodev-webserver02.altidev.net", "chef_environment": "chefdemodev", "normal": { "is_cdn": true, …. ``` Now we will have the cookbook dynamically search the environment for the node that has the **is_cdn** attribute set, so the recipe becomes: ``` include_recipe 'apache2' webcdn = '' search("node", "chef_environment:#{node.chef_environment} AND is_cdn:true").each do |server| webcdn << server['fqdn'] end template "#{node['apache']['docroot_dir']}/index.html" do source "index.html.erb" owner 'root' group node['apache']['root_group'] mode '0644' variables( :environment => node.chef_environment, :webcdn => webcdn ) end ``` The index.html.erb template will then become: ``` Chefdemo

Hi

This is <%= node['fqdn'] %>

This is the <%= @environment %> environment

image ``` Afterwards, we are going to increment the version in metadata.rb, do a berks install && berks upload once again and redeploy the cookbook with **Chef**-client -o "recipe[chefdemoweb]" on the chefdemodev-webserver01 vm. We can now see the change occurring in the **Chef** output. ``` - update content in file /var/www/html/index.html from 9cf426 to 336e5b --- /var/www/html/index.html 2016-04-11 11:49:00.020707241 +0000 +++ /tmp/chef-rendered-template20160411-11512-67feql 2016-04-11 11:49:31.556190114 +0000 @@ -9,6 +9,7 @@

Hi

This is chefdemodev-webserver01.altidev.net

This is the chefdemodev environment

+ image ``` The success of the deployment can be verified by reloading the browser: [](images/articles/tsm46/c3.png) Finally, we are going to change both nodes to be part of another environment, called **chefdemoprd** with the same settings as our current one. ``` knife environment create chefdemoprd knife node environment_set chefdemodev-webserver01.altidev.net chefdemoprd chefdemodev-webserver01.altidev.net: chef_environment: chefdemoprd knife node environment_set chefdemodev-webserver02.altidev.net chefdemoprd chefdemodev-webserver02.altidev.net: chef_environment: chefdemoprd ``` After re-running the **Chef**-client on both of the nodes and refreshing the browser for each of the http addresses, we will see that the information was updated dynamically, in accordance with the new environment name. [](images/articles/tsm46/c4.png) ### Conclusion **Chef** is a good tool for maintaining consistent configurations across environments, it is feature-rich and highly customizable. However, its flexibility is both a strong and weak point at the same time, in the sense that you shouldn't go too far creating complex and yet tangled code just because **Chef** allows you to. So, just like any other similar tools, you should bear in mind that **Chef** is basically "infrastructure as code" and should be treated accordingly. Remember, with great power comes great responsibility.

Conference

Sponsors

  • ntt data
  • 3PillarGlobal
  • Betfair
  • Telenav
  • Accenture
  • Siemens
  • Bosch
  • FlowTraders
  • MHP
  • Connatix
  • UIPatj
  • MetroSystems
  • Globant
  • MicroFocus
  • Colors in projects