Friday, November 7, 2014

Some vim goodness ...

As you would expect of a long term linux user, I am familiar with vim. Most who use it know that a PhD is needed to actually drive it with any amount of competency.

For this reason I am currently using Sublime Text 3 (on Mac), which by the way is awesome (http://www.sublimetext.com/3).

The team I am currently working with are highly skilled and are big users of vim so I have made it my mission to start the transition back to it and thought I would share some of the learnings that I have recently made.

1. Working with split screens


When working with things like Puppet and Ansible, it is often very useful to have 2 files open at once, either to copy something that was done before or when tracing down an issue.

So here is the scenario, I have file1 and file2 and I want to get them on the screen with a vertical split.

Firstly we open file1:

vim file1

Then split the screen vertically:

ESC + :vs

This will split the screen vertically, your cursour will be on the left hand side.

To move between the split you just use:

CTRL + w then L or R arrow

You can do the ":vs" more than once so can have more than 1 split going in a window, moving around it is the same.

To close down a split screen simply exit vim using something like:

ESC + :q (or :wq if you want to write changes)

2. Multi line changes


This is a very powerful action and one that I rely on every day in Sublime Text. This method uses vim's visual blocking or highlighting.

Here are a couple of examples on how do use this:

Indent a block of text 4 spaces

place the cursor on the first character of the first line
ctl+v (vertical select)
arrow down rows
shift+i
Press space 4 times
esc (apply to all lines)
In action: https://www.youtube.com/watch?v=o4ZvVZDyBV4

Amendment:
It has also been bought to my attention, if you want to just do a single indent (ie 2 spaces) you can use `>` (Thanks MichaelD)

Delete 4 spaces from the beginning of a block of text

ctl+v (vertical select)
use the arrows to highlight a block
press x (x usually is used to delete a single character)
In action: https://www.youtube.com/watch?v=fQrX4OYI2_w

3. vim customisation

There have been a number of things whilst using Sublime Text that I have really gotten use to, spaces used in tabs, line numbers etc etc.

vim has some very powerful customisations, this is controlled by a file called .vimrc and usually lives in the root of a users home directory ie ~/.vimrc

Similar to ruby's bundler vim has Vundle, this allows you to nicely manage plugins for vim.

https://github.com/gmarik/Vundle.vim

Here is a really good article on how to set it up

https://www.digitalocean.com/community/tutorials/how-to-use-vundle-to-manage-vim-plugins-on-a-linux-vps

Once you have vundle installed it is a simple matter of installing extra plugins for use. A good start is a colour theme, I am a fan of the colour scheme w0ng/vim-hybrid.

To install it you do:

vim then :PluginInstall w0ng/vim-hybrid

Then you add it to your ~/.vimrc.

Here is what my current file looks like:

Bundle 'w0ng/vim-hybrid'
color hybrid
syntax on
set number
set tabstop=2 softtabstop=0 expandtab shiftwidth=2
Hybrid is a really nice colour scheme that has great syntax highlighting for things like ruby, puppet and ansible. It is also really good when using vimdiff.

You can find a heap of plugins to install here

http://vim-scripts.org/vim/scripts.html

I also like to have the line numbers showing by default, this is done with `set number`. I also like to have softtabs set to 2 (ie tabs are achieved with spaces).

Given the complexity of vim you can do some amazing things with this customisation, many I have yet to discover but will share once I do.

Until next time.

Wednesday, November 5, 2014

Parsing JSON with Ansible ...

Recently, I have been doing a heap of work around the automation
and provisioning of resources with AWS cloud services. This entails working
frequently with the AWS API.

A useful filter we have been using in Ansible is taking output from a shell
action and turning it into something we can consume via variables.

An example playbook is below:

- shell: |
    export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY
    export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY
    lib/ec2.py --list --refresh-cache
  register: output
- set_fact:
    ec2_output: "{{ output.stdout|from_json }}"
- shell: "rm addresses.txt"
- shell: "echo '{{ item.value['ec2_private_ip_address'] }}' >> addresses.txt"
  when: item.value["ec2_tag_Name"] is defined and item.value["ec2_tag_Name"] == "deis-{{ environment_unique_id }}"
  with_dict: ec2_output._meta.hostvars
- shell: "cat addresses.txt"
  register: output
- add_host:
    hostname: "{{ output.stdout_lines.0 }}"
    groupname: "deis_node"
- debug: var={{ groups['deis_node'].0 }}

This example uses an Ansible plugin script ec2.py to retrieve a list of instances
running in an AWS account. It then filters out the ones that we want via the
`ec2_tag_Name` key and adds the private IP address to a file.

After running the following shell action, `output` holds a heap of information
about what the commands did and the output from stdout and stderr (if there was
any.)

- shell: |
    export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY
    export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY
    lib/ec2.py --list --refresh-cache
  register: output

The Ansible output for the registered variable output looks something like this:

TASK: [debug var=output] ******************************************************
ok: [localhost] => {
    "output": {
        "changed": true,
        "cmd": "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY \nexport AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY \nlib/ec2.py --list --refresh-cache",
        "delta": "0:00:05.268682",
        "end": "2014-10-05 11:43:05.235594",
        "invocation": {
            "module_args": "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY \nexport AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY \nlib/ec2.py --list --refresh-cache",
            "module_name": "shell"
        },
        "rc": 0,
        "start": "2013-10-05 11:42:59.966912",
        "stderr": "",
        "stdout": "<suppressed to keep it short>",
        "stdout_lines": [
            "{",
            "  \"_meta\": {",
            "    \"hostvars\": {",
            "      \"xx.xxx.xxx.xxx\": {",
            "        \"ec2__in_monitoring_element\": false, ",
            "        \"ec2_ami_launch_index\": \"0\", ",
            "        \"ec2_architecture\": \"x86_64\", ",
            "        \"ec2_client_token\": \"xxxxxxxx-xxxx-xxxx-xxxx-c55977fc0029_us-east-1a_1\", ",
            "        \"ec2_dns_name\": \"ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com\", ",
            "        \"ec2_ebs_optimized\": false, ",
            "        \"ec2_eventsSet\": \"\", ",
            "        \"ec2_group_name\": \"\", ",
            "        \"ec2_hypervisor\": \"xen\", ",
            "        \"ec2_id\": \"i-xxxxxx\", ",
            "        \"ec2_image_id\": \"ami-xxxxxx\", ",
            "        \"ec2_instance_profile\": \"\", ",
            "        \"ec2_instance_type\": \"m1.large\", ",
            "        \"ec2_ip_address\": \"xx.xxx.xxx.xxx\", ",
            "        \"ec2_item\": \"\", ",
            "        \"ec2_kernel\": \"aki-xxxxxxxx\", ",
            "        \"ec2_key_name\": \"ambari\", ",
            "        \"ec2_launch_time\": \"2013-10-01T04:53:00.000Z\", ",
            "        \"ec2_monitored\": true, ",
            "        \"ec2_monitoring\": \"\", ",
            "        \"ec2_monitoring_state\": \"enabled\", ",
            "        \"ec2_persistent\": false, ",
            "        \"ec2_placement\": \"us-east-1a\", ",
            "        \"ec2_platform\": \"\", ",
            "        \"ec2_previous_state\": \"\", ",
            "        \"ec2_previous_state_code\": 0, ",
            "        \"ec2_private_dns_name\": \"ip-xx-xx-x-xxx.ec2.internal\", ",
            "        \"ec2_private_ip_address\": \"xx.xx.x.xxx\", ",
            "        \"ec2_public_dns_name\": \"ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com\", ",
            "        \"ec2_ramdisk\": \"\", ",
            "        \"ec2_reason\": \"\", ",
            "        \"ec2_region\": \"us-east-1\", ",
            "        \"ec2_requester_id\": \"\", ",
            "        \"ec2_root_device_name\": \"/dev/sda1\", ",
            "        \"ec2_root_device_type\": \"ebs\", ",
            "        \"ec2_security_group_ids\": \"sg-xxxxxxxx\", ",
            "        \"ec2_security_group_names\": \"ambari\", ",
            "        \"ec2_sourceDestCheck\": \"true\", ",
            "        \"ec2_spot_instance_request_id\": \"\", ",
            "        \"ec2_state\": \"running\", ",
            "        \"ec2_state_code\": 16, ",
            "        \"ec2_state_reason\": \"\", ",
            "        \"ec2_subnet_id\": \"subnet-xxxxxxxx\", ",
            "        \"ec2_tag_Name\": \"hdpmaster1\", ",
            "        \"ec2_tag_aws_autoscaling_groupName\": \"ambari-LargeClusterGroup-148W3OVR9LSSE\", ",
            "        \"ec2_tag_aws_cloudformation_logical-id\": \"LargeClusterGroup\", ",
            "        \"ec2_tag_aws_cloudformation_stack-id\": \"arn:aws:cloudformation:us-east-1:167609138788:stack/ambari/xxxxxxxx-xxxx-xxxx-xxxx-50fa526be49c\", ",
            "        \"ec2_tag_aws_cloudformation_stack-name\": \"ambari\", ",
            "        \"ec2_tag_long_hostname\": \"hdpmaster1.domain.com\", ",
            "        \"ec2_virtualization_type\": \"paravirtual\", ",
            "        \"ec2_vpc_id\": \"vpc-xxxxxxxx\"",
            "      }, ",
            "}"
        ]
    }
}

As you can see stdout and stdout_lines aren't in a great format to do much with,
here enters from_json:

- set_fact:
    ec2_output: "{{ output.stdout|from_json }}"

This then allows us to do the following and `ec2_output` is now in a standard
JSON format that Ansible can then use to iterate over like so:

- shell: "echo '{{ item.value['ec2_private_ip_address'] }}' >> addresses.txt"
  when: item.value["ec2_tag_Name"] is defined and item.value["ec2_tag_Name"] == "deis-{{ environment_unique_id }}"
  with_dict: ec2_output._meta.hostvars

This Ansible task creates a file that contains all of the private IP addresses
of ec2 instances that match the `ec2_tag_Name` criteria.

We could then take the first of these IP addresses and use it later in our playbook:

- shell: "cat addresses.txt"
  register: output
- debug: var={{ output.stdout_lines.0 }}

This becomes a very powerful pattern when you mix in RESTful APIs, calls to them
and processing output from them.

The importance of the inspect and adaption cycle


At my work, we pride ourselves on our ability as an development team
to continually improve not only our engineering practices but also our planning,
testing, and communication.

Recently we performed an operation on one of our primary databases. This
involved creating a new slave database for our backups and other read only
services, promoting the current slave to be the primary as it had more resources
available to it, then decommission the old master.

This process was given a two hour window to be completed. The team involved
performed this work smoothly in just over an hour.

Post-work, we met as a team to perform a short retrospective; a common "meeting"
in the Agile/Scrum process which gives the team the chance to review what was
done, what worked, and what could be improved next time.

Here was the outcome from our retrospective:

Things we did well


  • Planning and documentation processes have improved immensely over the last couple of maintenance windows
  • Having the team onsite yet again proved to be superior to remote - communications and responsiveness to issues was great

Things we can improve


  • Obtaining a maintenance window was problematic. Given my work is a global business, finding a convenient and appropriate time for a maintenance window has always proved challenging
  • Generate and document success test cases that are going to be used to verify the system is ok in advance. This will help prevent us tripping over application bugs on the day, which could hamper the successful completion of maintenance

Actions


  • Develop and maintain a maintenance window calendar that the business has access to so that if a window is needed a call can be made without involving engineering
  • All maintenance windows should have a dress rehearsal done before a proposed window is requested. This is to ensure the team are across the process that needs to be done and more accurate timings are defined for the outage window

As agile practitioners its vital that we continually look at ways to improve on what was done so that next time it can be done better.