AWS CloudFormer VPC NAT Instance gotcha

I’ve been working pretty heavily w/ CloudFormation and AWS lately, automating the spin-up of large environments using CloudFormation and Puppet. More on that later, but wanted to note a little gotcha w/ CloudFormer.

CloudFormer is a BETA AWS appliance that presents a GUI and allows the user to select items in their account to be turned into a CFN template. It has a bit of intelligence where if you select an EC2 instance to be included in the JSON template output, it will auto-select the related Security Groups, and so on.

HOWEVER!!! As of this writing CloudFormer does not pick up on the SourceDestCheck true/false value that has to be turned off for NAT instances to do NAT. Per the AWS documentation at and the SourceDestCheck must be set to false for NAT to work. If you’re not familiar w/ this already you can spend days trying to figure out what’s wrong with NACLs, routes, SGs, etc. You can set this check yourself in the CloudFormer JSON template as shown here:

    "NATInstance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "DisableApiTermination": "FALSE",
        "ImageId": { "Fn::FindInMap" : [ "natmap", { "Ref" : "AWS::Region" }, "64"]},
        "InstanceType": {"Ref": "NATinstanceSize"},
        "KeyName": { "Ref" : "KeyName" },
        "SourceDestCheck" : "false",

Hope this helps!

Authenticating to puppetdb dashboard SSL using certificate auth

puppetdb dashboard has some performance monitoring built in. To access it via localhost is easy. No auth. To access it remotely via SSL, you have to generate a client cert for your browser, configure jetty to accept the puppet master CA cert, and add your client cert to a list of allowed certs. Here’s all the steps I needed to make this work on CentOS 6.4 (server) and Mac OSX 10.7 with Firefox 23.0.1.

[root@puppetmaster ssl]# puppet cert generate esakowski
Notice: esakowski has a waiting certificate request
Notice: Signed certificate request for esakowski
Notice: Removing file Puppet::SSL::CertificateRequest esakowski at '/etc/puppetlabs/puppet/ssl/ca/requests/esakowski.pem'
Notice: Removing file Puppet::SSL::CertificateRequest esakowski at '/etc/puppetlabs/puppet/ssl/certificate_requests/esakowski.pem'
Notice: You have 3 active and no inactive nodes.
Notice: You are currently licensed for 10 active nodes.
Notice: This Puppet Enterprise distribution is licensed to:
Notice:         N/A
You are using a complimentary ten node license provided free by Puppet Labs.
Notice: Your complimentary license does not include Support & Maintenance. If you
Notice: would like to obtain official Support & Maintenance, please contact us
Notice: for pricing, and to find out about volume discounts.
[root@puppetmaster ssl]# find . -name 'esakowski\.*'
[root@puppetmaster ssl]#

Next you have to smash all these keys into a pkcs12 format (enter nothing for the password):

[root@puppetmaster ssl]# cd /tmp
[root@puppetmaster tmp]# openssl pkcs12 -export -out esakowski.pfx -inkey /etc/puppetlabs/puppet/ssl/private_keys/esakowski.pem  -in /etc/puppetlabs/puppet/ssl/ca/signed/esakowski.pem -certfile /etc/puppetlabs/puppet/ssl/certs/ca.pem 
Enter Export Password:
Verifying - Enter Export Password:
[root@puppetmaster tmp]#
[root@puppetmaster tmp]# file esakowski.pfx
esakowski.pfx: data
[root@puppetmaster tmp]#

Download this .pfx file somewhere on your workstation filesystem so you can add it to FF.

To add it to your firefox keys, go to Preferences -> Advanced -> View Certificates -> Import
and select the appropriate file. Here again, you enter nothing for the password.

You have to tell jetty to trust the puppetmaster ca cert. Edit /etc/puppetlabs/puppetdb/conf.d/jetty.ini to add:
ssl-ca-cert =/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem

If you try to hit https://puppetmaster:8081 right now, it should let you connect, but it will tell you “You shall not pass!” — this is because your cert is trusted but it’s not yet in the whitelist of certs that are allowed to see statistics from the puppetdb dashboard. Edit the whitelist to add the name of your cert (whatever name you passed to puppet cert generate):

[root@puppetmaster conf.d]# cat /etc/puppetlabs/puppetdb/certificate-whitelist
[root@puppetmaster conf.d]#

Restart pe-puppetdb:

[root@puppetmaster conf.d]# service pe-puppetdb restart
Stopping pe-puppetdb:                                      [  OK  ]
Starting pe-puppetdb:                                      [  OK  ]
[root@puppetmaster conf.d]#

Now go to You should see something like the screengrab below. If not, check the contents of /var/log/pe-puppetdb/*.log

Good luck!

Screen Shot 2013-10-04 at 4.52.13 PM

Quick note: how to use create_resources to initialize a class in puppet

Quick note: how to use create_resources() to initialize a class in Puppet.

create_resources() is more commonly used in Puppet to initialize a defined type, but there is a lesser-known signature that allows initialization of classes using this function. I had to comb the usenet group puppet-users to find it, and there was a single match for what I wanted to do in a post by Nan Liu as almost a side comment. Using that posting by Nan I was encouraged enough to go read the Ruby source for create_resources and got what I needed. When searching now, I find a later usenet posting of my own as the top match, so I figured I’d formalize it here.

It sounds complicated, but you just have to structure your data properly. Given foo/manifests/init.pp:

class foo (
    $a = 'aaa',
    $b = 'bbb',
    $c = 'ccc'
) {
    notify { "a ${a} -- b ${b} -- c ${c} ": }

Let’s say you declare this in /etc/puppetlabs/puppet/manifests/site.pp and leave the defaults:

node 'foo-test' {
    class { foo: }

You’ll get output from puppet agent -t displaying:

notice: a aaa -- b bbb -- c ccc 
notice: /Stage[main]/Foo/Notify[a aaa -- b bbb -- c ccc ]/message: defined 'message' as 'a aaa -- b bbb -- c ccc '

Change your declaration in /etc/puppetlabs/puppet/manifests/site.pp like so:

node 'foo-test' {
  class { foo:
    a => 'asdf',
    b => 'qwer',
    c => 'zot'

And your output becomes:

notice: a asdf -- b qwer -- c zot 
notice: /Stage[main]/Foo/Notify[a asdf -- b qwer -- c zot ]/message: defined 'message' as 'a asdf -- b qwer -- c zot '

You want to initialize it w/ some other data source, probably from hiera. I’ll get into the hiera magic I concocted in a later post. For now, let’s stipulate that you get data from “somewhere” and you put it into a hash like so:

$foo_params = { foo => { a => 'alan', b => 'bill', c => 'carl' } }

Now you can initialize your class with create resources:

create_resources('class', $foo_params)

Here’s the full entry in /etc/puppetlabs/puppet/manifests/site.pp:

node 'foo-test' {
  $foo_params = { foo => { a => 'alan', b => 'bill', c => 'carl' } }
  create_resources('class', $foo_params)

This gives the following output:

notice: a alan -- b bill -- c carl 
notice: /Stage[main]/Foo/Notify[a alan -- b bill -- c carl ]/message: defined 'message' as 'a alan -- b bill -- c carl '

The key to using create_resources is that you have to have an anonymous hash that contains a key value pair where the key is the name of the class you want to create and the value is a hash itself containing key value pairs matching the parameter names of the class.

  foo => {
            a => 'a value',
            b => 'b value',
            c => 'c value'

Puppet: Managing SSL certs in a multiple master, single CA setup

Puppet: Managing SSL certs in a multiple master, single CA setup

I just spent quite a bit of time figuring all of this out and gathering from
multiple blogs and sources, so thought I’d share w/ the Internets. What I need
is a single CA for multiple masters. There are many ways to do it; this is
simply what worked for me. The bits that took the most time to figure out
were that the pe-dashboard has its own certs that need to be recreated and
re-signed, and that the non-CA master needs to still have a copy of the ca.crt.

The following assumes Puppet Enterprise 2.8.1: a reasonably experienced user should be able to adapt it to Puppet Community w/o too much difficulty.

Because I’m using Vagrant for my dev/test environment, all the hostnames are going
to have a ‘vagrant-‘ prefix. So, for this example we will have the CA be on the
‘first’ master, vagrant-pupmaster. The other master will be vagrant-pupmaster2.
The node will be called vagrant-cent64-pupnode. You will need to put these
names into /etc/hosts on all three: vagrant-pupmaster vagrant-pupmaster2 vagrant-cent64-pupnode

Destroy all the existing certs (don’t do this on a production system). On
vagrant-pupmaster, vagrant-pupmaster2, and vagrant-cent64-pupnode do:

mv /etc/puppetlabs/puppet/ssl /etc/puppetlabs/puppet/ssl.pre_ca_regenerate

Next, generate a new certs on the CA. On vagrant-pupmaster aka puppetca do:

/opt/puppet/sbin/puppetca --generate --dns_alt_names puppetca,puppet vagrant-pupmaster
/opt/puppet/sbin/puppetca --generate vagrant-pupmaster2

On vagrant-pupmaster2 create an empty ssl dir and chmod/chown it appropriately:

mkdir -p /etc/puppetlabs/puppet/ssl
chown pe-puppet:root /etc/puppetlabs/puppet/ssl 
chmod 771 /etc/puppetlabs/puppet/ssl
mkdir -p /etc/puppetlabs/puppet/ssl/certs
chmod 755 /etc/puppetlabs/puppet/ssl/certs/
mkdir -p /etc/puppetlabs/puppet/ssl/private_keys 
chmod 750 /etc/puppetlabs/puppet/ssl/private_keys

Copy the ca crt and the vagrant-pupmaster2 cert over to vagrant-pupmaster2. On
vagrant-pupmaster, do:

scp /etc/puppetlabs/puppet/ssl/private_keys/vagrant-pupmaster2.pem root@vagrant-pupmaster2:/etc/puppetlabs/puppet/ssl/private_keys
scp /etc/puppetlabs/puppet/ssl/certs/vagrant-pupmaster2.pem root@vagrant-pupmaster2:/etc/puppetlabs/puppet/ssl/certs
scp /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem root@vagrant-pupmaster2:/etc/puppetlabs/puppet/ssl/certs/ca_crl.pem

Recreate the puppet dashboard certs on the CA master. On vagrant-pupmaster, do:

su - puppet-dashboard
rake RAILS_ENV=production cert:create_key_pair
rake RAILS_ENV=production cert:request
rake RAILS_ENV=production cert:retrieve
rsync -avz /opt/puppet/share/puppet-dashboard/certs root@vagrant-pupmaster2:/opt/puppet/share/puppet-dashboard/

On vagrant-pupmaster2 edit /etc/puppetlabs/httpd/conf.d/puppetmaster.conf
and make sure the CRL is commented out or deleted and the ProxyPass for
all certs to go to the CA is present and enabled:

#    SSLCARevocationFile     /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem
SSLProxyEngine On
# Proxy all requests that start with things like /production/certificate to the CA
ProxyPassMatch ^/([^/]+/certificate.*)$ https://vagrant-pupmaster:8140/$1

On vagrant-pupmaster2 edit /etc/puppetlabs/puppet/puppet.conf. Make
sure ca = false:

[root@vagrant-pupmaster2 puppet]# grep 'ca =' /etc/puppetlabs/puppet/puppet.conf
    ca = false
[root@vagrant-pupmaster2 puppet]#

On vagrant-cent64-pupnode, edit /etc/puppetlabs/puppet/puppet.conf and set archive_file_server and
ca_server to vagrant-pupmaster, and set server to vagrant-pupmaster2:

    vardir = /var/opt/lib/pe-puppet
    logdir = /var/log/pe-puppet
    rundir = /var/run/pe-puppet
    modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
    user  = pe-puppet
    group = pe-puppet
    archive_files = true
    archive_file_server = vagrant-pupmaster
    certname = vagrant-cent64-pupnode
    server = vagrant-pupmaster2
    ca_server = vagrant-pupmaster
    report = true
    classfile = $vardir/classes.txt
    localconfig = $vardir/localconfig
    graph = true
    pluginsync = true

That should do it! Now restart puppet master and dashboard on both masters:

service pe-httpd restart;service pe-puppet-dashboard-workers restart

And run the agent on vagrant-cent64-pupnode:

puppet agent -t --debug --server vagrant-pupmaster2

Although the agent connects to vagrant-pupmaster2, the ProxyPass should
forward the CSR to vagrant-pupmaster (your CA). On vagrant-pupmaster do:

puppet cert --list

This should display the cert. The fact that it’s showing up on vagrant-pupmaster even
though you connected to vagrant-pupmaster2 as your server means the forwarding worked.
It should look something like below (the fingerprint will be different but it should have
the certname for the node):

[root@vagrant-pupmaster ssl]# puppet cert --list 
  "vagrant-cent64-pupnode" (F7:5C:EB:3D:D3:17:36:67:34:31:42:14:A0:4E:AC:DE)
[root@vagrant-pupmaster ssl]#

Now you need to sign it on the CA:

puppet cert --sign vagrant-cent64-pupnode

Once that’s done, run puppet agent on vagrant-cent64-pupnode:

puppet agent -t --debug --server vagrant-pupmaster

You should get a catalog and successful puppet run. If you get an error from
filebucket about getaddrinfo: Name or service not known, you probably need
to edit /etc/puppetlabs/puppet/manifests/site.pp to set filebucket to the
correct hostname:

    # Define filebucket 'main':
    filebucket { 'main':
      server => '<PUPPET MASTER'S DNS NAME>',
      path   => false,

Puppet: hpilo module to manage HP Integrated Lights Out on Centos/RHEL

I just finished a major refactor of Corey Osman’s hpilo puppet module. This module can be used to configure Hewlett Packard’s Integrated Lights Out (aka iLO) embedded card. It is now compatible with hiera either through use of create_resources(‘class’, {‘hpilo’ => $params}), or by loading it in puppet 3.0. You can grab the module from github at


Table of Contents

  1. Overview
  2. Module Description – What the module does and why it is useful
  3. Setup – The basics of getting started with [Modulename]
  4. Usage – Configuration options and additional functionality
  5. Reference – An under-the-hood peek at what the module is doing and how
  6. Limitations – OS compatibility, etc.
  7. Development – Guide for contributing to the module


hpilo configures Hewlett-Packard’s iLO card by creating ilosettings.xml and then using the
HP application hponcfg to load the settings file to the iLO.

Integrated Lights-Out, or iLO, is an embedded server management technology exclusive to
Hewlett-Packard but similar in functionality to the Lights out management (LOM) technology
of other vendors, for example Sun/Oracle’s ILOM, Dell DRAC and the IBM Remote Supervisor Adapter.

Tested with PE 2.8.1 and should work w/ Puppet 3.

Module Description

hpilo can setup iLO to use dhcp, user-selected static IP, or assign a static IP automatically
based on the system’s IP.


What hpilo affects

  • creates/modifies /etc/ilosettings.xml and /tmp/ilosettings.log
  • installs /sbin/hponcfg from HP Proliant support repo.

Setup Requirements

You will need the repo from
The hpilo module will install hponcfg from this repo.

Beginning with hpilo

class {'hpilo':  }


You can easily configure many parameters of iLo.


class { 'hpilo': 
  dhcp => true,
  ilouser => 'admin',
  ilouserpass => 'password'

For static:

class { 'hpilo':
  dns => '',
  gw => '',
  ip => '',
  netmask => '',
  shared => true,
  ilouser => 'admin',
  ilouserpass => 'password',

For autoip (Take $::ipaddress fact and sub 3rd octet with hpilo::ilonet value,
fx if ipaddress of system is ilo will get

class { 'hpilo':
  autoip => true,
  ilonet => '3',
  shared => true,
  ilouser => 'admin',
  ilouserpass => 'password',

To configure with yaml in hiera for Puppet 3:

hpilo::autoip: false

hpilo::dhcp: false

hpilo::dns: ‘’

hpilo::gw: ‘’

hpilo::gwbit: ‘240’

hpilo::ilonet: ‘3’

hpilo::ilouser: ‘admin’

hpilo::ilouserpass: ‘password’

hpilo::ip: ‘’

hpilo::logfile: ‘/tmp/ilosettings.log’

hpilo::netmask: ‘’

hpilo::settingsfile : ‘/etc/ilosettings.xml’

hpilo::shared: true



If you modify network settings but not the user after the initial run, hponcfg
will return an error exit code for trying to create a user that already exists,
but will still apply the new network settings.

Only tested on Centos 6.4 with Puppet Enterprise 2.8.1. Should probably work on RHEL and SuSE.