PaloAlto’s “fix” for making VM Series FW work with AWS ELB

I want to share my experience using Palo Alto FW in AWS ( using Terraform ).

Palo Alto has some good examples out there of how their stuff works with Terraform and AWS, such as this github repo for thier two-tier implementation using Terraform. The two tier design here is a small starting point upon which to code out your platform. However, it does not take into account how the PA’s work with multiple AZs or Load Balancers, both of which will most likely be needed in your implementation.

Multiple AZ’s are not hard to do, but if you want Elastic Load Balancers in the mix, there is a Management Interface swap that needs to happen in order for the Palo Alto VM FW to work with AWS ELB.

This seemingly tiny, insignificant detail has been challenging to integrate into Terraform, and although PaloAlto does provide some documentation, it is not enough. Here is what I learned:

  • An EIP association is also needed for eth1 ( what will become mgmt. interface ) if you are not using a Bastion host to connect initially.
  • Another EIP association is also needed in Terraform for FW eth0 so device can read and get bootstrap from S3 Bucket. ( this EIP can later be torn down – but yes, you need TWO EIPs at boot time to make this work! ) [ UPDATE: a VPC endpoint w/ routing can replace the need for this second EIP ]
  • FW Initial Ruleset must permit ssh / web to the IP of Ethernet 1/1 new mgmt ( that’s obvious but included here so not be be over looked )
  • Ethernet 1/1 was not assigned to a Zone in PAN-OS initial config bootstrap when I got it working.
  • Terraform interface templates for Palo Alto need to have mgmt. interface associated to device_index = 1 ( instead of zero ) in initial config.  Configuring for Terraform, thus AWS, the actual interface assignments are abstracted, so you configure them as you will use them AFTER the swap.

Also of note, the PAN-OS does not “see” this change in the GUI. You will still implement your rules and zones as though there was not an interface change. Eth 1/1 shows up in Network interfaces…  The only place to see this is in the CLI with this lengthy command:

Fig 1. Normal boot:

admin@PA-VM> debug show vm-series interfaces all

Interface_name       Base-OS_port       Base-OS_MAC             PCI-ID         Driver
mgt                     eth0          06:82:4e:66:99:9a       0000:00:03.0      ixgbevf
Ethernet1/1             eth1          06:1a:e7:12:01:e0       0000:00:04.0      ixgbevf
Ethernet1/2             eth2          06:39:13:1b:e9:d4       0000:00:05.0      ixgbevf

Fig 2. Booting with the command ‘op-command-modes=mgmt-interface-swap’ in the init.cfg

admin@sample-cft-fw> debug show vm-series interfaces all

Interface_name       Base-OS_port       Base-OS_MAC             PCI-ID         Driver
mgt (interface-swap)    eth0          06:db:de:02:e5:22       0000:00:04.0      ixgbevf
Ethernet1/1             eth1          06:9f:0e:8b:de:ec       0000:00:03.0      ixgbevf
Ethernet1/2             eth2          06:5d:86:02:b1:4e       0000:00:05.0      ixgbevf

It’s hard to notice a difference, but if it worked, you’ll see the (interface-swap) after mgt. ( above ) .  Palo Alto provides this graphic to explain it, but it does not really line up well with the output above since the VM interfaces are extracted into AWS.

In Terraform, it lines up as so:

device_index = 0 will be eth0  in AWS, which is initial mgmt in Palo Alto (before swap), communication to the S3 bucket for bootstrap happens from this interface.

device_index = 1 will be  eth1 in AWS, (which would be new mgmt if swapped)

device_index = 2 will be eth2 in AWS, not affected by swap

Also, before getting it to work in Terraform, I tried the command to swap interfaces on a FW VM series running n AWS that had booted normally; using Palo’s CLI:

set system setting mgmt-interface-swap enable yes

It responded with:

Reboot system to take effect new changes. After reboot use IP address of eth1 (external to VM) for management

After it booted again, I had trouble getting back into the Firewall with  ssh. Got connection refused. Could have been various reasons, but at this stage, I think it was that I did not wait long enough. Another Engineer I am working with said that this worked for him but he had to wait a long time after boot.

This Management interface swap feels very much like the Palo Alto VM Firewall has been ‘shoe-horned’ to work in and with AWS.  I have always loved Palo Alto, but not really a fan of this fix.

I’ll be writing more as I learn. I hope this helps you! Oh, before I go, here are some handy PAN-OS CLI commands you will use if you are doing this same thing:

op-command-modes=mgmt-interface-swap # for bootstrap
set system setting mgmt-interface-swap enable yes # PA cli
debug show vm-series interfaces all # show your stuff
set mgt-config users admin password #you'll need this to get to the web GUI
save config
This entry was posted in AWS, Palo Alto FW, Palo Alto Networks. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s