Troubleshooting Network Connectivity in OpenStack
Network connectivity issues are among the most common challenges when managing OpenStack instances. This guide provides a systematic troubleshooting methodology to diagnose and resolve connectivity problems efficiently. We'll cover the most frequent scenarios and walk through solutions using both the Horizon dashboard and command-line tools.
Understanding the Troubleshooting Methodology
The key to effective network troubleshooting in OpenStack is working from the inside out. Start by verifying the instance itself, then move outward through security groups, network configuration, subnet settings, and finally router configuration. This approach helps isolate the problem quickly and prevents unnecessary changes to your infrastructure.
When troubleshooting, always check the most common causes first: security group rules, floating IP associations, and router gateway configurations. These three elements account for the majority of connectivity issues in OpenStack deployments.
Instance Cannot Reach the Internet
When an instance cannot reach the internet, several components could be misconfigured. Let's start with checks in the Horizon dashboard.
Checking with Horizon Dashboard
First, verify the router gateway configuration. Navigate to Network > Routers and select your router. Check the "Gateway" section to ensure an external network is attached. If no gateway appears, your router lacks internet connectivity. Click "Set Gateway" and select your external network from the dropdown menu.
Next, confirm subnet attachment to the router. Still in the router details page, click the "Interfaces" tab. Verify that your instance's subnet appears in the list. If missing, click "Add Interface" and select the correct subnet. The interface should show an IP address from your subnet range.
Security group configuration is critical for outbound connectivity. Navigate to Network > Security Groups and select the security group assigned to your instance. Check the "Egress Rules" tab. By default, OpenStack allows all outbound traffic, but custom rules may block internet access. Ensure you have a rule allowing "Egress" traffic with "IP Protocol: Any" and "Remote: CIDR 0.0.0.0/0".
Finally, verify DNS configuration for your subnet. Go to Network > Networks, select your network, and click the subnet name. Check the "DNS Name Servers" field. If empty or incorrect, edit the subnet and add reliable DNS servers like 8.8.8.8 and 8.8.4.4. DNS issues often manifest as domain name resolution failures while direct IP access works.
Using the Horizon Console
The Horizon console provides direct access to test connectivity from within the instance. Navigate to Compute > Instances and click your instance name. Select the "Console" tab to access the instance's virtual console.
Once logged in, test basic connectivity with ping commands. Try "ping 8.8.8.8" to test raw IP connectivity to the internet. If this works but "ping google.com" fails, you have a DNS issue. Check your subnet's DNS settings as described above.
If ping to external IPs fails entirely, verify the instance's default gateway with "ip route show". The default route should point to your subnet's gateway IP, typically the first IP in your subnet range. If missing, your instance may have DHCP issues or the subnet configuration is incorrect.
CLI Diagnostic Commands
For deeper diagnosis, use the OpenStack CLI to examine network configuration. First, verify router gateway status:
1openstack router show <router-name>
Look for the "external_gateway_info" field. It should reference your external network. If null, set the gateway:
1openstack router set --external-gateway <external-network> <router-name>
Check router interfaces to confirm subnet attachment:
1openstack router show <router-name> --fit-width2openstack port list --router <router-name>
Examine security group rules for outbound traffic:
1openstack security group rule list <security-group-name>
Verify that egress rules exist for the protocols you need. If not, add appropriate rules:
1openstack security group rule create --egress --protocol any <security-group-name>
Cannot SSH into Instance
SSH connectivity failures are typically caused by missing floating IPs, incorrect security group rules, or instance configuration issues.
Horizon Dashboard Checks
Start by verifying floating IP association. Navigate to Compute > Instances and check the IP addresses column. You should see both a fixed private IP and a floating public IP. If only the private IP appears, you need to associate a floating IP.
Click the dropdown arrow next to "Create Snapshot" and select "Associate Floating IP". If no floating IPs are available, click the plus icon to allocate one from your external network pool. Select the instance's port and assign the floating IP.
Next, examine security group rules for SSH access. Go to Network > Security Groups and select the security group attached to your instance. In the "Ingress Rules" tab, verify that a rule exists allowing TCP port 22. The rule should show "Direction: Ingress", "IP Protocol: TCP", "Port Range: 22", and "Remote: CIDR" with your allowed IP range.
If no SSH rule exists, click "Add Rule", select "SSH" from the Rule dropdown, and specify the source IP range. For testing purposes, you can use 0.0.0.0/0, but restrict this to your specific IP addresses for production security.
Confirm the instance is running by checking its state in Compute > Instances. The instance must show "Active" status and "Running" power state. If the instance is shutoff or in error state, SSH will fail regardless of network configuration.
Horizon Console Verification
Access the instance console as described earlier. Once logged in, verify that the SSH service is running:
1sudo systemctl status sshd
If the service is stopped, start it with "sudo systemctl start sshd". Check that SSH is listening on port 22 with "sudo ss -tlnp | grep :22".
Verify that the instance received a floating IP by checking the network interface configuration with "ip addr show". You should see the private IP on the network interface, though the floating IP won't appear here as it's NAT'd at the router level.
CLI Troubleshooting
List floating IPs and their associations:
1openstack floating ip list
If your instance lacks a floating IP, create and associate one:
1openstack floating ip create <external-network>2openstack server add floating ip <instance-name> <floating-ip-address>
Verify security group rules allow SSH:
1openstack security group rule list <security-group-name> | grep 22
Add an SSH rule if missing:
1openstack security group rule create --protocol tcp --dst-port 22 --remote-ip 0.0.0.0/0 <security-group-name>
Check the instance console log for boot errors:
1openstack console log show <instance-name>
This can reveal SSH key issues, network configuration failures, or other problems preventing SSH access.
Instances on Same Network Cannot Communicate
Inter-instance communication issues usually stem from security group restrictions or port security settings.
Horizon Dashboard Resolution
Navigate to Network > Security Groups and select the security group used by your instances. For instances to communicate, you must explicitly allow traffic between them.
Click "Add Rule" and create an ingress rule with the source set to the same security group. Select "Remote: Security Group" and choose the current security group from the dropdown. This creates a self-referencing rule that allows all instances in the group to communicate with each other.
If you need to allow specific protocols only, create targeted rules. For example, to allow ping between instances, add a rule with "Custom ICMP Rule", "Type: -1", "Code: -1", and "Remote: Security Group" set to your group.
For port security issues, navigate to Network > Networks, select your network, and click the "Ports" tab. Select a port and check if "Port Security" is enabled. If disabled, instances on that port can't receive traffic filtered by security groups. Click "Edit Port" and enable port security if needed.
CLI Verification and Fixes
Check security group rules between instances:
1openstack security group rule list <security-group-name>
Add a self-referencing rule to allow inter-instance communication:
1openstack security group rule create --protocol any --remote-group <security-group-id> <security-group-name>
Verify port security settings:
1openstack port list --network <network-name>2openstack port show <port-id>
If port security is disabled when it should be enabled:
1openstack port set --enable-port-security <port-id>
Floating IP Not Working
Floating IP issues typically involve router gateway problems or incorrect associations.
Horizon Dashboard Diagnosis
First, verify the router has an external gateway as described in the internet connectivity section. Without a gateway, floating IPs cannot route traffic.
Next, confirm the floating IP association. Go to Network > Floating IPs. Find your floating IP in the list and check the "Mapped Fixed IP Address" column. It should show the instance's private IP. If it shows a dash, the floating IP isn't associated. Click "Associate" and select the target instance port.
Check that the floating IP belongs to your external network. In the same Floating IPs page, verify the "Pool" column matches your external network name. If you accidentally allocated a floating IP from the wrong network, it won't work. Disassociate it and allocate a new one from the correct pool.
CLI Floating IP Management
List all floating IPs and their status:
1openstack floating ip list
Check router gateway configuration:
1openstack router show <router-name>
Associate a floating IP to an instance:
1openstack server add floating ip <instance-name> <floating-ip-address>
If the floating IP is from the wrong network, delete and recreate it:
1openstack floating ip delete <floating-ip-address>2openstack floating ip create <correct-external-network>
Verify the floating IP can be reached by checking router NAT rules:
1openstack router show <router-name> -f json | grep floating
Additional Troubleshooting Tips
When standard checks don't reveal the issue, consider these additional factors. Network quotas may prevent you from creating necessary resources. Check your project quotas with "openstack quota show" in the CLI.
MTU mismatches can cause subtle connectivity problems where some traffic works but large packets fail. Verify MTU settings on your network match your physical infrastructure requirements.
If you're using provider networks or VLAN segmentation, ensure the network type and segmentation ID are correctly configured for your infrastructure. These settings must match your physical network configuration.
For persistent issues, examine Neutron agent status to ensure all networking services are operational. In the CLI, use "openstack network agent list" to verify all agents show "Alive: True".
By following this systematic approach and checking each component methodically, you can resolve most OpenStack networking issues efficiently. Remember to document your findings and solutions for future reference, as networking configurations often require similar troubleshooting steps across different instances and projects.