Viewing and Analyzing Instance Logs in OpenStack
What Are Instance Logs?
Instance logs in OpenStack capture the console output from your virtual machine (instance), providing a window into what happens during boot, system operations, and application runtime. These logs are essential for diagnosing issues when an instance fails to start, becomes unresponsive, or exhibits unexpected behavior.
OpenStack preserves console output even when you cannot access the instance through SSH or other remote connections. This makes logs your primary troubleshooting tool when standard access methods fail.
Accessing Instance Logs Through Horizon
The Horizon dashboard provides the most straightforward method for viewing instance logs.
Step-by-Step Process:
- Log into your OpenStack Horizon dashboard
- Navigate to Project > Compute > Instances
- Locate the instance you want to inspect
- Click the dropdown menu in the Actions column for that instance
- Select View Log from the menu
The console log will open in a new window or panel, displaying the most recent console output (typically the last 35 lines by default).
Viewing Full Console Output:
The initial log view shows only the tail end of the console output. To see the complete log:
- In the log viewer, look for the View Full Log button or link
- Click it to display the entire console history
- Use your browser's search function (Ctrl+F or Cmd+F) to find specific error messages or events
Accessing Instance Logs via OpenStack CLI
For administrators who prefer command-line access or need to retrieve logs programmatically, the OpenStack CLI provides flexible log retrieval options.
Installation Requirements:
First, ensure you have the OpenStack command-line client installed:
1pip install python-openstackclient
Viewing Console Logs:
To display the console log for a specific instance:
1openstack console log show <instance-name-or-id>
This command retrieves the last 35 lines of console output by default.
Retrieving Full Console Output:
To capture the complete console log:
1openstack console log show <instance-name-or-id> --lines 0
Setting --lines 0 returns all available console output, not just the most recent entries.
Saving Logs to a File:
For detailed analysis or record-keeping, redirect the output to a file:
1openstack console log show <instance-name-or-id> --lines 0 > instance-console.log
You can then open this file in your preferred text editor or log analysis tool.
Understanding Console Log Contents
Instance logs contain chronological system messages that reveal what your instance is doing at various stages.
Boot Sequence Information:
During startup, the console log captures:
- Kernel initialization messages: Hardware detection, driver loading, memory allocation
- Boot loader output: GRUB or similar bootloader messages
- Systemd or init output: Service startup sequences and status
- Network configuration: DHCP requests, IP address assignment, DNS resolution
Application Output:
Depending on your instance configuration, you may see:
- Application startup messages from services configured to log to the console
- Cloud-init execution output (particularly valuable for troubleshooting initialization scripts)
- System service status messages
Error Indicators:
Look for these common patterns that signal problems:
- Lines containing
ERROR,FAIL,Failed, orpanic - Kernel errors beginning with
kernel:followed by error descriptions - Service failures indicated by
systemdorinitmessages - Timeout messages suggesting network or storage issues
- Segmentation fault messages indicating application crashes
Troubleshooting Common Issues with Instance Logs
Instance logs excel at diagnosing several categories of problems.
Boot Failures
Symptoms: Instance status shows "Active" but is unreachable; instance never becomes accessible.
What to Look For in Logs:
- Kernel panic messages indicating the system crashed during boot
- Disk or filesystem errors suggesting corrupted storage
- Missing initramfs or kernel modules preventing proper startup
- Timeout errors during boot sequence
Example Error Pattern:
1[ 12.345678] Kernel panic - not syncing: VFS: Unable to mount root fs
This indicates the instance cannot find or mount its root filesystem, often caused by incorrect partitioning, corrupted images, or missing drivers.
Cloud-Init Failures
Cloud-init handles instance initialization tasks like user creation, SSH key injection, and running custom startup scripts.
What to Look For:
- Lines beginning with
cloud-initfollowed by error messages - Script execution failures if you provided user data
- SSH key injection problems
- Network metadata service timeouts
Example Error Pattern:
1cloud-init[1234]: ERROR: Failed to fetch metadata from http://169.254.169.254
This suggests the instance cannot reach the OpenStack metadata service, which provides configuration data.
Application or Service Failures
What to Look For:
- Service-specific error messages (nginx, apache, database errors)
- Port binding failures indicating another process is using the port
- Permission denied errors suggesting incorrect file ownership
- Out-of-memory errors indicating resource constraints
Example Error Pattern:
1systemd[1]: nginx.service: Failed with result 'exit-code'
This indicates the nginx service failed to start. Follow-up steps would involve examining nginx-specific logs via SSH (if accessible) or investigating the configuration.
Network Connectivity Issues
What to Look For:
- DHCP failures preventing IP address assignment
- DNS resolution timeouts
- Routing errors or gateway configuration problems
- Firewall or security group blocking messages
Example Error Pattern:
1[ 45.678901] DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 102[ 55.789012] No DHCPOFFERS received
This indicates the instance is broadcasting DHCP requests but receiving no responses, suggesting network configuration or OpenStack networking issues.
Best Practices for Log Analysis
Effective log analysis requires a systematic approach.
Start with the Most Recent Entries:
When troubleshooting active issues, begin with the tail end of the log (most recent output). This shows the current state and most recent errors.
Search for Known Error Patterns:
Use your browser or text editor's search function to locate:
- "error" or "ERROR"
- "fail" or "FAIL"
- "panic"
- "timeout"
- "refused"
- "denied"
Understand the Timeline:
Console logs include timestamps (in brackets) showing when events occurred. This helps you:
- Identify how far the boot process progressed before failing
- Correlate errors with specific initialization stages
- Determine if issues are immediate or delayed
Compare with Known-Good Instances:
If you have a working instance from the same image, compare its console log with the problematic instance. Differences often point directly to the issue.
Save Logs Before Rebooting:
Console logs typically reset when you reboot an instance. Always save the log content before attempting a reboot, as it contains evidence of what went wrong.
Limitations of Console Logs
Console logs have specific constraints you should understand.
Size Limitations:
OpenStack retains a limited amount of console output (typically the most recent 100KB). Very verbose systems or long-running instances may exceed this limit, causing earlier log entries to be discarded.
Not All Output Appears:
Some application logs do not write to the system console. Applications that log only to files (like /var/log/application.log) will not appear in the console log. You'll need SSH access to view those.
Reboot Behavior:
Console logs reset when you perform a hard reboot. Soft reboots (reboot command from within the instance) may preserve some output, but hard reboots (via Horizon or CLI) typically clear the log.
When to Escalate Beyond Console Logs
Console logs are powerful, but some situations require additional investigation.
SSH Access Available:
If you can connect via SSH, examine detailed application and system logs in /var/log/:
/var/log/syslogor/var/log/messagesfor general system logs/var/log/auth.logfor authentication events/var/log/nginx/or/var/log/apache2/for web server logs- Application-specific log directories
Persistent Boot Failures:
If console logs show recurring boot failures that you cannot resolve:
- Verify the instance flavor provides adequate resources (vCPUs, RAM, disk)
- Check the image integrity and ensure it's compatible with your environment
- Review volume attachment if using persistent volumes
- Contact InMotion Cloud support with the saved console log
Performance Issues Without Errors:
Console logs primarily capture errors and system messages. For performance investigation (high CPU, memory exhaustion, slow response times), you'll need:
- SSH access to run performance monitoring tools (
top,htop,vmstat) - OpenStack metrics and monitoring integration
- Application performance monitoring tools
Summary
Instance logs in OpenStack provide essential visibility into virtual machine behavior, particularly when troubleshooting boot failures and system issues. Accessing these logs through Horizon or the OpenStack CLI gives you immediate insight into console output, error messages, and system initialization.
Effective log analysis involves searching for error patterns, understanding boot sequences, and correlating timestamps with specific events. While console logs have limitations around size and scope, they remain the first line of defense when instances fail to start or become unresponsive.
For complex issues or when console logs don't provide sufficient detail, escalate to system-level logs via SSH or contact InMotion Cloud support with your saved console output for further investigation.
