Pause, Suspend & Shelve: Instance States

Introduction

Managing instance states effectively is a core skill when working with OpenStack. Whether you need to temporarily halt an instance to save resources, perform maintenance, or prepare for migration, understanding the differences between pause, suspend, and shelve operations ensures you choose the right approach for your specific situation.

Each state change affects your instance differently in terms of resource consumption, billing, recovery time, and data persistence. This guide breaks down these three operations, explaining what happens under the hood, when to use each one, and the tradeoffs involved.

Understanding OpenStack Instance States

OpenStack provides several power states that control how instances consume compute resources. The three most commonly used temporary states are pause, suspend, and shelve. While all three stop instance execution, they differ significantly in resource allocation, storage behavior, and intended use cases.

Active State (Baseline)

Before diving into the temporary states, it helps to understand the active state. An active instance runs normally with:

vCPUs actively processing workloads
RAM fully allocated and in use
Root disk accessible and mounted
Network interfaces connected and routing traffic
Full billing for compute, memory, and storage resources

When you move an instance to pause, suspend, or shelve, you change which resources remain allocated and how quickly you can return to the active state.

Pause: Quick Freeze with Full Resource Allocation

What Happens When You Pause

Pausing an instance freezes its execution state in RAM while keeping all resources allocated. The hypervisor stops the instance's processes but maintains the memory contents exactly as they were. Think of it like hitting the pause button on a video, the frame stays on screen but nothing moves forward.

Specifically, pause:

Freezes all vCPU activity instantly
Keeps the instance's RAM allocated on the hypervisor
Maintains all memory contents in place
Preserves network configuration and IP assignments
Continues to allocate the instance to specific compute node resources

Resource Implications

What You Still Pay For:

RAM allocation (memory remains reserved)
Compute node allocation (vCPU slots reserved)
Root disk storage
Attached volumes

What Stops:

CPU cycles and active processing
Network traffic generation
Active I/O operations

When to Use Pause

Pause works best for very short interruptions where you need near-instant recovery:

Appropriate Scenarios:

Brief maintenance window (5-15 minutes)
Debugging a running process that needs to be frozen in place
Temporarily halting an instance to examine its exact state
Quick resource adjustments on the host
Emergency freeze to prevent runaway processes

Recovery Time: Typically 1-3 seconds. Unpausing simply resumes execution from the frozen state.

When NOT to Use Pause

Pause is inappropriate for:

Extended downtime (more than 30 minutes)
Cost savings, you still pay for allocated resources
Freeing up compute capacity for other instances
Long-term storage of instance state

If you pause an instance and leave it that way for hours or days, you waste allocated resources that could serve other workloads while still incurring costs.

Suspend: Save State to Disk for Medium-Term Holds

What Happens When You Suspend

Suspending an instance writes its entire memory state to disk storage, then deallocates the RAM. This creates a snapshot of the instance's exact execution state which can be restored later. The hypervisor saves the memory dump to the compute node's local storage.

Specifically, suspend:

Writes RAM contents to disk (memory snapshot)
Deallocates RAM from the compute node
Stops all vCPU activity
Maintains the instance's allocated position on the compute node
Preserves network configuration
Keeps root disk and attached volumes accessible

Resource Implications

What You Still Pay For:

Root disk storage
Attached volumes
Memory snapshot storage (additional disk space)
Compute node reservation (instance slot remains allocated)

What You Save:

Active RAM allocation on the host
vCPU cycles

When to Use Suspend

Suspend suits scenarios where you need guaranteed quick recovery but can tolerate a few seconds of downtime:

Appropriate Scenarios:

Planned maintenance lasting 30 minutes to several hours
Temporarily freeing RAM on a busy compute node
Creating a recoverable checkpoint before risky operations
Overnight or weekend shutdowns where quick Monday morning recovery matters
Reducing active resource consumption during low-usage periods

Recovery Time: Typically 10-30 seconds, depending on instance RAM size. The hypervisor must read the memory snapshot from disk and restore it to RAM.

When NOT to Use Suspend

Suspend is inappropriate for:

Long-term storage (weeks or months), the memory snapshot consumes disk space
Migrating instances between hosts, the memory state is tied to the original compute node
Severe resource constraints where even disk space is limited
Instances with massive RAM allocations, the snapshot can be very large

Shelve: Long-Term Storage with Maximum Resource Release

What Happens When You Shelve

Shelving an instance creates a complete snapshot of its root disk, then completely deallocates all compute resources. The instance effectively disappears from the compute node, freeing up vCPU, RAM, and host allocation. Only the disk snapshot remains stored.

Specifically, shelve:

Creates a snapshot image of the root disk
Deallocates all vCPUs from the compute node
Deallocates all RAM from the compute node
Removes the instance from its original compute node
Stores the snapshot in the image storage backend (typically Glance)
Releases the instance slot on the compute node

Resource Implications

What You Still Pay For:

Snapshot storage in the image backend
Attached volumes (if any)

What You Save:

All compute resources (vCPU, RAM)
Compute node allocation
Active root disk on the hypervisor

Cost Impact: Shelving provides the maximum cost reduction of the three options since you only pay for snapshot storage, which is typically much cheaper than active compute resources.

When to Use Shelve

Shelve is the best choice for long-term inactivity or when you need to free up significant resources:

Appropriate Scenarios:

Development instances not needed for weeks or months
Seasonal workloads that run only certain times of year
Backup instances kept on hand for disaster recovery
Test environments used sporadically
Projects on hold waiting for budget or approval
Migrating instances between regions or availability zones

Recovery Time: Typically 2-5 minutes. Unshelving requires:

Allocating compute resources on an available node
Restoring the snapshot to a new root disk
Booting the instance from the restored disk
Reconnecting network interfaces

When NOT to Use Shelve

Shelve is inappropriate for:

Short-term pauses (under 1 hour), the recovery overhead is too high
Instances that must maintain exact memory state, shelve only preserves disk data
Situations requiring instant or very fast recovery
Instances with large root disks where snapshot creation takes excessive time

Comparison Matrix

| Aspect | Pause | Suspend | Shelve |
|--------|-------|---------|--------|
| vCPU Allocation | Reserved | Reserved | Deallocated |
| RAM Allocation | Reserved | Deallocated | Deallocated |
| Memory State | In RAM | Saved to disk | Lost |
| Compute Node | Allocated | Allocated | Deallocated |
| Root Disk | Active | Active | Snapshot created |
| Recovery Time | 1-3 seconds | 10-30 seconds | 2-5 minutes |
| Best Duration | Minutes | Hours | Weeks to months |
| Cost Savings | Minimal | Moderate | Maximum |
| Use Case | Quick freeze | Medium-term hold | Long-term storage |

How to Manage Instance States in Horizon Dashboard

The Horizon dashboard provides a straightforward interface for managing instance states. All state management operations are accessible through the instance actions menu.

Pausing an Instance in Horizon

Pausing an instance through Horizon freezes its execution while keeping all resources allocated. This operation completes instantly.

Steps:

Log into the Horizon dashboard at your InMotion Cloud portal.
Navigate to Project > Compute > Instances.
Locate the instance you want to pause in the instances list.
Click the dropdown arrow on the right side of the instance row.
Select Pause Instance from the actions menu.
Horizon immediately changes the instance status to "Paused".

To Unpause:

Return to Project > Compute > Instances.
Locate the paused instance (status shows "Paused").
Click the dropdown arrow on the instance row.
Select Resume Instance from the actions menu.
The instance returns to "Active" status within 1-3 seconds.

What You'll See:

The instance status column displays "Paused" while the instance is frozen. The power state indicator shows the instance is not consuming CPU cycles but remains allocated on the compute node.

Suspending an Instance in Horizon

Suspending saves the instance's memory state to disk and deallocates RAM. This operation takes a few seconds depending on the instance's memory allocation.

Steps:

Log into the Horizon dashboard.
Navigate to Project > Compute > Instances.
Locate the instance you want to suspend.
Click the dropdown arrow on the instance row.
Select Suspend Instance from the actions menu.
Horizon begins writing the memory state to disk.
The status changes to "Suspended" when the operation completes.

To Resume:

Return to Project > Compute > Instances.
Locate the suspended instance (status shows "Suspended").
Click the dropdown arrow on the instance row.
Select Resume Instance from the actions menu.
Horizon restores the memory state from disk.
The instance returns to "Active" status in 10-30 seconds.

What You'll See:

During suspension, the status temporarily shows "Suspending" before changing to "Suspended". During resume, you'll see "Resuming" before the instance becomes "Active". Progress indicators appear for instances with large memory allocations.

Shelving an Instance in Horizon

Shelving creates a complete snapshot and deallocates all compute resources. This operation takes longer than pause or suspend since it must create and store a full disk snapshot.

Steps:

Log into the Horizon dashboard.
Navigate to Project > Compute > Instances.
Locate the instance you want to shelve.
Click the dropdown arrow on the instance row.
Select Shelve Instance from the actions menu.
Horizon begins creating a snapshot of the root disk.
The status changes to "Shelved Offloaded" when complete.

To Unshelve:

Return to Project > Compute > Instances.
Locate the shelved instance (status shows "Shelved Offloaded").
Click the dropdown arrow on the instance row.
Select Unshelve Instance from the actions menu.
Horizon finds available compute resources and restores the instance from the snapshot.
The instance returns to "Active" status in 2-5 minutes.

What You'll See:

The shelving process shows several status transitions: "Active" → "Shelving" → "Shelved" → "Shelved Offloaded". The shelved instance remains in your instances list but consumes no compute resources. During unshelving, you'll see "Spawning" as the instance allocates resources and restores from the snapshot.

Note About IP Addresses:

Shelved instances typically lose their fixed IP addresses. After unshelving, the instance receives a new IP address from the network pool. Plan for this when shelving production instances. Floating IPs may also be released depending on your cloud configuration.

Monitoring State Changes in Horizon

The Instances page provides real-time status updates:

Status Column: Shows the current state (Active, Paused, Suspended, Shelved Offloaded, etc.).

Task Column: Displays ongoing operations (Pausing, Suspending, Shelving, Spawning, etc.).

Power State Column: Indicates the underlying hypervisor state.

Horizon automatically refreshes the instances list every few seconds. For immediate updates, click the refresh button at the top of the instances table.

Horizon Dashboard Tips

Bulk Operations: You can select multiple instances using the checkboxes and perform batch operations. However, state changes are applied sequentially, not simultaneously.

Instance Details: Click an instance name to view detailed information including current resource usage, attached volumes, network interfaces, and complete state history.

Filtering: Use the search box to filter instances by name or status. This helps when managing large deployments.

Error Handling: If a state change fails, Horizon displays an error message at the top of the page. Check the instance details for more information about the failure reason.

How to Manage Instance States Using OpenStack CLI

For users who prefer command-line interfaces or need to automate state management operations, the OpenStack CLI provides direct access to all instance state controls.

Pausing an Instance via CLI

To Pause:

1openstack server pause <instance-name-or-id>

Example:

1openstack server pause web-server-01

To Unpause:

1openstack server unpause <instance-name-or-id>

Example:

1openstack server unpause web-server-01

Verify Status:

1openstack server show <instance-name-or-id> -c status -c OS-EXT-STS:power_state

The status field shows "PAUSED" and power state shows "Paused".

Suspending an Instance via CLI

To Suspend:

1openstack server suspend <instance-name-or-id>

Example:

1openstack server suspend database-server-01

To Resume:

1openstack server resume <instance-name-or-id>

Example:

1openstack server resume database-server-01

Verify Status:

1openstack server show <instance-name-or-id> -c status -c OS-EXT-STS:power_state

The status field shows "SUSPENDED" and power state shows "Suspended".

Shelving an Instance via CLI

To Shelve:

1openstack server shelve <instance-name-or-id>

Example:

1openstack server shelve dev-environment-01

To Unshelve:

1openstack server unshelve <instance-name-or-id>

Example:

1openstack server unshelve dev-environment-01

Unshelve to Specific Availability Zone:

1openstack server unshelve --availability-zone <zone-name> <instance-name-or-id>

By default, unshelve places the instance on any available compute node. Use the availability zone option to control placement.

Verify Status:

1openstack server show <instance-name-or-id> -c status -c OS-EXT-STS:power_state

The status field shows "SHELVED_OFFLOADED" when fully shelved.

CLI Automation Tips

Batch Operations:

1for instance in web-01 web-02 web-03; do
2  openstack server pause $instance
3done

Scheduled State Changes:

Combine with cron for scheduled operations:

1# Suspend dev instances at 6 PM
20 18 * * * openstack server suspend dev-environment-01
3# Resume dev instances at 8 AM
40 8 * * * openstack server resume dev-environment-01

Status Monitoring:

1watch -n 5 'openstack server list --status ACTIVE --status PAUSED --status SUSPENDED'

This refreshes the instance list every 5 seconds, showing current states.

Important Considerations

Data Persistence

All three operations preserve:

Root disk data (files, configurations, installed software)
Attached volumes and their contents
Instance metadata and properties

Only suspend preserves:

Exact memory contents and execution state
Running processes and their states
Open file handles and network connections

What none of them preserve:

Ephemeral storage on shutdown (if configured)
In-memory data that was never written to disk

Always ensure critical data is written to persistent storage before changing instance states.

Network and IP Behavior

Pause and Suspend:

Maintain the instance's IP address assignments
Preserve security group rules
Keep network ports allocated

Shelve:

May release floating IPs (depending on configuration)
Releases fixed IPs from the network pool
On unshelve, the instance typically receives a new IP address
Plan for IP changes when using shelve for production instances

Billing and Resource Quotas

Different cloud providers implement billing differently for paused, suspended, and shelved instances:

InMotion Cloud Approach:

Paused instances continue billing at full rate (resources remain allocated)
Suspended instances bill for storage and reduced compute allocation
Shelved instances bill only for snapshot storage

Check your specific provider's billing policies before choosing a state change strategy.

Security Considerations

Paused Instances:

Memory contents remain in RAM on the host
Sensitive data in memory is not encrypted by default
Paused state provides limited additional security

Suspended Instances:

Memory snapshot is written to disk
Disk storage may or may not be encrypted
Snapshot contains the complete memory state including sensitive data

Shelved Instances:

Only disk contents are preserved in snapshot
Sensitive data in memory is lost
Snapshot storage follows the same security as image storage

For instances handling sensitive data, consider security implications before suspending or shelving.

Choosing the Right State

Ask yourself these questions:

How long will the instance be inactive?

Under 30 minutes: Pause
30 minutes to 8 hours: Suspend
Days to months: Shelve

Do you need to preserve exact memory state?

Yes: Suspend (or pause for very short duration)
No: Shelve acceptable

How important is recovery speed?

Critical (under 5 seconds): Pause
Important (under 1 minute): Suspend
Acceptable (a few minutes): Shelve

Do you need to free up compute resources?

No: Pause acceptable
RAM only: Suspend
All resources: Shelve

Is cost reduction a primary goal?

No: Any option works
Moderate savings: Suspend
Maximum savings: Shelve

Common Mistakes to Avoid

Leaving Instances Paused for Days

This wastes resources and money. Paused instances hold onto RAM and vCPU allocations that could serve other workloads. If you need an instance inactive for more than an hour, suspend or shelve it instead.

Shelving Instances That Need Quick Recovery

Shelve introduces several minutes of recovery time. If you need an instance back online in under 30 seconds, use suspend instead.

Forgetting IP Address Changes After Unshelve

Shelved instances typically receive new IP addresses when unshelved. Update DNS records, firewall rules, and application configurations accordingly.

Suspending Instances with Massive RAM

An instance with 256GB of RAM creates a 256GB memory snapshot on disk when suspended. Ensure adequate disk space is available on the compute node before suspending large instances.

Troubleshooting Common Issues

Pause Operation Fails

Symptom: Pause command returns an error or instance remains in active state.

Common Causes:

Hypervisor does not support pause (rare with modern OpenStack)
Instance is already in a transitional state
Compute node has insufficient resources

Resolution: Check instance status with openstack server show and review compute node logs.

Resume Takes Longer Than Expected

Symptom: Suspended instance takes minutes to resume instead of seconds.

Common Causes:

Large RAM allocation requires time to read from disk
Compute node experiencing high I/O load
Disk backend is slow or congested

Resolution: Monitor compute node performance and consider shelving instead for long-term inactivity.

Unshelve Fails or Hangs

Symptom: Instance remains in "shelved_offloaded" state after unshelve command.

Common Causes:

No compute nodes have sufficient resources available
Snapshot is corrupted or missing
Image storage backend is unavailable

Resolution: Check compute node availability, verify snapshot exists in image storage, and review scheduler logs.

Conclusion

Understanding the differences between pause, suspend, and shelve gives you precise control over instance lifecycle management in OpenStack. Each operation serves specific use cases with distinct tradeoffs in resource allocation, recovery time, and cost.

Use pause for brief freezes lasting minutes where instant recovery matters. Use suspend for medium-term holds lasting hours where you need memory state preservation and quick recovery. Use shelve for long-term storage lasting days or months where maximum cost savings and resource release are priorities.

Choosing the right state change for your situation optimizes both resource utilization and operational costs while ensuring instances return to service as quickly as your requirements demand.

For more information about managing OpenStack instances with InMotion Cloud, visit our support documentation or contact our cloud architecture team.