Troubleshoot Slow & Freezing Craft CMS Imports

Having issues with slow imports and freezes in your Craft CMS project, especially when using Feed Me? You're not alone! Let's dive into some common causes and solutions to get your imports running smoothly again. We will cover everything from server configuration to plugin settings, ensuring a comprehensive approach to resolving your import woes.

Understanding the Problem

So, you're trying to import content into your Craft CMS site, and it's running slower than a snail in molasses. Initially, you were importing thousands of entries per day, but now you're struggling to get even a few hundred in. The Queue Manager shows the import getting stuck in a Pending state, and you have to refresh the page constantly to keep it going. Frustrating, right? You've already tried increasing the max_execution_time and max_input_vars in your php.ini file, but it's still not cutting it. Plus, you're not even updating entries; you're just creating new ones using a single ID field as the unique identifier. Let's figure out what's going on.

Initial Checks: Server Configuration and Resources

PHP Configuration

First things first, let's double-check your PHP configuration. While you've increased max_execution_time and max_input_vars, ensure these values are sufficiently high. A good starting point for max_execution_time is 300 (seconds), but for large imports, you might need even more. For max_input_vars, try setting it to at least 5000 or higher. Also, make sure you've restarted your web server (e.g., Apache, Nginx) after making these changes for them to take effect. Additionally, confirm that the memory_limit is adequately set (e.g., 256M or 512M), as imports can be memory-intensive.

Database Server

Your database server is a crucial component. Ensure your MySQL server has enough resources (CPU, RAM, disk I/O). Slow queries can bottleneck the import process. Use tools like mysqltop or the performance monitoring features of your database management tool (e.g., phpMyAdmin, Sequel Pro) to check for slow queries or resource constraints. Optimizing your database configuration (e.g., innodb_buffer_pool_size for InnoDB) can also significantly improve performance. Regularly check your database server's error logs for any warnings or errors that could indicate underlying issues.

Web Server

Your web server's configuration can also impact import speed. Ensure it's properly configured to handle concurrent requests. For Apache, check the MaxRequestWorkers setting in your httpd.conf file. For Nginx, review the worker_processes and worker_connections settings in your nginx.conf file. Using a caching mechanism (e.g., Redis, Memcached) can reduce the load on your database server by caching frequently accessed data.

Plugin-Specific Optimizations: Feed Me

Feed Me Settings

Feed Me is a fantastic plugin, but its settings can significantly impact performance. Here are a few key areas to investigate:

• Batch Size: You mentioned experimenting with different batch sizes (500 vs. 5000). A smaller batch size can reduce memory usage and prevent timeouts, but it might increase the overall import time due to the overhead of starting and stopping the import process more frequently. Experiment to find the optimal balance for your specific data and server configuration.
• Data Mapping: Ensure your data mapping is as efficient as possible. Avoid complex transformations or lookups within Feed Me if you can preprocess the data beforehand. Using custom fields efficiently and avoiding unnecessary field mappings can also help.
• Element Queries: If you're using element queries within your Feed Me configuration (e.g., to relate entries), ensure these queries are optimized. Complex queries can slow down the import process significantly. Consider simplifying the queries or using alternative methods to establish relationships.

Feed Me and Queue Manager

The interaction between Feed Me and the Queue Manager is crucial. The "stuck in Pending" issue suggests that the queue worker might be timing out or encountering errors. Here’s what to check:

• Queue Worker Configuration: Ensure your queue worker is properly configured and running. Check the Craft CMS logs (storage/logs/) for any errors related to the queue worker. You might need to increase the queue worker's timeout settings to allow it to complete long-running tasks.
• Overlapping Tasks: Avoid running multiple resource-intensive tasks concurrently. If you have other queue tasks running at the same time as the Feed Me import, they could be competing for resources and causing slowdowns. Try running the Feed Me import in isolation to see if it improves performance.
• Database Transactions: Feed Me uses database transactions to ensure data integrity. Long-running transactions can lock tables and prevent other operations from running. Breaking the import into smaller batches can reduce the duration of these transactions and improve concurrency.

Database Considerations

Indexing

Proper indexing is essential for database performance. Ensure that the fields you're using as unique identifiers (and any fields you're querying during the import) are properly indexed. Use the EXPLAIN statement in MySQL to analyze your queries and identify missing indexes. Adding indexes to relevant columns can dramatically speed up queries.

Database Optimization

Regularly optimize your database tables to reclaim space and improve performance. Use the OPTIMIZE TABLE command in MySQL to rebuild indexes and defragment tables. Consider using a database optimization tool or script to automate this process. Regularly backing up your database is also crucial, especially before performing large imports or updates.

Connection Limits

Ensure your database server has enough connection slots available. If you're hitting the maximum number of connections, new connections will be blocked, causing slowdowns. Check the max_connections setting in your MySQL configuration and increase it if necessary.

Code-Level Debugging

Profiling

Use a profiling tool (e.g., Xdebug) to identify bottlenecks in your code. Profiling can help you pinpoint the exact lines of code that are consuming the most time and resources. Focus on optimizing these areas to improve overall performance. Tools like Blackfire.io can also provide valuable insights into your application's performance.

Logging

Add detailed logging to your Feed Me configuration and custom code. Log the start and end times of each import step, as well as any relevant data or errors. This can help you track down the source of slowdowns and identify patterns. Use Craft CMS's built-in logging capabilities to record important events and errors.

Event Listeners and Hooks

If you're using any event listeners or hooks during the import process, ensure they're not causing performance issues. Disable them temporarily to see if it improves import speed. Optimize any custom code within these listeners or hooks to minimize their impact on performance.

Final Checklist and Best Practices

• Keep Craft CMS and Plugins Updated: You've already updated Craft CMS and Feed Me, which is excellent. Always stay on the latest stable versions to benefit from performance improvements and bug fixes.
• Monitor Server Resources: Continuously monitor your server's CPU, RAM, disk I/O, and network usage during the import process. This can help you identify resource bottlenecks and optimize your server configuration.
• Test in a Staging Environment: Always test large imports in a staging environment before running them in production. This allows you to identify and resolve any issues without impacting your live site.
• Optimize Images: If you're importing images, ensure they're optimized for the web. Large, unoptimized images can significantly slow down the import process.
• Use Caching: Implement caching strategies (e.g., using Redis or Memcached) to reduce the load on your database server and improve overall performance.

By systematically addressing these areas, you should be able to diagnose and resolve the slow import and freezing issues in your Craft CMS project. Good luck, and happy importing!