Memcached crashes after W3TC 0.15.0 update | BoldGrid
Viewing 16 posts - 1 through 16 (of 16 total)
  • Author
    Posts
  • #27562 Reply
    Alex Billington
    Guest

    I’m currently running WordPress version 5.4.2, PHP version 7.3.22. I’ve been using the W3TC for years, no issues at all until today. After updating the plugin from 0.14.4 to the latest 0.15.0 version, we experienced a MAJOR server issue today when Memcached ended up overloaded. My sysadmin discovered that “the connection count was increasing along with the newly created webserver processes on the frontend servers and as these were not terminated, they are staying connected to the memcached server.” We have never experienced this before. It occurred after publishing a new post, and suddenly the entire site crashed and memcached was overloaded.

    I am also experiencing another weird caching and draft/revision issue with the WordPress backend. I can’t tell if it’s related to W3TC, but it only started acting this way after upgrading the plugin yesterday. I notice that if I write in a post, save the draft, the post sometimes reverts to an early draft that is from the previous save. However, it is saved in the database. But then if I refresh the page and try to load the “previous auto-draft” nothing new comes up. It shows up in the “revisions” section, but nothing happens when I try to restore this draft. Eventually it’s completely gone and there’s no way I can recover the previous text until I write and save the draft all over again. I have no idea why this is happening or how to fix it.

    #27567 Reply
    Jesse Owens
    Keymaster

    Hello Alex-

    Thank you very much for the report. This is the first report we’ve had relating to Memcached overloading, so I’ve informed the developers about the issue.

    For now, I’d recommend reverting to 0.14.4, using a plugin like WP Rollback.

    After you revert, check to see if the issues with post revisions are fixed as well.

    I’ll update you here as soon as I have news from the development team for you.

    #27569 Reply
    Alex Billington
    Guest

    Hi. I did roll back the plugin to 0.14.4 for now. I think the Memcached issues are resolved, for the moment. Something about out configuration must cause issues with the updated plugin (since I noticed there was a change to the Memached configuration in this update). But that’s okay for now, and eventually I’ll update our WordPress version, PHP version, and try updating this plugin again in a few months once you have another new patched version.

    As for the issues with the auto-draft in posts, it is still happening even after changing the plugin back. I have no idea why. I don’t even know where to look or where to start. Does it have to do with browser cache? Or does it have to do with the databases and the way it’s saving drafts/revisions while editing? I don’t know…

    One other important cache question – I am already using the W3TC plugin with browser cache enabled through the plugin. My sysadmin also recently turned on the browser cache settings directly ON the server so it does this through its own configuration. Will this double setup cause problems? Will it override one cache setup? Is this causing issues or can we leave both browser cache settings enabled and it will still run okay?

    #27595 Reply
    Jesse Owens
    Keymaster

    Hi Alex-

    Autosaves are stored in your browser, but I can’t think of a way that browser cache would affect the creation of autosaves. One thing you can try is to define the autosave interval in your wp-config.php file, like so:

    define('AUTOSAVE_INTERVAL', 120);

    Revisions aren’t saved to the database until you either save a draft or update the post. You mentioned that you are saving them, and it’s being included in the database, but the post isn’t updating to the newest revision. I have to say I’ve never heard of an issue like that before.

    If the issue is related to W3 Total Cache in any way, you may be able to resolve it by checking the options to Don’t cache pages for logged in users in Performance > Page Cache and Disable minify for logged in users in Performance > Minify. That being said, W3 Total Cache doesn’t cache administration screens like the editor.

    For your second question about browser cache, W3 Total Cache handles this by making modifications to your site’s main .htaccess file. Generally speaking, changes in .htaccess will override your server’s settings, so you can consult your sysadmin on which settings they’re setting up and make sure that your W3 Total Cache settings in Performance > Browser Cache match.

    #27664 Reply
    Alex Billington
    Guest

    Thank you for the answers. I’ve never heard of this problem either! Very confusing, no idea why it was acting this way. So far since reverting back to 0.14.4 it has been running okay again. I also cleaned up and optimized the databases, incase there might’ve been some corruption.

    We’re running a nginx server so there’s no htaccess file. But we have already activated the browser cache through W3TC (I’m not exactly sure how it does this on nginx?) and also directly on the server. I’ll ask my sysadmin to doublecheck the values. I noticed that the browser cache settings from W3TC are saved in the “nginx.conf” file, which I assume is fine, but I don’t know if this conflicts with the actual server-side browser caching systems. Perhaps it all works together and makes one setting redundant, but I’m not entirely sure if there are some sort of browser errors because of duplicate cache requests from the server.

    #27666 Reply
    Jesse Owens
    Keymaster

    Hi Alex-
    Similar to the way that Apache treats local .htaccess files, NGINX will use a local nginx.conf file to override the server-level settings on a per-directory basis.

    That means it won’t create any conflicts with your server-level settings, but it might change them if the plugin settings differ from what your System Administrator is setting up.

    #27833 Reply
    Jesse Owens
    Keymaster

    Hi Alex-

    I wanted to let you know that W3 Total Cache 0.15.1 has been released with a bugfix for the memcached issue. Please let us know if you have any more issues!

    #27841 Reply
    Alex Billington
    Guest

    Thanks for the note! I’ll run the update weekend and do some testing to see if there’s any other issues. Hopefully this patched the problem – at least with memcache.

    #27929 Reply
    Alex Billington
    Guest

    Nope. Even more problems with memcache in this latest version. Not sure what happened, but today all of the memcache servers crashed. The plugin is giving me this error now:

    The following memcached servers are not responding or not running:

    Database Cache: 127.0.0.1:11211.
    Object Cache: 127.0.0.1:11211.
    Page Cache: 127.0.0.1:11211.

    This message will automatically disappear once the issue is resolved.

    No idea why but something isn’t right with the way memcache works with this latest version of the plugin. Both updates keep causing problems.

    #27955 Reply
    Jesse Owens
    Keymaster

    Hi Alex-

    You mentioned that the servers had crashed, this error would indicate that the communication to the server is failing altogether. Double-check your addresses and credentials. You mentioned that there were external Memcached servers (multiple) but these addresses all refer to a Memcached server running locally on your webserver.

    Going back to your first message,

    “the connection count was increasing along with the newly created webserver processes on the frontend servers and as these were not terminated, they are staying connected to the memcached server.”

    Have you tried disabling the option for Persistent connection? Your sysadmin’s diagnosis would seem to indicate that the persistent connection might be causing issues.

    #27958 Reply
    Alex Billington
    Guest

    This time no the server itself was able to run. But the memcache servers are crashing entirely once they get overloaded. Everything worked fine for YEARS without any issue with memcache and the plugin. We only ever started experiencing issues like this with this server after upgrading to 0.15.0. I don’t know if there is some compatibility problem or something else misconfigured, but again, all of this began only after the upgrade to 0.15.0 (and 0.15.1). Considering the most recent version updates have made some significant changes to memcache and the way things are handled, it leads me to believe it’s all connected to the plugin’s latest updates and how things are working. But we can find any other errors or anything else in the logs to help clarify and explain the issues. All that we see are crashing memcache servers which (sometimes) leads to taking down the entire server (due to overloads or some other issue like that).

    #27961 Reply
    Jesse Owens
    Keymaster

    Hi Alex-
    Thank you very much for the additional info, I’m consulting with the developers and the Support Team to see what else we can check for you. We’ll update you here when we have some more information for you.

    #27974 Reply
    Jesse Owens
    Keymaster

    Hi Alex-

    So far as we know, the bugfixes in 0.15.1 resolved the memcached problems for all the other users who reported issues in 0.15.0, so we’d like to get a little more information to help troubleshoot.

    First, can you run the following commands, and reply back with the output?

    memcached-tool 127.0.0.1:11211 display
    memcached-tool 127.0.0.1:11211 stats

    If you’re not seeing any output from those, check to see if the service is running, and check if you can connect to it:

    ps afux | grep memc
    telnet 127.0.0.1 11211
    > stats

    If you see the service running but not responding to status commands, try restarting the server:

    service memcached restart

     

    #28321 Reply
    Alex Billington
    Guest

    Here you go. I already reverted the plugin back to 0.14.4 again – just to be safe. Especially since it’s entirely stable on my WordPress and doesn’t cause any problems. I’m running on WP 5.4.2 and with this plugin. Here’s the memcache results. Everything runs fine and everything is smooth with this 0.14.4 version. But updating to anything 0.15+ just causes problems – both with the backend/saving drafts, and with the memcache crashing again. Still no idea why…

    # Item_Size Max_age Pages Count Full? Evicted Evict_Time OOM
    2 120B 113s 1 7 yes 0 0 0
    4 192B 86392s 53 108591 yes 1 475439 0
    5 240B 45599s 1 2791 yes 0 0 0
    6 304B 127127s 1 2130 yes 0 0 0
    7 384B 86379s 11 25579 yes 0 0 0
    8 480B 274392s 24 43914 yes 0 0 0
    9 600B 86392s 17 27959 yes 0 0 0
    10 752B 86281s 34 44029 yes 0 0 0
    11 944B 86373s 13 11875 yes 0 0 0
    12 1.2K 86401s 13 9353 yes 0 0 0
    13 1.4K 86401s 48 30682 yes 0 0 0
    14 1.8K 86373s 54 28527 yes 0 0 0
    15 2.3K 86373s 87 36385 yes 0 0 0
    16 2.8K 58527s 66 18009 yes 0 0 0
    17 3.5K 81817s 116 30340 yes 0 0 0
    18 4.4K 7740s 52 1646 yes 0 0 0
    19 5.5K 35222s 24 2032 yes 0 0 0
    20 6.9K 3555s 4 213 yes 0 0 0
    21 8.7K 85982s 156 17848 yes 7 86382 0
    22 10.8K 43802s 104 5186 yes 0 0 0
    23 13.6K 67093s 630 18672 yes 1 465122 0
    24 16.9K 43802s 131 3656 yes 0 0 0
    25 21.2K 3621s 45 405 yes 1 27265 0
    26 26.5K 3645s 49 262 yes 0 0 0
    27 33.1K 3316s 16 82 yes 0 0 0
    28 41.4K 3651s 14 43 yes 0 0 0
    29 51.7K 38820s 28 113 yes 0 0 0
    30 64.7K 60655s 42 163 yes 0 0 0
    31 80.9K 3319s 41 15 yes 0 0 0
    32 101.1K 3401s 70 8 yes 0 0 0
    33 126.3K 61109s 42 269 yes 0 0 0
    34 157.9K 61206s 41 232 yes 0 0 0
    35 197.4K 61235s 26 121 yes 0 0 0
    36 246.8K 2825s 4 1 yes 0 0 0
    37 308.5K 13659s 2 1 yes 0 0 0
    38 385.6K 12056s 2 1 yes 0 0 0
    39 482.0K 13659s 3 2 yes 0 0 0
    40 602.5K 12056s 3 1 yes 0 0 0
    41 753.1K 12057s 3 1 yes 0 0 0
    
    ----
    
    #127.0.0.1:11211 Field Value
    accepting_conns 1
    auth_cmds 0
    auth_errors 0
    bytes 1041799960
    bytes_read 44152075210
    bytes_written 331445781756
    cas_badval 0
    cas_hits 0
    cas_misses 0
    cmd_flush 0
    cmd_get 72482221
    cmd_set 11693559
    cmd_touch 0
    conn_yields 0
    connection_structures 32
    curr_connections 28
    curr_items 471212
    decr_hits 0
    decr_misses 0
    delete_hits 53369
    delete_misses 112172
    evicted_unfetched 9
    evictions 10
    expired_unfetched 2272958
    get_hits 54505775
    get_misses 17976446
    hash_bytes 4194304
    hash_is_expanding 0
    hash_power_level 19
    incr_hits 0
    incr_misses 0
    libevent 2.0.21-stable
    limit_maxbytes 2147483648
    listen_disabled_num 0
    pid 23411
    pointer_size 64
    reclaimed 2435690
    reserved_fds 20
    rusage_system 1120.630942
    rusage_user 484.521307
    threads 4
    time 1602723062
    total_connections 265185
    total_items 11693559
    touch_hits 0
    touch_misses 0
    uptime 723666
    version 1.4.15

    Let me know if any of this helps clear up issues?

    #28359 Reply
    Mikhail
    Guest

    Hi,
    I am experience absolutely same issue. Memcacheed is overloaded. Using 0.15.1 version. Restarting memcache helps but only temporary.
    Rolled back now to 0.14.4

    #28365 Reply
    Jesse Owens
    Keymaster

    Hi Alex and Mikhail-

    Thanks for the additional info and report. I’ve sent this information to the developers for more investigation. I’ll update you here as soon as I’ve heard back.

Viewing 16 posts - 1 through 16 (of 16 total)
Reply To: Memcached crashes after W3TC 0.15.0 update
Your information: