Redis Junk response from redis

Hi,

We are using a ecommerce website which is Magento base. For caching purpose using redis(v=5.0.5). During processing of an order we had junk response from redis and further that transaction was failed.

Looks all communication with redis for affected order have junk response from redis. 1st stack trace we are trying to save something in redis and we have junk response. For 2nd and 3rd stack trace looks while getting data have junk response.

2020-03-02T11:20:22+00:00 ERROR (400): 
CredisException: Invalid response: !gÀ/õ∏©LùPã„Æ˚&ŸiZ?\!Ôæ‹}wÃjF§±ÄäYìÔyı€¡…‰Ω<“Nì®RÚ±AêÇ4¶—Vî*= in /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Credis/Client.php:1199
Stack trace:
#0 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Credis/Client.php(904): Credis_Client->read_reply('hget')
#1 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Cm/Cache/Backend/Redis.php(300): Credis_Client->__call('hget', Array)
#2 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Zend/Cache/Core.php(390): Cm_Cache_Backend_Redis->save('a:3799:{i:1;a:4...', 'f92_c225e465710...', Array, 86400)
#3 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Varien/Cache/Core.php(145): Zend_Cache_Core->save('a:3799:{i:1;a:4...', 'f92_c225e465710...', Array, 86400, 8)
#4 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Cache/Model/Cache.php(2): Varien_Cache_Core->save('a:3799:{i:1;a:4...', 'c225e46571060c1...', Array, 86400, 8)
#5 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Configurator/Helper/Expression.php(2): Dyna_Cache_Model_Cache->save('a:3799:{i:1;a:4...', 'category_names_...', Array, 86400)
#6 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Configurator/Helper/Expression.php(2): Dyna_Configurator_Helper_Expression->setCategories()
#7 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Configurator/Helper/Expression.php(2): Vfde_Configurator_Helper_Expression->setCategories()
#8 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(555): Dyna_Configurator_Helper_Expression->__construct()
#9 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Configurator/Model/Expression/Cart.php(2): Mage::helper('dyna_configurat...')
#10 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/Config.php(1357): Dyna_Configurator_Model_Expression_Cart->__construct(Array)
#11 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(471): Mage_Core_Model_Config->getModelInstance('dyna_configurat...', Array)
#12 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/PriceRules/Model/Validator.php(2): Mage::getModel('dyna_configurat...')
#13 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Varien/Object.php(112): Dyna_PriceRules_Model_Validator->_construct()
#14 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/Config.php(1357): Varien_Object->__construct(Array)
#15 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(471): Mage_Core_Model_Config->getModelInstance('salesrule/valid...', Array)
#16 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(485): Mage::getModel('salesrule/valid...', Array)
#17 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/SalesRule/Model/Quote/Freeshipping.php(40): Mage::getSingleton('salesrule/valid...')
#18 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/Config.php(1357): Mage_SalesRule_Model_Quote_Freeshipping->__construct(Array)
#19 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(471): Mage_Core_Model_Config->getModelInstance('Omnius_Checkout...', Array)
#20 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Sales/Model/Quote/Address/Total/Collector.php(116): Mage::getModel('Omnius_Checkout...')
#21 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Sales/Model/Quote/Address/Total/Collector.php(145): Mage_Sales_Model_Quote_Address_Total_Collector->_initModelInstance('Omnius_Checkout...', 'freeshipping', Object(Mage_Core_Model_Config_Element))
#22 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Sales/Model/Quote/Address/Total/Collector.php(81): Mage_Sales_Model_Quote_Address_Total_Collector->_initModels()
#23 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/Config.php(1357): Mage_Sales_Model_Quote_Address_Total_Collector->__construct(Array)
#24 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(471): Mage_Core_Model_Config->getModelInstance('sales/quote_add...', Array)
#25 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(485): Mage::getModel('sales/quote_add...', Array)
#26 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Sales/Model/Quote/Address.php(987): Mage::getSingleton('sales/quote_add...', Array)
#27 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Sales/Model/Quote/Address.php(1012): Mage_Sales_Model_Quote_Address->getTotalCollector()
#28 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Checkout/Model/Sales/Quote.php(2): Mage_Sales_Model_Quote_Address->collectTotals()
#29 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Sales/Model/Quote/Payment.php(148): Dyna_Checkout_Model_Sales_Quote->collectTotals()
#30 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Checkout/Model/Type/Multishipping.php(398): Mage_Sales_Model_Quote_Payment->importData(Object(Varien_Object))
#31 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Validators/Helper/SubmitOrder.php(2): Mage_Checkout_Model_Type_Multishipping->setPaymentMethod(Array)
#32 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Validators/Helper/SubmitOrder.php(2): Vfde_Validators_Helper_SubmitOrder->setDefaultPayment()
#33 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Checkout/controllers/IndexController.php(2): Vfde_Validators_Helper_SubmitOrder->superOrderCreatePreValidations()
#34 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Checkout/Model/Order.php(2): Vfde_Checkout_IndexController->createOrder('5172')
#35 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/Consumers/SubmitOrderConsumer.php(2): Vfde_Checkout_Model_Order->createOrder(Array)
#36 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/Dispatcher.php(2): Dyna_Job_Model_Consumers_SubmitOrderConsumer->process(Object(Dyna_Job_Model_Jobs_SubmitOrder))
#37 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/StandaloneJobRunner.php(2): Dyna_Job_Model_Dispatcher->dispatch(Object(Dyna_Job_Model_Jobs_SubmitOrder))
#38 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/shell/job_processor_runner.php(40): Dyna_Job_Model_StandaloneJobRunner->run('{"type":"submit...')
#39 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/shell/job_processor_runner.php(62): Dyna_Job_Processor_Runner->run()
#40 {main}
2020-03-02T11:20:22+00:00 ERROR (400): 
Vfde_Validators_Model_Exception_SubmitOrder: Invalid response: !gÀ/õ∏©LùPã„Æ˚&ŸiZ?\!Ôæ‹}wÃjF§±ÄäYìÔyı€¡…‰Ω<“Nì®RÚ±AêÇ4¶—Vî*= in /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/Config.php:1357
Stack trace:
#0 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(471): Mage_Core_Model_Config->getModelInstance('vfde_validators...', 'Invalid respons...')
#1 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Validators/Helper/SubmitOrder.php(2): Mage::getModel('vfde_validators...', 'Invalid respons...')
#2 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Validators/Helper/SubmitOrder.php(2): Vfde_Validators_Helper_SubmitOrder->setDefaultPayment()
#3 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Checkout/controllers/IndexController.php(2): Vfde_Validators_Helper_SubmitOrder->superOrderCreatePreValidations()
#4 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Checkout/Model/Order.php(2): Vfde_Checkout_IndexController->createOrder('5172')
#5 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/Consumers/SubmitOrderConsumer.php(2): Vfde_Checkout_Model_Order->createOrder(Array)
#6 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/Dispatcher.php(2): Dyna_Job_Model_Consumers_SubmitOrderConsumer->process(Object(Dyna_Job_Model_Jobs_SubmitOrder))
#7 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/StandaloneJobRunner.php(2): Dyna_Job_Model_Dispatcher->dispatch(Object(Dyna_Job_Model_Jobs_SubmitOrder))
#8 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/shell/job_processor_runner.php(40): Dyna_Job_Model_StandaloneJobRunner->run('{"type":"submit...')
#9 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/shell/job_processor_runner.php(62): Dyna_Job_Processor_Runner->run()
#10 {main}
2020-03-02T11:20:23+00:00 ERROR (400): 
CredisException: Invalid response: ÒQ˘E @7üLCµ,àUÙN>ø‹›‡dÚ™jd†ûB¥Ë
π3%5Ipê›Öh∏‡Òò—bXO'|hıAx≠îIHÌ∫¸Œi~π≠ìÔπ‹Ì‡dÚê^™ûk9?d+d…∫ä!R]‡ 8øûÏëÓ¨ìéò⁄J∂ôg oıͶ3œ{Íùº±ÒÁı3flN&Ô: eË€; in /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Credis/Client.php:1199
Stack trace:
#0 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Credis/Client.php(904): Credis_Client->read_reply('hget')
#1 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Cm/Cache/Backend/Redis.php(198): Credis_Client->__call('hget', Array)
#2 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Zend/Cache/Core.php(306): Cm_Cache_Backend_Redis->load('f92_AUDIT_CONFI...', false)
#3 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/lib/Varien/Cache/Core.php(158): Zend_Cache_Core->load('f92_AUDIT_CONFI...', false, false)
#4 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/Cache.php(379): Varien_Cache_Core->load('AUDIT_CONFIG')
#5 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/App.php(1153): Mage_Core_Model_Cache->load('audit_config')
#6 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Omnius/Audit/Model/Processor.php(2): Mage_Core_Model_App->loadCache('audit_config')
#7 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Omnius/Audit/Model/Processor.php(2): Omnius_Audit_Model_Processor->getConfig()
#8 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Omnius/Audit/Model/Processor.php(2): Omnius_Audit_Model_Processor->getEntityHandler()
#9 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Omnius/Audit/Model/Observer.php(2): Omnius_Audit_Model_Processor->processEntity(Object(Varien_Event_Observer), 'save')
#10 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Omnius/Audit/Model/Observer.php(2): Omnius_Audit_Model_Observer->handleEntityEvent(Object(Varien_Event_Observer), 'save')
#11 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/App.php(1358): Omnius_Audit_Model_Observer->handleEntitySaveEvent(Object(Varien_Event_Observer))
#12 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/App.php(1337): Mage_Core_Model_App->_callObserverMethod(Object(Dyna_Audit_Model_Observer), 'handleEntitySav...', Object(Varien_Event_Observer))
#13 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(456): Mage_Core_Model_App->dispatchEvent('model_save_befo...', Array)
#14 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Mage/Core/Model/Abstract.php(396): Mage::dispatchEvent('model_save_befo...', Array)
#15 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Sales/Model/Quote.php(323): Mage_Core_Model_Abstract->_beforeSave()
#16 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Omnius/Checkout/Model/Sales/Quote.php(2): Mage_Sales_Model_Quote->_beforeSave()
#17 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Checkout/Model/Sales/Quote.php(2): Omnius_Checkout_Model_Sales_Quote->_beforeSave()
#18 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Checkout/Model/Sales/Quote.php(2): Dyna_Checkout_Model_Sales_Quote->_beforeSave()
#19 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Mage/Core/Model/Abstract.php(322): Vfde_Checkout_Model_Sales_Quote->_beforeSave()
#20 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Sales/Model/Quote.php(1966): Mage_Core_Model_Abstract->save()
#21 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/community/Omnius/Checkout/Model/Sales/Quote.php(2): Mage_Sales_Model_Quote->save()
#22 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Configurator/Model/Observer.php(2): Omnius_Checkout_Model_Sales_Quote->save()
#23 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/App.php(1358): Vfde_Configurator_Model_Observer->initQuote(Object(Varien_Event_Observer))
#24 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Core/Model/App.php(1337): Mage_Core_Model_App->_callObserverMethod(Object(Vfde_Configurator_Model_Observer), 'initQuote', Object(Varien_Event_Observer))
#25 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/Mage.php(456): Mage_Core_Model_App->dispatchEvent('checkout_quote_...', Array)
#26 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Checkout/Model/Session.php(159): Mage::dispatchEvent('checkout_quote_...', Array)
#27 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/core/Mage/Checkout/Model/Session.php(369): Mage_Checkout_Model_Session->getQuote()
#28 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Vfde/Customer/Helper/Data.php(2): Mage_Checkout_Model_Session->clear()
#29 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/Consumers/SubmitOrderConsumer.php(2): Vfde_Customer_Helper_Data->unloadCustomer()
#30 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/Dispatcher.php(2): Dyna_Job_Model_Consumers_SubmitOrderConsumer->process(Object(Dyna_Job_Model_Jobs_SubmitOrder))
#31 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/app/code/local/Dyna/Job/Model/StandaloneJobRunner.php(2): Dyna_Job_Model_Dispatcher->dispatch(Object(Dyna_Job_Model_Jobs_SubmitOrder))
#32 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/shell/job_processor_runner.php(40): Dyna_Job_Model_StandaloneJobRunner->run('{"type":"submit...')
#33 /opt/SP/WEB_DATA/zend/var/apps/http/osf-telesales-int/80/6.2.1-7_19311908/shell/job_processor_runner.php(62): Dyna_Job_Processor_Runner->run()
#34 {main}

Comment From: mgravell

This feels like something that is most likely going to be a "client" issue than a "server" issue. The first question here is likely to be:

how do we repro this?

Without that, it is virtually impossible to comment. Personally, the first thing I'd suspect is : 2 threads fighting over the same client/socket instance at the same time, and getting in a mess. But again, without a repro...