Elasticsearch crash due to corrupted data

Published by bolmaster2

Elasticsearch doesn’t start? First thing to remember is where is it started from? Homebrew services? Let’s check that…

brew services list

Name              Status  User Plist
[email protected] started joel /Users/joel/Library/LaunchAgents/[email protected]

Realize elasticsearch is yellow ☝️…

Ok. So let’s check if it’s running?

ps aux | grep elasticsearch

Nope… Nothing there.

Ok, so let’s see what the logs say? Check the launch agent plist file. They are located under ~/Library/LaunchAgents/ according to the homebrew documentation. You can also find it from the brew services list command.

cat ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch\@2.4.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>KeepAlive</key>
    <false/>
    <key>Label</key>
    <string>[email protected]</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/opt/[email protected]/bin/elasticsearch</string>
    </array>
    <key>EnvironmentVariables</key>
    <dict>
    </dict>
    <key>RunAtLoad</key>
    <true/>
    <key>WorkingDirectory</key>
    <string>/usr/local/var</string>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/log/[email protected]</string>
    <key>StandardOutPath</key>
    <string>/usr/local/var/log/[email protected]</string>
  </dict>
</plist>

There it is (StandardErrorPath), let’s check that out…

tail -f -n 1000 /usr/local/var/log/[email protected]

Realize there’s broken files ☝️ which CorruptStateException implies:

Exception in thread "main" ElasticsearchException[failed to read [id:29, legacy:false, file:/usr/local/var/elasticsearch/elasticsearch_joel/nodes/0/_state/global-29.st]]; nested: IOException[failed to read [id:29, legacy:false, file:/usr/local/var/elasticsearch/elasticsearch_joel/nodes/0/_state/global-29.st]]; nested: CorruptStateException[Format version is not supported (resource SimpleFSIndexInput(path="/usr/local/var/elasticsearch/elasticsearch_joel/nodes/0/_state/global-29.st")): 1 (needs to be between 0 and 0)];
Likely root cause: org.elasticsearch.gateway.CorruptStateException: Format version is not supported (resource SimpleFSIndexInput(path="/usr/local/var/elasticsearch/elasticsearch_joel/nodes/0/_state/global-29.st")): 1 (needs to be between 0 and 0)

Delete shit

rm -rf /usr/local/var/elasticsearch

Restart it!

brew services restart [email protected]

Check if it’s running:

ps aux | grep elasticsearch
joel             97960   0,1  1,6  6920728 263744   ??  S    10:23am   0:06.96 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/local/Cellar/[email protected]/2.4.6/libexec -cp /usr/local/Cellar/[email protected]/2.4.6/libexec/lib/elasticsearch-2.4.6.jar:/usr/local/Cellar/[email protected]/2.4.6/libexec/lib/* org.elasticsearch.bootstrap.Elasticsearch start
joel             98305   0,0  0,0  4267752    716 s006  R+   10:23am   0:00.00 grep elasticsearch

Boom 💥. Work completed 😅.