In order for offline speech recognition to work, you will need to be running a Jelly Bean or higher device (e.g. Android 4.1+).
If you are using a Jelly Bean or higher device, make sure to download the data required for offline speech recognition.
Below is a screen shot showing the setting in your devices Settings > Voice input and output > Recognition options:
Make sure to install data for each language you use on your device.