I suppose you mean "training performance". For simpler problems, this doesn't really seem to matter. If it takes 10 seconds or 1 minute to train, who cares. It will only train once. It is not like deep networks, which take days on a eight GPUs rig sometimes.

If you mean "inference performance", there might be an advantage over SVMs, as inference runs as many times as needed, and each cost some dollars. However, I assume a properly train neural net would be as accurate and faster at inference, as it has less operations (and can be pruned for extra savings).