When you enable save game manager and add the directory to the plugin manager, the save files are moved from the save folder in the documents folder. I think plugin manager moves the files into its folder in the game directory. Then if you remove the directory from within plugin manager, it restores the save files to the normal save folder in the documents folder. But if you disable save game manager before you remove the directory from withing the plugin manager, your save are lost forever it seems.
In a way it's my own fault for losing my save files. But mainly, I think the Plugin Manager is to blame. It shouldnt be possible for the Plugin Manager to permanently lose save files. This needs to be adressed. A safer way of handling the save files is needed or at least a warning that the save files should be backed up manually before enabling save game manager.

Anyways, if someone still has any ideas how to restore my save files, please let me know.