While removing Site Content Type you got the error that "The content type "My Content Type" is part of an application feature". The fix is rather easy and simple.

The error looked like this:

Exception calling "Delete" with "0" argument(s): "The content type "CT_fss" is part of an application feature."
At line:1 char:16
+ $cons[0].Delete <<<< ()
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : DotNetMethodException

This error is because the Content Type, that you want to remove, is used by some SharePoint list instance. If the list with this Content Type was the only one than I should give you the advice just to remove list and to clean Recycle Bin on Site Collection level after that.