“Type ‘Microsoft.Owin.OwinContext’ in assembly ‘Microsoft.Owin, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ is not marked as serializable.”

If you’re using Topshelf, you can configure what the Windows service does when the self-hosted process shuts down. See: http://docs.topshelf-project.com/en/latest/configuration/config_api.html#id1

Self-hosting should allow you to use those authentication methods (and any feature, really) as long as its functionality comes from application code (i.e. not an IIS module) and doesn’t have a dependency on HttpContext.Current.

Does self-hosting allow for the usual authentication methods such as Kerberos, SAML, etc?

You definitely can (and should) self-host ASP.NET Core MVC applications. It gives you the ability to develop in and deploy to non-Windows environments, since you wouldn’t be depending on IIS. Even hosting in IIS simply forwards the IIS request to the self-hosted Kestrel server.

That being said, I wouldn’t use the Kestrel server on the edge: I would have something like nginx sitting in front of it as a proxy, as it has been in production for much longer, and you get much more out of the box like load balancing.