Exporting HTML Files

If I am exporting .html files from my App do I need to add that type to myfiletypes? If so what would that entry look like?
Also same question for the Windows export how do I associate that file to Explorer?

To just create file, you don’t need a file type set.
You can use BinaryStream and write your html text into it.

here are examples for files types in dialog used as filter.
at windows html should be assigned with the default browser, if not you can use open with …

Will Windows automatically allocate an icon or do I have to do that in Innosetup?

if you create a .html file, you get the default icon.

But InnoSetup can create new icons for your own file extensions.