In your sites template you can adjust the desired doctype to your liking by setting the value of the the property:
    page.config.doctype

There are several options which TYPO3 accepts by default and will give appropriate settings for, these are:
    xhtml_trans
    xhtml_frames
    xhtml_strict
    xhtml_11
    xhtml_2
    none

If none of these are the doctype you wish to use, you can set your doctype directly into the value of page.config.doctype

Furthermore, in order to ensure that ie does not run in quirks mode (which can ofcourse cause problems in the output your users browsers may recieve) we can set the following property to disable the display of the opening XML doctype declaration:
pafe.config.xmlprologue = none