patch 9.1.1339: missing out-of-memory checks for enc_to_utf16()/utf16_to_enc()

Problem:  missing out-of-memory checks for enc_to_utf16() and
          utf16_to_enc()
Solution: Add out-of-memory checks and fix a few other minor issues
          (John Marriott)

This change does:
-  add missing out-of-memory checks for enc_to_utf16() and
   utf16_to_enc()
-  add a small optimisation in mch_errmsg_c() and mch_msg_c() (in
   message.c) to only call STRLEN() if needed.
-  fix a memory leak in winpty_term_and_job_init() (in terminal.c).

closes: #17191

Signed-off-by: John Marriott <basilisk@internode.on.net>
Signed-off-by: Christian Brabandt <cb@256bit.org>
diff --git a/src/fileio.c b/src/fileio.c
index 0ccfd2d..f6e35ab 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -5476,6 +5476,8 @@
     // "sh".  NOTE: This also checks 'shellcmdflag' to help those people who
     // didn't set 'shellslash' but only if not using PowerShell.
     retval = utf16_to_enc(itmp, NULL);
+    if (retval == NULL)
+	return NULL;
     shname = gettail(p_sh);
     if ((*p_shcf == '-' && !(strstr((char *)shname, "powershell") != NULL
 			     || strstr((char *)shname, "pwsh") != NULL ))