[vlc-devel] [PATCH] linux: vlc_memfd: fallback to POSIX implementation

[Thomas Guillem]

>From man (2) open:

O_TMPFILE  requires  support by the underlying filesystem; only a subset of
Linux filesystems provide that support.  In the initial implementation, support
was provided in  the  ext2,  ext3,  ext4,  UDF, Minix,  and shmem filesystems.
Support for other filesystems has subsequently been added as follows: XFS
(Linux 3.15); Btrfs (Linux 3.16); F2FS (Linux 3.16); and ubifs (Linux 4.9)

One must check for two different error codes, EISDIR and ENOENT, when trying to
determine whether the kernel supports O_TMPFILE functionality.

Fixes #22250
---
 src/linux/filesystem.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/src/linux/filesystem.c b/src/linux/filesystem.c
index 6634c3d374..2e3f975179 100644
--- a/src/linux/filesystem.c
+++ b/src/linux/filesystem.c
@@ -36,11 +36,22 @@
 
 int vlc_memfd(void)
 {
+    int fd;
 #ifdef HAVE_MEMFD_CREATE
-    int fd = memfd_create(PACKAGE_NAME"-memfd",
-                          MFD_CLOEXEC | MFD_ALLOW_SEALING);
+    fd = memfd_create(PACKAGE_NAME"-memfd", MFD_CLOEXEC | MFD_ALLOW_SEALING);
     if (fd != -1 || errno != ENOSYS)
         return fd;
 #endif
-    return open("/tmp", O_RDWR | O_CLOEXEC | O_TMPFILE, S_IRUSR | S_IWUSR);
+    /* Fallback to open with O_TMPFILE, */
+    fd = open("/tmp", O_RDWR | O_CLOEXEC | O_TMPFILE, S_IRUSR | S_IWUSR);
+
+    if (fd == -1 && (errno == EISDIR || errno == ENOENT))
+    {
+        /* Fallback to POSIX implementation if O_TMPFILE is not supported */
+        char bufpath[] = "/tmp/"PACKAGE_NAME"XXXXXX";
+        fd = vlc_mkstemp(bufpath);
+        if (fd != -1)
+            unlink(bufpath);
+    }
+    return fd;
 }
-- 
2.20.1