Server-side file copying between smb shares on linux

2023-05-04

I needed to do a simple task: copy a file between two SMB shares residing on the same server. It would be wasteful to transfer this file back and forth between server and client machine using the network, so why not use the server-side copy feature?

Trying regular 'cp' command copies the file across the network, so there must be some other way. After a little googling, I've found on Samba wiki (https://wiki.samba.org/index.php/Server-Side_Copy) that Linux in fact does support server-side copying. The site mentioned 'cp --reflink' - which did not work for me - along with a program called 'cloner' - which I could find for the life of me. It's time for a little digging.

Samba wiki mentions that one must issue FSCTL_SRV_COPYCHUNK_WRITE request on the server, by using CIFS_IOC_COPYCHUNK_FILE ioctl.

grep-ing the cifs source files in the kernel 'fs' directory for this particular ioctl return two entries:

$ grep CIFS_IOC_COPYCHUNK_FILE * -r
cifs_ioctl.h:#define CIFS_IOC_COPYCHUNK_FILE  _IOW(CIFS_IOCTL_MAGIC, 3, int)
ioctl.c:              case CIFS_IOC_COPYCHUNK_FILE:

_IOW is a macro defined in include/uapi/asm-generic/ioctl.h

#define _IOW(type,nr,size) _IOC(_IOC_WRITE,(type),(nr),(_IOC_TYPECHECK(size)))

This macro basically translates given arguments into a 32-bit long message which is sent to the device. The type here is CIFS_IOCTL_MAGIC, which is defined in cifs_ioctl.h as 0xCF (and can also be found in the Documentation/ioctl/ioctl-number.txt:

0xCF  02      fs/cifs/ioctl.c

Presumably, CF is abbreviated CIFS - neat :)

The 'nr' in the macro is a 'request' argument as described in the ioctl manual page (in this case it's 3).

In the fs/cifs/ioctl.c we can see, that when this ioctl command is issued, the 'cifs_ioctl_copychunk' function is called.

	`case CIFS_IOC_COPYCHUNK_FILE:`
	`rc = cifs_ioctl_copychunk(xid, filep, arg);`

This in turn calls 'cifs_file_copychunk_range' from 'cifsfs.c'

	`rc = cifs_file_copychunk_range(xid, src_file.file, 0, dst_file, 0,`
	`src_inode->i_size, 0);`

Let's try a simple program:

#include <stdio.h>
#include <sys/ioctl.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>

#define CIFS_IOCTL_MAGIC        0xCF
#define CIFS_IOC_COPYCHUNK_FILE _IOW(CIFS_IOCTL_MAGIC, 3, int)

int main( int argc, char *argv[] )  {

   if( argc == 3 ) {
      printf("Starting copying...\n", argv[1]);
   }
   else if( argc > 3 ) {
      printf("Too many arguments supplied.\n");
      return 1;
   }
   else {
      printf("Two arguments expected - src and dest.\n");
      return 1;
   }

   char *src_file = argv[1];
   int src_fd = open(src_file, O_RDONLY);
   char *dst_file = argv[2];
   int dst_fd = open(dst_file, O_CREAT | O_WRONLY, 0644);

   int ret = ioctl(dst_fd, CIFS_IOC_COPYCHUNK_FILE, src_fd);

   ret = errno;
   if (ret != 0){
        fprintf(stderr, "Error %i: %s\n", ret, strerror( ret ));
   }
   return ret;
}

After compiling it with a name 'copy', I've mounted two smb shares on /tmp/test1 and /tmp/test2.

Let's create a test file and check our new program:

$ dd if=/dev/urandom of=/tmp/test1/file-1mb bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52753 s, 2.0 MB/s
$ ./copy /tmp/test1/file-1mb /tmp/test2/file-1mb
Starting copying...
$ ls -alh /tmp/test2
-rwxr-xr-x  1 user 1666 1.0M May  4 18:15 file-1mb

Looks like it's working. Next step is to copy a larger file and monitor the network usage:

$ dd if=/dev/urandom of=/tmp/test1/file-10gb bs=1M count=10000
./copy /tmp/test1/file-10gb /tmp/test2/file-10gb
Starting copying...
Error 22: Invalid argument

Hmm, something is not right. But upon checking the /tmp/test2 directory, we can see that the file is actually there. Just not the whole file:

ls -alh /tmp/test2
-rwxr-xr-x  1 user 1666 2.0G May  4 19:37 file-10gb

Using instructions from samba wiki (https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting) I've enabled the cifs debug to inspect this issue:

# echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
# echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
# echo 7 > /proc/fs/cifs/cifsFYI

And I've found this in the dmesg:

CIFS: Status code returned 0xc000000d STATUS_INVALID_PARAMETER
CIFS: fs/cifs/smb2maperror.c: Mapping SMB2 status code 0xc000000d to POSIX err -22
CIFS: fs/cifs/smb2ops.c: MaxChunks 256 BytesChunk 1048576 MaxCopy 16777216
CIFS: fs/cifs/ioctl.c: VFS: leaving cifs_ioctl (xid = 881) rc = -22

Further investigating the 'cifs_file_copychunk_range' which is defined in 'cifsfs.c', we see:

	`ssize_t cifs_file_copychunk_range(unsigned int xid,`
	`struct file *src_file, loff_t off,`
	`struct file *dst_file, loff_t destoff,`
	`size_t len, unsigned int flags)`

	`rc = target_tcon->ses->server->ops->copychunk_range(xid,`
	`smb_file_src, smb_file_target, off, len, destoff);`

Checking the structures 'ses', 'server' and 'ops' we find, that the last mentioned is defined in 'smb2ops.c' for mutliple smb versions. Fortunately, all those versions use the same function for 'copychunk_range' method, for example:

	`struct smb_version_operations smb21_operations = {`
	`(...)`

.copychunk_range = smb2_copychunk_range,

smb2_copychunk_range is defined in the same file and uses offsets just like 'cifs_file_copychunk_range'. But going back to the 'cifs_file_copychunk_range' function, we see that it is called with the value '0' for both offsets. However, smb2_copychung_range iterates over file regions of size 'len' (which is the size of part of the file not copied yet) or 'tcon->max_bytes_chunk' (declared by the server) - whichever is the smallest:

smb2_copychunk_range(const unsigned int xid,
              struct cifsFileInfo *srcfile,
              struct cifsFileInfo *trgtfile, u64 src_off,
              u64 len, u64 dest_off)
{

(...)

        while (len > 0) {
              pcchunk->SourceOffset = cpu_to_le64(src_off);
              pcchunk->TargetOffset = cpu_to_le64(dest_off);
              pcchunk->Length =
                      cpu_to_le32(min_t(u32, len, tcon->max_bytes_chunk));

              /* Request server copy to target from src identified by key */
              kfree(retbuf);
              retbuf = NULL;
              rc = SMB2_ioctl(xid, tcon, trgtfile->fid.persistent_fid,
                      trgtfile->fid.volatile_fid, FSCTL_SRV_COPYCHUNK_WRITE,
                      (char *)pcchunk, sizeof(struct copychunk_ioctl),
              CIFSMaxBufSize, (char **)&retbuf, &ret_data_len);

So what's the problem?

After sniffing around with wireshark, I've noticed that the host sends requests to the server, telling it to copy files chunk at the given offset and of the given length. But at some point, the requested transfer length is 0, which causes the SMB server to respond with an -EINVAL error and closes the transmission.

SMB2 (Server Message Block Protocol version 2)
  SMB2 Header
  Ioctl Request (0x0b)
      StructureSize: 0x0039
      Reserved: 0000
      Function: FSCTL_SRV_COPYCHUNK_WRITE (0x001480f2)
      GUID handle File: file-5gb
      Max Ioctl In Size: 0
      Max Ioctl Out Size: 16384
      Flags: 0x00000001
      Reserved: 00000000
      Blob Offset: 0x00000000
      Blob Length: 0
      Out Data: NO DATA
      Blob Offset: 0x00000078
      Blob Length: 56
      In Data
          ResumeKey: Opaque Data
          Chunk Count: 1
          Reserved: 00000000
          Chunk
              Source Offset: 947912704
              Target Offset: 947912704
              Transfer Length: 0
              Reserved: 00000000

After recompiling the CIFS module several times with various debug messages, I've finally found the culprit. It was the above-mentioned 'min_t' function, which casts the u64 value of 'len' into u32, transforming it into 0 when you didn't expect it.

After changing the line to:

pcchunk->Length =
        cpu_to_le32(min_t(u64, len, tcon->max_bytes_chunk));

it was finally functioning as expected and I could go on with my life (after submitting the patch to the cifs community of course :) )