If you don't already have it, THIS is the support site for your Dell Alienware 17 R3 17.3" laptop.

According to its original system configuration, it came with 8 GB(4 GB x 2) of DDR4-2133 MHz RAM and a 1 TB SATA3 7200 RPM hard disk drive.

According to its service manual, it can accommodate a single 2.5" hard disk drive or solid state drive, or it can accommodate two M.2 PCIe solid state drives.
It cannot accommodate 2.5" and M.2 PCIe storage drives at the same time.

From what I can determine, it supports a maximum of 32 GB of DDR4 RAM.
Since yours has only 8 GB, you should definitely increase it to at least 16 GB.

I know nothing about M.2 and NVMe solid state drives, so I'll pass to others here who can address that for you.

-------------------------------------------------

 

As you can read from the other support member, the laptop hardware will not support both hard drives at the same time. You are definitely likely to see better speed and throughput with the NVMe versus a SATA M.2 drive.

Since the hardware will not support both drives being connected at the same time, you are going to need a cloning process and software. You would use the cloning software to make an image of the current OS install, then install the new hardware, boot a recovery tool, and apply the image back to the new hardware. There are many cloning options available. I would suggest you research recommendations specific for the new NVMe drive since that is where the image will be installed.

One final option would be to do a clean installation of Windows on the new NVMe drive, then install your software, special software from Alienware, and copy over your personal files and data. I already know from personal experience that this is likely to be the longer route. But it also has the advantage of cleaning out stuff that tends to accumulate in an installation after it's been used for a length of time.

If you are interested, I can make two recommendations on cloning and backup software. I will defer that recommendation until requested.

 

I want to install a 2 TB NVMe 2280, clone the 2.5" SATA onto it for my boot drive and use the 2.5" SATA as a secondary storage drive

Cannot be done unless you install the sata drive in an external housing and use it that way. The sata ports are disabled when a M.2 type drive is installed. Because of this, you would need some extra hardware to clone your existing sata drive to a M.2. You are going to have to purchase an enclosure for the sata drive and connect via usb; preferably a powered enclosure. Your other option would be to make an image of your existing drive. Store that on an external. Boot with the boot usb that the imaging program makes for you and restore the image. Either way you need an external drive.

Just an opinion however I would not spend any money; I would just clean install windows on the M.2. Up to you.