CVX Solver: Exceeding Maximum Array Size with Large Dataset

1

I am using the CVX solver to solve an optimisation problem with a dataset of input and output vectors of size 1400x1. When I take a subsample of 200 datapoints, this results in 40,002 equality constraints. Ideally I would use all 1400 datapoints for my problem, but I then receive the error:

Requested 1962801x1401 (20.5GB) array exceeds maximum array size preference (16.0GB). This might cause MATLAB to become unresponsive.

I’ve looked into “Preferences” > “Workspace” > “MatLab Array Size Limit” > checked off “Limit the maximum array size to a percentage of RAM” (already set to 100%).

However, I have seen other posts where people have worked with over 2 million equality constraints, albeit with a runtime of 40 minutes. Is this simply a problem where I need to use a computer with more memory and a better processor, or is there any way that I can modify the settings to still solve the optimisation problem, even if the runtime may take longer?

2

It appears that the array in question increases in size (number of elements) as the cube of your input vector length. So the requested array is about 343 times larger for 1400 data points than for 200 data points.

Perhaps if you showed your program, including the sizes of all input data, someone could provide additional assessment and advice.