Your problem consist of 2 smaler problems: 1)Witch is the greatest number of rectangles of given dimensions that fits in a rectangle of fixed dimension;
2)Finding the right size of rectangles for a given paper size to meet some requirements(like proportion, paper waste, e.t.c.)
A starter for 1) is backtraking. Remember maximum number of rectangles that fits is div(container area, rectangle area). A unrefined idea is to use a matrix of [a =cmmd(greatest common divisor for calculation use Euclid's algoritm)(containerX,rectangleX,containerY,rectangleY)] (containerX/a, containerY/a) to simulate the space ocupied by rectangles. Rectangles could have 2 positions horizontal or vertical. Your basic purpose is to find the greatest number of rectangles that fit and save that solution positions(left top corner and horizontal/vertical attribute) for rectangles.
Backtraking is a consumer, so optimize the search. Good luck! Let me know what you do.