Is It Rude to Refuse Food at Someones House?

Is It Rude to Refuse Food at Someones House? The act of refusing food at someone