The problem is that you're explicitly formatting 1M+ cells (cells which ClosedXML needs to create). You have to either format the entire column or only format the cells used. There are many ways you can do this but here are some examples:
ws.Column(3).Style.NumberFormat.Format = "$ #,##0.00";
ws.RangeUsed().Column(3).Style.NumberFormat.Format = "$ #,##0.00";
ws.Column(3).Column(2, ws.LastRowUsed().RowNumber()).Style.NumberFormat.Format = "$ #,##0.00";
ws.Range(2, 3, ws.LastRowUsed().RowNumber(), 3).Style.NumberFormat.Format = "$ #,##0.00";