Drain olives and place them in a large bowl.
To the olives, add the oregano, rosemary, thyme, garlic, red pepper flakes, orange zest, and lemon zest. Add salt, as needed (See NOTES).
Add enough oil to coat the olives, about ½ cup. Gently stir the olives with the aromatics until they are fully coated. If possible, transfer the olives to a large jar with a lid, or a couple of smaller jars with lids. Add more oil to just cover the tops of the olives. If keeping in the bowl, cover with plastic wrap, if using jars, screw on the lids.
Place in the refrigerator for 12 hours, or preferably 24 hours, or up to 1 month. Before serving, allow the olives to come to room temperature, a couple of hours.
To serve, pour the olives with the oil into a large bowl. Use a slotted spoon to place the olives in a serving bowl. Scoop out some of the herbs, zest, and garlic from the oil and drizzle over the olives. If serving whole olives, be sure to include a bowl for guests to place the pits.