Everyone seems to be focusing on the wrong thing here.
The points to consider are the following:
1) They tested to see if they could get a speedboat within contact / point blank attacking range, and
2) They attempted to get the US to shoot at them, regardless of what weapon.
All the business about whether you can point-blank fire missiles is moot, as that probably wouldn't be how Tehran would respond.
They are rattling sabres and hoping the US will draw and strike first so they can start messing with Uncle Sam directly. At the moment they want the US out of the Gulf ASAP so they can put their stranglehold around the Shatt and secondarily try to control Iraq. They know there is a Shiite majority there and that works for them, perhaps even a step towards annexing the entire country - who knows? That is too far down the line to care about right now.
The Iranians think that the US engaging Iranian boats would cause the West to mount further pressure to rush the US out of Iraq. An open conflict by the US with Iran would cause another series of crises in the area, and potentially add Iranian military to the current issues that the US has in Iraq and the Gulf.
The transmissions possibly did not come from the boats, but by another vessel or ground station nearby. In any case, this could be the defence put forward by the Revolutionary Guard if there was an altercation, even if they did send them from the boats or from a directly supporting establishment or vessel.
Unfortunately, the microscope is on the US everywhere right now, and this incident is nothing but a footnote compared to whenever the US make a simple honest mistake. The Iranians know this and may continue harassing the US until someone starts a deadset real shooting war.
The Iranians are smarter than to carry on like they did in the 80's, and are trying to draw the US into looking like it is taking an unprovoked action, achieving this by presenting scenarios that they can dismiss any apparent bad behaviour on their part with some type of excuse.
The boxes would be explained away as 'trash dumped overboard' or 'loose fishing cartons being left unattended'. Any ship needs to avoid any collision with an underwater object, inert objects can cause immense damage to hulls, sonar equipment, countermeasures, props and rudders. Explosive or not, I'd cite this as a deliberate action by the Iranians - one would think that there is countless acres of aquatic real estate for things to come off a boat. For it to happen in the path of a US Navy vessel accompanied with all the other goings-on I'd be gathering evidence of it happening for later.
So, Iran is probing the US to see how far they can push them, how close they can get, and how ready Uncle Sam is to start flinging lead.
As for what the US were using, they were about to engage with an M240B according to some sources (and the 25mm would have been the next item on the "to use" list). They were definitely well within the minimum range for the missile system, and the 5" would have been overkill.
As for the Iranians, they have a lot of anti-shipping missiles onshore.
They have the Kowsar, Silkworm and the Noor, all can easily reach targets in the Straits while being miles inshore. Next, they have buckets of fast missile craft. They are all about saturation attacks from either or both, and quite simply they can put a vessel out there to follow the US for intel gathering and position reporting, and there isn't a lot that can be done about it. If the US shoots a threatening gunboat/boghammer/patrol boat/missile boat, then it just might be that Tehran could decide to take matters into their own hands right afterwards without resorting to the public arena to voice concerns. It may even be that Tehran sits on it and strikes later; revenge being served cold and whatnot.
Effectiveness can be argued all day about whether they can fire enough missiles, whether the missiles (like the Silkworm) will work, whether the AEGIS can blot them out before they get within burst range or hit, and whether the Iranians would actually try.
Whatever your flavour, and hit percentages and so on, if that happens there will be a lot more than that initial engagement to worry about. I need not remind everyone of the USS Stark, USS Samuel B. Roberts and USS Vincennes incidents.
As far as the Iranians are concerned, this is a "mission accomplished".