Paint the wood. They only go for raw wood surfaces. Then get a can of carpenter bee spray from your big box store, stick the tube up in the hole and foam it up. When it drys then get a tube of some kine of caulk and squirt it in the hole to seal it off.

When you paint treated wood it take something like Kills for it to stick to the wood and the green crap not to bleed through. Ask the paint dept at your big box what to use to paint treated wood.